From 8afe64f60f6e5262c1f98b53c7b5179110b8ddee Mon Sep 17 00:00:00 2001 From: Katon Minhas Date: Thu, 5 Mar 2026 19:38:37 +0000 Subject: [PATCH] Merged in DEV (pull request #895) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit DEV * Merged in bugfix/parser-downstream-improvements (pull request #875) Bugfix/parser downstream improvements * Refactor: Implement field-aware JSON parsers with centralized normalization This refactor introduces a robust system for normalizing LLM output based on field format mappings, ensuring consistent data types throughout the pipeline. Key Changes: - Add FIELD_FORMAT_MAPPING constant defining expected formats for all fields - Create format_normalization.py utility for type-aware normalization - Update json_utils.py parsers to accept field_names/field_name parameters - Refactor prompt_templates.py to use parser factories (_create_json_dict_parser, _create_json_list_parser) that bind field metadata for automatic normalization - Update prompt_calls.py to pass field names to parsers, eliminating redundant normalization logic - Remove parse_json_dict_or_list (unused, ambiguous function) - Simplify METHODOLOGY_BREAKOUT and REIMBURSEMENT_PRIMARY to use helper functions - Add comprehensive integration tests… * Update other lists * Black format * Remove prints * Update qa_qc to account for lists in is_empty check * Resolve issues * Black format * Merged main into DEV * Merged in feature/claim-type-only-runner (pull request #876) Feature/claim type only runner * Add specific_fields config for running extraction on field groups ## What Changed 4 files modified: 1. src/config.py - Added configuration for field-specific extraction: - SPECIFIC_FIELDS arg (default: 'all') - pass field group name or comma-separated field names - FIELD_GROUPS dict - predefined groups: claim_type, dates, provider - get_specific_fields_list() - resolves config to actual field list 2. src/prompts/fieldset.py - Added filter_by_names() method to FieldSet class to filter fields by a list of names 3. src/pipelines/shared/extraction/one_to_n_funcs.py - Updated exhibit_level() to accept specific_fields parameter and skip prompts for fields not in the list 4. src/pipelines/saas/file_processing.py - Passes specific_fields through the call chain to both one_to_n and one_to_one extraction ## How It Works When specific_fields is set to something other than 'all': 1. Config resolves the field l… * Merged in bugfix/generic_lesser_of (pull request #880) Bugfix/generic lesser of * Updated lesser of ditribution and lesser of check prompt * Remove excessive logging * Updated prompt for combining and formatting lesser of statement and reimb term * Code clean upt * Removed unneeded instruction on output format * style: format prompt_calls.py with black Approved-by: Katon Minhas * Merged in feature/new_output_format (pull request #879) Feature/new output format * bugfix * Merge branch 'DEV' into Optimize/DAIP2-1474-restructure-postprocess * requested postprocessing changes * prompt changes reverted * fix pipeline issues * fix pipeline issues * fix pipeline issues * fixed formatting * fixed formatting * Merge branch 'DEV' into Optimize/DAIP2-1474-restructure-postprocess * Merge branch 'DEV' into Optimize/DAIP2-1474-restructure-postprocess * save dashboard and cc output separately * save dashboard output in s3 * pipeline error fixed * json list through postprocessing * Merged DEV into Optimize/DAIP2-1474-restructure-postprocess * Merge remote-tracking branch 'origin/Optimize/DAIP2-1474-restructure-postprocess' into feature/new_output_format * Restructure output file organization and add standard field sanitization Output Structure Changes: - Reorganize output files into hierarchical directory structure: - full_outputs/cc_results/ for consolidated CC results - full… * Merged in bugfix/dynamic_issues_feb12 (pull request #882) Bugfix/dynamic issues feb12 * prompt changes reverted * fix pipeline issues * fix pipeline issues * fix pipeline issues * fixed formatting * fixed formatting * Merge branch 'DEV' into Optimize/DAIP2-1474-restructure-postprocess * Merge branch 'DEV' into Optimize/DAIP2-1474-restructure-postprocess * save dashboard and cc output separately * save dashboard output in s3 * pipeline error fixed * json list through postprocessing * Merged DEV into Optimize/DAIP2-1474-restructure-postprocess * Merge remote-tracking branch 'origin/Optimize/DAIP2-1474-restructure-postprocess' into feature/new_output_format * Restructure output file organization and add standard field sanitization Output Structure Changes: - Reorganize output files into hierarchical directory structure: - full_outputs/cc_results/ for consolidated CC results - full_outputs/dashboard_results/ for consolidated dashboard results - full_outputs/ for error files - automation_q… * Merged in feature/DAIP2-1562-add-aarete_derived_payer_name (pull request #867) Feature/DAIP2-1562 add aarete derived payer name * black format * Merged DEV into feature/DAIP2-1562-add-aarete_derived_payer_name * llm_choose_derived_payer_name function added * threshold updated * Merge branch 'DEV' into feature/DAIP2-1562-add-aarete_derived_payer_name * aarete_derived_payer_name column added * prompt structure updated * state_flag added * pipeline error fixed * pipeline error fixed * Merged DEV into feature/DAIP2-1562-add-aarete_derived_payer_name * remove debug print statement for similarity matrix in clustering function * Merged DEV into feature/DAIP2-1562-add-aarete_derived_payer_name * state logic added in clustering * removed print statements * pull request updates * black format fix * Merged DEV into feature/DAIP2-1562-add-aarete_derived_payer_name * added scalability feature and optimization * black format * Pull Request Changes * updated config parameters * updated main * no pay… * Merged in bugfix/DAIP2-1701-issue-tracker-fixes-methodology-breakout (pull request #885) Bugfix/DAIP2-1701 issue tracker fixes methodology breakout * testing fee schedules issues * logging debug added for lesser of * Merged DEV into bugfix/DAIP2-1701-issue-tracker-fixes-methodology-breakout * special case term fix * updated lesser of distribution prompt * Merged DEV into bugfix/DAIP2-1701-issue-tracker-fixes-methodology-breakout * refactor: use f-string for special case term concatenation Approved-by: Siddhant Medar * Merged in bugfix/DAIP2-1699-issue-tracker-fixes-one-to-one-dates-prov-info (pull request #881) bugfix/DAIP2-1699-issue-tracker-fixes-one-to-one-dates-prov-info * contract title fixes * Add instruction to strip hyphens from TIN and NPI * prompt update * Merge branch 'bugfix/auto-renewal-term' into bugfix/DAIP2-1699-issue-tracker-fixes-one-to-one-dates-prov-info * removed print statements * pipeline error fixed * signed_ind logic changed * signatory_ind logic updated * prompt update * prompt update * pipeline error fixed * Merge branch 'DEV' into bugfix/DAIP2-1699-issue-tracker-fixes-one-to-one-dates-prov-info * Merge branch 'DEV' into bugfix/DAIP2-1699-issue-tracker-fixes-one-to-one-dates-prov-info * requested changes applied * test case added * prompt update * contract title prompt update * latest prompt added Approved-by: Siddhant Medar * Merged in bugfix/default_ind_postprocess (pull request #887) Bugfix/default ind postprocess * Add logic to standardize UNIT_OF_MEASURE for flat-rate reimbursement methods - Implemented functionality in `standardize_reimb_method_and_fee_schedule` to set UNIT_OF_MEASURE to blank for rows where DEFAULT_IND is 'Y' and AARETE_DERIVED_REIMB_METHOD is 'flat rate'. - Added unit tests to verify behavior for various scenarios, including case insensitivity and non-default conditions. - Ensured that UNIT_OF_MEASURE remains unchanged for non-flat rate methods. * Enhance child rank handling and ensure column consistency in parent-child mapping - Added initialization for the `child_rank` column in both parents and children DataFrames to prevent KeyError during concatenation when no children exist. - Updated `cols_to_keep` in `parent_child_mapping` to filter out columns not present in `pc_df`, ensuring robustness in data processing. * Ran Black * made a small change in code_last_check, fixed so it returns string and n… * Apply formatting fix * Merged in DAIP2-1898-one-to-one-issues-filename-tin-auto-renewal-and-contract-title (pull request #888) DAIP2-1898 one to one issues filename tin auto renewal and contract title * contract title fixes * Add instruction to strip hyphens from TIN and NPI * prompt update * Merge branch 'bugfix/auto-renewal-term' into bugfix/DAIP2-1699-issue-tracker-fixes-one-to-one-dates-prov-info * removed print statements * pipeline error fixed * signed_ind logic changed * signatory_ind logic updated * prompt update * prompt update * pipeline error fixed * Merge branch 'DEV' into bugfix/DAIP2-1699-issue-tracker-fixes-one-to-one-dates-prov-info * Merge branch 'DEV' into bugfix/DAIP2-1699-issue-tracker-fixes-one-to-one-dates-prov-info * requested changes applied * test case added * prompt update * contract title prompt update * prompt update * Merge branch 'DEV' into DAIP2-1898-one-to-one-issues-filename-tin-auto-renewal-and-contract-title Approved-by: Siddhant Medar * Merged in feature/DAIP2-1803-add-aarete-derived-provider-name (pull request #884) Feature/DAIP2-1803 add aarete derived provider name * Merged DEV into feature/DAIP2-1562-add-aarete_derived_payer_name * state logic added in clustering * removed print statements * pull request updates * black format fix * Merged DEV into feature/DAIP2-1562-add-aarete_derived_payer_name * added scalability feature and optimization * black format * Pull Request Changes * generalised funcs from aarete derived payer name * updated remove states and updated docstring for build similarity matrix * derived payer name functionality added * config and main file changes * sync dev into ADD AARETE DERIVED PROVIDER GROUP NAME FULL * updated derived_provider name * pipeline fixes * Merged DEV into feature/DAIP2-1803-add-aarete-derived-provider-name * config changes * field name updated to AARETE_DERIVED_PROVIDER_NAME * Merged DEV into feature/DAIP2-1803-add-aarete-derived-provider-name * black format fix * Fix docstr… * Merged in improve-logging (pull request #886) Improve logging * Add timing blocks for comprehensive pipeline logging - Add timing_utils.timed_block() to key extraction functions in one_to_n_funcs.py: - exhibit_level sub-functions (prompt_exhibit_level, dynamic_primary, etc.) - reimbursement_level extraction and cleaning - breakout functions (methodology_breakout, special_case_breakout) - carveout_and_special_case parallel processing - one_to_n_cleaning sub-steps (crosswalk, lob_relationship, split_reimb_dates) - Add timing to dynamic_funcs.py: - dynamic_assignment parallel processing - Add timing to postprocess.py: - standard_postprocess, contract_config_postprocess, dashboard_postprocess - Add timing to preprocess.py: - exhibit_chunking sub-steps (get_exhibit_pages, link_exhibit_pages, chunk_by_exhibit) - Add info-level logging statements for completed operations with row/item counts * Standardize logging levels and remove deprecated code DAIP2-1798: Ensure logging statements are at … * Merged in bugfix/DAIP2-1913-lesser-of-indicator-issues (pull request #889) Bugfix/DAIP2-1913 lesser of indicator issues * contract title fixes * Add instruction to strip hyphens from TIN and NPI * prompt update * Merge branch 'bugfix/auto-renewal-term' into bugfix/DAIP2-1699-issue-tracker-fixes-one-to-one-dates-prov-info * removed print statements * pipeline error fixed * signed_ind logic changed * signatory_ind logic updated * prompt update * prompt update * pipeline error fixed * Merge branch 'DEV' into bugfix/DAIP2-1699-issue-tracker-fixes-one-to-one-dates-prov-info * Merge branch 'DEV' into bugfix/DAIP2-1699-issue-tracker-fixes-one-to-one-dates-prov-info * requested changes applied * test case added * prompt update * contract title prompt update * prompt update * Merge branch 'DEV' into DAIP2-1898-one-to-one-issues-filename-tin-auto-renewal-and-contract-title * handle nested cases of lesser_of and greater_of * Merge branch 'DEV' into bugfix/DAIP2-1913-lesser-of-indicator-issues * pi… * Merged in bugfix/reimb_primary (pull request #890) Update reimbursement prompts for consistent PMPM handling, time-unit dedup, and validation improvements * Update reimbursement prompts for consistent PMPM handling, time-unit dedup, and validation improvements - REIMBURSEMENT_PRIMARY: append unit of measure (PMPM, per visit, etc.) from table headers into REIMB_TERM - METHODOLOGY_BREAKOUT: collapse mathematically equivalent time-unit rates into single entry - VALIDATE_REIMBURSEMENTS: add umbrella terms as valid service terms, accept CPT/HCPCS percentage as concrete rate, narrow COB disqualifier to primary subject only Co-Authored-By: Claude Opus 4.6 Approved-by: Siddhant Medar * Merged in feature/update-postprocessing-columns (pull request #893) move column order to final step * move column order to final step * Black Approved-by: Siddhant Medar * Merged in bugfix/update-postprocessing-paths (pull request #896) Update postprocessing paths * Update postprocessing paths Approved-by: Siddhant Medar * Merged in feature/TIN_stats_update (pull request #898) Feature/TIN stats update * Tin_stats_report * black formatting * minor changes * black format applied * Addressed the katons comments * Merged DEV into feature/TIN_stats_update * black reformat Approved-by: Katon Minhas --- .gitignore | 1 + src/codes/code_funcs.py | 7 +- src/config.py | 75 ++- src/constants/investment_columns.py | 2 + src/constants/regex_patterns.py | 3 + src/parent_child/__main__.py | 35 +- src/parent_child/pipeline.py | 6 + .../clients/bcbs_promise/file_processing.py | 8 +- .../clients/clover/file_processing.py | 8 +- src/pipelines/runner.py | 171 +++-- src/pipelines/saas/file_processing.py | 236 ++++--- src/pipelines/saas/main.py | 189 ++++-- src/pipelines/saas/prompts/prompt_calls.py | 135 +++- .../shared/extraction/dynamic_funcs.py | 65 +- .../shared/extraction/one_to_n_funcs.py | 280 ++++---- .../shared/extraction/one_to_one_funcs.py | 377 +++++++++++ .../shared/postprocessing/postprocess.py | 204 +++++- .../postprocessing/postprocessing_funcs.py | 288 +++++++- .../hybrid_smart_chunking_funcs.py | 5 +- .../shared/preprocessing/preprocess.py | 66 +- src/prompts/fieldset.py | 18 + src/prompts/investment_prompts.json | 15 +- src/prompts/prompt_templates.py | 206 +++++- src/qc_qa/__main__.py | 19 +- src/qc_qa/pipeline.py | 269 +++++++- src/scripts/postprocess_existing_output.py | 14 +- src/tests/test_derived_names.py | 251 +++++++ src/tests/test_io_utils.py | 265 +++++++- src/tests/test_postprocess.py | 626 +++++++++++++++++- src/tests/test_qc_qa_pipeline.py | 135 +++- src/utils/formatting_utils.py | 4 +- src/utils/io_utils.py | 459 ++++++++++++- src/utils/logging_utils.py | 18 +- src/utils/string_utils.py | 286 ++++++++ 34 files changed, 4247 insertions(+), 499 deletions(-) create mode 100644 src/tests/test_derived_names.py diff --git a/.gitignore b/.gitignore index b09252d..8af674e 100644 --- a/.gitignore +++ b/.gitignore @@ -108,3 +108,4 @@ TEST*.xml # PRD Documents *.prd *prd.md +.git.instructions.md diff --git a/src/codes/code_funcs.py b/src/codes/code_funcs.py index 58d889f..b991be3 100644 --- a/src/codes/code_funcs.py +++ b/src/codes/code_funcs.py @@ -489,8 +489,11 @@ def code_last_check(service, filename): ) try: llm_answer_final = _parser(llm_answer_raw) - return llm_answer_final[0] - except: + return llm_answer_final if llm_answer_final else "Generic" + except Exception: + logging.error( + "Failed to parse LLM response in code_last_check: %s", llm_answer_raw + ) return "Generic" diff --git a/src/config.py b/src/config.py index 180807d..2fdb804 100644 --- a/src/config.py +++ b/src/config.py @@ -95,6 +95,11 @@ RUN_MODE = get_arg_value("run_mode", "ec2") # Valid: local or ec2 READ_MODE = get_arg_value("read_mode", "s3") # Valid: local or s3 BATCH_ID = get_arg_value("batch_id", None) # BATCH_ID is mandatory +# Logging settings +LOG_LEVEL = get_arg_value( + "log_level", "INFO" +).upper() # Standard logging level for console output + # Per-file logging settings (for debugging) ENABLE_PER_FILE_LOGGING = get_arg_value("enable_per_file_logging", "False") == "True" PER_FILE_LOG_LEVEL = get_arg_value( @@ -108,6 +113,48 @@ ENABLE_RUNTIME_ROTATION = False # Set to True to enable multi-runtime failover FIELDS = get_arg_value("fields", "all") # Valid: one_to_one, one_to_n, all FIELD_JSON_PATH = "src/prompts/investment_prompts.json" +# Specific field or field group to extract (comma-separated) +# Valid: "all", "claim_type", or specific field names like "CLAIM_TYPE_CD,CONTRACT_TITLE" +# "claim_type" is a shortcut for CLAIM_TYPE_CD + AARETE_DERIVED_CLAIM_TYPE_CD + CONTRACT_TITLE +SPECIFIC_FIELDS = get_arg_value("specific_fields", "all") + +# Field groups - predefined sets of related fields +FIELD_GROUPS = { + "claim_type": [ + "CLAIM_TYPE_CD", + "AARETE_DERIVED_CLAIM_TYPE_CD", + "CONTRACT_TITLE", + ], + "dates": [ + "EFFECTIVE_DT", + "AARETE_DERIVED_EFFECTIVE_DT", + "TERMINATION_DT", + "AARETE_DERIVED_TERMINATION_DT", + ], + "provider": [ + "PROV_GROUP_TIN", + "PROV_GROUP_NPI", + "PROV_GROUP_NAME_FULL", + "PROV_OTHER_TIN", + "PROV_OTHER_NPI", + "PROV_OTHER_NAME_FULL", + ], +} + + +def get_specific_fields_list(): + """Get the list of specific fields to extract based on config.""" + if SPECIFIC_FIELDS == "all": + return None # None means extract all fields + + # Check if it's a field group + if SPECIFIC_FIELDS in FIELD_GROUPS: + return FIELD_GROUPS[SPECIFIC_FIELDS] + + # Otherwise, treat as comma-separated list of field names + return [f.strip() for f in SPECIFIC_FIELDS.split(",") if f.strip()] + + # Client-specific prompts CLOVER_PROMPTS_PATH = "src/prompts/clover_prompts.json" BCBS_PROMISE_PROMPTS_PATH = "src/prompts/bcbs_promise_prompts.json" @@ -301,9 +348,11 @@ MIN_TABLE_ROWS_TO_SPLIT = int(get_arg_value("min_table_rows_to_split", "20")) # Table splitting method (currently only "cells" is supported) TABLE_SPLITTING_METHOD = get_arg_value("table_splitting_method", "cells") -######################################## EC2 ######################################## +######################################## OUTPUT FILE SPLITTING SETTINGS ######################################## +# Maximum rows per split file (files are split to keep filenames together) +MAX_ROWS_PER_SPLIT = int(get_arg_value("max_rows_per_split", "70000")) -# DEPRECATED: MODEL_STATS and GLOBAL_STATS are no longer updated. +######################################## EC2 ######################################## def assume_role_and_get_runtime(account_id, role_name): @@ -338,11 +387,10 @@ DOCZY_PDF_FILES_BUCKET_S3_URL = f"https://{DOCZY_PDF_FILES_BUCKET_NAME}.s3.us-ea ENABLE_VISION = get_arg_value("enable_vision", "False") == "True" # False by default ############## PARENT CHILD CONFIG ############## -PERFORM_PARENT_CHILD_MAPPING = get_arg_value("perform_pc", "False") == "True" +PERFORM_PARENT_CHILD_MAPPING = get_arg_value("perform_pc", "True") == "True" DOCZY_OUTPUT_FOR_PC = get_arg_value( "doczy_output_for_pc", "" ) # Pass either S3 URI path, or local path -WRITE_PC_TO_S3 = get_arg_value("write_pc_to_s3", "False") == "True" ############## DOCUMENT TYPE CLASSIFICATION SETTINGS ############## PERFORM_DTC = ( @@ -357,6 +405,25 @@ DTC_OUTPUT_FILE = ( ) DTC_JSON_OUTPUT_FOLDER = "dtc_json_results" # Folder for individual JSON results +############## DASHBOARD POSTPROCESSING ############## +# Run dashboard postprocessing (comma-separated list format). Default: CC only. +# Set to True via run_dashboard arg when dashboard output is needed. +RUN_DASHBOARD_POSTPROCESSING = get_arg_value("run_dashboard", "False") == "True" + ############## QC/QA VALIDATION SETTINGS ############## # Enable QC/QA validation in pipeline ENABLE_QC_QA = get_arg_value("enable_qc_qa", "True") == "True" + +############## ADD_AARETE_DERIVED_PAYER_NAME SETTINGS ############ +# STATE_FLAG controls whether payer names are treated as state-specific. +# +# True → Preserve state distinctions. +# Example: "Molina Healthcare of Texas" and +# "Molina Healthcare of Utah" remain separate. +# +# False → Collapse regional state suffixes into national brand. +# Example: "Molina Healthcare of Texas" and +# "Molina Healthcare of Utah" both become "Molina Healthcare". +# Leading state-based brand names (e.g., "Oklahoma Complete Health") +# are preserved. +STATE_FLAG = True diff --git a/src/constants/investment_columns.py b/src/constants/investment_columns.py index 0f127a3..4b730ea 100644 --- a/src/constants/investment_columns.py +++ b/src/constants/investment_columns.py @@ -31,12 +31,14 @@ FIELD_FORMAT_MAPPING = { "AARETE_DERIVED_AMENDMENT_NUM": "str", "CLIENT_NAME": "str", "PAYER_NAME": "str", + "AARETE_DERIVED_PAYER_NAME": "str", "PAYER_STATE": "list[str]", "PROVIDER_STATE": "list[str]", "FILENAME_TIN": "list[str]", "PROV_GROUP_TIN": "list[str]", "PROV_GROUP_NPI": "list[str]", "PROV_GROUP_NAME_FULL": "list[str]", + "AARETE_DERIVED_PROVIDER_NAME": "str", "PROV_OTHER_TIN": "list[str]", "PROV_OTHER_NPI": "list[str]", "PROV_OTHER_NAME_FULL": "list[str]", diff --git a/src/constants/regex_patterns.py b/src/constants/regex_patterns.py index 44b0af8..a3082d3 100644 --- a/src/constants/regex_patterns.py +++ b/src/constants/regex_patterns.py @@ -73,3 +73,6 @@ OCR_SUBSTITUTIONS = { "B": "8", # Letter B to eight "b": "8", } + +# dba patterns +DBA_PATTERNS = [r"\bD/B/A\b", r"\bDBA\b", r"\bDOING BUSINESS AS\b", r"\bD B A\b"] diff --git a/src/parent_child/__main__.py b/src/parent_child/__main__.py index 6b6a257..a69e449 100644 --- a/src/parent_child/__main__.py +++ b/src/parent_child/__main__.py @@ -79,7 +79,11 @@ def write_pc_excel( return output_path -def main(doczy_output_for_pc: str, client: Optional[str] = None) -> int: +def main( + doczy_output_for_pc: str, + client: Optional[str] = None, + output_dir: Optional[str] = None, +) -> tuple[int, str]: """ Main entry point for parent-child mapping. @@ -88,15 +92,20 @@ def main(doczy_output_for_pc: str, client: Optional[str] = None) -> int: client: Optional client identifier. If provided, takes precedence over CLI argument and path-based detection. Supported values: bcbs, caresource, molina, cnc, clover_health + output_dir: Directory to write output. When provided by pipeline/batch + run, PC results go to the same run directory as other outputs. + When None (standalone run), uses outputs/parent_child/{run_timestamp}. Returns: - Number of rows processed + Tuple of (row_count, local_path) where local_path is the written Excel file. """ # Extract client name from input file for output naming client_name = extract_client_name(doczy_output_for_pc) - # Generate timestamp for this run - run_timestamp = f"run_{datetime.now().strftime('%Y%m%d_%H-%M')}_{client_name}" + # Pipeline/batch run passes output_dir; standalone uses outputs/parent_child + if output_dir is None: + run_timestamp = f"run_{datetime.now().strftime('%Y%m%d_%H-%M')}_{client_name}" + output_dir = os.path.join("outputs", "parent_child", run_timestamp) try: # Read input file @@ -145,21 +154,13 @@ def main(doczy_output_for_pc: str, client: Optional[str] = None) -> int: original_df_with_pc, pc_df = parent_child_mapping(cleaned_df, original_df) # Save detailed results - output_dir = os.path.join("outputs/parent_child", run_timestamp) pc_df1 = qc_main(pc_df, client) output_filename = f"{client_name}_generic_hierarchy_lineage_output.xlsx" - if config.WRITE_PC_TO_S3: - # Write locally first, then upload to S3 - write_pc_excel(df_read, pc_df1, output_dir, client_name) - # TODO: Add S3 upload logic if needed - logging.info(f"PC results saved: {run_timestamp}/{output_filename}") - else: - write_pc_excel(df_read, pc_df1, output_dir, client_name) - logging.info(f"PC results saved locally: {output_dir}/{output_filename}") + local_path = write_pc_excel(df_read, pc_df1, output_dir, client_name) - # TODO: check with dev - return pc_df.shape[0] if pc_df is not None else mapped_df.shape[0] + row_count = pc_df.shape[0] if pc_df is not None else mapped_df.shape[0] + return row_count, local_path except Exception as e: logging.error(f"Parent child mapping failed due to {e}", exc_info=True) raise @@ -173,7 +174,7 @@ if __name__ == "__main__": # Generate timestamp for this run run_timestamp = f"run_{datetime.now().strftime('%Y%m%d_%H-%M')}_{config.BATCH_ID}" if config.DOCZY_OUTPUT_FOR_PC: - results = main(config.DOCZY_OUTPUT_FOR_PC) - logging.info(f"PC Mapping complete. Processed {results} files.") + row_count, _ = main(config.DOCZY_OUTPUT_FOR_PC) + logging.info(f"PC Mapping complete. Processed {row_count} files.") else: logging.warning("No input file specified in config.DOCZY_OUTPUT_FOR_PC") diff --git a/src/parent_child/pipeline.py b/src/parent_child/pipeline.py index ba471eb..1b92bd1 100644 --- a/src/parent_child/pipeline.py +++ b/src/parent_child/pipeline.py @@ -743,6 +743,11 @@ def assign_child_ranks(df, grouper_col="grouping_key"): parents = df[df["parent"]].copy() children = df[~df["parent"]].copy() + # Ensure child_rank column exists so concat result has it when there are no + # children (otherwise result["child_rank"].astype(str) raises KeyError). + parents["child_rank"] = "" + children["child_rank"] = "" + # For each child, find their actual parent's identity # Then group children by the parent they're actually assigned to children["_parent_identity"] = None @@ -1096,6 +1101,7 @@ def parent_child_mapping( "parent_child_flag", "parent_name", ] + cols_to_keep = [col for col in cols_to_keep if col in pc_df.columns] # Ensure consistent types for merge - convert both to string to avoid datetime/object mismatch pc_df["fixed_effective_date"] = pc_df["fixed_effective_date"].astype(str) diff --git a/src/pipelines/clients/bcbs_promise/file_processing.py b/src/pipelines/clients/bcbs_promise/file_processing.py index ba2477a..9753204 100644 --- a/src/pipelines/clients/bcbs_promise/file_processing.py +++ b/src/pipelines/clients/bcbs_promise/file_processing.py @@ -161,19 +161,19 @@ def process_file(file_object, constants: Constants, run_timestamp): # POSTPROCESS with timing_utils.timed_block("postprocess", context=filename): - final_df = postprocess.postprocess(final_results, constants) + cc_df, dashboard_df = postprocess.postprocess(final_results, constants) logging.info(f"{datetime_str()} Postprocessing Complete - {filename}") ################## WRITE INDIVIDUAL ################## with timing_utils.timed_block("write_individual", context=filename): if config.WRITE_TO_S3: - io_utils.write_s3(final_df, filename, run_timestamp, "individual") + io_utils.write_s3(cc_df, filename, run_timestamp, "individual_cc") else: - io_utils.write_local(final_df, filename, "", "individual") + io_utils.write_local(cc_df, filename, "", "individual_cc") logging.info(f"{datetime_str()} Writing Complete - {filename}") - return final_df + return cc_df, dashboard_df def run_one_to_one_prompts( diff --git a/src/pipelines/clients/clover/file_processing.py b/src/pipelines/clients/clover/file_processing.py index 04f3b2f..86ebb3d 100644 --- a/src/pipelines/clients/clover/file_processing.py +++ b/src/pipelines/clients/clover/file_processing.py @@ -161,19 +161,19 @@ def process_file(file_object, constants: Constants, run_timestamp): # POSTPROCESS with timing_utils.timed_block("postprocess", context=filename): - final_df = postprocess.postprocess(final_results, constants) + cc_df, dashboard_df = postprocess.postprocess(final_results, constants) logging.info(f"{datetime_str()} Postprocessing Complete - {filename}") ################## WRITE INDIVIDUAL ################## with timing_utils.timed_block("write_individual", context=filename): if config.WRITE_TO_S3: - io_utils.write_s3(final_df, filename, run_timestamp, "individual") + io_utils.write_s3(cc_df, filename, run_timestamp, "individual_cc") else: - io_utils.write_local(final_df, filename, "", "individual") + io_utils.write_local(cc_df, filename, "", "individual_cc") logging.info(f"{datetime_str()} Writing Complete - {filename}") - return final_df + return cc_df, dashboard_df def run_one_to_one_prompts( diff --git a/src/pipelines/runner.py b/src/pipelines/runner.py index f328a5c..6f61eae 100644 --- a/src/pipelines/runner.py +++ b/src/pipelines/runner.py @@ -13,6 +13,7 @@ Usage: import concurrent.futures import importlib import logging +import os import traceback import warnings from datetime import datetime @@ -68,14 +69,19 @@ def safe_process_file(item, constants, run_timestamp, file_processing): file_processing: The file_processing module to use Returns: - pd.DataFrame: Either: - - DataFrame with extracted field results (success case) - - Single-row DataFrame with error details (failure case) + tuple: Either: + - (cc_df, dashboard_df) tuple with extracted field results (success case) + - (error_df, error_df) tuple with error details (failure case) + + Note: + Returns both CC and dashboard versions from file processing. """ file_id = item[0] if isinstance(item, (list, tuple)) and len(item) > 0 else item try: - result = file_processing.process_file(item, constants, run_timestamp) - return result + cc_df, dashboard_df = file_processing.process_file( + item, constants, run_timestamp + ) + return cc_df, dashboard_df except Exception as e: error_type = type(e).__name__ error_message = str(e) @@ -86,7 +92,7 @@ def safe_process_file(item, constants, run_timestamp, file_processing): logging.error(f"Error Message: {error_message}") logging.error(f"Full traceback:\n{full_traceback}") - return pd.DataFrame( + error_df = pd.DataFrame( [ { "FILE_NAME": file_id, @@ -96,6 +102,7 @@ def safe_process_file(item, constants, run_timestamp, file_processing): } ] ) + return error_df, error_df # Return tuple for consistency def main(client: str = "saas", testing=False, test_params={}): @@ -150,7 +157,7 @@ def main(client: str = "saas", testing=False, test_params={}): k: v for k, v in input_dict.items() if k in test_params["input_files"] } - logging.info(f"Total Input Files : {len(input_dict)}") + logging.debug(f"Total Input Files : {len(input_dict)}") # ========== COMMON: Document Type Classification ========== if config.PERFORM_DTC: @@ -172,12 +179,15 @@ def main(client: str = "saas", testing=False, test_params={}): llm_utils.warm_prompt_cache( instruction=instruction, cache_key=cache_key, model_id="sonnet_latest" ) - logging.info(f"Successfully warmed {len(cacheable_instructions)} prompt caches") + logging.debug( + f"Successfully warmed {len(cacheable_instructions)} prompt caches" + ) except Exception as e: logging.warning(f"Error warming caches (will proceed anyway): {str(e)}") # ========== CLIENT-SPECIFIC: File Processing ========== - successful_results = [] + successful_results_cc = [] + successful_results_dashboard = [] error_results = [] with concurrent.futures.ThreadPoolExecutor(max_workers=max_workers) as executor: futures = [ @@ -189,49 +199,71 @@ def main(client: str = "saas", testing=False, test_params={}): for future in concurrent.futures.as_completed(futures): try: - results = future.result() - if "error" in results.columns: - error_results.append(results) + cc_result, dashboard_result = future.result() + if "error" in cc_result.columns: + error_results.append(cc_result) else: - successful_results.append(results) + successful_results_cc.append(cc_result) + if ( + config.RUN_DASHBOARD_POSTPROCESSING + and dashboard_result is not None + ): + successful_results_dashboard.append(dashboard_result) except Exception as e: logging.error(f"Error processing future: {str(e)}") - error_results.append(pd.DataFrame([{"error": str(e)}])) + error_df = pd.DataFrame([{"error": str(e)}]) + error_results.append(error_df) # Combine results - if len(successful_results) > 0: - FINAL_RESULT_DF = pd.concat(successful_results) + if len(successful_results_cc) > 0: + FINAL_RESULT_DF_CC = pd.concat(successful_results_cc, ignore_index=True) + if successful_results_dashboard: + FINAL_RESULT_DF_DASHBOARD = pd.concat( + successful_results_dashboard, ignore_index=True + ) + else: + FINAL_RESULT_DF_DASHBOARD = pd.DataFrame() else: - FINAL_RESULT_DF = pd.DataFrame() + FINAL_RESULT_DF_CC = pd.DataFrame() + FINAL_RESULT_DF_DASHBOARD = pd.DataFrame() if len(error_results) > 0: - ERROR_RESULT_DF = pd.concat(error_results) + ERROR_RESULT_DF = pd.concat(error_results, ignore_index=True) else: ERROR_RESULT_DF = pd.DataFrame() # ========== COMMON: QC/QA Validation ========== - if config.ENABLE_QC_QA and not FINAL_RESULT_DF.empty: + # Run QC/QA validation by default (preserves automatic behavior for single files and batches) + # QC/QA runs on CC version only (dashboard version and error files do not need QC/QA) + validated_cc_df = None + qc_qa_stats_df = None + + # Run QC/QA on CC version if we have successful results + if not FINAL_RESULT_DF_CC.empty: logging.info("=" * 80) - logging.info("Running QC/QA Validation Pipeline...") + logging.info("Running QC/QA Validation Pipeline on CC results...") logging.info("=" * 80) try: - validated_df = run_qc_qa_pipeline(FINAL_RESULT_DF.copy(), stats_df=None) - logging.info("QC/QA validation completed successfully") + # Ensure unique index before QC/QA (safety check) + if FINAL_RESULT_DF_CC.index.has_duplicates: + logging.warning( + "Detected duplicate indices in FINAL_RESULT_DF_CC, resetting index..." + ) + FINAL_RESULT_DF_CC = FINAL_RESULT_DF_CC.reset_index(drop=True) - logging.info("Generating QC/QA statistics...") - qc_qa_stats_df = generate_statistics(validated_df) - logging.info("QC/QA statistics generated successfully") - - save_qc_qa_outputs( - validated_df=validated_df, - stats_df=qc_qa_stats_df, - run_timestamp=run_timestamp, - write_to_s3=config.WRITE_TO_S3, + validated_cc_df = run_qc_qa_pipeline( + FINAL_RESULT_DF_CC.copy(), stats_df=None ) + logging.debug("QC/QA validation on CC results completed successfully") + + # Generate validation statistics from CC version + logging.debug("Generating QC/QA statistics...") + qc_qa_stats_df = generate_statistics(validated_cc_df) + logging.debug("QC/QA statistics generated successfully") except Exception as e: - logging.error(f"QC/QA validation failed: {str(e)}") + logging.error(f"QC/QA validation on CC results failed: {str(e)}") logging.error(f"Full traceback:\n{traceback.format_exc()}") - logging.warning("Continuing with original unvalidated results...") + logging.warning("Continuing with original unvalidated CC results...") logging.info("=" * 80) # ========== COMMON: Output & Parent-Child ========== @@ -241,32 +273,93 @@ def main(client: str = "saas", testing=False, test_params={}): ) if config.WRITE_TO_S3: + # Write CC version doczy_output_for_pc = io_utils.write_s3( - FINAL_RESULT_DF, "", run_timestamp, "final" + FINAL_RESULT_DF_CC, "", run_timestamp, "cc_results_full" ) + # Write dashboard version (only when dashboard postprocessing is enabled) + if ( + config.RUN_DASHBOARD_POSTPROCESSING + and not FINAL_RESULT_DF_DASHBOARD.empty + ): + io_utils.write_s3( + FINAL_RESULT_DF_DASHBOARD, + "", + run_timestamp, + "dashboard_results_full", + ) + # Write error file if not ERROR_RESULT_DF.empty: io_utils.write_s3(ERROR_RESULT_DF, "", run_timestamp, "error") + # Write QC/QA outputs (only for CC version, not dashboard) + if validated_cc_df is not None: + io_utils.write_s3(validated_cc_df, "", run_timestamp, "qc_qa_cc_full") + if qc_qa_stats_df is not None: + io_utils.write_s3(qc_qa_stats_df, "", run_timestamp, "qc_qa_stats") + if not per_file_usage_df.empty and not batch_summary_df.empty: - logging.info("Exporting usage and cost tracking data to S3...") + logging.debug("Exporting usage and cost tracking data to S3...") io_utils.write_s3(per_file_usage_df, "", run_timestamp, "usage") io_utils.write_s3(batch_summary_df, "", run_timestamp, "usage_summary") else: + # Write CC version doczy_output_for_pc = io_utils.write_local( - FINAL_RESULT_DF, "", run_timestamp, "final" + FINAL_RESULT_DF_CC, "", run_timestamp, "cc_results_full" ) + # Write dashboard version (only when dashboard postprocessing is enabled) + if ( + config.RUN_DASHBOARD_POSTPROCESSING + and not FINAL_RESULT_DF_DASHBOARD.empty + ): + io_utils.write_local( + FINAL_RESULT_DF_DASHBOARD, + "", + run_timestamp, + "dashboard_results_full", + ) + # Write error file if not ERROR_RESULT_DF.empty: io_utils.write_local(ERROR_RESULT_DF, "", run_timestamp, "error") + # Write QC/QA outputs (only for CC version, not dashboard) + if validated_cc_df is not None: + io_utils.write_local( + validated_cc_df, "", run_timestamp, "qc_qa_cc_full" + ) + if qc_qa_stats_df is not None: + io_utils.write_local(qc_qa_stats_df, "", run_timestamp, "qc_qa_stats") + if config.PERFORM_PARENT_CHILD_MAPPING: - parent_child_main.main(doczy_output_for_pc, client=client) + if FINAL_RESULT_DF_CC.empty: + logging.warning( + "Skipping Parent-Child mapping: No successful results to process (FINAL_RESULT_DF_CC is empty)" + ) + else: + logging.debug( + f"Starting Parent-Child mapping with {len(FINAL_RESULT_DF_CC)} records from: {doczy_output_for_pc}" + ) + pc_output_dir = os.path.join( + config.CONSOLIDATED_OUTPUT_DIRECTORY, + run_timestamp, + "parent-child", + ) + _, pc_local_path = parent_child_main.main( + doczy_output_for_pc, + client=client, + output_dir=pc_output_dir, + ) + if config.WRITE_TO_S3 and pc_local_path: + io_utils.upload_local_file_to_s3( + pc_local_path, run_timestamp, "parent-child" + ) if not per_file_usage_df.empty and not batch_summary_df.empty: - logging.info("Exporting usage and cost tracking data locally...") + logging.debug("Exporting usage and cost tracking data locally...") io_utils.write_local(per_file_usage_df, "", run_timestamp, "usage") io_utils.write_local(batch_summary_df, "", run_timestamp, "usage_summary") else: - return FINAL_RESULT_DF, ERROR_RESULT_DF + return FINAL_RESULT_DF_CC, FINAL_RESULT_DF_DASHBOARD, ERROR_RESULT_DF if __name__ == "__main__": diff --git a/src/pipelines/saas/file_processing.py b/src/pipelines/saas/file_processing.py index 7aec330..eb1e276 100644 --- a/src/pipelines/saas/file_processing.py +++ b/src/pipelines/saas/file_processing.py @@ -40,7 +40,7 @@ def process_file(file_object, constants: Constants, run_timestamp): # With this, all logging calls in this thread will now route to logs/{filename}.log # This includes logging from all called functions (preprocess, one_to_n_funcs, etc.) logging_utils.set_current_file(filename) - logging.info(f"{datetime_str()} Processing {filename}...") + logging.debug(f"{datetime_str()} Processing {filename}...") # Set default values dynamic_one_to_one_fields = FieldSet() @@ -77,15 +77,17 @@ def process_file(file_object, constants: Constants, run_timestamp): filename=filename, ) ) - logging.info(f"{datetime_str()} Preprocessing Complete - {filename}") + logging.debug(f"{datetime_str()} Preprocessing Complete - {filename}") one_to_n_results = pd.DataFrame([{"FILE_NAME": filename}]) # Initialize here if string_utils.contains_reimbursement(contract_text): - logging.info( + logging.debug( f"{datetime_str()} Starting One-to-N extraction for {len(exhibit_chunk_mapping)} exhibits - {filename}" ) with timing_utils.timed_block("one_to_n_extraction", context=filename): + # Get specific fields to extract (if configured) + specific_fields = config.get_specific_fields_list() ( one_to_n_results, dynamic_one_to_one_fields, @@ -97,6 +99,7 @@ def process_file(file_object, constants: Constants, run_timestamp): all_exhibit_headers=all_exhibit_headers, constants=constants, filename=filename, + specific_fields=specific_fields, ) if not one_to_n_results.empty: one_to_n_results["FILE_NAME"] = filename @@ -106,22 +109,24 @@ def process_file(file_object, constants: Constants, run_timestamp): one_to_n_results = postprocessing_funcs.generate_reimb_ids( one_to_n_results ) - logging.info(f"{datetime_str()} One to N Complete - {filename}") + logging.debug(f"{datetime_str()} One to N Complete - {filename}") else: first_reimbursement_page = "1" - logging.info( + logging.debug( f"{datetime_str()} No Reimbursement Found, Skipping - {filename}" ) final_results = one_to_n_results # Set as final results else: first_reimbursement_page = "1" - logging.info( + logging.debug( f"{datetime_str()} Fields not configured for One to N, Skipping - {filename}" ) # ONE TO ONE PROCESSING if process_one_to_one: + # Get specific fields to extract (if configured) + specific_fields = config.get_specific_fields_list() with timing_utils.timed_block("one_to_one_extraction", context=filename): one_to_one_results = run_one_to_one_prompts( filename, @@ -131,9 +136,10 @@ def process_file(file_object, constants: Constants, run_timestamp): dynamic_one_to_one_fields, first_reimbursement_page, constants, + specific_fields=specific_fields, ) one_to_one_results["FILE_NAME"] = filename - logging.info(f"{datetime_str()} One to One Complete - {filename}") + logging.debug(f"{datetime_str()} One to One Complete - {filename}") # Decide how to handle one_to_one results with timing_utils.timed_block( @@ -148,7 +154,7 @@ def process_file(file_object, constants: Constants, run_timestamp): # ONLY one_to_one processed - convert dict to DataFrame final_results = pd.DataFrame([one_to_one_results]) else: - logging.info( + logging.debug( f"{datetime_str()} Fields not configured for One to One, Skipping - {filename}" ) @@ -157,23 +163,23 @@ def process_file(file_object, constants: Constants, run_timestamp): with timing_utils.timed_block("code_processing", context=filename): results_with_code = code_funcs.code_breakout(final_results, constants) final_results = code_funcs.grouper_breakout(results_with_code) - logging.info(f"{datetime_str()} Codes Complete - {filename}") + logging.debug(f"{datetime_str()} Codes Complete - {filename}") # POSTPROCESS with timing_utils.timed_block("postprocess", context=filename): - final_df = postprocess.postprocess(final_results, constants) - logging.info(f"{datetime_str()} Postprocessing Complete - {filename}") + cc_df, dashboard_df = postprocess.postprocess(final_results, constants) + logging.debug(f"{datetime_str()} Postprocessing Complete - {filename}") ################## WRITE INDIVIDUAL ################## with timing_utils.timed_block("write_individual", context=filename): if config.WRITE_TO_S3: - io_utils.write_s3(final_df, filename, run_timestamp, "individual") + io_utils.write_s3(cc_df, filename, run_timestamp, "individual_cc") else: - io_utils.write_local(final_df, filename, "", "individual") + io_utils.write_local(cc_df, filename, "", "individual_cc") - logging.info(f"{datetime_str()} Writing Complete - {filename}") + logging.debug(f"{datetime_str()} Writing Complete - {filename}") - return final_df + return cc_df, dashboard_df def run_one_to_one_prompts( @@ -184,6 +190,7 @@ def run_one_to_one_prompts( dynamic_one_to_one_fields: FieldSet, first_reimbursement_page: str, constants: Constants, + specific_fields: Optional[list] = None, ): ################## INITIALIZE FIELDS ################## @@ -191,31 +198,54 @@ def run_one_to_one_prompts( relationship="one_to_one", file_path=config.FIELD_JSON_PATH ).combine(dynamic_one_to_one_fields) + # Filter fields if specific_fields is provided + if specific_fields is not None: + one_to_one_fields = one_to_one_fields.filter_by_names(specific_fields) + + # Check if we need provider info fields + provider_fields = set(config.FIELD_GROUPS.get("provider", [])) + run_provider_info = specific_fields is None or any( + f in provider_fields for f in specific_fields + ) + ################## RUN PROVIDER INFO ################## - with timing_utils.timed_block("one_to_one.provider_info", context=filename): - one_to_one_results = tin_npi_funcs.run_provider_info_fields( - contract_text, text_dict, filename, payer_name="" - ) + one_to_one_results = {} + if run_provider_info: + with timing_utils.timed_block("one_to_one.provider_info", context=filename): + one_to_one_results = tin_npi_funcs.run_provider_info_fields( + contract_text, text_dict, filename, payer_name="" + ) ################## RUN HYBRID SMART CHUNKED PROMPTS ################## # RAG function loads retrieval questions internally and matches with investment_prompts.json - with timing_utils.timed_block("one_to_one.hybrid_smart_chunking", context=filename): - hybrid_smart_chunked_answers_dict = ( - hybrid_smart_chunking_funcs.run_hybrid_smart_chunked_fields( - one_to_one_fields, constants, contract_text, filename, text_dict + # Only run if there are fields to extract + hybrid_smart_chunked_answers_dict = {} + if one_to_one_fields.contains_fields(): + with timing_utils.timed_block( + "one_to_one.hybrid_smart_chunking", context=filename + ): + hybrid_smart_chunked_answers_dict = ( + hybrid_smart_chunking_funcs.run_hybrid_smart_chunked_fields( + one_to_one_fields, constants, contract_text, filename, text_dict + ) ) - ) + logging.debug( + f"Hybrid Smart Chunked Answers for {filename}: {hybrid_smart_chunked_answers_dict}" + ) ################## RUN FULL CONTEXT PROMPTS ################## - with timing_utils.timed_block("one_to_one.full_context", context=filename): - full_context_answers_dict = one_to_one_funcs.run_full_context_fields( - one_to_one_fields, - contract_text, - text_dict, - first_reimbursement_page, - constants, - filename, - ) + # Only run if there are fields to extract + full_context_answers_dict = {} + if one_to_one_fields.contains_fields(): + with timing_utils.timed_block("one_to_one.full_context", context=filename): + full_context_answers_dict = one_to_one_funcs.run_full_context_fields( + one_to_one_fields, + contract_text, + text_dict, + first_reimbursement_page, + constants, + filename, + ) ################## COMBINE ANSWERS ################ # Prefer HSC answers over full_context when HSC value is not N/A for key, value in full_context_answers_dict.items(): @@ -228,19 +258,25 @@ def run_one_to_one_prompts( # Add HSC's N/A if field doesn't exist yet one_to_one_results[key] = value - one_to_one_results = tin_npi_funcs.merge_provider_info_with_one_to_one( - one_to_one_results, filename - ) + if run_provider_info: + one_to_one_results = tin_npi_funcs.merge_provider_info_with_one_to_one( + one_to_one_results, filename + ) ################## ADD AD FIELDS ################ - one_to_one_results = one_to_one_funcs.get_aarete_derived_dates( - one_to_one_results, text_dict, filename - ) + # Only run if date fields are requested + date_fields = set(config.FIELD_GROUPS.get("dates", [])) + if specific_fields is None or any(f in date_fields for f in specific_fields): + one_to_one_results = one_to_one_funcs.get_aarete_derived_dates( + one_to_one_results, text_dict, filename + ) ################## ADD SIGNATURE COUNT ################## - one_to_one_results = one_to_one_funcs.add_signature_count( - one_to_one_results, contract_text - ) + # Only run if signature count is requested + if specific_fields is None or "NUM_SIGNED_SIGNATORY_LINE_COUNT" in specific_fields: + one_to_one_results = one_to_one_funcs.add_signature_count( + one_to_one_results, contract_text + ) ################## Crosswalk Fields ################## # All field format normalization is handled at prompt_calls level via field-aware parsers @@ -264,6 +300,7 @@ def process_page_reimbursements( exhibit_page_nums: Optional[list[str]] = None, constants: Optional[Constants] = None, filename: Optional[str] = None, + specific_fields: Optional[list] = None, ): """ STEP 1 HELPER: Extract reimbursements from a single page (without exhibit-level fields). @@ -277,6 +314,9 @@ def process_page_reimbursements( exhibit_page_nums: List of page numbers in the exhibit (for backward compatibility) constants: Constants object filename: Name of the file being processed + specific_fields: Optional list of specific field names to extract. + If provided with exhibit-level-only fields (like claim_type), + skip detailed reimbursement extraction. """ # Get page text - prefer using exhibit.get_page_text() if available if exhibit.pages is not None: @@ -325,6 +365,18 @@ def process_page_reimbursements( if not reimbursement_level_answers: return [], [] + # If specific_fields is set and doesn't include reimbursement-level fields, + # skip detailed extraction (carveouts, lesser_of, breakout) - just return + # minimal info to trigger exhibit_level processing + if specific_fields is not None: + # Check if any reimbursement-level fields are requested + reimbursement_fields = {"SERVICE_TERM", "REIMB_TERM", "REIMB_PAGE"} + if not any(f in reimbursement_fields for f in specific_fields): + # Just return minimal reimbursement info to indicate this page has reimbursements + for answer_dict in reimbursement_level_answers: + answer_dict["REIMB_PAGE"] = page_num + return reimbursement_level_answers, [] + ################################ Carveouts and Special Case ################################ reimbursement_level_answers, special_case_answers = ( one_to_n_funcs.carveout_and_special_case( @@ -365,6 +417,7 @@ def run_one_to_n_prompts( all_exhibit_headers: Optional[dict[str, str]] = None, constants: Optional[Constants] = None, filename: Optional[str] = None, + specific_fields: Optional[list] = None, ): """ 3-STEP APPROACH (per exhibit): @@ -379,6 +432,8 @@ def run_one_to_n_prompts( all_exhibit_headers: Dictionary mapping exhibit pages to their headers constants: Constants object filename: Name of the file being processed + specific_fields: Optional list of specific field names to extract. + If provided, only these fields will be extracted at exhibit level. """ if exhibit_chunk_mapping is None: exhibit_chunk_mapping = {} @@ -420,6 +475,7 @@ def run_one_to_n_prompts( exhibit_pages, constants, filename, + specific_fields, ): page_num for page_num in exhibit_pages } @@ -450,6 +506,7 @@ def run_one_to_n_prompts( exhibit.exhibit_page, constants, filename, + specific_fields=specific_fields, ) ) exhibit.set_exhibit_level_data( @@ -459,49 +516,68 @@ def run_one_to_n_prompts( f"Exhibit Level Answers for {filename}, Page {exhibit.exhibit_page}: {exhibit_level_answers}" ) - # STEP 3: dynamic assignment & combine for this exhibit - # Get dynamic fields from previous exhibit if needed (for future use) - if exhibit.dynamic_reimbursement_fields: - dynamic_fields = exhibit.dynamic_reimbursement_fields - elif ( - exhibit.prev_exhibit - and not exhibit.prev_exhibit.has_reimbursements - and exhibit.get_previous_exhibit_dynamic_fields().fields - ): - dynamic_fields = exhibit.get_previous_exhibit_dynamic_fields() - else: - dynamic_fields = None + # Check if we need reimbursement-level processing + # If specific_fields is set and doesn't include reimbursement fields, skip Step 3 + reimbursement_fields = {"SERVICE_TERM", "REIMB_TERM", "REIMB_PAGE"} + skip_reimbursement_processing = specific_fields is not None and not any( + f in reimbursement_fields for f in specific_fields + ) - # Run dynamic assignment for ALL reimbursement rows in this exhibit - # Note: dynamic_assignment expects a list of rows and returns a list of rows - reimbursement_rows_with_dynamic = exhibit.reimbursement_rows - if exhibit.reimbursement_rows and dynamic_fields is not None: - reimbursement_rows_with_dynamic = dynamic_funcs.dynamic_assignment( - exhibit.reimbursement_rows, - dynamic_fields, - pages_dict, - text_dict, - exhibit, - constants, + if skip_reimbursement_processing: + # For exhibit-level-only extraction (like claim_type), just create minimal rows + # with exhibit_level_answers to preserve the data + if exhibit.exhibit_level_answers: + minimal_row = exhibit.exhibit_level_answers.copy() + minimal_row["REIMB_PAGE"] = exhibit.exhibit_page + # Apply crosswalk mapping for derived fields (e.g., CLAIM_TYPE_CD -> AARETE_DERIVED_CLAIM_TYPE_CD) + minimal_rows_with_crosswalk = aarete_derived.get_crosswalk_fields( + [minimal_row], constants + ) + one_to_n_results.extend(minimal_rows_with_crosswalk) + else: + # STEP 3: dynamic assignment & combine for this exhibit + # Get dynamic fields from previous exhibit if needed (for future use) + if exhibit.dynamic_reimbursement_fields: + dynamic_fields = exhibit.dynamic_reimbursement_fields + elif ( + exhibit.prev_exhibit + and not exhibit.prev_exhibit.has_reimbursements + and exhibit.get_previous_exhibit_dynamic_fields().fields + ): + dynamic_fields = exhibit.get_previous_exhibit_dynamic_fields() + else: + dynamic_fields = None + + # Run dynamic assignment for ALL reimbursement rows in this exhibit + # Note: dynamic_assignment expects a list of rows and returns a list of rows + reimbursement_rows_with_dynamic = exhibit.reimbursement_rows + if exhibit.reimbursement_rows and dynamic_fields is not None: + reimbursement_rows_with_dynamic = dynamic_funcs.dynamic_assignment( + exhibit.reimbursement_rows, + dynamic_fields, + pages_dict, + text_dict, + exhibit, + constants, + filename, + ) + + # Combine reimbursement rows with exhibit-level answers + combined_rows = row_funcs.combine_one_to_n( + exhibit.exhibit_text, + reimbursement_rows_with_dynamic, + exhibit.special_case_rows, + exhibit.exhibit_level_answers, filename, ) - # Combine reimbursement rows with exhibit-level answers - combined_rows = row_funcs.combine_one_to_n( - exhibit.exhibit_text, - reimbursement_rows_with_dynamic, - exhibit.special_case_rows, - exhibit.exhibit_level_answers, - filename, - ) + # Run cleaning + combined_rows = one_to_n_funcs.one_to_n_cleaning( + combined_rows, exhibit.exhibit_text, constants, filename + ) - # Run cleaning - combined_rows = one_to_n_funcs.one_to_n_cleaning( - combined_rows, exhibit.exhibit_text, constants, filename - ) - - exhibit.final_rows = combined_rows - one_to_n_results.extend(combined_rows) + exhibit.final_rows = combined_rows + one_to_n_results.extend(combined_rows) # Track first reimbursement page if first_reimbursement_page == "1" and exhibit.has_reimbursements: diff --git a/src/pipelines/saas/main.py b/src/pipelines/saas/main.py index 0a74473..9eb9ffb 100644 --- a/src/pipelines/saas/main.py +++ b/src/pipelines/saas/main.py @@ -1,5 +1,6 @@ import concurrent.futures import logging +import os import random import time import traceback @@ -10,6 +11,9 @@ import pandas as pd random.seed(42) import src.pipelines.saas.file_processing as file_processing +from src.pipelines.shared.extraction import one_to_one_funcs +from src.pipelines.shared.postprocessing import postprocessing_funcs +from src.constants.investment_columns import FIELD_FORMAT_MAPPING import src.prompts.prompt_templates as prompt_templates import src.utils.io_utils as io_utils import src.utils.llm_utils as llm_utils @@ -22,6 +26,8 @@ from src.parent_child import main as parent_child_main from src.document_classification import main as dtc_main from src.qc_qa.pipeline import ( generate_statistics, + generate_tin_contract_summary, + generate_tin_statistics, run_qc_qa_pipeline, save_qc_qa_outputs, ) @@ -40,20 +46,22 @@ def safe_process_file(item, constants, run_timestamp): run_timestamp: Timestamp string for output file naming and tracking Returns: - pd.DataFrame: Either: - - DataFrame with extracted field results (success case) - - Single-row DataFrame with error details (failure case) + tuple: Either: + - (cc_df, dashboard_df) tuple with extracted field results (success case) + - (error_df, error_df) tuple with error details (failure case) Note: - Always returns a DataFrame to maintain consistent output format for pd.concat. + Returns both CC and dashboard versions from file processing. With FAISS migration, no special resource cleanup is needed. """ file_id = ( item[0] if isinstance(item, (list, tuple)) and len(item) > 0 else item ) # unpack file_id from item (passed in as a tuple below) try: - result = file_processing.process_file(item, constants, run_timestamp) - return result + cc_df, dashboard_df = file_processing.process_file( + item, constants, run_timestamp + ) + return cc_df, dashboard_df except ( Exception ) as e: # When there's an issue with the processing inside the future @@ -67,7 +75,7 @@ def safe_process_file(item, constants, run_timestamp): logging.error(f"Full traceback:\n{full_traceback}") - return pd.DataFrame( + error_df = pd.DataFrame( [ { "FILE_NAME": file_id, @@ -81,6 +89,7 @@ def safe_process_file(item, constants, run_timestamp): } ] ) # Return a single-row dataframe so we can still concat it later + return error_df, error_df # Return tuple for consistency def main(testing=False, test_params={}): @@ -160,7 +169,8 @@ def main(testing=False, test_params={}): logging.warning(f"Error warming caches (will proceed anyway): {str(e)}") # Process files concurrently - each thread gets its own per-file logging context - successful_results = [] + successful_results_cc = [] + successful_results_dashboard = [] error_results = [] logging.info(f"Starting parallel file processing with {max_workers} workers...") @@ -178,11 +188,16 @@ def main(testing=False, test_params={}): completed_count = 0 for future in concurrent.futures.as_completed(futures): try: - results = future.result() - if "error" in results.columns: - error_results.append(results) + cc_result, dashboard_result = future.result() + if "error" in cc_result.columns: + error_results.append(cc_result) else: - successful_results.append(results) + successful_results_cc.append(cc_result) + if ( + config.RUN_DASHBOARD_POSTPROCESSING + and dashboard_result is not None + ): + successful_results_dashboard.append(dashboard_result) completed_count += 1 if completed_count % 5 == 0: # Log progress every 5 files @@ -194,54 +209,97 @@ def main(testing=False, test_params={}): ) as e: # When there's an issue with the future itself (not the processing inside the future) logging.error(f"Error processing future: {str(e)}") # Coerce this to the format expected by pd.concat (so a single-row dataframe) - error_results.append(pd.DataFrame([{"error": str(e)}])) + error_df = pd.DataFrame([{"error": str(e)}]) + error_results.append(error_df) # only concat if we have results with timing_utils.timed_block("concat_results", log_level="INFO"): - if len(successful_results) > 0: - FINAL_RESULT_DF = pd.concat(successful_results, ignore_index=True) + if len(successful_results_cc) > 0: + FINAL_RESULT_DF_CC = pd.concat(successful_results_cc, ignore_index=True) + if successful_results_dashboard: + FINAL_RESULT_DF_DASHBOARD = pd.concat( + successful_results_dashboard, ignore_index=True + ) + else: + FINAL_RESULT_DF_DASHBOARD = pd.DataFrame() else: - FINAL_RESULT_DF = pd.DataFrame() + FINAL_RESULT_DF_CC = pd.DataFrame() + FINAL_RESULT_DF_DASHBOARD = pd.DataFrame() + + if not FINAL_RESULT_DF_CC.empty: + FINAL_RESULT_DF_CC = one_to_one_funcs.add_aarete_derived_payer_name( + FINAL_RESULT_DF_CC, config.STATE_FLAG + ) # add aarete derived payer name to cc version + FINAL_RESULT_DF_CC = one_to_one_funcs.add_aarete_derived_provider_name( + FINAL_RESULT_DF_CC, config.STATE_FLAG + ) # add aarete derived provider name to cc version + if not FINAL_RESULT_DF_DASHBOARD.empty: + FINAL_RESULT_DF_DASHBOARD = one_to_one_funcs.add_aarete_derived_payer_name( + FINAL_RESULT_DF_DASHBOARD, config.STATE_FLAG + ) # add aarete derived payer name to dashboard version + FINAL_RESULT_DF_DASHBOARD = ( + one_to_one_funcs.add_aarete_derived_provider_name( + FINAL_RESULT_DF_DASHBOARD, config.STATE_FLAG + ) + ) # add aarete derived provider name to dashboard version + + FINAL_RESULT_DF_CC = postprocessing_funcs.reorder_columns( + FINAL_RESULT_DF_CC, FIELD_FORMAT_MAPPING + ) # Reorder columns in CC version only + FINAL_RESULT_DF_DASHBOARD = postprocessing_funcs.reorder_columns( + FINAL_RESULT_DF_DASHBOARD, FIELD_FORMAT_MAPPING + ) # Reorder columns in dashboard version only if len(error_results) > 0: ERROR_RESULT_DF = pd.concat(error_results, ignore_index=True) else: ERROR_RESULT_DF = pd.DataFrame() - # Run QC/QA validation if enabled (creates separate validated copy) - if config.ENABLE_QC_QA and not FINAL_RESULT_DF.empty: + # Run QC/QA validation by default (preserves automatic behavior for single files and batches) + # QC/QA runs on CC version only (dashboard version and error files do not need QC/QA) + validated_cc_df = None + qc_qa_stats_df = None + + # Run QC/QA on CC version if we have successful results + if not FINAL_RESULT_DF_CC.empty: logging.info("=" * 80) - logging.info("Running QC/QA Validation Pipeline...") + logging.info("Running QC/QA Validation Pipeline on CC results...") logging.info("=" * 80) with timing_utils.timed_block("qc_qa_validation", log_level="INFO"): try: # Ensure unique index before QC/QA (safety check) - if FINAL_RESULT_DF.index.has_duplicates: + if FINAL_RESULT_DF_CC.index.has_duplicates: logging.warning( - "Detected duplicate indices in FINAL_RESULT_DF, resetting index..." + "Detected duplicate indices in FINAL_RESULT_DF_CC, resetting index..." ) - FINAL_RESULT_DF = FINAL_RESULT_DF.reset_index(drop=True) + FINAL_RESULT_DF_CC = FINAL_RESULT_DF_CC.reset_index(drop=True) - # Create a copy for validation - preserve original FINAL_RESULT_DF - validated_df = run_qc_qa_pipeline(FINAL_RESULT_DF.copy(), stats_df=None) - logging.info("QC/QA validation completed successfully") + # Create a copy for validation - preserve original FINAL_RESULT_DF_CC + validated_cc_df = run_qc_qa_pipeline( + FINAL_RESULT_DF_CC.copy(), stats_df=None + ) + logging.info("QC/QA validation on CC results completed successfully") - # Generate validation statistics + # Generate validation statistics from CC version logging.info("Generating QC/QA statistics...") - qc_qa_stats_df = generate_statistics(validated_df) + qc_qa_stats_df = generate_statistics(validated_cc_df) + tin_stats_df = generate_tin_statistics(validated_cc_df) + tin_contract_summary_df = generate_tin_contract_summary(validated_cc_df) logging.info("QC/QA statistics generated successfully") # Save QC/QA outputs (local + S3) - separate from main results save_qc_qa_outputs( - validated_df=validated_df, + validated_df=validated_cc_df, stats_df=qc_qa_stats_df, run_timestamp=run_timestamp, write_to_s3=config.WRITE_TO_S3, + tin_stats_df=tin_stats_df, + tin_contract_summary_df=tin_contract_summary_df, ) except Exception as e: - logging.error(f"QC/QA validation failed: {str(e)}") + logging.error(f"QC/QA validation on CC results failed: {str(e)}") logging.error(f"Full traceback:\n{traceback.format_exc()}") - logging.warning("Continuing with original unvalidated results...") + logging.warning("Continuing with original unvalidated CC results...") logging.info("=" * 80) if not testing: @@ -252,15 +310,36 @@ def main(testing=False, test_params={}): with timing_utils.timed_block("write_outputs", log_level="INFO"): if config.WRITE_TO_S3: + # Write CC version doczy_output_for_pc = io_utils.write_s3( - FINAL_RESULT_DF, "", run_timestamp, "final" + FINAL_RESULT_DF_CC, "", run_timestamp, "cc_results_full" ) + # Write dashboard version (only when dashboard postprocessing is enabled) + if ( + config.RUN_DASHBOARD_POSTPROCESSING + and not FINAL_RESULT_DF_DASHBOARD.empty + ): + io_utils.write_s3( + FINAL_RESULT_DF_DASHBOARD, + "", + run_timestamp, + "dashboard_results_full", + ) + # Write error file if not ERROR_RESULT_DF.empty: io_utils.write_s3(ERROR_RESULT_DF, "", run_timestamp, "error") logging.warning( f"{len(ERROR_RESULT_DF)} files failed processing - error results written to S3" ) + # Write QC/QA outputs (only for CC version, not dashboard) + if validated_cc_df is not None: + io_utils.write_s3( + validated_cc_df, "", run_timestamp, "qc_qa_cc_full" + ) + if qc_qa_stats_df is not None: + io_utils.write_s3(qc_qa_stats_df, "", run_timestamp, "qc_qa_stats") + # Export usage and cost tracking data to S3 only if both dataframes have data if not per_file_usage_df.empty and not batch_summary_df.empty: logging.info("Exporting usage and cost tracking data to S3...") @@ -269,23 +348,46 @@ def main(testing=False, test_params={}): batch_summary_df, "", run_timestamp, "usage_summary" ) else: + # Write CC version doczy_output_for_pc = io_utils.write_local( - FINAL_RESULT_DF, "", run_timestamp, "final" + FINAL_RESULT_DF_CC, "", run_timestamp, "cc_results_full" ) + # Write dashboard version (only when dashboard postprocessing is enabled) + if ( + config.RUN_DASHBOARD_POSTPROCESSING + and not FINAL_RESULT_DF_DASHBOARD.empty + ): + io_utils.write_local( + FINAL_RESULT_DF_DASHBOARD, + "", + run_timestamp, + "dashboard_results_full", + ) + # Write error file if not ERROR_RESULT_DF.empty: io_utils.write_local(ERROR_RESULT_DF, "", run_timestamp, "error") logging.warning( f"{len(ERROR_RESULT_DF)} files failed processing - error results written locally" ) + # Write QC/QA outputs (only for CC version, not dashboard) + if validated_cc_df is not None: + io_utils.write_local( + validated_cc_df, "", run_timestamp, "qc_qa_cc_full" + ) + if qc_qa_stats_df is not None: + io_utils.write_local( + qc_qa_stats_df, "", run_timestamp, "qc_qa_stats" + ) + if config.PERFORM_PARENT_CHILD_MAPPING: - if FINAL_RESULT_DF.empty: + if FINAL_RESULT_DF_CC.empty: logging.warning( - "Skipping Parent-Child mapping: No successful results to process (FINAL_RESULT_DF is empty)" + "Skipping Parent-Child mapping: No successful results to process (FINAL_RESULT_DF_CC is empty)" ) else: logging.info( - f"Starting Parent-Child mapping with {len(FINAL_RESULT_DF)} records from: {doczy_output_for_pc}" + f"Starting Parent-Child mapping with {len(FINAL_RESULT_DF_CC)} records from: {doczy_output_for_pc}" ) with timing_utils.timed_block("parent_child_mapping", log_level="INFO"): client = ( @@ -295,7 +397,20 @@ def main(testing=False, test_params={}): and config.CLIENT[0] != "None" else None ) - parent_child_main.main(doczy_output_for_pc, client=client) + pc_output_dir = os.path.join( + config.CONSOLIDATED_OUTPUT_DIRECTORY, + run_timestamp, + "parent-child", + ) + _, pc_local_path = parent_child_main.main( + doczy_output_for_pc, + client=client, + output_dir=pc_output_dir, + ) + if config.WRITE_TO_S3 and pc_local_path: + io_utils.upload_local_file_to_s3( + pc_local_path, run_timestamp, "parent-child" + ) # Export usage and cost tracking data locally only if both dataframes have data if not per_file_usage_df.empty and not batch_summary_df.empty: @@ -310,7 +425,7 @@ def main(testing=False, test_params={}): logging.info("=" * 80) timing_utils.print_hierarchical_timing_summary() else: - return FINAL_RESULT_DF, ERROR_RESULT_DF + return FINAL_RESULT_DF_CC, FINAL_RESULT_DF_DASHBOARD, ERROR_RESULT_DF if __name__ == "__main__": diff --git a/src/pipelines/saas/prompts/prompt_calls.py b/src/pipelines/saas/prompts/prompt_calls.py index 3c0f53d..eb06b3e 100644 --- a/src/pipelines/saas/prompts/prompt_calls.py +++ b/src/pipelines/saas/prompts/prompt_calls.py @@ -87,10 +87,12 @@ def prompt_exhibit_level_breakout( def prompt_dynamic_primary( exhibit_text: str, field: Field, constants: Constants, filename: str, TEMPLATE ): + """Extract dynamic primary field from exhibit text.""" prompt, _parser = TEMPLATE( exhibit_text, field.field_name, field.get_prompt(constants) ) logging.debug(f"Dynamic primary prompt for {filename}; {field}: {prompt}") + llm_answer_raw = llm_utils.invoke_claude( prompt, "sonnet_latest", @@ -99,9 +101,8 @@ def prompt_dynamic_primary( instruction=prompt_templates.DYNAMIC_PRIMARY_INSTRUCTION(), ) logging.debug(f"Claude answer for {filename}; {field}: {llm_answer_raw}") - llm_answer_final = _parser(llm_answer_raw) - # Normalization is already done in the JSON parser + llm_answer_final = _parser(llm_answer_raw) return llm_answer_final @@ -411,13 +412,15 @@ def prompt_full_context( # Extract field names for field-aware normalization field_names = full_context_fields.list_fields() + context_text = contract_text[ + 0 : min( + config.MAX_CONTEXT_LENGTH - len(prompt_questions), + len(contract_text) - 1, + ) + ] + full_context_prompt, _parser = prompt_templates.ONE_TO_ONE_MULTI_FIELD_TEMPLATE( - context=contract_text[ - 0 : min( - config.MAX_CONTEXT_LENGTH - len(prompt_questions), - len(contract_text) - 1, - ) - ], + context=context_text, questions=prompt_questions, field_names=field_names, ) @@ -433,6 +436,7 @@ def prompt_full_context( ) full_context_answers_dict = _parser(claude_answer_raw) + # Field-aware normalization is already done in the JSON parser # No additional normalization needed for fields in FIELD_FORMAT_MAPPING except Exception as e: @@ -756,6 +760,9 @@ def prompt_lesser_of_distribution( instruction=prompt_templates.LESSER_OF_DISTRIBUTION_INSTRUCTION(), usage_label="LESSER_OF_DISTRIBUTION", ) + logging.debug( + f"LLM raw output for lesser-of distribution on page {page_num} for service '{service_term}': {llm_answer_raw}" + ) # Check if LLM determined no lesser-of applies if "N/A" in llm_answer_raw: @@ -1076,3 +1083,115 @@ def prompt_provider_info( logging.error(f"Raw response: {llm_answer_raw}") # Return a default value return [{"TIN": "N/A", "NPI": "N/A", "NAME": "N/A"}] + + +def prompt_aarete_derived_payer_name(cluster_rows: list[dict], state_flag: bool) -> str: + """ + Select the canonical payer name from a cluster of similar names (e.g., merging 'AvMed, Inc' and 'AvMed Health Plans'). + Expects a list of standardized payer name dictionaries. + + Args: + cluster_rows (list[dict]): A list of dictionaries, each containing standardized payer name information. + + Returns: + str: The canonical payer name selected by the LLM, or "N/A" if selection fails. + """ + + # If only one record in cluster, avoid LLM call + if len(cluster_rows) == 1: + base_name = cluster_rows[0].get("base_name", "UNKNOWN") + return str(base_name).upper() + + prompt, _parser = prompt_templates.AARETE_DERIVED_PAYER_NAME( + cluster_rows, state_flag + ) + logging.debug(f"Aarete Derived Payer Name Prompt: {prompt}") + llm_answer_raw = llm_utils.invoke_claude( + prompt, + "sonnet_latest", + "all_files", + cache=True, + instruction=prompt_templates.AARETE_DERIVED_PAYER_NAME_INSTRUCTION(), + usage_label="AARETE_DERIVED_PAYER_NAME", + ) + logging.debug(f"LLM Response for Aarete Derived Payer Name: {llm_answer_raw}") + try: + llm_answer_final = _parser(llm_answer_raw) + if ( + llm_answer_final + and "AARETE_DERIVED_PAYER_NAME" in llm_answer_final + and not string_utils.is_empty(llm_answer_final["AARETE_DERIVED_PAYER_NAME"]) + ): + derived_name = ( + llm_answer_final["AARETE_DERIVED_PAYER_NAME"] + .replace("```", "") + .replace("**", "") + .strip() + .upper() + ) + return derived_name + else: + # If LLM returns N/A or empty, fallback to the programmatic base_name + fallback = cluster_rows[0].get("base_name", "UNKNOWN") + return str(fallback).upper() + + except Exception as e: + logging.error( + f"Error parsing LLM response for Aarete Derived Payer Name: {str(e)}" + ) + return str(cluster_rows[0].get("base_name", "UNKNOWN")).upper() + + +def prompt_aarete_derived_provider_name( + cluster_rows: list[dict], state_flag: bool +) -> str: + """ + Select canonical provider name using LLM. + """ + + # Skip LLM if only one record + if len(cluster_rows) == 1: + base_name = cluster_rows[0].get("base_name", "UNKNOWN") + return str(base_name).upper() + + prompt, _parser = prompt_templates.AARETE_DERIVED_PROVIDER_NAME( + cluster_rows, state_flag + ) + logging.debug(f"Aarete Derived Provider Name Prompt: {prompt}") + llm_answer_raw = llm_utils.invoke_claude( + prompt, + "sonnet_latest", + "all_files", + cache=True, + instruction=prompt_templates.AARETE_DERIVED_PROVIDER_NAME_INSTRUCTION(), + usage_label="AARETE_DERIVED_PROVIDER_NAME", + ) + logging.debug(f"LLM Response for Aarete Derived Provider Name: {llm_answer_raw}") + try: + llm_answer_final = _parser(llm_answer_raw) + if ( + llm_answer_final + and "AARETE_DERIVED_PROVIDER_NAME" in llm_answer_final + and not string_utils.is_empty( + llm_answer_final["AARETE_DERIVED_PROVIDER_NAME"] + ) + ): + derived_name = ( + llm_answer_final["AARETE_DERIVED_PROVIDER_NAME"] + .replace("```", "") + .replace("**", "") + .strip() + .upper() + ) + return derived_name + + # Explicit fallback + fallback = cluster_rows[0].get("base_name", "UNKNOWN") + return str(fallback).upper() + + except Exception as e: + logging.error( + f"Error parsing LLM response for Aarete Derived Provider Name: {str(e)}" + ) + fallback = cluster_rows[0].get("base_name", "UNKNOWN") + return str(fallback).upper() diff --git a/src/pipelines/shared/extraction/dynamic_funcs.py b/src/pipelines/shared/extraction/dynamic_funcs.py index ad51855..188e472 100644 --- a/src/pipelines/shared/extraction/dynamic_funcs.py +++ b/src/pipelines/shared/extraction/dynamic_funcs.py @@ -4,7 +4,7 @@ from typing import TYPE_CHECKING, Optional from src.pipelines.saas.prompts import prompt_calls from src.pipelines.shared.preprocessing import preprocessing_funcs -from src.utils import string_utils +from src.utils import string_utils, timing_utils from src.constants.constants import Constants from src import config from src.prompts import prompt_templates @@ -42,7 +42,7 @@ def dynamic_primary( dynamic_reimbursement_fields = FieldSet() exhibit_level_answer_dict = {} - for field in dynamic_primary_fields.fields: + for field in list(dynamic_primary_fields.fields): exhibit_text_answer = prompt_calls.prompt_dynamic_primary( exhibit_text, field, @@ -219,8 +219,16 @@ def dynamic_assignment( return updated max_workers = min(len(reimbursement_primary_answers), 8) - with concurrent.futures.ThreadPoolExecutor(max_workers=max_workers) as executor: - return list(executor.map(_assign_single, reimbursement_primary_answers)) + with timing_utils.timed_block( + "dynamic_assignment.parallel_processing", context=filename + ): + with concurrent.futures.ThreadPoolExecutor(max_workers=max_workers) as executor: + results = list(executor.map(_assign_single, reimbursement_primary_answers)) + + logging.debug( + f"Dynamic assignment complete: {len(results)} rows processed with {len(dynamic_reimbursement_fields.fields)} dynamic fields - {filename}" + ) + return results def add_one_to_one_field( @@ -230,25 +238,26 @@ def add_one_to_one_field( field_to_add.relationship = "one_to_one" # Special Handling for Program, Product, and Network: - # If any LOB value has been found, skip PROGRAM, PRODUCT, and NETWORK - # Only search for these fields when NO LOB has been found + # Skip when full LOB (all rows have LOB). When partial LOB, pass PROGRAM/PRODUCT; + # merge fills only empty cells, so 1:N values are preserved. if field_to_add.field_name in ["PROGRAM", "PRODUCT", "NETWORK"]: - # LOB can be a string or a list (from JSON format) - # Extract all LOB values, handling both string and list formats lob_values = [] for answer_dict in answer_dicts: lob_value = answer_dict.get("LOB", "") if isinstance(lob_value, list): - # If it's a list, extract each item and ensure it's a string for item in lob_value: if not string_utils.is_empty(item): lob_values.append(str(item)) elif not string_utils.is_empty(lob_value): - # If it's a string, add it directly (already hashable) lob_values.append(lob_value) - unique_lobs = set(lob_values) - if len(unique_lobs) > 0: + has_lob = len(set(lob_values)) > 0 + lob_empty_count = sum( + 1 for d in answer_dicts if string_utils.is_empty(d.get("LOB")) + ) + partial_lob = has_lob and lob_empty_count > 0 + # Skip only when full LOB; allow PROGRAM, PRODUCT when partial + if has_lob and not partial_lob: return one_to_one_fields if field_to_add.field_name == "CLAIM_TYPE_CD": @@ -313,6 +322,13 @@ def get_dynamic_one_to_one_fields( unique_lobs = set(lob_values) has_lob = len(unique_lobs) > 0 + # Partial LOB: some rows have LOB, some don't (e.g. stripped headers) + # When partial, pass LOB+PROGRAM+PRODUCT to 1:1; merge fills only empty cells + lob_empty_count = sum( + 1 for d in answer_dicts if string_utils.is_empty(d.get("LOB")) + ) + partial_lob = has_lob and lob_empty_count > 0 + # Handle ALL empty fields for field in all_empty_fields.fields: field_name = field.field_name @@ -325,11 +341,25 @@ def get_dynamic_one_to_one_fields( if string_utils.is_empty(answer_dict.get(field_name)) ) total_count = len(answer_dicts) + base = field.base_field if field.base_field else field_name - if empty_count == total_count: - # Skip PROGRAM, PRODUCT, NETWORK if any LOB has been found - # Only search for these fields when NO LOB has been found - if has_lob and field_name in ["PROGRAM", "PRODUCT", "NETWORK"]: + # LOB: pass when empty in ANY row (partial detection due to stripped headers) + # PROGRAM, PRODUCT: also pass when partial_lob and empty in any row (LOB relationship) + # Others: pass only when empty in ALL rows + should_pass_to_1to1 = ( + empty_count == total_count + or (base == "LOB" and empty_count > 0) + or (partial_lob and base in ["PROGRAM", "PRODUCT"] and empty_count > 0) + ) + + if should_pass_to_1to1: + # Skip PROGRAM, PRODUCT, NETWORK when full LOB (all rows have LOB) + # Do NOT skip when partial_lob; pass PROGRAM, PRODUCT for consistency + if ( + has_lob + and not partial_lob + and base in ["PROGRAM", "PRODUCT", "NETWORK"] + ): continue field_to_add = Field.load_from_file( @@ -339,8 +369,5 @@ def get_dynamic_one_to_one_fields( one_to_one_fields = add_one_to_one_field( one_to_one_fields, field_to_add, answer_dicts, constants ) - else: - # Field found in some rows - keep in 1:N - pass return one_to_one_fields diff --git a/src/pipelines/shared/extraction/one_to_n_funcs.py b/src/pipelines/shared/extraction/one_to_n_funcs.py index 56c129a..ac37b3f 100644 --- a/src/pipelines/shared/extraction/one_to_n_funcs.py +++ b/src/pipelines/shared/extraction/one_to_n_funcs.py @@ -4,7 +4,7 @@ import logging from src.pipelines.shared.extraction import dynamic_funcs from src.pipelines.saas.prompts import prompt_calls from src.pipelines.shared.postprocessing import aarete_derived, postprocessing_funcs -from src.utils import llm_utils, string_utils +from src.utils import llm_utils, string_utils, timing_utils import src.prompts.prompt_templates as prompt_templates from src.constants.constants import Constants from src import config @@ -18,6 +18,7 @@ def exhibit_level( exhibit_page: str, constants: Constants, filename: str, + specific_fields: list = None, ) -> tuple[dict, FieldSet]: """ Processes exhibit-level fields, including dynamic fields. @@ -28,6 +29,8 @@ def exhibit_level( reimbursement_level_fields (FieldSet): A FieldSet object containing reimbursement-level field prompts. constants (Constants): A Constants object filename (str): The name of the file + specific_fields (list): Optional list of specific field names to extract. + If provided, only these fields will be extracted. Returns: dict: All Exhibit-Level answers FieldSet: Updated reimbursement_level_fields with any dynamically added fields @@ -41,41 +44,75 @@ def exhibit_level( ) dynamic_code_fields = FieldSet(config.FIELD_JSON_PATH, field_type="dynamic_code") + # Filter fields if specific_fields is provided + if specific_fields is not None: + exhibit_level_fields = exhibit_level_fields.filter_by_names(specific_fields) + dynamic_primary_fields = dynamic_primary_fields.filter_by_names(specific_fields) + dynamic_reimb_info_fields = dynamic_reimb_info_fields.filter_by_names( + specific_fields + ) + dynamic_code_fields = dynamic_code_fields.filter_by_names(specific_fields) + # True Exhibit Level - these apply to the entire Exhibit - exhibit_level_answers = prompt_calls.prompt_exhibit_level( - exhibit_text, exhibit_level_fields, constants, filename - ) + exhibit_level_answers = {} + if exhibit_level_fields.contains_fields(): + with timing_utils.timed_block( + "exhibit_level.prompt_exhibit_level", context=filename + ): + exhibit_level_answers = prompt_calls.prompt_exhibit_level( + exhibit_text, exhibit_level_fields, constants, filename + ) - exhibit_level_answers = prompt_calls.prompt_exhibit_level_breakout( - exhibit_level_answers, filename - ) + with timing_utils.timed_block( + "exhibit_level.prompt_exhibit_level_breakout", context=filename + ): + exhibit_level_answers = prompt_calls.prompt_exhibit_level_breakout( + exhibit_level_answers, filename + ) - # Dynamic Primary - exhibit_level_answers, dynamic_reimbursement_fields = dynamic_funcs.dynamic_primary( - exhibit_text, exhibit_level_answers, constants, filename, dynamic_primary_fields - ) + # Dynamic Primary (skip if filtering to specific fields that don't include dynamic_primary) + dynamic_reimbursement_fields = FieldSet() + if dynamic_primary_fields.contains_fields(): + with timing_utils.timed_block( + "exhibit_level.dynamic_primary", context=filename + ): + exhibit_level_answers, dynamic_reimbursement_fields = ( + dynamic_funcs.dynamic_primary( + exhibit_text, + exhibit_level_answers, + constants, + filename, + dynamic_primary_fields, + ) + ) - # Dynamic Codes - exhibit_level_answers, dynamic_reimbursement_fields = dynamic_funcs.dynamic( - exhibit_text, - exhibit_header, - exhibit_level_answers, - constants, - filename, - dynamic_code_fields, - dynamic_reimbursement_fields, - ) + # Dynamic Codes (skip if filtering to specific fields) + if dynamic_code_fields.contains_fields(): + with timing_utils.timed_block("exhibit_level.dynamic_codes", context=filename): + exhibit_level_answers, dynamic_reimbursement_fields = dynamic_funcs.dynamic( + exhibit_text, + exhibit_header, + exhibit_level_answers, + constants, + filename, + dynamic_code_fields, + dynamic_reimbursement_fields, + ) - # Dynamic Reimb Info - exhibit_level_answers, dynamic_reimbursement_fields = dynamic_funcs.dynamic( - exhibit_text, - exhibit_header, - exhibit_level_answers, - constants, - filename, - dynamic_reimb_info_fields, - dynamic_reimbursement_fields, - ) + # Dynamic Reimb Info (skip if filtering to specific fields) + if dynamic_reimb_info_fields.contains_fields(): + with timing_utils.timed_block( + "exhibit_level.dynamic_reimb_info", context=filename + ): + exhibit_level_answers, dynamic_reimbursement_fields = dynamic_funcs.dynamic( + exhibit_text, + exhibit_header, + exhibit_level_answers, + constants, + filename, + dynamic_reimb_info_fields, + dynamic_reimbursement_fields, + ) # add exhibit identifiers exhibit_level_answers["EXHIBIT_PAGE"] = exhibit_page @@ -89,14 +126,20 @@ def reimbursement_level( ) -> list[dict[str, str]]: # Reimbursement Primary - reimbursement_primary_answers = prompt_calls.prompt_reimbursement_primary( - page_text, filename - ) + with timing_utils.timed_block( + "reimbursement_level.prompt_reimbursement_primary", context=filename + ): + reimbursement_primary_answers = prompt_calls.prompt_reimbursement_primary( + page_text, filename + ) if reimbursement_primary_answers: - reimbursement_primary_answers = clean_reimbursement_primary( - reimbursement_primary_answers, constants, filename - ) + with timing_utils.timed_block( + "reimbursement_level.clean_reimbursement_primary", context=filename + ): + reimbursement_primary_answers = clean_reimbursement_primary( + reimbursement_primary_answers, constants, filename + ) return reimbursement_primary_answers @@ -154,12 +197,14 @@ def breakout( - list[dict]: The processed reimbursement primary answers with breakout details. """ - reimbursement_level_answers = methodology_breakout( - reimbursement_level_answers, constants, filename - ) # Including base methodology breakout, grouper, and fee schedule + with timing_utils.timed_block("breakout.methodology_breakout", context=filename): + reimbursement_level_answers = methodology_breakout( + reimbursement_level_answers, constants, filename + ) # Including base methodology breakout, grouper, and fee schedule # Special Case Breakout - special_case_answers = special_case_breakout(special_case_answers, filename) + with timing_utils.timed_block("breakout.special_case_breakout", context=filename): + special_case_answers = special_case_breakout(special_case_answers, filename) return reimbursement_level_answers, special_case_answers @@ -215,7 +260,7 @@ def process_single_carveout( return ("reimbursement", answer_dict) elif reimbursement_categorization in special_case_definitions.keys(): - answer_dict[reimbursement_categorization] = reimb_term + answer_dict[reimbursement_categorization] = f"{service_term} - {reimb_term}" answer_dict.pop("REIMB_TERM", None) answer_dict.pop("SERVICE_TERM", None) return ("special_case", answer_dict) @@ -245,30 +290,36 @@ def carveout_and_special_case( # Parallelize carveout processing if reimbursement_breakout_answers: - with concurrent.futures.ThreadPoolExecutor( - max_workers=min(len(reimbursement_breakout_answers), 5) - ) as executor: - futures = [ - executor.submit( - process_single_carveout, - answer_dict.copy(), - carveout_definitions, - special_case_definitions, - filename, - ) - for answer_dict in reimbursement_breakout_answers - ] + with timing_utils.timed_block( + "carveout_and_special_case.parallel_processing", context=filename + ): + with concurrent.futures.ThreadPoolExecutor( + max_workers=min(len(reimbursement_breakout_answers), 5) + ) as executor: + futures = [ + executor.submit( + process_single_carveout, + answer_dict.copy(), + carveout_definitions, + special_case_definitions, + filename, + ) + for answer_dict in reimbursement_breakout_answers + ] - for future in concurrent.futures.as_completed(futures): - try: - result_type, result_dict = future.result() - if result_type == "reimbursement": - reimbursement_level_answers.append(result_dict) - elif result_type == "special_case": - special_case_answers.append(result_dict) - except Exception as e: - logging.error(f"Error in carveout processing: {str(e)}") + for future in concurrent.futures.as_completed(futures): + try: + result_type, result_dict = future.result() + if result_type == "reimbursement": + reimbursement_level_answers.append(result_dict) + elif result_type == "special_case": + special_case_answers.append(result_dict) + except Exception as e: + logging.error(f"Error in carveout processing: {str(e)}") + logging.debug( + f"Carveout processing complete: {len(reimbursement_level_answers)} reimbursements, {len(special_case_answers)} special cases - {filename}" + ) return reimbursement_level_answers, special_case_answers @@ -771,16 +822,32 @@ def one_to_n_cleaning( all_exhibit_rows: list[dict], exhibit_text: str, constants: Constants, filename: str ): ################################ Crosswalk Fields ################################ - all_exhibit_rows = aarete_derived.get_crosswalk_fields(all_exhibit_rows, constants) + with timing_utils.timed_block( + "one_to_n_cleaning.crosswalk_fields", context=filename + ): + all_exhibit_rows = aarete_derived.get_crosswalk_fields( + all_exhibit_rows, constants + ) ################################ Determine LOB Relationship ################################ - all_exhibit_rows = get_lob_relationship(all_exhibit_rows, exhibit_text, filename) + with timing_utils.timed_block( + "one_to_n_cleaning.lob_relationship", context=filename + ): + all_exhibit_rows = get_lob_relationship( + all_exhibit_rows, exhibit_text, filename + ) ################################ Fill NA Mapping ################################ - all_exhibit_rows = aarete_derived.fill_na_mapping(all_exhibit_rows) + with timing_utils.timed_block( + "one_to_n_cleaning.fill_na_mapping", context=filename + ): + all_exhibit_rows = aarete_derived.fill_na_mapping(all_exhibit_rows) ################################ Split REIMB_DATES ################################ - all_exhibit_rows = split_reimb_dates(all_exhibit_rows, filename) + with timing_utils.timed_block( + "one_to_n_cleaning.split_reimb_dates", context=filename + ): + all_exhibit_rows = split_reimb_dates(all_exhibit_rows, filename) ################################ Normalize Single-Value Fields to Strings ################################ # REIMB_TERM, SERVICE_TERM should always be strings (not lists) @@ -909,80 +976,3 @@ def lesser_of_distribution( final_reimbursement_level_answers.append(answer_dict) return final_reimbursement_level_answers - - -def check_and_combine_exhibit_inheritance( - previous_exhibit: dict | None, - current_exhibit_page_nums: list[str], - current_exhibit_text: str, - current_dynamic_fields: FieldSet, -) -> tuple[bool, list[str], FieldSet]: - """ - Check if inheritance should happen and combine exhibits if needed. - - Conditions for inheritance: - 1. Previous exhibit exists - 2. Current exhibit's dynamic_reimbursement_fields is empty FieldSet - 3. Previous exhibit's dynamic_reimbursement_fields is NOT empty - 4. Previous exhibit had NO reimbursement rows - - When inheritance happens: - - Replace current dynamic_reimbursement_fields with previous exhibit's dynamic_reimbursement_fields - - Combine exhibit_page_nums (previous + current) to provide context through simplify_exhibit - - Args: - previous_exhibit: Dict with keys: 'exhibit_page_nums', 'dynamic_reimbursement_fields', 'had_reimbursements' - current_exhibit_page_nums: List of page numbers for current exhibit - current_exhibit_text: Full text of current exhibit (not used in return, but kept for consistency) - current_dynamic_fields: FieldSet of dynamic fields found in current exhibit - - Returns: - Tuple of: - - should_inherit: bool indicating if inheritance happened - - combined_page_nums: list[str] (combined if inheritance, original otherwise) - - updated_dynamic_fields: FieldSet (replaced with previous if inheritance, original otherwise) - """ - # Check if previous exhibit exists - if previous_exhibit is None: - return False, current_exhibit_page_nums, current_dynamic_fields - - # Check if current exhibit's dynamic_reimbursement_fields is empty - current_field_names = current_dynamic_fields.list_fields() - current_is_empty = len(current_field_names) == 0 - - if not current_is_empty: - return False, current_exhibit_page_nums, current_dynamic_fields - - # Check if previous exhibit's dynamic_reimbursement_fields is NOT empty - prev_dynamic_fields = previous_exhibit["dynamic_reimbursement_fields"] - prev_field_names = prev_dynamic_fields.list_fields() - prev_is_not_empty = len(prev_field_names) > 0 - - if not prev_is_not_empty: - return False, current_exhibit_page_nums, current_dynamic_fields - - # Check if previous exhibit had NO reimbursement rows - prev_no_reimbursements = not previous_exhibit.get("had_reimbursements", True) - - if not prev_no_reimbursements: - return False, current_exhibit_page_nums, current_dynamic_fields - - # All conditions met - inheritance should happen - # Combine exhibit_page_nums (previous + current) - # This ensures simplify_exhibit will include context from both exhibits - combined_page_nums = ( - previous_exhibit["exhibit_page_nums"] + current_exhibit_page_nums - ) - - # Replace current dynamic_reimbursement_fields with previous exhibit's - updated_dynamic_fields = prev_dynamic_fields - - # Minimal logging: inheritance activated, pages being merged, and fields inherited - prev_pages = ", ".join(previous_exhibit.get("exhibit_page_nums", [])) - current_pages = ", ".join(current_exhibit_page_nums) - fields_str = ", ".join(prev_field_names) - logging.debug( - f"Exhibit inheritance activated: merging pages {prev_pages} → {current_pages}, inheriting fields: {fields_str}" - ) - - return True, combined_page_nums, updated_dynamic_fields diff --git a/src/pipelines/shared/extraction/one_to_one_funcs.py b/src/pipelines/shared/extraction/one_to_one_funcs.py index 3e32d88..bac922a 100644 --- a/src/pipelines/shared/extraction/one_to_one_funcs.py +++ b/src/pipelines/shared/extraction/one_to_one_funcs.py @@ -2,6 +2,10 @@ import logging import os import re import json +import numpy as np +from typing import Any +from rapidfuzz import fuzz +from collections import defaultdict import pandas as pd import src.pipelines.shared.postprocessing.postprocessing_funcs as postprocessing_funcs @@ -395,3 +399,376 @@ def add_signature_count(one_to_one_results: dict, contract_txt: str) -> dict: one_to_one_results["NUM_SIGNED_SIGNATORY_LINE_COUNT"] = total_signatures return one_to_one_results + + +def add_aarete_derived_payer_name(df: pd.DataFrame, state_flag: bool) -> pd.DataFrame: + """ + Derive canonical payer names from PAYER_NAME column. + + This function clusters similar payer names, standardizes them, and selects + the most appropriate canonical name for each cluster. + + Args: + df: Input DataFrame containing a PAYER_NAME column. + state_flag: Controls whether payer names are treated as state-specific. + Always True (confirmed with business team). + True → Preserve state distinctions. + Example: "Molina Healthcare of Texas" and + "Molina Healthcare of Utah" remain separate. + False → Collapse regional state suffixes into national brand. + Example: "Molina Healthcare of Texas" and + "Molina Healthcare of Utah" both become "Molina Healthcare". + Leading state-based brand names (e.g., "Oklahoma Complete Health") + are preserved. + + Returns: + DataFrame with added AARETE_DERIVED_PAYER_NAME column. + """ + df_payer = df.copy() + # Check if PAYER_NAME column exists in the DataFrame, if not add an empty AARETE_DERIVED_PAYER_NAME column and return the DataFrame + if "PAYER_NAME" not in df_payer.columns: + logging.warning( + "PAYER_NAME column not found in DataFrame. skipping AARETE_DERIVED_PAYER_NAME column." + ) + return df_payer + # 1. Extract unique payer names + payer_names = df_payer["PAYER_NAME"].dropna().unique().tolist() + unique_payer_names = [ + payer for payer in payer_names if not string_utils.is_empty(payer) + ] + if not unique_payer_names: + df_payer["AARETE_DERIVED_PAYER_NAME"] = "" + return df_payer + + # 2. Standardize OR normalizing unique payer names (list of dicts) + standardized_unique_payer_name_rows = standardize_unique_names( + unique_payer_names, state_flag + ) + + standardized_unique_payer_name_rows = propagate_dba_to_legal( + standardized_unique_payer_name_rows + ) + + # Combining Step`s 3-4: Build similarity matrix and cluster with blocking to handle large volumes efficiently + clusters = cluster_with_blocking( + standardized_unique_payer_name_rows, + state_flag=state_flag, + threshold=90, + ) + + # 5. Build mapping using LLM placeholder + mapping = {} + for cluster in clusters: + derived_payer_name = prompt_calls.prompt_aarete_derived_payer_name( + cluster, state_flag + ) + + for record in cluster: + mapping[record["original_name"]] = derived_payer_name + + df_payer["AARETE_DERIVED_PAYER_NAME"] = df_payer["PAYER_NAME"].map(mapping) + + return df_payer + + +def standardize_unique_names( + unique_names: list[str], state_flag: bool +) -> list[dict[str, Any]]: + """ + Normalize payer names and enrich them with DBA and token signatures. + Runs only on unique payer names for efficiency. + + Block key is derived from the FIRST token of the base payer name + (not from sorted token signature) to preserve brand identity. + """ + rows = [] + + for name in unique_names: + clean = string_utils.normalize_text(name) + + state = string_utils.detect_state(clean) + + legal, dba = string_utils.extract_dba_name(clean) + base = dba if dba else clean + legal = legal if legal else clean + + base = string_utils.remove_legal_suffixes(base) + legal = string_utils.remove_legal_suffixes(legal) + + if not state_flag: + base = string_utils.remove_states(base) + legal = string_utils.remove_states(legal) + + signature = string_utils.token_signature(base) + tokens = base.split() + block_key = tokens[0] if tokens else "" + + rows.append( + { + "original_name": name, + "base_name": base, + "signature": signature if signature else (), + "has_dba_name": dba is not None, + "dba_name": dba, + "legal_name": legal, + "state": state, + "block_key": block_key, + } + ) + + return rows + + +def build_similarity_matrix(rows: list[dict]) -> np.ndarray: + """ + Create a pairwise similarity matrix for standardized entity names. + + Computes fuzzy similarity scores between each pair of rows using + both base_name and legal_name fields. The maximum score from multiple + fuzzy comparisons is used as the final similarity value. + + Args: + rows : list[dict] + List of standardized entity dictionaries containing + "base_name" and "legal_name". + + Returns: + np.ndarray + An (n x n) symmetric matrix of similarity scores (0–100), + where n is the number of rows. + """ + # Number of payer name records + n = len(rows) + + # Initialize an empty similarity matrix + sim_matrix = np.zeros((n, n)) + + # Compute pairwise similarities + for i in range(n): + for j in range(i, n): + if i == j: + # A string is always 100% similar to itself + sim_matrix[i, j] = 100 + else: + # Compute similarity using multiple fuzzy matching methods and different combinations + score = max( + fuzz.ratio(rows[i]["base_name"], rows[j]["base_name"]), + fuzz.token_set_ratio(rows[i]["base_name"], rows[j]["base_name"]), + fuzz.ratio(rows[i]["legal_name"], rows[j]["base_name"]), + fuzz.token_set_ratio(rows[i]["legal_name"], rows[j]["base_name"]), + fuzz.ratio(rows[i]["base_name"], rows[j]["legal_name"]), + fuzz.token_set_ratio(rows[i]["base_name"], rows[j]["legal_name"]), + ) + + # Fill both symmetric positions in the matrix + sim_matrix[i, j] = score + sim_matrix[j, i] = score + + return sim_matrix + + +def cluster_from_similarity( + rows: list[dict[str, Any]], + sim_matrix: np.ndarray, + state_flag: bool, + threshold: int = 90, +) -> list[list[dict[str, Any]]]: + """ + Cluster payer records using a similarity matrix and threshold. + + Each row is grouped with other rows whose similarity score + meets or exceeds the given threshold. Once a row is assigned + to a cluster, it is not reconsidered. + + Args: + rows (list[dict]): List of row dictionaries containing payer data. + sim_matrix (np.ndarray): Pairwise similarity matrix (n x n). + threshold (int): Minimum similarity score required to cluster. + + Returns: + list[list[dict]]: A list of clusters, where each cluster + is a list of row dictionaries. + """ + n = len(rows) + visited: set[int] = set() + clusters: list[list[dict[str, Any]]] = [] + + # Iterate through each record + for i in range(n): + # Skip rows that have already been clustered + if i in visited: + continue + + # Start a new cluster with the current index + cluster_indices = {i} + + # Add all rows similar enough to the current one + for j in range(n): + if sim_matrix[i, j] >= threshold: + + # --- STATE LOGIC --- + if state_flag: + state_i = rows[i]["state"] + state_j = rows[j]["state"] + + # If one has state and the other doesn't → do NOT cluster + if state_i != state_j: + continue + cluster_indices.add(j) + + # Mark clustered indices as visited + visited.update(cluster_indices) + + # Build the cluster from the original rows + clusters.append([rows[idx] for idx in cluster_indices]) + + return clusters + + +def cluster_with_blocking( + rows: list[dict[str, Any]], + state_flag: bool, + threshold: int = 90, +) -> list[list[dict[str, Any]]]: + """ + Cluster payer names using first-token blocking to scale to large volumes. + """ + + # 1. Group rows by block key + blocks = defaultdict(list) + for row in rows: + blocks[row["block_key"]].append(row) + + all_clusters = [] + + # 2. Process each block independently + for block_rows in blocks.values(): + + # Small blocks don't need matrix + if len(block_rows) == 1: + all_clusters.append(block_rows) + continue + + sim_matrix = build_similarity_matrix(block_rows) + + block_clusters = cluster_from_similarity( + block_rows, + sim_matrix, + state_flag=state_flag, + threshold=threshold, + ) + + all_clusters.extend(block_clusters) + + return all_clusters + + +def propagate_dba_to_legal(rows: list[dict]) -> list[dict]: + """ + If a legal entity has an associated DBA in any row, + propagate that DBA to all rows sharing the same legal entity. + """ + + legal_to_dba = {} + + # First pass: build mapping + for row in rows: + legal = row.get("legal_name") + dba = row.get("dba_name") + + if legal and dba: + legal_to_dba[legal] = dba + + # Second pass: propagate + for row in rows: + legal = row.get("legal_name") + + if legal in legal_to_dba: + new_base = legal_to_dba[legal] + row["base_name"] = new_base + + # Update block key + tokens = new_base.split() + row["block_key"] = tokens[0] if tokens else "" + + return rows + + +def add_aarete_derived_provider_name( + df: pd.DataFrame, state_flag: bool +) -> pd.DataFrame: + """ + Normalize extracted AARETE_DERIVED_PROVIDER_NAME values into canonical form. + """ + df_copy = df.copy() + + # Check if AARETE_DERIVED_PROVIDER_NAME column exists, if not add empty column and return + if "AARETE_DERIVED_PROVIDER_NAME" not in df_copy.columns: + logging.warning( + "AARETE_DERIVED_PROVIDER_NAME column not found in DataFrame. Adding empty column." + ) + df_copy["AARETE_DERIVED_PROVIDER_NAME"] = "" + return df_copy + + # 1. Extract unique provider group names + raw_values = df_copy["AARETE_DERIVED_PROVIDER_NAME"].dropna().tolist() + + flattened_names = [] + + for value in raw_values: + if isinstance(value, list): + flattened_names.extend(value) + else: + flattened_names.append(value) + + # Remove empties + unique_names = list( + {name for name in flattened_names if not string_utils.is_empty(name)} + ) + + if not unique_names: + df_copy["AARETE_DERIVED_PROVIDER_NAME"] = "" + return df_copy + + # 2. Standardize + standardized_rows = standardize_unique_names(unique_names, state_flag) + + # 3. DBA propagation (kept separate in case rules diverge later) + standardized_rows = propagate_dba_to_legal(standardized_rows) + + # 4. Cluster with blocking + clusters = cluster_with_blocking( + standardized_rows, + state_flag=state_flag, + threshold=90, + ) + + # 5. Build mapping + mapping = {} + + for cluster in clusters: + derived_name = prompt_calls.prompt_aarete_derived_provider_name( + cluster, state_flag + ) + + for record in cluster: + mapping[record["original_name"]] = derived_name + + def map_value(value): + if isinstance(value, list): + derived_values = {mapping.get(v) for v in value if mapping.get(v)} + + # If all collapse to same canonical, return single string + if len(derived_values) == 1: + return derived_values.pop() + + # If multiple canonicals (rare), return list + return list(derived_values) + + return mapping.get(value) + + df_copy["AARETE_DERIVED_PROVIDER_NAME"] = df_copy[ + "AARETE_DERIVED_PROVIDER_NAME" + ].apply(map_value) + + return df_copy diff --git a/src/pipelines/shared/postprocessing/postprocess.py b/src/pipelines/shared/postprocessing/postprocess.py index 88addaf..542a106 100644 --- a/src/pipelines/shared/postprocessing/postprocess.py +++ b/src/pipelines/shared/postprocessing/postprocess.py @@ -1,10 +1,49 @@ +import json +import logging + +import pandas as pd + import src.config as config from src.constants.constants import Constants from src.constants.investment_columns import FIELD_FORMAT_MAPPING from src.pipelines.shared.postprocessing import postprocessing_funcs +from src.utils import timing_utils def postprocess(df, constants: Constants): + """ + Postprocess the extracted data into CC (Contract Config) and optionally Dashboard format. + + Dashboard postprocessing is skipped unless config.RUN_DASHBOARD_POSTPROCESSING is True. + + Args: + df: Raw extracted data DataFrame + constants: Constants object + + Returns: + tuple: (cc_df, dashboard_df) - CC-formatted and Dashboard-formatted DataFrames. + dashboard_df is None when RUN_DASHBOARD_POSTPROCESSING is False. + """ + # Standard Postprocessing Steps for All Outputs + with timing_utils.timed_block("postprocess.standard_postprocess"): + df = standard_postprocess(df, constants) + + # Postprocessing Steps unique to Contract Config Output + with timing_utils.timed_block("postprocess.contract_config_postprocess"): + cc_df = contract_config_postprocess(df, constants) + + # Postprocessing Steps unique to Dashboard Output (optional, off by default) + if config.RUN_DASHBOARD_POSTPROCESSING: + with timing_utils.timed_block("postprocess.dashboard_postprocess"): + dashboard_df = dashboard_postprocess(df, constants) + else: + dashboard_df = None + + logging.debug(f"Postprocessing complete: {len(cc_df)} rows in CC output") + return cc_df, dashboard_df + + +def standard_postprocess(df, constants: Constants): if df.shape[0] == 0: return df @@ -23,9 +62,19 @@ def postprocess(df, constants: Constants): sort_columns.append("REIMB_LESSER_OF_ID") df = df.sort_values(by=sort_columns).reset_index(drop=True) - # Format rate fields + # Normalize currency fields and format rate fields if "REIMB_FEE_RATE" in df.columns: df["REIMB_FEE_RATE"] = df["REIMB_FEE_RATE"].apply( + postprocessing_funcs.normalize_currency + ) + df["REIMB_FEE_RATE"] = df["REIMB_FEE_RATE"].apply( + postprocessing_funcs.format_rate_fields_with_commas + ) + if "REIMB_CONVERSION_FACTOR" in df.columns: + df["REIMB_CONVERSION_FACTOR"] = df["REIMB_CONVERSION_FACTOR"].apply( + postprocessing_funcs.normalize_currency + ) + df["REIMB_CONVERSION_FACTOR"] = df["REIMB_CONVERSION_FACTOR"].apply( postprocessing_funcs.format_rate_fields_with_commas ) if "REIMB_PCT_RATE" in df.columns: @@ -87,7 +136,158 @@ def postprocess(df, constants: Constants): df = postprocessing_funcs.attach_sid_column(df) + df["FILE_NAME"] = df["FILE_NAME"].apply(postprocessing_funcs.standardize_file_name) + + # This will remove non-standard characters from every cell in the DataFrame + df = df.replace(r"[^ -~]+", "", regex=True) + + # Remove N/A values from all columns (standard cleaning for all outputs) + df = postprocessing_funcs.clean_na_values(df) + # Standardize output column order - this should ALWAYS be the final postprocessing step df = postprocessing_funcs.reorder_columns(df, FIELD_FORMAT_MAPPING) - return df + + +def contract_config_postprocess(df, constants: Constants): + cc_df = df.copy() + + if cc_df.shape[0] == 0: + return cc_df + + # CC only: blank UNIT_OF_MEASURE for default flat-rate rows (not for dashboard). + cc_df = postprocessing_funcs.blank_uom_for_default_flat_rate_cc(cc_df) + + # Contract Config specific postprocessing steps: + # Apply json formatting + cc_df["PAYER_STATE"] = cc_df["PAYER_STATE"].apply( + postprocessing_funcs.format_as_json_list + ) + cc_df["PROVIDER_STATE"] = cc_df["PROVIDER_STATE"].apply( + postprocessing_funcs.format_as_json_list + ) + cc_df["LOB"] = cc_df["LOB"].apply(postprocessing_funcs.format_as_json_list) + cc_df["AARETE_DERIVED_LOB"] = cc_df["AARETE_DERIVED_LOB"].apply( + postprocessing_funcs.format_as_json_list + ) + cc_df["NETWORK"] = cc_df["NETWORK"].apply(postprocessing_funcs.format_as_json_list) + cc_df["AARETE_DERIVED_NETWORK"] = cc_df["AARETE_DERIVED_NETWORK"].apply( + postprocessing_funcs.format_as_json_list + ) + cc_df["PRODUCT"] = cc_df["PRODUCT"].apply(postprocessing_funcs.format_as_json_list) + cc_df["AARETE_DERIVED_PRODUCT"] = cc_df["AARETE_DERIVED_PRODUCT"].apply( + postprocessing_funcs.format_as_json_list + ) + cc_df["PROGRAM"] = cc_df["PROGRAM"].apply(postprocessing_funcs.format_as_json_list) + cc_df["AARETE_DERIVED_PROGRAM"] = cc_df["AARETE_DERIVED_PROGRAM"].apply( + postprocessing_funcs.format_as_json_list + ) + cc_df["REIMB_PROV_TIN"] = cc_df["REIMB_PROV_TIN"].apply( + postprocessing_funcs.format_as_json_list + ) + cc_df["REIMB_PROV_NPI"] = cc_df["REIMB_PROV_NPI"].apply( + postprocessing_funcs.format_as_json_list + ) + cc_df["REIMB_PROV_NAME"] = cc_df["REIMB_PROV_NAME"].apply( + postprocessing_funcs.format_as_json_list + ) + cc_df["PROV_TAXONOMY_CD"] = cc_df["PROV_TAXONOMY_CD"].apply( + postprocessing_funcs.format_as_json_list + ) + cc_df["PROV_TAXONOMY_CD_DESC"] = cc_df["PROV_TAXONOMY_CD_DESC"].apply( + postprocessing_funcs.format_as_json_list + ) + cc_df["PROV_SPECIALTY_CD"] = cc_df["PROV_SPECIALTY_CD"].apply( + postprocessing_funcs.format_as_json_list + ) + cc_df["PROV_SPECIALTY_CD_DESC"] = cc_df["PROV_SPECIALTY_CD_DESC"].apply( + postprocessing_funcs.format_as_json_list + ) + cc_df["PLACE_OF_SERVICE_CD"] = cc_df["PLACE_OF_SERVICE_CD"].apply( + postprocessing_funcs.format_as_json_list + ) + cc_df["PLACE_OF_SERVICE_CD_DESC"] = cc_df["PLACE_OF_SERVICE_CD_DESC"].apply( + postprocessing_funcs.format_as_json_list + ) + cc_df["BILL_TYPE_CD"] = cc_df["BILL_TYPE_CD"].apply( + postprocessing_funcs.format_as_json_list + ) + cc_df["BILL_TYPE_CD_DESC"] = cc_df["BILL_TYPE_CD_DESC"].apply( + postprocessing_funcs.format_as_json_list + ) + + return cc_df + + +def dashboard_postprocess(df, constants: Constants): + dashboard_df = df.copy() + + if dashboard_df.shape[0] == 0: + return dashboard_df + + # Dashboard specific postprocessing steps can be added here + dashboard_df["PAYER_STATE"] = dashboard_df["PAYER_STATE"].apply( + lambda x: ", ".join(x) if isinstance(x, list) else x + ) + dashboard_df["PROVIDER_STATE"] = dashboard_df["PROVIDER_STATE"].apply( + lambda x: ", ".join(x) if isinstance(x, list) else x + ) + dashboard_df["LOB"] = dashboard_df["LOB"].apply( + lambda x: ", ".join(x) if isinstance(x, list) else x + ) + dashboard_df["AARETE_DERIVED_LOB"] = dashboard_df["AARETE_DERIVED_LOB"].apply( + lambda x: ", ".join(x) if isinstance(x, list) else x + ) + dashboard_df["NETWORK"] = dashboard_df["NETWORK"].apply( + lambda x: ", ".join(x) if isinstance(x, list) else x + ) + dashboard_df["AARETE_DERIVED_NETWORK"] = dashboard_df[ + "AARETE_DERIVED_NETWORK" + ].apply(lambda x: ", ".join(x) if isinstance(x, list) else x) + dashboard_df["PRODUCT"] = dashboard_df["PRODUCT"].apply( + lambda x: ", ".join(x) if isinstance(x, list) else x + ) + dashboard_df["AARETE_DERIVED_PRODUCT"] = dashboard_df[ + "AARETE_DERIVED_PRODUCT" + ].apply(lambda x: ", ".join(x) if isinstance(x, list) else x) + dashboard_df["PROGRAM"] = dashboard_df["PROGRAM"].apply( + lambda x: ", ".join(x) if isinstance(x, list) else x + ) + dashboard_df["AARETE_DERIVED_PROGRAM"] = dashboard_df[ + "AARETE_DERIVED_PROGRAM" + ].apply(lambda x: ", ".join(x) if isinstance(x, list) else x) + dashboard_df["REIMB_PROV_TIN"] = dashboard_df["REIMB_PROV_TIN"].apply( + lambda x: ", ".join(x) if isinstance(x, list) else x + ) + dashboard_df["REIMB_PROV_NPI"] = dashboard_df["REIMB_PROV_NPI"].apply( + lambda x: ", ".join(x) if isinstance(x, list) else x + ) + dashboard_df["REIMB_PROV_NAME"] = dashboard_df["REIMB_PROV_NAME"].apply( + lambda x: ", ".join(x) if isinstance(x, list) else x + ) + dashboard_df["PROV_TAXONOMY_CD"] = dashboard_df["PROV_TAXONOMY_CD"].apply( + lambda x: ", ".join(x) if isinstance(x, list) else x + ) + dashboard_df["PROV_TAXONOMY_CD_DESC"] = dashboard_df["PROV_TAXONOMY_CD_DESC"].apply( + lambda x: ", ".join(x) if isinstance(x, list) else x + ) + dashboard_df["PROV_SPECIALTY_CD"] = dashboard_df["PROV_SPECIALTY_CD"].apply( + lambda x: ", ".join(x) if isinstance(x, list) else x + ) + dashboard_df["PROV_SPECIALTY_CD_DESC"] = dashboard_df[ + "PROV_SPECIALTY_CD_DESC" + ].apply(lambda x: ", ".join(x) if isinstance(x, list) else x) + dashboard_df["PLACE_OF_SERVICE_CD"] = dashboard_df["PLACE_OF_SERVICE_CD"].apply( + lambda x: ", ".join(x) if isinstance(x, list) else x + ) + dashboard_df["PLACE_OF_SERVICE_CD_DESC"] = dashboard_df[ + "PLACE_OF_SERVICE_CD_DESC" + ].apply(lambda x: ", ".join(x) if isinstance(x, list) else x) + dashboard_df["BILL_TYPE_CD"] = dashboard_df["BILL_TYPE_CD"].apply( + lambda x: ", ".join(x) if isinstance(x, list) else x + ) + dashboard_df["BILL_TYPE_CD_DESC"] = dashboard_df["BILL_TYPE_CD_DESC"].apply( + lambda x: ", ".join(x) if isinstance(x, list) else x + ) + + return dashboard_df diff --git a/src/pipelines/shared/postprocessing/postprocessing_funcs.py b/src/pipelines/shared/postprocessing/postprocessing_funcs.py index aecfb66..5fd9bb6 100644 --- a/src/pipelines/shared/postprocessing/postprocessing_funcs.py +++ b/src/pipelines/shared/postprocessing/postprocessing_funcs.py @@ -6,6 +6,7 @@ import re from datetime import datetime import pandas as pd from src.utils import string_utils +import json # Determine the base directory for the project BASE_DIR = os.path.dirname(os.path.dirname(os.path.dirname(os.path.abspath(__file__)))) @@ -40,16 +41,141 @@ def normalize_indicator_field(value: str) -> str: """ Normalize the indicator field value to 'Y' or 'N'. + Explicitly handles empty strings, None, NaN, and blank values by setting them to 'N'. + Args: value (str): The input value from an _IND field. Returns: str: 'Y' if the value is 'Y', otherwise 'N'. """ - if isinstance(value, str) and value.strip().upper() == "Y": - return "Y" + # Handle None, NaN, and empty values + if value is None or (isinstance(value, float) and pd.isna(value)): + return "N" + + # Handle empty strings and whitespace-only strings + if isinstance(value, str): + value_stripped = value.strip() + if not value_stripped: + return "N" + if value_stripped.upper() == "Y": + return "Y" + + # Everything else becomes 'N' return "N" +def clean_na_values(df: pd.DataFrame) -> pd.DataFrame: + """ + Remove cells containing ONLY "N/A", "['N/A']", "[]", "['[]']", "UNKNOWN", or similar placeholder values. + + This function removes placeholder values only when they are the sole value in a cell. + If a placeholder value is part of a list with other non-placeholder values, it is kept. + Empty list representations (e.g. "[]", "['[]']") are replaced with blank to avoid + clutter in the output. + + Args: + df (pd.DataFrame): Input DataFrame to clean. + + Returns: + pd.DataFrame: DataFrame with placeholder values removed from cells where they are the only value. + """ + if df.empty: + return df + + df_cleaned = df.copy() + + # Patterns to match placeholder values (case-insensitive) + # Include [] and ['[]'] so empty list representations are blanked instead of shown + placeholder_patterns = [ + r"^N/A$", + r"^NA$", + r"^NaN$", + r"^UNKNOWN$", + r"^None$", + r"^null$", + r"^\['N/A'\]$", + r"^\[\"N/A\"\]$", + r"^\[N/A\]$", + r"^\[\]$", # Empty list "[]" + r"^\['\[\]'\]$", # Python repr of list containing "[]" + r"^\[\"\[\]\"\]$", # JSON list containing "[]" + ] + + def is_placeholder_only(value): + """Check if value is only a placeholder.""" + # Handle None, NaN, and pd.NA + if value is None: + return True + try: + if pd.isna(value): + return True + except (TypeError, ValueError): + # pd.isna might raise TypeError for some types, continue checking + pass + + if isinstance(value, str): + value_stripped = value.strip() + if not value_stripped: + return True + + # Try to parse as JSON list first + if value_stripped.startswith("[") and value_stripped.endswith("]"): + try: + parsed_list = json.loads(value_stripped) + if isinstance(parsed_list, list): + # If list contains only placeholder values, consider it placeholder + if len(parsed_list) == 0: + return True + all_placeholders = True + for item in parsed_list: + item_str = str(item).strip() + is_placeholder = False + for pattern in placeholder_patterns: + if re.match(pattern, item_str, re.IGNORECASE): + is_placeholder = True + break + if not is_placeholder and item_str: + all_placeholders = False + break + return all_placeholders + except (json.JSONDecodeError, ValueError): + # If JSON parsing fails, continue with string matching + pass + + # Check against placeholder patterns for string values + for pattern in placeholder_patterns: + if re.match(pattern, value_stripped, re.IGNORECASE): + return True + + # Handle list types (Python lists) + if isinstance(value, list): + if len(value) == 0: + return True + # If list contains only placeholder values, consider it placeholder + all_placeholders = True + for item in value: + item_str = str(item).strip() + is_placeholder = False + for pattern in placeholder_patterns: + if re.match(pattern, item_str, re.IGNORECASE): + is_placeholder = True + break + if not is_placeholder and item_str: + all_placeholders = False + break + return all_placeholders + + return False + + # Apply cleaning to all columns + for col in df_cleaned.columns: + df_cleaned[col] = df_cleaned[col].apply( + lambda x: "" if is_placeholder_only(x) else x + ) + + return df_cleaned + + def format_rate_fields_with_commas(value: str) -> str: """ Format the rate with commas and two decimal places. @@ -69,6 +195,32 @@ def format_rate_fields_with_commas(value: str) -> str: return "" +def normalize_currency(value: str) -> str: + """ + Normalize currency-like strings by removing $ and commas. + + Args: + value (str): A string representing a currency value. + Returns: + str: Normalized numeric string or empty string if invalid. + """ + if string_utils.is_empty(value): + return "" + + if isinstance(value, (int, float)) and not pd.isna(value): + return str(value) + + if not isinstance(value, str): + return "" + + text = value.strip() + if text.upper() in {"N/A", "NA", "NAN"}: + return "" + + text = text.replace("$", "").replace(",", "").strip() + return text + + def remove_hyphens(value): """ Remove hyphens from the input value. @@ -238,7 +390,7 @@ def normalize_cpt_fields(value): else: result = [str(value).strip()] # Default case for other types - return str(result) # Return the list directly + return json.dumps(result) # Return JSON list string def generate_reimb_ids(df: pd.DataFrame) -> pd.DataFrame: @@ -461,6 +613,27 @@ def standardize_reimb_method_and_fee_schedule( return df +def blank_uom_for_default_flat_rate_cc(df: pd.DataFrame) -> pd.DataFrame: + """ + CC output only: blank UNIT_OF_MEASURE when DEFAULT_IND='Y' and method is flat rate. + + Ensures default flat-rate reimbursements are not shown as per-unit in CC output. + Dashboard output is not modified (call this only in contract_config_postprocess). + """ + if not all( + c in df.columns + for c in ("DEFAULT_IND", "AARETE_DERIVED_REIMB_METHOD", "UNIT_OF_MEASURE") + ): + return df + default_ind_y = df["DEFAULT_IND"].astype(str).str.strip().str.upper() == "Y" + flat_rate_method = ( + df["AARETE_DERIVED_REIMB_METHOD"].astype(str).str.strip().str.lower() + == "flat rate" + ) + df.loc[default_ind_y & flat_rate_method, "UNIT_OF_MEASURE"] = "" + return df + + def process_patient_age_range(df): """ Process the 'PATIENT_AGE_RANGE' column into 'PATIENT_AGE_MIN' and 'PATIENT_AGE_MAX'. @@ -509,6 +682,30 @@ def add_aarete_derived_amendment_num(df: pd.DataFrame) -> pd.DataFrame: pd.DataFrame: The updated DataFrame with the 'AARETE_DERIVED_AMENDMENT_NUM' column. """ + def derive_amendment_num(value): + if value is None or (isinstance(value, float) and pd.isna(value)): + return "0" + + if not isinstance(value, str): + value = str(value) + + text = value.strip() + if string_utils.is_empty(text): + return "0" + + lowered = text.lower() + if lowered in {"n/a", "na", "nan", "unknown", "unspecified", "none", "null"}: + return "0" + + if any(token in lowered for token in ["base", "original", "master"]): + return "N/A" + + numbers = re.findall(r"\b\d+\b", text) + if numbers: + return str(int(numbers[-1])) + + return "0" + if "CONTRACT_AMENDMENT_NUM" in df.columns: df["AARETE_DERIVED_AMENDMENT_NUM"] = ( df["CONTRACT_AMENDMENT_NUM"] @@ -517,6 +714,10 @@ def add_aarete_derived_amendment_num(df: pd.DataFrame) -> pd.DataFrame: .fillna(df["CONTRACT_AMENDMENT_NUM"]) ) + df["AARETE_DERIVED_AMENDMENT_NUM"] = df["CONTRACT_AMENDMENT_NUM"].apply( + derive_amendment_num + ) + return df @@ -818,7 +1019,8 @@ def add_aarete_derived_signatory_complete_ind(df: pd.DataFrame) -> pd.DataFrame: Creates column: AARETE_DERIVED_SIGNATORY_COMPLETE_IND = 'Y' if - NUM_AVAILABLE_SIGNATORY_LINE_COUNT == NUM_SIGNED_SIGNATORY_LINE_COUNT + ((NUM_AVAILABLE_SIGNATORY_LINE_COUNT <= NUM_SIGNED_SIGNATORY_LINE_COUNT) and + (NUM_SIGNED_SIGNATORY_LINE_COUNT >= 1)) else 'N'. Parameters @@ -849,10 +1051,13 @@ def add_aarete_derived_signatory_complete_ind(df: pd.DataFrame) -> pd.DataFrame: # Create indicator column df["AARETE_DERIVED_SIGNATORY_COMPLETE_IND"] = ( - df["NUM_AVAILABLE_SIGNATORY_LINE_COUNT"] - .eq(df["NUM_SIGNED_SIGNATORY_LINE_COUNT"]) - .map({True: "Y", False: "N"}) - ) + ( + df["NUM_AVAILABLE_SIGNATORY_LINE_COUNT"].le( + df["NUM_SIGNED_SIGNATORY_LINE_COUNT"] + ) + ) + & (df["NUM_SIGNED_SIGNATORY_LINE_COUNT"].ge(1)) + ).map({True: "Y", False: "N"}) return df @@ -861,6 +1066,24 @@ def add_aarete_derived_signatory_complete_ind(df: pd.DataFrame) -> pd.DataFrame: return df +def standardize_file_name(file_name: str) -> str: + """ + Standardizes the given file name by applying a series of regex replacements + to normalize terms related to time periods. + + Args: + file_name (str): The original file name string. + + Returns: + str: The standardized file name string. + """ + + # remove .txt from the end if present + file_name = re.sub(r"\.txt$", "", file_name, flags=re.IGNORECASE) + + return file_name + + def fill_claim_type_from_title(df: pd.DataFrame) -> pd.DataFrame: """ Fill empty AARETE_DERIVED_CLAIM_TYPE_CD values using two strategies: @@ -978,3 +1201,52 @@ def fill_claim_type_from_title(df: pd.DataFrame) -> pd.DataFrame: df["AARETE_DERIVED_CLAIM_TYPE_CD"] = df.apply(infer_from_title, axis=1) return df + + +def _is_empty_list_placeholder(item) -> bool: + """Return True if item is "[]", "['[]']", or similar empty-list representation.""" + if item is None or (isinstance(item, float) and pd.isna(item)): + return True + s = str(item).strip() + return s in ("[]", "['[]']", '["[]"]') or s == "" + + +def format_as_json_list(val): + """ + Format a value as a JSON list string. Returns blank for empty lists + instead of "[]" to avoid clutter in output. + """ + # 1. Handle actual lists or Nulls + if isinstance(val, list): + filtered = [ + str(i).strip() + for i in val + if i is not None and not _is_empty_list_placeholder(i) + ] + return json.dumps(filtered) if filtered else "" + if pd.isna(val) or val == "" or val == "[]": + return "" + + # 2. Cleanup & Parsing + text = str(val).strip() + + # Try to parse as JSON if it looks like a list + if text.startswith("[") and text.endswith("]"): + try: + parsed = json.loads(text) + if isinstance(parsed, list): + filtered = [ + str(i).strip() + for i in parsed + if i is not None and not _is_empty_list_placeholder(i) + ] + return json.dumps(filtered) if filtered else "" + except (json.JSONDecodeError, ValueError): + text = text.strip("[]") # Strip brackets if it's a "fake" list like ['A'] + + # 3. Handle comma-separated strings & Clean special characters + # This regex keeps letters, numbers, commas, and spaces + clean_text = re.sub(r"[^a-zA-Z0-9, ]+", "", text) + items = [i.strip() for i in clean_text.split(",") if i.strip()] + + return json.dumps(items) if items else "" diff --git a/src/pipelines/shared/preprocessing/hybrid_smart_chunking_funcs.py b/src/pipelines/shared/preprocessing/hybrid_smart_chunking_funcs.py index 9a5bc9c..5314bee 100644 --- a/src/pipelines/shared/preprocessing/hybrid_smart_chunking_funcs.py +++ b/src/pipelines/shared/preprocessing/hybrid_smart_chunking_funcs.py @@ -367,6 +367,8 @@ def prompt_hsc_single_field( llm_prompt = field.get_prompt_dict( constants ) # Returns dict of {field_name : prompt} + if field.field_name == "CONTRACT_TITLE": + context = text_dict["1"] prompt, _parser = prompt_templates.ONE_TO_ONE_SINGLE_FIELD_TEMPLATE( context, llm_prompt, field_name=field.field_name ) @@ -382,6 +384,7 @@ def prompt_hsc_single_field( llm_answer_raw = llm_utils.invoke_claude( prompt, "sonnet_latest", filename, max_tokens=8192 ) + logging.debug(f"Raw LLM answer for {field.field_name}: {llm_answer_raw}") try: llm_answer_final = _parser(llm_answer_raw) # returns dict field_value = llm_answer_final.get(field.field_name) @@ -396,7 +399,7 @@ def prompt_hsc_single_field( else "N/A" ) ) - if field_value_str == "N/A": + if string_utils.is_empty(field_value_str): field.field_type = "full_context" if field.field_name == "PAYER_NAME": field.prompt = ( diff --git a/src/pipelines/shared/preprocessing/preprocess.py b/src/pipelines/shared/preprocessing/preprocess.py index 8b666cc..7b9b2d8 100644 --- a/src/pipelines/shared/preprocessing/preprocess.py +++ b/src/pipelines/shared/preprocessing/preprocess.py @@ -3,6 +3,7 @@ from typing import TYPE_CHECKING from src import config from src.pipelines.shared.preprocessing import preprocessing_funcs +from src.utils import timing_utils if TYPE_CHECKING: from src.pipelines.shared.extraction.page_funcs import Page @@ -105,32 +106,53 @@ def one_to_n_exhibit_chunking( """ # Determine which dictionary to use if pages_dict is not None: - exhibit_pages, all_exhibit_headers = preprocessing_funcs.get_exhibit_pages( - pages_dict=pages_dict, - EXHIBIT_HEADER_MARKERS=EXHIBIT_HEADER_MARKERS, - filename=filename, - ) - exhibit_pages, all_exhibit_headers = preprocessing_funcs.link_exhibit_pages( - all_exhibit_headers, pages_dict=pages_dict, filename=filename - ) - exhibit_chunk_mapping = preprocessing_funcs.chunk_by_exhibit( - pages_dict=pages_dict, exhibit_pages=exhibit_pages - ) + with timing_utils.timed_block( + "exhibit_chunking.get_exhibit_pages", context=filename + ): + exhibit_pages, all_exhibit_headers = preprocessing_funcs.get_exhibit_pages( + pages_dict=pages_dict, + EXHIBIT_HEADER_MARKERS=EXHIBIT_HEADER_MARKERS, + filename=filename, + ) + with timing_utils.timed_block( + "exhibit_chunking.link_exhibit_pages", context=filename + ): + exhibit_pages, all_exhibit_headers = preprocessing_funcs.link_exhibit_pages( + all_exhibit_headers, pages_dict=pages_dict, filename=filename + ) + with timing_utils.timed_block( + "exhibit_chunking.chunk_by_exhibit", context=filename + ): + exhibit_chunk_mapping = preprocessing_funcs.chunk_by_exhibit( + pages_dict=pages_dict, exhibit_pages=exhibit_pages + ) elif text_dict is not None: - exhibit_pages, all_exhibit_headers = preprocessing_funcs.get_exhibit_pages( - text_dict=text_dict, - EXHIBIT_HEADER_MARKERS=EXHIBIT_HEADER_MARKERS, - filename=filename, - ) - exhibit_pages, all_exhibit_headers = preprocessing_funcs.link_exhibit_pages( - all_exhibit_headers, text_dict=text_dict, filename=filename - ) - exhibit_chunk_mapping = preprocessing_funcs.chunk_by_exhibit( - text_dict=text_dict, exhibit_pages=exhibit_pages - ) + with timing_utils.timed_block( + "exhibit_chunking.get_exhibit_pages", context=filename + ): + exhibit_pages, all_exhibit_headers = preprocessing_funcs.get_exhibit_pages( + text_dict=text_dict, + EXHIBIT_HEADER_MARKERS=EXHIBIT_HEADER_MARKERS, + filename=filename, + ) + with timing_utils.timed_block( + "exhibit_chunking.link_exhibit_pages", context=filename + ): + exhibit_pages, all_exhibit_headers = preprocessing_funcs.link_exhibit_pages( + all_exhibit_headers, text_dict=text_dict, filename=filename + ) + with timing_utils.timed_block( + "exhibit_chunking.chunk_by_exhibit", context=filename + ): + exhibit_chunk_mapping = preprocessing_funcs.chunk_by_exhibit( + text_dict=text_dict, exhibit_pages=exhibit_pages + ) else: raise ValueError("Either text_dict or pages_dict must be provided") + logging.debug( + f"Exhibit chunking complete: {len(exhibit_chunk_mapping)} exhibits identified - {filename}" + ) return exhibit_chunk_mapping, all_exhibit_headers diff --git a/src/prompts/fieldset.py b/src/prompts/fieldset.py index 5f9d21c..e3373b5 100644 --- a/src/prompts/fieldset.py +++ b/src/prompts/fieldset.py @@ -378,3 +378,21 @@ class FieldSet: if field.field_name not in self.list_fields(): combined_fields.append(field) return FieldSet(fields=combined_fields) + + def filter_by_names(self, field_names: list): + """ + Filter fields to only include those in the provided list of field names. + + Args: + field_names: List of field names to keep. If None, returns all fields. + + Returns: + A new FieldSet containing only fields with names in the provided list. + """ + if field_names is None: + return FieldSet(fields=self.fields.copy()) + + filtered_fields = [ + field for field in self.fields if field.field_name in field_names + ] + return FieldSet(fields=filtered_fields) diff --git a/src/prompts/investment_prompts.json b/src/prompts/investment_prompts.json index 5b65111..5824a66 100644 --- a/src/prompts/investment_prompts.json +++ b/src/prompts/investment_prompts.json @@ -185,7 +185,7 @@ "field_name": "CONTRACT_TITLE", "relationship": "one_to_one", "field_type": "smart_chunked", - "prompt": "What is the complete title of the contract or the document? It is generally written in CAPITAL LETTERS and is generally found at the beginning of the document before a paragraph containing provider and payer name. It typically contains the words AGREEMENT, AMENDMENT, CONTRACT, ADDENDUM, MEMORANDUM, LETTER, PROVIDER or REQUEST. In case of agreement, typical structure is the provider name, followed by the phrase with agreement, followed potentially by a number signifying a version of the contract. In case of amendment, typical structure is word amendment followed by number, followed by the phrase with agreement. Extract the complete title of the contract or document as accurately as possible. Ensure: All leading characters are included for each word (e.g. 'NETWORK' instead of 'ETWORK'). Spaces between words are preserved in phrases (e.g., 'PROVIDER COLLABORATION' instead of 'PROVIDERCOLLABORATION'). If text appears incomplete or lacks spacing, use logical patterns like capitalization or common formatting rules to infer and correct the output. Replace new lines and line breaks with space.", + "prompt": "What is the complete title of the contract or the document present in the top of the given page text? It is generally written in CAPITAL LETTERS and is generally found at the beginning of the document before the opening recital or first main paragraph. It typically contains the words AGREEMENT, AMENDMENT, CONTRACT, ADDENDUM, MEMORANDUM, LETTER, PROVIDER or REQUEST. In case of agreement, typical structure is the provider name, followed by the phrase with agreement, followed potentially by a number signifying a version of the contract. In case of amendment, typical structure is word amendment followed by number, followed by the phrase with agreement. Extract the complete title of the contract or document as accurately as possible. Ensure: All leading characters are included for each word (e.g. 'NETWORK' instead of 'ETWORK'). Spaces between words are preserved in phrases (e.g., 'PROVIDER COLLABORATION' instead of 'PROVIDERCOLLABORATION'). If text appears incomplete or lacks spacing, use logical patterns like capitalization or common formatting rules to infer and correct the output. Replace new lines and line breaks with space. Prioritize the title on the top of the page", "keywords": [ "AGREEMENT", "AMENDMENT", @@ -240,7 +240,7 @@ "field_name": "PAYER_NAME", "relationship": "one_to_one", "field_type": "smart_chunked", - "prompt": "What is the name of the payer that is a party to the contract as stated in the Preamble?", + "prompt": "What is the name of the payer that is a party to the contract as stated in the Preamble? Extract the name exactly as it appears in the contract.", "retrieval_question": "What is the name of the payer, health plan, insurance company, or payor that is a party to this contract? Payer name in the preamble, parties section, or beginning of agreement." }, { @@ -268,6 +268,13 @@ "prompt": "Extract the PROVIDER organization name(s) from the retrieved contract text.\n\nThe provider is the healthcare organization entering into the agreement (NOT the payer/insurance company like Highmark, Aetna, UnitedHealthcare, Blue Cross Blue Shield, etc.).\n\nGUIDELINES:\n1. Look for the provider name in phrases like:\n - 'This Agreement is between [PAYER] and [PROVIDER]'\n - 'Agreement between [PAYER] and [PROVIDER]'\n - 'by and between [PAYER] and [PROVIDER]'\n - '[PROVIDER] (hereinafter referred to as \"Provider\")'\n - 'Provider: [PROVIDER]'\n2. The provider name often appears in the opening paragraph or preamble of the contract.\n3. Return the EXACT provider name as it appears in the document, including legal suffixes (LLC, Inc., P.C., etc.).\n4. If the provider name includes 'd/b/a' (doing business as), return the FULL name including the d/b/a portion.\n5. If MULTIPLE provider organization names appear in the contract (e.g., multiple entities entering the agreement), return ALL of them as a JSON list.\n - Example: [\"Provider Group A, LLC\", \"Provider Group B, P.C.\", \"Provider Group C d/b/a City Medical\"]\n6. Do NOT return:\n - The payer/insurance company name\n - Individual physician names (unless that is the practice name)\n - Department names only\n - Addresses or other metadata\n7. If the provider name cannot be determined from the retrieved text, return 'N/A'.\n\nBriefly explain your reasoning. Return your answer as a valid JSON dictionary with the key \"PROVIDER_NAME\". If multiple provider names are found, the value should be a JSON list. If a single provider name is found, the value should be a string. Example: {\"PROVIDER_NAME\": [\"Provider Group A, LLC\", \"Provider Group B, P.C.\"]} or {\"PROVIDER_NAME\": \"Provider Group A, LLC\"} or {\"PROVIDER_NAME\": \"N/A\"}", "retrieval_question": "What is the provider organization name? Agreement between payer and provider. This agreement is made between. By and between. Hereinafter referred to as Provider. Provider name in contract preamble. Provider party to the agreement. Healthcare organization entering agreement." }, + { + "field_name": "AARETE_DERIVED_PROVIDER_NAME", + "relationship": "one_to_one", + "field_type": "full_context", + "prompt": "Extract the PROVIDER organization name from the retrieved contract text.\n\nThe provider is the healthcare organization entering into the agreement (NOT the payer/insurance company like Highmark, Aetna, UnitedHealthcare, Blue Cross Blue Shield, etc.).\n\nGUIDELINES:\n1. Look for the provider name in phrases like:\n - 'This Agreement is between [PAYER] and [PROVIDER]'\n - 'Agreement between [PAYER] and [PROVIDER]'\n - 'by and between [PAYER] and [PROVIDER]'\n - '[PROVIDER] (hereinafter referred to as \"Provider\")'\n - 'Provider: [PROVIDER]'\n2. The provider name often appears in the opening paragraph or preamble of the contract.\n3. Return the EXACT provider name as it appears in the document, including legal suffixes (LLC, Inc., P.C., etc.).\n4. If the provider name includes 'd/b/a' (doing business as), return the FULL name including the d/b/a portion.\n5. If multiple provider organization names appear in the contract (for example, if multiple entities are entering into the agreement), identify and return the single most appropriate provider name based on the overall contract context. Carefully analyze all provider names mentioned in the contract. If the extracted provider name contains the word 'and', split the name into separate entities and determine which one is the most relevant primary provider for the agreement. Always return only one provider name in the output. The final result must not contain the word 'and'.\n6. Do NOT return:\n - The payer/insurance company name\n - Individual physician names (unless that is the practice name)\n - Department names only\n - Addresses or other metadata\n7. If the provider name cannot be determined from the retrieved text, return 'N/A'.\n\nBriefly explain your reasoning.", + "retrieval_question": "What is the provider organization name? Agreement between payer and provider. This agreement is made between. By and between. Hereinafter referred to as Provider. Provider name in contract preamble. Provider party to the agreement. Healthcare organization entering agreement." + }, { "field_name": "REIMB_PROV_TIN", "relationship": "one_to_n", @@ -342,7 +349,7 @@ "field_name": "AUTO_RENEWAL_IND", "relationship": "one_to_one", "field_type": "smart_chunked", - "prompt": "This pertains to the term and termination section of the contract. If this contract automatically renews, answer Y. If this contract will remain in effect unless terminated, answer Y. Otherwise, answer N. If you cannot determine answer N.", + "prompt": "This pertains to the term and termination section of the contract. If this contract automatically renews (keyword \"automatically\" should be mentioned) , answer Y. Otherwise, answer N. If you cannot determine answer N.", "keywords": [ "Renew", "renew", @@ -360,7 +367,7 @@ "field_name": "AUTO_RENEWAL_TERM", "relationship": "one_to_one", "field_type": "smart_chunked", - "prompt": "If the contract automatically renews, what is the length of the renewal term? This pertains to the term and termination section of the contract. The response should be a numerical representation of the renewal period with no punctuation. Acceptable formats are '1 year' or '12 months' (both representing the same duration). 'Year to year' and 'year' should be interpreted as '1 year'. If the contract automatically renews but the length of the renewal term is not specified, default to '1 year'. Only return the length of time for the renewal with no additional context.", + "prompt": "If the contract automatically renews, what is the length of the renewal term? This pertains to the term and termination section of the contract. The response should be a numerical representation of the renewal period with no punctuation. Acceptable formats are '1 year' or '12 months' (both representing the same duration). 'Year to year' and 'year' should be interpreted as '1 year'. Only return the length of time for the renewal with no additional context.", "keywords": [ "Renew", "renew", diff --git a/src/prompts/prompt_templates.py b/src/prompts/prompt_templates.py index 3c057c4..1eef9f9 100644 --- a/src/prompts/prompt_templates.py +++ b/src/prompts/prompt_templates.py @@ -253,14 +253,14 @@ def EXHIBIT_LEVEL_INSTRUCTION() -> str: Extract attributes from a section of a contract. [GENERAL INSTRUCTIONS] -- For each field, return ALL correct answers found. There may be more than one correct answer. If there are multiple, return them as a JSON list within the dictionary value (e.g., "field_name": ["value1", "value2"]). +- For each field, return the correct answer. There will only be one correct answer. - If a list of valid values is provided, ONLY answer with the EXACT value from that list. - If there are no correct answers for a given field, write 'N/A' as the answer for that field. - Be consistent and deterministic; avoid creative paraphrasing. [OUTPUT FORMAT] Briefly explain your answer, then put the final answer in the properly-formatted JSON dictionary. -{JSON_DICT_WITH_LISTS_FORMAT_INSTRUCTIONS}""" +{JSON_DICT_FORMAT_INSTRUCTIONS}""" def EXHIBIT_LEVEL( @@ -584,6 +584,7 @@ REIMB_TERM: Describes the method by which the price of the Service is calculated - NEVER return a partial sentence - include the entire sentence even if only part of the sentence is necessary. - It's ok if the SERVICE_TERM shows up in the REIMB_TERM. Prioritize maintaining complete sentences. - Do not leave out any relevant information from either the SERVICE_TERM or REIMB_TERM even if it is not present in the vicinity. e.g. answer "$100 per visit" rather than just "$100". If table headers OR table rows provide ANY relevant information (like code type, provider type, hospital name, provider name, Tax ID, legal name, etc), include that in the SERVICE_TERM or REIMB_TERM as appropriate. +- When a unit of measure (e.g., "PMPM", "per member per month", "per visit", "per diem", "per case") appears in a table header, column label, or surrounding context, always append it to the REIMB_TERM for each rate in that table. For example, if the header says "Payment PMPM" and a row shows "$31.51", the REIMB_TERM should be "$31.51 PMPM", not just "$31.51". - Include all SERVICE_TERM and REIMB_TERM pairs mentioned, even if they are redundant or overlapping. There should be at least one entry for each SERVICE_TERM and REIMB_TERM pair mentioned. - If different reimbursement percentages or dollar values are mentioned for different effective dates, create separate entries for each effective date. - For the tables: Extract EVERY ROW as a separate entry, including all line items, subtotals, totals, adjustments, charges, and calculated ratios. When rows involve different providers/entities, create separate entries for EACH provider/entity and include the provider/entity name in the SERVICE_TERM (e.g., "Provider Name – Service Category"). @@ -650,26 +651,62 @@ DECISION LOGIC: Keep only statements whose scope matches SERVICE: - "All services in this exhibit" / "All services" / "All covered services" → applies to any service - "Inpatient services only" → does NOT apply to outpatient services - - Lesser-of statements containing "listed below", "rates below", "set forth below" + - Lesser-of statements containing "listed below", "rates below", "set forth below" → applies to ANY service rate that appears after that statement in the exhibit text, - regardless of section headers - + regardless of section headers + + EXCEPTION CLAUSE HANDLING: + - If a lesser-of statement has an exception clause (e.g., "except for services in Table X", + "excluding procedures listed below"), and SERVICE appears in that exception list: + + a) The ENTIRE payment rule in that sentence/paragraph does NOT apply to SERVICE. + This includes all comparisons, caps, and conditions within that grammatical unit. + + b) ONLY overarching constraints still apply to SERVICE. + Test: Can this constraint be read independently without referencing the excepted statement? + If YES → it still applies + If NO → it's part of the excepted rule and does NOT apply + + c) ALWAYS re-scan the exhibit text for standalone constraints after excluding the payment rule. + Do not stop at step (a) — even if the main rule is excluded, independent constraints MUST be applied. + + Examples: + - SEPARATE sentence: "In no event shall reimbursement exceed the amount billed by Provider." + → Still applies (passes independence test — standalone constraint) + - SAME sentence: "Compensation shall be the lesser of 80% of the scheduled fee or usual + and customary charges, capped at the Provider's submitted amount" + → Does NOT apply (entire grammatical unit is the excepted rule) + If nothing applies → return N/A + CONDITIONAL METHODOLOGY RULE: + -If a “lesser of” or “not to exceed” statement: + →Applies only under a specific condition (e.g., “where there is no rate,” “if not listed,” “when no fee schedule exists”), and + →Defines an alternative reimbursement method rather than limiting the base METHODOLOGY, + -Treat it as a separate reimbursement methodology. + -Do NOT apply it as a constraint to the primary METHODOLOGY. + -Do NOT merge it into the output. + -Only include “lesser of” or “not to exceed” language when it operates as an overarching ceiling that applies to the base METHODOLOGY itself. + 3. COMBINE & FORMAT - Substitution rules: + Substitution: - Replace generic "rates" / "contracted rates" with METHODOLOGY value - - Preserve original "lesser of" phrasing (don't add if not present) - - Join multiple terms: "lesser of X, Y, or Z" + - Preserve original phrasing exactly — do not paraphrase, abbreviate, or alter wording + - Do not include service code or description in the output + - Deduplicate identical constraints + - Join multiple terms preserving meaning through nesting or other structures: e.g., "[the lesser of [the greater of A or B] and C]", "lesser of X, Y, or Z". Ensure brackets and nesting are used appropriately to maintain the original meaning. -OUTPUT FORMAT: -Return a JSON list with a single string containing the combined statement. - -Examples: -- ["Lab Test paid at lesser of $100, billed charges, or 110% Medicare"] -- ["Radiology paid at $250 or charges"] -- ["N/A"] + Validation: + - The output string MUST contain METHODOLOGY value + - If METHODOLOGY is a simple rate (e.g., "$30.00") and is missing from the final string, prepend it + - If METHODOLOGY is a complex formula (e.g., "80% of State Medicaid fee schedule", "the lesser of (i) X or (ii) Y"), verify it appears as the base of the output + + Examples: + - ["lesser of $100, billed charges, or 110% Medicare"] + - ["$30.00 not to exceed Provider's billed charges"] + - ["the lesser of [the greater of 100% of the State Medicaid fee schedule or Affinity's standard fee schedule] and billed charges"] + - ["N/A"] Example — Intra-exhibit statement references another exhibit → N/A: - METHODOLOGY: "per visit basis at payment rates established in Attachment A" @@ -919,6 +956,7 @@ Analyze a given term from a Payer-Provider contract and extract key fields. - Carefully extract **every individual reimbursement clause within the comparison**. - For example: "Lesser of (i) 80% of billed charges, (ii) Medicare rate, or (iii) $400" should yield **three separate JSON dictionaries**. - **Do not skip any clause**. Every item compared under "lesser of", "greater of", "equal to", or similar comparative logic must be represented as its own dictionary in the output list. + - If the comparative structure includes nested comparisons (e.g., "the lesser of [the greater of A or B] and C"), each one of A,B and C must be extracted as its own dictionary. LESSER_OF and GREATER_OF are determined only by the immediate parent comparison. So even if it is part of outer LESSER_OF comparison, LESSER_OF should be "N" if GREATER_OF is "Y". In the case of the above example, dictionaries A and B should have GREATER_OF as "Y" and LESSER_OF as "N". Dictionary C should have LESSER_OF as "Y" and GREATER_OF as "N". - **IMPORTANT**: For each component, classify the `AARETE_DERIVED_REIMB_METHOD` based on the INDIVIDUAL component, not the overall structure. - If the methodology includes **additional reimbursement terms** beyond the "lesser of" structure (e.g., per diem charges, or carve-out payments): - **These must also be extracted as separate JSON dictionaries** in the same list. @@ -926,7 +964,7 @@ Analyze a given term from a Payer-Provider contract and extract key fields. - For each dictionary, only one of the following fields should be present: REIMB_FEE_RATE, REIMB_PCT_RATE, or REIMB_CONVERSION_FACTOR. The others should be "N/A". - Do not repeat the same reimbursement information in different output entries. For example, if 'cost of vaccines plus a $5 administration fee' is present and $5 is already captured as an addition, do not create a separate dictionary with $5 as a flat fee. Include it only once as an addition. - For direct calculations like a flat fee of "$150.00 + $8.00", calculate the total ($158.00) and include only this flat fee rate in one dictionary. - +- When multiple rates are expressed in different time units and are mathematically equivalent via a standard time conversion (e.g., "$50 per 30 min" and "$100 per hour" where $50 × 2 = $100), treat them as a single reimbursement methodology. Extract only the most granular (smallest) time unit. If the values are not mathematically equivalent, extract each as a separate dictionary. If AARETE_DERIVED_REIMB_METHOD is Grouper, the dollar amount refers to REIMB_CONVERSION_FACTOR. If AARETE_DERIVED_REIMB_METHOD is Flat Rate or similar, the dollar amount refers to a REIMB_FEE_RATE. [FIELDS] @@ -1251,12 +1289,14 @@ Extract explicit procedure, revenue, diagnosis, and other healthcare codes from [INSTRUCTIONS] - For each field, return a list of all codes EXPLICITLY written for that field. The code itself must be written, not language that describes the code. +- FIELD ASSIGNMENT: Assign each code to exactly one field that matches its format. Do not put diagnosis codes (ICD-10: letter followed by digits, e.g. Z00.00, A01.1) in PROCEDURE_CD. Do not put procedure codes (CPT 5 digits or HCPCS 1 letter + 4 digits) in DIAG_CD. Revenue codes (3-4 digits, may end in X) belong only in REVENUE_CD. When in doubt, use format: PROCEDURE_CD = CPT/HCPCS only; DIAG_CD = ICD-10 only. - Note that codes may be present, but not explicitly labeled as codes. For example, you may simply see "J1098", which is a PROCEDURE_CD. You may see "155" which is a REVENUE_CD, etc. - If the text gives a range of codes, without listing each code in the range individually, return the range in the format 'LowestCode-HighestCode'. - If the text describes an entire Code Category (not one specific code) based on a start letter, return the Category in the form "Category: X". e.g. "A-Codes" --> "Category: A", "K-Codes" --> "Category K", etc. - If the text explicitly says that there is no published or established rate, write "NOT_ESTABLISHED" for the PROCEDURE_CD value. - If any of the codes are not found, do not write a list for that field. Simply populate the field with an empty list []. - If a standalone 3-digit code appears without explicit labels (e.g., "DRG," "revenue," "Rev code"), analyze the surrounding context for clues. Look for any direct or indirect references—no matter how subtle—that may suggest a connection to either a Revenue Code or a Grouper Code. If any contextual clues imply relevance to either classification, categorize the code accordingly. +- Return only valid codes that match the stated format (e.g. CPT exactly 5 digits, HCPCS 1 letter + 4 digits); do not invent or guess codes. [FIELDS] Populate a JSON dictionary with the following fields: @@ -1421,16 +1461,19 @@ Examples: ["Specific"] or ["Generic"] {JSON_LIST_FORMAT_INSTRUCTIONS}""" -def CODE_LAST_CHECK(service) -> Tuple[str, Callable[[str], list]]: +def CODE_LAST_CHECK(service) -> Tuple[str, Callable[[str], Any]]: """Call CODE_LAST_CHECK_INSTRUCTION() separately for the cached instruction. - Returns: - Tuple of (prompt_string, parser_function) where parser expects JSON list output. + Uses format-aware list parser with expected_format='str' so single-item list + (e.g. ['Specific'] or ['Generic']) is normalized to a string. Parser returns str. """ prompt = f"""[SERVICE TO ANALYZE] {service}""" - return (prompt, _json_list_parser) + return ( + prompt, + _create_json_list_parser(field_name="CODE_LAST_CHECK", expected_format="str"), + ) def FILL_BILL_TYPE_INSTRUCTION() -> str: @@ -1573,6 +1616,7 @@ Briefly explain your reasoning. After your explanation, include the heading "FIN - All provider objects must have these exact keys: "TIN", "NPI", "NAME" - ALl dict values must be strings - if there are multiple different TINs or NPIs for an identical NAME, put them in different dictionaries. - Use "N/A" for any missing values +- Strip ALL non-digit characters from any found TIN or NPI (e.g., remove hyphens, spaces, parentheses). - A provider record is considered valid as long as either a TIN, NPI, or NAME is present. If any one or two of those fields are missing, use "N/A" for those fields. - Your final JSON array must begin with an opening bracket '[' and end with a closing bracket ']' - Make sure the JSON array is correctly formatted with commas between objects @@ -1916,7 +1960,7 @@ Default is NO. Return YES only if ALL three conditions pass: Must identify a billable service, procedure, or care category. -Valid examples: service names, procedure types, CPT/HCPCS codes, care categories +Valid examples: service names, procedure types, CPT/HCPCS codes, care categories, umbrella terms like "Covered Services" or "All Services" Invalid - return NO if SERVICE TERM is: - Clinical definitions or level-of-care descriptions (acuity, intervention frequency, co-morbidities) @@ -1943,7 +1987,7 @@ Valid payment methods: - Special types: OUTLIER, STOP_LOSS, SEQUESTRATION, DISCOUNT, PREMIUM, RATE_ESCALATOR, FACILITY_ADJUSTMENT Invalid - return NO if REIMBURSEMENT TERM is: -- Empty reimbursements: structure of a payment method but actual rate/percentage is not stated or references another location (e.g., "paid at rates in the next section", "the percentage listed below", "per the attached schedule", "as set forth in Exhibit A"). Note: formulas that define calculation methodology (e.g., "base rate multiplied by DRG weight") ARE valid even if specific dollar amounts aren't stated. +- Empty reimbursements: structure of a payment method but actual rate/percentage is not stated or references another location (e.g., "paid at rates in the next section", "the percentage listed below", "per the attached schedule", "as set forth in Exhibit A"). Note: formulas that define calculation methodology (e.g., "base rate multiplied by DRG weight") ARE valid even if specific dollar amounts aren't stated. Also valid: a percentage tied to a specific CPT/HCPCS code (e.g., "100% of S5125-U3") — the percentage IS the concrete rate. - Vague or undefined amounts: uses non-specific language like "appropriate amount", "applicable rate", "as determined", "reasonable charges", "customary charges" without a concrete percentage or dollar figure. A fee schedule reference is only valid if paired with a specific percentage (e.g., "100% of fee schedule" is valid; "appropriate amount under fee schedule" is NOT valid). - Service definitions without payment terms - Authorization or coverage requirements only @@ -1957,7 +2001,7 @@ Invalid - return NO if REIMBURSEMENT TERM is: ## CONDITION 3: No Disqualifiers Return NO if EITHER term contains (even if valid payment method exists): -- Coordination of Benefits (COB) +- Coordination of Benefits (COB) as the primary subject. Note: standard payment reductions (e.g., "less co-payments, deductibles, or amounts paid by third parties") within a reimbursement formula are NOT COB disqualifiers. - Risk-sharing, surplus/deficit distributions, settlement payments - Performance bonuses, incentives, administrative compensation - Audit/timeliness metrics (e.g., "pay 85% of claims within 30 days") @@ -2352,3 +2396,119 @@ Reimbursement: {reimbursement_dict["REIMB_TERM"]} Return the integer of the {special_case_term} that is most associated with the REIMBURSEMENT TERM.""" return (prompt, _json_list_parser) + + +def AARETE_DERIVED_PAYER_NAME_INSTRUCTION(): + return f"""You are an expert in healthcare entity data normalization. + Your goal is to take a list of similar payer names and determine the single, + most accurate canonical legal or brand name. + + Given variants string contains variations of a single Payer (insurance company) name + extracted from different contracts. They all refer to the same entity. + + TASK: + 1. Select or derive the most professional canonical name. + 2. If given STATE_FLAG is FALSE: + - Collapse regional entities into the national brand (e.g., 'Molina Healthcare'). + - DO NOT remove the word 'State' if it is part of a proper noun (e.g., 'New York State Catholic Health Plan'). + 3. If given STATE_FLAG is TRUE: + - Maintain state distinctions (e.g., 'Molina Healthcare of Texas'). + 4. Prefer the 'DBA' (Doing Business As) name if it is present. + 5. Cleaned name should be preferred over original name, but use your judgment to determine if the cleaned name is actually more accurate or if it has removed important context. + 6. Remove excessive noise like "a corporation of...", "an Illinois company", or "d/b/a" prefixes. + 7. Maintain essential legal identifiers only if they are part of the standard brand identity. + 8. Return only the cleaned canonical name without any explanation. + + Example Variants Input: + VARIANTS: + - AvMed Health Plan, Inc + - AvMED, INC., d/b/a AvMED HEALTH PLANS + Example Output: + {{"AARETE_DERIVED_PAYER_NAME":"AvMed Health Plans"}} + + [OUTPUT FORMAT] + Return ONLY the JSON dictionary with no explanatory text. Use "AARETE_DERIVED_PAYER_NAME" as the key. + {JSON_DICT_FORMAT_INSTRUCTIONS} + """ + + +def AARETE_DERIVED_PAYER_NAME( + cluster_rows: list[dict], state_flag: bool +) -> Tuple[str, Callable[[str], dict]]: + """ + Constructs the prompt using both original and cleaned names for maximum context. + """ + variant_list = [] + for r in cluster_rows: + # Using your preferred logic to show the LLM both states of the data + variant = f"- Original: {r['original_name']} (Cleaned: {r['base_name']})" + variant_list.append(variant) + + variants_string = "\n".join(variant_list) + prompt = f""" + VARIANTS: + {variants_string} + STATE_FLAG: + {state_flag} + """ + return (prompt, _json_dict_parser) + + +def AARETE_DERIVED_PROVIDER_NAME_INSTRUCTION(): + return f"""You are an expert in healthcare provider group name normalization. + +Your goal is to take a list of similar provider group names and determine +the single most accurate canonical provider group name. + +The given variants refer to the same Provider Group entity. + +TASK: +1. Select or derive the most professional canonical name. +2. If given STATE_FLAG is FALSE: + - Collapse regional entities into the national brand (e.g., 'Molina Healthcare'). + - DO NOT remove the word 'State' if it is part of a proper noun (e.g., 'New York State Catholic Health Plan'). +3. If given STATE_FLAG is TRUE: + - Maintain state distinctions (e.g., 'Molina Healthcare of Texas'). +4. Prefer the 'DBA' (Doing Business As) name if it is present. +5. Cleaned name should be preferred over original name, but use your judgment to determine if the cleaned name is actually more accurate or if it has removed important context. +6. Remove excessive noise like "a corporation of...", "an Illinois company", or "d/b/a" prefixes. +7. Maintain essential legal identifiers only if they are part of the standard brand identity. +8. Return only the cleaned canonical name without any explanation. + +Example Variants Input: +VARIANTS: +- Texas Health Physicians Group +- THPG +Example Output: +{{"AARETE_DERIVED_PROVIDER_NAME":"Texas Health Physicians Group"}} + +[OUTPUT FORMAT] +Return ONLY the JSON dictionary with key: +"AARETE_DERIVED_PROVIDER_NAME" +{JSON_DICT_FORMAT_INSTRUCTIONS} +""" + + +def AARETE_DERIVED_PROVIDER_NAME( + cluster_rows: list[dict], state_flag: bool +) -> Tuple[str, Callable[[str], dict]]: + """ + Construct LLM prompt for provider group canonical selection. + """ + + variant_list = [] + + for r in cluster_rows: + variant = f"- Original: {r['original_name']} (Cleaned: {r['base_name']})" + variant_list.append(variant) + + variants_string = "\n".join(variant_list) + + prompt = f""" +VARIANTS: +{variants_string} +STATE_FLAG: +{state_flag} +""" + + return (prompt, _json_dict_parser) diff --git a/src/qc_qa/__main__.py b/src/qc_qa/__main__.py index 22860d3..39bb585 100644 --- a/src/qc_qa/__main__.py +++ b/src/qc_qa/__main__.py @@ -24,6 +24,8 @@ from src.config import get_arg_value from src.qc_qa.pipeline import ( convert_excel_to_csv, generate_statistics, + generate_tin_contract_summary, + generate_tin_statistics, read_file_by_extension, run_qc_qa_pipeline, save_qc_qa_outputs, @@ -33,6 +35,8 @@ from src.qc_qa.pipeline import ( def main(): """Main entry point for standalone QC/QA validation.""" + logging.basicConfig(level=logging.INFO, format="%(levelname)s:%(name)s:%(message)s") + # Get arguments input_file = get_arg_value("input_file", None) output_file = get_arg_value("output_file", None) @@ -84,7 +88,8 @@ def main(): # Generate statistics logging.info("Generating statistics...") output_stats_df = generate_statistics(validated_df) - + tin_stats_df = generate_tin_statistics(validated_df) + tin_contract_summary_df = generate_tin_contract_summary(validated_df) # Save to qa_qc_output folder (local) and S3 using centralized function logging.info("Saving outputs to qa_qc_output and S3...") @@ -93,6 +98,8 @@ def main(): stats_df=output_stats_df, run_timestamp=run_timestamp, write_to_s3=config.WRITE_TO_S3, + tin_stats_df=tin_stats_df, + tin_contract_summary_df=tin_contract_summary_df, ) # Legacy: Save to user-specified output file if provided (supports CSV and Excel) @@ -121,8 +128,14 @@ def main(): logging.warning(f"Warning: Could not remove temporary file {csv_file}: {e}") logging.info("QC/QA Validation Complete!") - logging.info(f"\nOutputs saved to:") - logging.info(f" - Local: {config.QC_QA_OUTPUT_DIRECTORY}/{run_timestamp}/") + logging.info( + f"\nOutputs saved to: {config.QC_QA_OUTPUT_DIRECTORY}/{run_timestamp}/" + ) + logging.info(f" - {config.BATCH_ID}-QC-QA-VALIDATED.csv") + logging.info( + f" - {config.BATCH_ID}-QC-QA-STATS.xlsx " + f"(sheets: QC-QA-STATS, QC-QA-TIN-STATS, QC-QA-TIN-CONTRACT-SUMMARY)" + ) if config.WRITE_TO_S3: logging.info( f" - S3: s3://{config.S3_OUTPUT_BUCKET}/{config.BATCH_ID}/{run_timestamp}/automation_qa-qc/" diff --git a/src/qc_qa/pipeline.py b/src/qc_qa/pipeline.py index ce70fb9..98e17d3 100644 --- a/src/qc_qa/pipeline.py +++ b/src/qc_qa/pipeline.py @@ -21,6 +21,7 @@ import pandas as pd from src import config from src.utils import qa_qc_utils as qc +from src.utils.string_utils import is_filled, extract_nested_values import logging # Suppress pandas FutureWarnings @@ -599,6 +600,8 @@ def save_qc_qa_outputs( stats_df: pd.DataFrame, run_timestamp: str, write_to_s3: bool = True, + tin_stats_df: pd.DataFrame = None, + tin_contract_summary_df: pd.DataFrame = None, ) -> None: """ Save QC/QA outputs to local qa_qc_output folder and optionally to S3. @@ -618,28 +621,271 @@ def save_qc_qa_outputs( from src.utils import io_utils import logging + output_dir = os.path.join(config.QC_QA_OUTPUT_DIRECTORY, run_timestamp) + os.makedirs(output_dir, exist_ok=True) + # Save validated data locally io_utils.write_local(validated_df, "", run_timestamp, "qc_qa_validated") logging.info( - f"Saved QC/QA validated data to: {config.QC_QA_OUTPUT_DIRECTORY}/{run_timestamp}/{config.BATCH_ID}-QC-QA-VALIDATED.csv" + f"Saved QC/QA validated data to: {output_dir}/{config.BATCH_ID}-QC-QA-VALIDATED.csv" ) - # Save statistics locally - io_utils.write_local(stats_df, "", run_timestamp, "qc_qa_stats") - logging.info( - f"Saved QC/QA statistics to: {config.QC_QA_OUTPUT_DIRECTORY}/{run_timestamp}/{config.BATCH_ID}-QC-QA-STATS.csv" - ) + # Save combined stats Excel with both stats as separate sheets + stats_excel_path = os.path.join(output_dir, f"{config.BATCH_ID}-QC-QA-STATS.xlsx") + with pd.ExcelWriter(stats_excel_path, engine="openpyxl") as writer: + stats_df.to_excel(writer, sheet_name="QC-QA-STATS", index=False) + if tin_stats_df is not None: + tin_stats_df.to_excel(writer, sheet_name="QC-QA-TIN-STATS", index=False) + if tin_contract_summary_df is not None: + tin_contract_summary_df.to_excel( + writer, sheet_name="QC-QA-TIN-CONTRACT-SUMMARY", index=False + ) + logging.info(f"Saved combined stats Excel to: {stats_excel_path}") # Upload to S3 if enabled if write_to_s3: logging.info("Uploading QC/QA outputs to S3...") io_utils.write_s3(validated_df, "", run_timestamp, "qc_qa_validated") - io_utils.write_s3(stats_df, "", run_timestamp, "qc_qa_stats") + s3_key = f"{config.BATCH_ID}/{run_timestamp}/automation_qa-qc/{config.BATCH_ID}-QC-QA-STATS.xlsx" + try: + with open(stats_excel_path, "rb") as f: + config.S3_CLIENT.put_object( + Bucket=config.S3_OUTPUT_BUCKET, Key=s3_key, Body=f.read() + ) + logging.info(f"Uploaded combined stats Excel to S3: {s3_key}") + except Exception as e: + logging.warning(f"Failed to upload combined Excel to S3: {e}") logging.info( f"Uploaded to S3: {config.S3_OUTPUT_BUCKET}/{config.BATCH_ID}/{run_timestamp}/automation_qa-qc/" ) +def generate_tin_statistics(df: pd.DataFrame) -> pd.DataFrame: + """ + Generate summary statistics for the validated DataFrame. + + Args: + df: Validated DataFrame + + Returns: + DataFrame containing statistics + """ + import ast + + cols = [ + "FILENAME_TIN", + "PROV_GROUP_TIN", + "PROV_GROUP_NPI", + "PROV_GROUP_NAME_FULL", + "PROV_OTHER_TIN", + "PROV_OTHER_NPI", + "PROV_OTHER_NAME_FULL", + "PROV_INFO_JSON", + ] + + prov_cols_to_validate = [ + "PROV_GROUP_TIN", + "PROV_GROUP_NPI", + "PROV_GROUP_NAME_FULL", + "PROV_OTHER_TIN", + "PROV_OTHER_NPI", + "PROV_OTHER_NAME_FULL", + ] + + lob_product_program_cols = [ + "LOB", + "PRODUCT", + "PROGRAM", + "AARETE_DERIVED_LOB", + "AARETE_DERIVED_PROGRAM", + "NETWORK", + "AARETE_DERIVED_NETWORK", + ] + + effective_date_cols = [ + "AARETE_DERIVED_EFFECTIVE_DT", + "REIMB_EFFECTIVE_DT", + "EFFECTIVE_DT", + ] + + rows = [] + + def append_metric(metric: str, filled: Optional[int], total: Optional[int]) -> None: + blank = None if filled is None or total is None else int(total - filled) + rows.append( + { + "Metric": metric, + "Filled": filled, + "Blank": blank, + "Total": total, + } + ) + + # Rows filled count per column + for col in cols: + if col not in df.columns: + append_metric(col, None, len(df)) + continue + filled = df[col].apply(is_filled).sum() + append_metric(col, int(filled), len(df)) + + # PROV_INFO_JSON validation: check if each prov col's values appear in JSON + if "PROV_INFO_JSON" in df.columns: + col_filled_mask = { + col: df[col].apply(is_filled) + for col in prov_cols_to_validate + if col in df.columns + } + json_blank_mask = ~df["PROV_INFO_JSON"].apply(is_filled) + json_blank_total = int(json_blank_mask.sum()) + + for check_col in prov_cols_to_validate: + if check_col not in df.columns: + continue + + col_filled = col_filled_mask[check_col] + checkable = int( + col_filled.sum() + ) # denominator = all filled rows for this column + + def value_in_json(row, _col=check_col): + if not is_filled(row["PROV_INFO_JSON"]): + return False # JSON empty — not a match + values = extract_nested_values(row[_col]) + if not values: + return False # no extractable values — not a match + json_str = str(row["PROV_INFO_JSON"]) + return all(v in json_str for v in values if v) + + matched = df[col_filled].apply(value_in_json, axis=1).sum() + append_metric( + f"{check_col} value found in PROV_INFO_JSON (of filled {check_col} rows)", + int(matched), + checkable, + ) + + append_metric( + f"{check_col} filled when PROV_INFO_JSON is blank", + int((col_filled & json_blank_mask).sum()), + json_blank_total, + ) + + # LOB / Product / Program filled counts + for col in lob_product_program_cols: + if col not in df.columns: + append_metric(col, None, len(df)) + continue + filled = df[col].apply(is_filled).sum() + append_metric(col, int(filled), len(df)) + + # Effective date filled counts + for col in effective_date_cols: + if col not in df.columns: + append_metric(col, None, len(df)) + continue + filled = df[col].apply(is_filled).sum() + append_metric(col, int(filled), len(df)) + + result = pd.DataFrame(rows) + for col in ["Filled", "Blank", "Total"]: + result[col] = result[col].astype("Int64") + return result + + +def generate_tin_contract_summary(df: pd.DataFrame) -> pd.DataFrame: + """ + Generate one summary row per contract for provider fields filled when JSON is blank. + + Args: + df: Validated DataFrame + + Returns: + DataFrame containing per-contract summary statistics + """ + contract_col = qc.QC_CONTRACT_NAME + agreement_col = qc.QC_AGREEMENT_NAME + provider_cols = [ + "PROV_GROUP_TIN", + "PROV_GROUP_NPI", + "PROV_GROUP_NAME_FULL", + "PROV_OTHER_TIN", + "PROV_OTHER_NPI", + "PROV_OTHER_NAME_FULL", + ] + + if contract_col not in df.columns: + return pd.DataFrame() + + summary_rows = [] + for contract_name, contract_df in df.groupby(contract_col, dropna=False): + row = { + "FILE_NAME": contract_name, + "Row Count": len(contract_df), + } + + if agreement_col in contract_df.columns: + agreement_values = ( + contract_df[agreement_col].dropna().astype(str).map(str.strip) + ) + agreement_values = agreement_values[agreement_values != ""] + row[agreement_col] = ( + agreement_values.iloc[0] if not agreement_values.empty else "" + ) + + json_filled_mask = ( + contract_df["PROV_INFO_JSON"].apply(is_filled) + if "PROV_INFO_JSON" in contract_df.columns + else pd.Series(False, index=contract_df.index) + ) + json_blank_mask = ~json_filled_mask + row["PROV_INFO_JSON Filled Rows"] = int(json_filled_mask.sum()) + row["PROV_INFO_JSON Blank Rows"] = int(json_blank_mask.sum()) + + for col in provider_cols: + if col not in contract_df.columns: + row[f"{col} Filled Rows"] = None + row[f"{col} Filled When JSON Blank"] = None + continue + + col_filled_mask = contract_df[col].apply(is_filled) + row[f"{col} Filled Rows"] = int(col_filled_mask.sum()) + row[f"{col} Filled When JSON Blank"] = int( + (col_filled_mask & json_blank_mask).sum() + ) + + summary_rows.append(row) + + result = pd.DataFrame(summary_rows) + preferred_order = [ + "FILE_NAME", + agreement_col, + "Row Count", + "PROV_GROUP_TIN Filled Rows", + "PROV_GROUP_NPI Filled Rows", + "PROV_GROUP_NAME_FULL Filled Rows", + "PROV_OTHER_TIN Filled Rows", + "PROV_OTHER_NPI Filled Rows", + "PROV_OTHER_NAME_FULL Filled Rows", + "PROV_INFO_JSON Filled Rows", + "PROV_INFO_JSON Blank Rows", + "PROV_GROUP_TIN Filled When JSON Blank", + "PROV_GROUP_NPI Filled When JSON Blank", + "PROV_GROUP_NAME_FULL Filled When JSON Blank", + "PROV_OTHER_TIN Filled When JSON Blank", + "PROV_OTHER_NPI Filled When JSON Blank", + "PROV_OTHER_NAME_FULL Filled When JSON Blank", + ] + result = result[[col for col in preferred_order if col in result.columns]] + + int_cols = [ + col for col in result.columns if col.endswith("Rows") or col == "Row Count" + ] + for col in int_cols: + if col in result.columns: + result[col] = result[col].astype("Int64") + return result + + def generate_statistics(df: pd.DataFrame) -> pd.DataFrame: """ Generate summary statistics for the validated DataFrame. @@ -652,7 +898,14 @@ def generate_statistics(df: pd.DataFrame) -> pd.DataFrame: """ length_stats = {} stats = {} - stats["percentage_filled"] = (df.notna().sum() / len(df) * 100).round(2).to_dict() + filled_mask = df.apply( + lambda col: col.map( + lambda x: x is not None + and not (isinstance(x, float) and pd.isna(x)) + and str(x).strip() != "" + ) + ) + stats["percentage_filled"] = (filled_mask.sum() / len(df) * 100).round(2).to_dict() if qc.QC_CONTRACT_NAME in df.columns: stats["distinct_contract_count"] = df[qc.QC_CONTRACT_NAME].nunique() diff --git a/src/scripts/postprocess_existing_output.py b/src/scripts/postprocess_existing_output.py index 40c2356..fa7e017 100644 --- a/src/scripts/postprocess_existing_output.py +++ b/src/scripts/postprocess_existing_output.py @@ -1,14 +1,14 @@ import pandas as pd -from constants.constants import Constants -from src.investment.postprocess import postprocess +from src.constants.constants import Constants +from src.pipelines.shared.postprocessing.postprocess import postprocess # File paths -input_excel_path = "input.xlsx" +input_excel_path = "input.csv" output_csv_path = "output.csv" # Read Excel file into DataFrame print(f"Reading Excel file: {input_excel_path}") -df = pd.read_excel(input_excel_path, dtype=str) +df = pd.read_csv(input_excel_path, dtype=str) print(f"Loaded {len(df)} rows from Excel") # Initialize Constants @@ -16,10 +16,10 @@ constants = Constants() # Process the DataFrame print("Processing data through postprocess()...") -df_processed = postprocess(df, constants) -print(f"Processing complete. Output has {len(df_processed)} rows") +cc_df, dashboard_df = postprocess(df, constants) +print(f"Processing complete. Output has {len(cc_df)} rows") # Write to CSV print(f"Writing CSV file: {output_csv_path}") -df_processed.to_csv(output_csv_path, index=False) +cc_df.to_csv(output_csv_path, index=False) print("Done!") diff --git a/src/tests/test_derived_names.py b/src/tests/test_derived_names.py new file mode 100644 index 0000000..9929502 --- /dev/null +++ b/src/tests/test_derived_names.py @@ -0,0 +1,251 @@ +import unittest +import pandas as pd +import numpy as np +from unittest.mock import patch + +from src.utils.string_utils import ( + extract_dba_name, + normalize_text, + remove_legal_suffixes, + remove_states, + token_signature, +) +from src.pipelines.shared.extraction.one_to_one_funcs import ( + add_aarete_derived_provider_name, + standardize_unique_names, + add_aarete_derived_payer_name, + cluster_from_similarity, + build_similarity_matrix, + cluster_with_blocking, + propagate_dba_to_legal, +) + + +class TestPayerNameTextUtilities(unittest.TestCase): + + def test_normalize_payer_name_text(self): + text = "Molina Healthcare, Inc." + result = normalize_text(text) + self.assertEqual(result, "molina healthcare inc") + + def test_extract_dba_payer_name_from_text(self): + text = "ABC Corp d/b/a XYZ Health" + normalized = normalize_text(text) + result = extract_dba_name(normalized) + self.assertEqual(result, ("abc corp", "xyz health")) + + def test_remove_legal_suffixes_from_payer_name(self): + text = "molina healthcare inc" + result = remove_legal_suffixes(text) + self.assertEqual(result, "molina healthcare") + + def test_generate_token_signature_for_payer_name(self): + text = "molina healthcare of utah" + result = token_signature(text) + self.assertEqual(result, ("healthcare", "molina", "utah")) + + +class TestPayerNameStandardization(unittest.TestCase): + + def test_standardize_unique_payer_names(self): + names = ["Molina Healthcare of Utah, Inc."] + rows = standardize_unique_names(names, state_flag=True) + + self.assertEqual(len(rows), 1) + self.assertEqual(rows[0]["original_name"], names[0]) + self.assertIn("base_name", rows[0]) + self.assertIn("signature", rows[0]) + self.assertIn("block_key", rows[0]) + self.assertFalse(rows[0]["has_dba_name"]) + + +class TestRemoveStatesFunction(unittest.TestCase): + + def test_remove_trailing_state_suffix(self): + self.assertEqual( + remove_states("community health choice texas"), "community health choice" + ) + + self.assertEqual(remove_states("fidelis care new york"), "fidelis care") + + def test_preserve_leading_state_brand(self): + self.assertEqual( + remove_states("oklahoma complete health"), "oklahoma complete health" + ) + + self.assertEqual( + remove_states("texas childrens health plan"), "texas childrens health plan" + ) + + def test_remove_of_state_pattern(self): + self.assertEqual( + remove_states("molina healthcare of utah"), "molina healthcare" + ) + + self.assertEqual( + remove_states("molina healthcare of texas"), "molina healthcare" + ) + + def test_no_state_present(self): + self.assertEqual(remove_states("centene corporation"), "centene corporation") + + +class TestPropagateDbaToLegal(unittest.TestCase): + + def test_propagate_dba_to_matching_legal_entity(self): + rows = [ + { + "original_name": "NY State Catholic Health Plan dba Fidelis Care New York", + "base_name": "fidelis care new york", + "legal_name": "new york state catholic health plan", + "dba_name": "fidelis care new york", + "block_key": "fidelis", + }, + { + "original_name": "New York State Catholic Health Plan", + "base_name": "new york state catholic health plan", + "legal_name": "new york state catholic health plan", + "dba_name": None, + "block_key": "new", + }, + ] + + updated = propagate_dba_to_legal(rows) + + # Both should now share same base_name + self.assertEqual(updated[0]["base_name"], "fidelis care new york") + self.assertEqual(updated[1]["base_name"], "fidelis care new york") + + # Block key should also update + self.assertEqual(updated[1]["block_key"], "fidelis") + + def test_no_propagation_when_no_dba_present(self): + rows = [ + { + "original_name": "ODS Health Plan", + "base_name": "ods health plan", + "legal_name": "ods health plan", + "dba_name": None, + "block_key": "ods", + }, + { + "original_name": "Moda Health Plan", + "base_name": "moda health plan", + "legal_name": "moda health plan", + "dba_name": None, + "block_key": "moda", + }, + ] + + updated = propagate_dba_to_legal(rows) + + # Should remain unchanged + self.assertEqual(updated[0]["base_name"], "ods health plan") + self.assertEqual(updated[1]["base_name"], "moda health plan") + + +class TestPayerNameClustering(unittest.TestCase): + + def test_cluster_payer_names_with_state_flag_true(self): + names = ["Molina Healthcare of Utah", "Molina Healthcare"] + + rows = standardize_unique_names(names, state_flag=True) + + clusters = cluster_with_blocking(rows, state_flag=True, threshold=85) + + # Should remain separate clusters + self.assertEqual(len(clusters), 2) + + def test_cluster_payer_names_with_state_flag_false(self): + names = ["Molina Healthcare of Utah", "Molina Healthcare"] + + rows = standardize_unique_names(names, state_flag=False) + + clusters = cluster_with_blocking(rows, state_flag=False, threshold=85) + + # Should cluster together + self.assertEqual(len(clusters), 1) + + +class TestPayerNameEndToEndPipeline(unittest.TestCase): + + @patch("src.pipelines.saas.prompts.prompt_calls.prompt_aarete_derived_payer_name") + def test_derive_payer_names_end_to_end(self, mock_llm): + + data = { + "PAYER_NAME": [ + "NEW YORK STATE CATHOLIC HEALTH PLAN, INC. dba FIDELIS CARE NEW YORK", + "NEW YORK STATE CATHOLIC HEALTH PLAN, INC.", + "NEW YORK STATE CATHOLIC HEALTH PLAN, INC. d/b/a FIDELIS CARE NEW YORK", + ] + } + + # Mock LLM to return canonical name + mock_llm.return_value = "FIDELIS CARE NEW YORK" + + df = pd.DataFrame(data) + + result = add_aarete_derived_payer_name(df, state_flag=True) + + self.assertIn("AARETE_DERIVED_PAYER_NAME", result.columns) + + # All rows should map to same canonical value + self.assertEqual( + result.loc[0, "AARETE_DERIVED_PAYER_NAME"], "FIDELIS CARE NEW YORK" + ) + + self.assertEqual( + result.loc[1, "AARETE_DERIVED_PAYER_NAME"], "FIDELIS CARE NEW YORK" + ) + + self.assertEqual( + result.loc[2, "AARETE_DERIVED_PAYER_NAME"], "FIDELIS CARE NEW YORK" + ) + + +class TestProvGroupEndToEnd(unittest.TestCase): + + def test_derived_provider_names_column_missing(self): + """Test that missing column is added as empty and function returns early.""" + data = {"OTHER_COLUMN": ["value1", "value2"]} + df = pd.DataFrame(data) + + result = add_aarete_derived_provider_name(df, state_flag=True) + + self.assertIn("AARETE_DERIVED_PROVIDER_NAME", result.columns) + self.assertEqual(result.loc[0, "AARETE_DERIVED_PROVIDER_NAME"], "") + self.assertEqual(result.loc[1, "AARETE_DERIVED_PROVIDER_NAME"], "") + + @patch( + "src.pipelines.saas.prompts.prompt_calls.prompt_aarete_derived_provider_name" + ) + def test_derived_provider_names_end_to_end(self, mock_llm): + + data = { + "AARETE_DERIVED_PROVIDER_NAME": [ + "Texas Health Physicians Group, Inc.", + "Texas Health Physicians Group", + ] + } + + mock_llm.return_value = "TEXAS HEALTH PHYSICIANS GROUP" + + df = pd.DataFrame(data) + + result = add_aarete_derived_provider_name(df, state_flag=True) + + self.assertIn("AARETE_DERIVED_PROVIDER_NAME", result.columns) + + self.assertEqual( + result.loc[0, "AARETE_DERIVED_PROVIDER_NAME"], + "TEXAS HEALTH PHYSICIANS GROUP", + ) + + self.assertEqual( + result.loc[1, "AARETE_DERIVED_PROVIDER_NAME"], + "TEXAS HEALTH PHYSICIANS GROUP", + ) + + +if __name__ == "__main__": + unittest.main() diff --git a/src/tests/test_io_utils.py b/src/tests/test_io_utils.py index a94b427..ec5040a 100644 --- a/src/tests/test_io_utils.py +++ b/src/tests/test_io_utils.py @@ -14,6 +14,8 @@ from src.utils.io_utils import ( read_s3_csv, read_xlsb, remove_txt_extension, + split_dataframe_by_filename, + write_local, write_s3, ) @@ -373,7 +375,7 @@ class TestIOUtils: call_kwargs = mock_s3_client.put_object.call_args[1] assert ( call_kwargs["Key"] - == "test_batch_123/20240101120000/test_batch_123-ERRORS.csv" + == "test_batch_123/20240101120000/full_outputs/test_batch_123-ERRORS.csv" ) def test_write_s3_usage(self, mocker): @@ -516,6 +518,263 @@ class TestIOUtils: mock_to_csv.assert_called_once() call_args = mock_to_csv.call_args[0][0] - # Check output path includes outputs/qc_qa folder and correct filename - assert "outputs/qc_qa" in call_args + # Check output path includes automation_qa-qc folder and correct filename + assert "automation_qa-qc" in call_args assert "QC-QA-STATS.csv" in call_args + + # Tests for splitting logic + def test_split_dataframe_by_filename_empty(self): + """Test split_dataframe_by_filename with empty DataFrame.""" + df = pd.DataFrame() + result = split_dataframe_by_filename(df, 70000) + assert len(result) == 1 + assert result[0].empty + + def test_split_dataframe_by_filename_no_file_name_column(self): + """Test split_dataframe_by_filename raises error without FILE_NAME column.""" + df = pd.DataFrame({"col1": [1, 2], "col2": [3, 4]}) + with pytest.raises(ValueError, match="FILE_NAME"): + split_dataframe_by_filename(df, 70000) + + def test_split_dataframe_by_filename_single_file_small(self): + """Test split_dataframe_by_filename with single file under limit.""" + df = pd.DataFrame( + { + "FILE_NAME": ["file1.txt"] * 10, + "col1": range(10), + "col2": range(10, 20), + } + ) + result = split_dataframe_by_filename(df, 70000) + assert len(result) == 1 + assert len(result[0]) == 10 + assert all(result[0]["FILE_NAME"] == "file1.txt") + + def test_split_dataframe_by_filename_single_file_large(self): + """Test split_dataframe_by_filename with single file over limit.""" + # Create a DataFrame with one file that exceeds the limit + df = pd.DataFrame( + { + "FILE_NAME": ["large_file.txt"] * 80000, + "col1": range(80000), + } + ) + result = split_dataframe_by_filename(df, 70000) + # Should still be one split since we can't break up a filename + assert len(result) == 1 + assert len(result[0]) == 80000 + + def test_split_dataframe_by_filename_multiple_files_sorted(self): + """Test split_dataframe_by_filename sorts by row count ascending.""" + # Create files with different row counts + # small.txt: 10 rows total (5 + 5) + # medium.txt: 40 rows total (20 + 20) + # large.txt: 50 rows + df = pd.DataFrame( + { + "FILE_NAME": ( + ["small.txt"] * 5 + + ["medium.txt"] * 20 + + ["large.txt"] * 50 + + ["small.txt"] * 5 + + ["medium.txt"] * 20 + ), + "col1": range(100), + } + ) + result = split_dataframe_by_filename(df, 30) + # With max_rows=30: + # - small (10) fits in first split + # - medium (40) alone exceeds 30, so new split + # - large (50) alone exceeds 30, so new split + # Result: 3 splits + assert len(result) == 3 + # First split should have small (10 rows) + assert len(result[0]) == 10 + assert set(result[0]["FILE_NAME"].unique()) == {"small.txt"} + # Second split should have medium (40 rows) + assert len(result[1]) == 40 + assert set(result[1]["FILE_NAME"].unique()) == {"medium.txt"} + # Third split should have large (50 rows) + assert len(result[2]) == 50 + assert set(result[2]["FILE_NAME"].unique()) == {"large.txt"} + + def test_split_dataframe_by_filename_multiple_splits(self): + """Test split_dataframe_by_filename creates multiple splits correctly.""" + # Create multiple files that will need splitting + df = pd.DataFrame( + { + "FILE_NAME": ( + ["file1.txt"] * 40000 + + ["file2.txt"] * 40000 + + ["file3.txt"] * 40000 + ), + "col1": range(120000), + } + ) + result = split_dataframe_by_filename(df, 70000) + # Each file is 40k, so: file1 (40k) fits, file2 (40k) would make 80k > 70k, so separate + # So we should get 3 splits + assert len(result) == 3 + assert all(len(split) == 40000 for split in result) + + def test_write_local_cc_results_full_single_split(self, mocker, tmp_path): + """Test write_local with cc_results_full that doesn't need splitting.""" + mocker.patch("src.config.BATCH_ID", "test_batch") + mocker.patch("src.config.CONSOLIDATED_OUTPUT_DIRECTORY", str(tmp_path)) + mocker.patch("src.config.PERFORM_PARENT_CHILD_MAPPING", False) + mocker.patch("src.config.MAX_ROWS_PER_SPLIT", 70000) + + df = pd.DataFrame( + { + "FILE_NAME": ["file1.txt"] * 10, + "col1": range(10), + } + ) + result = write_local(df, "", "run_20240101_10-00_test", "cc_results_full") + + # Should create FULL file + expected_file = ( + tmp_path + / "run_20240101_10-00_test" + / "full_outputs" + / "cc_results" + / "test_batch-RESULTS-FULL.csv" + ) + assert expected_file.exists() + assert result is None # No parent-child mapping + + def test_write_local_cc_results_full_multiple_splits(self, mocker, tmp_path): + """Test write_local with cc_results_full that needs splitting.""" + mocker.patch("src.config.BATCH_ID", "test_batch") + mocker.patch("src.config.CONSOLIDATED_OUTPUT_DIRECTORY", str(tmp_path)) + mocker.patch("src.config.PERFORM_PARENT_CHILD_MAPPING", True) + mocker.patch("src.config.MAX_ROWS_PER_SPLIT", 30) + + # Create data that will need splitting + df = pd.DataFrame( + { + "FILE_NAME": ( + ["file1.txt"] * 20 + ["file2.txt"] * 20 + ["file3.txt"] * 20 + ), + "col1": range(60), + } + ) + result = write_local(df, "", "run_20240101_10-00_test", "cc_results_full") + + # Should create SPLIT files + output_dir = ( + tmp_path / "run_20240101_10-00_test" / "full_outputs" / "cc_results" + ) + split1_file = output_dir / "test_batch-RESULTS-SPLIT1.csv" + split2_file = output_dir / "test_batch-RESULTS-SPLIT2.csv" + + assert split1_file.exists() + assert split2_file.exists() + # Should return first split for parent-child mapping + assert result == str(split1_file) + + def test_write_s3_cc_results_full_single_split(self, mocker): + """Test write_s3 with cc_results_full that doesn't need splitting.""" + mock_s3_client = mocker.MagicMock() + mocker.patch("src.config.S3_CLIENT", mock_s3_client) + mocker.patch("src.config.BATCH_ID", "test_batch") + mocker.patch("src.config.S3_OUTPUT_BUCKET", "test-bucket") + mocker.patch("src.config.PERFORM_PARENT_CHILD_MAPPING", False) + mocker.patch("src.config.MAX_ROWS_PER_SPLIT", 70000) + mocker.patch("src.utils.io_utils.logging") + + df = pd.DataFrame( + { + "FILE_NAME": ["file1.txt"] * 10, + "col1": range(10), + } + ) + write_s3(df, "", "run_20240101_10-00_test", "cc_results_full") + + # Should upload FULL file + mock_s3_client.put_object.assert_called_once() + call_kwargs = mock_s3_client.put_object.call_args[1] + assert "full_outputs/cc_results" in call_kwargs["Key"] + assert "RESULTS-FULL.csv" in call_kwargs["Key"] + + def test_write_s3_cc_results_full_multiple_splits(self, mocker): + """Test write_s3 with cc_results_full that needs splitting.""" + mock_s3_client = mocker.MagicMock() + mocker.patch("src.config.S3_CLIENT", mock_s3_client) + mocker.patch("src.config.BATCH_ID", "test_batch") + mocker.patch("src.config.S3_OUTPUT_BUCKET", "test-bucket") + mocker.patch("src.config.PERFORM_PARENT_CHILD_MAPPING", True) + mocker.patch("src.config.MAX_ROWS_PER_SPLIT", 30) + mocker.patch("src.utils.io_utils.logging") + + # Create data that will need splitting + df = pd.DataFrame( + { + "FILE_NAME": ( + ["file1.txt"] * 20 + ["file2.txt"] * 20 + ["file3.txt"] * 20 + ), + "col1": range(60), + } + ) + result = write_s3(df, "", "run_20240101_10-00_test", "cc_results_full") + + # Should upload multiple SPLIT files + assert mock_s3_client.put_object.call_count >= 2 + # Check that SPLIT files were uploaded + call_keys = [ + call[1]["Key"] for call in mock_s3_client.put_object.call_args_list + ] + assert any("SPLIT1" in key for key in call_keys) + assert any("SPLIT2" in key for key in call_keys) + # Should return first split path for parent-child mapping + assert result is not None + assert "SPLIT1" in result + + def test_write_s3_dashboard_results_full_splits(self, mocker): + """Test write_s3 with dashboard_results_full that needs splitting.""" + mock_s3_client = mocker.MagicMock() + mocker.patch("src.config.S3_CLIENT", mock_s3_client) + mocker.patch("src.config.BATCH_ID", "test_batch") + mocker.patch("src.config.S3_OUTPUT_BUCKET", "test-bucket") + mocker.patch("src.config.MAX_ROWS_PER_SPLIT", 30) + mocker.patch("src.utils.io_utils.logging") + + df = pd.DataFrame( + { + "FILE_NAME": (["file1.txt"] * 20 + ["file2.txt"] * 20), + "col1": range(40), + } + ) + write_s3(df, "", "run_20240101_10-00_test", "dashboard_results_full") + + # Should upload multiple SPLIT files + assert mock_s3_client.put_object.call_count >= 2 + call_keys = [ + call[1]["Key"] for call in mock_s3_client.put_object.call_args_list + ] + assert any("dashboard-SPLIT" in key for key in call_keys) + + def test_write_s3_qc_qa_cc_full_splits(self, mocker): + """Test write_s3 with qc_qa_cc_full that needs splitting.""" + mock_s3_client = mocker.MagicMock() + mocker.patch("src.config.S3_CLIENT", mock_s3_client) + mocker.patch("src.config.BATCH_ID", "test_batch") + mocker.patch("src.config.S3_OUTPUT_BUCKET", "test-bucket") + mocker.patch("src.config.MAX_ROWS_PER_SPLIT", 30) + mocker.patch("src.utils.io_utils.logging") + + df = pd.DataFrame( + { + "FILE_NAME": (["file1.txt"] * 20 + ["file2.txt"] * 20), + "col1": range(40), + } + ) + write_s3(df, "", "run_20240101_10-00_test", "qc_qa_cc_full") + + # Should upload multiple SPLIT files + assert mock_s3_client.put_object.call_count >= 2 + call_keys = [ + call[1]["Key"] for call in mock_s3_client.put_object.call_args_list + ] + assert any("QC-QA-SPLIT" in key for key in call_keys) diff --git a/src/tests/test_postprocess.py b/src/tests/test_postprocess.py index e5e1e29..97e3a87 100644 --- a/src/tests/test_postprocess.py +++ b/src/tests/test_postprocess.py @@ -1,12 +1,19 @@ +import json +import os +import tempfile import unittest from unittest.mock import MagicMock, patch import pandas as pd import pytest from src.pipelines.shared.postprocessing.postprocessing_funcs import ( + blank_uom_for_default_flat_rate_cc, + clean_na_values, deduplicate_provider_columns, fill_claim_type_from_title, + add_aarete_derived_signatory_complete_ind, flatten_singleton_string_list, + format_as_json_list, format_rate_fields_with_commas, normalize_auto_renewal_term, normalize_cpt_fields, @@ -15,18 +22,36 @@ from src.pipelines.shared.postprocessing.postprocessing_funcs import ( remove_hyphens, remove_redundant_reimb_info, rename_columns, + standardize_reimb_method_and_fee_schedule, validate_and_reformat_date, ) +from src.pipelines.shared.postprocessing.postprocess import standard_postprocess +from src.constants.constants import Constants class TestPostprocessFunctions(unittest.TestCase): def test_normalize_indicator_field(self): + """Test normalize_indicator_field with various inputs including blanks/None/NaN.""" self.assertEqual(normalize_indicator_field("Y"), "Y") self.assertEqual(normalize_indicator_field("y"), "Y") self.assertEqual(normalize_indicator_field("N"), "N") self.assertEqual(normalize_indicator_field(""), "N") self.assertEqual(normalize_indicator_field(None), "N") + # Test with whitespace-only strings + self.assertEqual(normalize_indicator_field(" "), "N") + self.assertEqual(normalize_indicator_field("\t\n"), "N") + # Test with NaN (float) + import numpy as np + + self.assertEqual(normalize_indicator_field(np.nan), "N") + self.assertEqual(normalize_indicator_field(float("nan")), "N") + # Test with pd.NA + self.assertEqual(normalize_indicator_field(pd.NA), "N") + # Test that non-Y values become N + self.assertEqual(normalize_indicator_field("X"), "N") + self.assertEqual(normalize_indicator_field("Yes"), "N") + self.assertEqual(normalize_indicator_field("1"), "N") def test_format_rate_fields_with_commas(self): # Test numeric values @@ -48,6 +73,17 @@ class TestPostprocessFunctions(unittest.TestCase): self.assertEqual(flatten_singleton_string_list("invalid"), "invalid") self.assertEqual(flatten_singleton_string_list(None), "") + def test_format_as_json_list_returns_blank_for_empty(self): + """Test format_as_json_list returns blank instead of [] for empty lists.""" + self.assertEqual(format_as_json_list([]), "") + self.assertEqual(format_as_json_list("[]"), "") + self.assertEqual(format_as_json_list(None), "") + self.assertEqual(format_as_json_list(""), "") + self.assertEqual(format_as_json_list([""]), "") + self.assertEqual(format_as_json_list(["[]"]), "") + self.assertEqual(format_as_json_list(["Valid"]), '["Valid"]') + self.assertEqual(format_as_json_list(["A", "B"]), '["A", "B"]') + def test_rename_columns(self): """Tests the rename_columns function to ensure it correctly renames specified columns. @@ -108,15 +144,15 @@ class TestPostprocessFunctions(unittest.TestCase): self.assertEqual(normalize_auto_renewal_term(None), "") def test_normalize_cpt_fields(self): - self.assertEqual(normalize_cpt_fields("[123, 456]"), "['123', '456']") - self.assertEqual(normalize_cpt_fields("123-456"), "['123-456']") + self.assertEqual(normalize_cpt_fields("[123, 456]"), '["123", "456"]') + self.assertEqual(normalize_cpt_fields("123-456"), '["123-456"]') self.assertEqual( normalize_cpt_fields("['T0000-T9999, S0000-S9999']"), - "['T0000-T9999', 'S0000-S9999']", + '["T0000-T9999", "S0000-S9999"]', ) - self.assertEqual(normalize_cpt_fields("123"), "['123']") + self.assertEqual(normalize_cpt_fields("123"), '["123"]') self.assertEqual(normalize_cpt_fields(None), "") - self.assertEqual(normalize_cpt_fields(123), "['123']") + self.assertEqual(normalize_cpt_fields(123), '["123"]') def test_process_patient_age_range(self): """Tests the process_patient_age_range function with various age range formats. @@ -662,6 +698,586 @@ class TestPostprocessFunctions(unittest.TestCase): # Empty list should be treated as empty and filled with mode or inferred assert result_df.loc[0, "AARETE_DERIVED_CLAIM_TYPE_CD"] == "M" + def test_add_aarete_derived_signatory_complete_ind(self): + """Test the derivation of the signatory complete indicator for fully signed, partially signed, and zero-signature scenarios.""" + df = pd.DataFrame( + { + "NUM_AVAILABLE_SIGNATORY_LINE_COUNT": [2, 2, 0], + "NUM_SIGNED_SIGNATORY_LINE_COUNT": [2, 1, 0], + } + ) + result = add_aarete_derived_signatory_complete_ind(df) + + self.assertEqual( + result["AARETE_DERIVED_SIGNATORY_COMPLETE_IND"].tolist(), ["Y", "N", "N"] + ) + + def test_clean_na_values_string_placeholders(self): + """Test clean_na_values removes placeholder strings when they're the only value.""" + # Test with string placeholders + input_df = pd.DataFrame( + { + "FIELD1": ["N/A", "Valid Value", "UNKNOWN", "NA"], + "FIELD2": ["Valid", "N/A", "Another Valid", "null"], + } + ) + + result_df = clean_na_values(input_df) + + # N/A, UNKNOWN, NA, null should be removed (become empty strings) + self.assertEqual(result_df.loc[0, "FIELD1"], "") + self.assertEqual(result_df.loc[1, "FIELD1"], "Valid Value") + self.assertEqual(result_df.loc[2, "FIELD1"], "") + self.assertEqual(result_df.loc[3, "FIELD1"], "") + + self.assertEqual(result_df.loc[0, "FIELD2"], "Valid") + self.assertEqual(result_df.loc[1, "FIELD2"], "") + self.assertEqual(result_df.loc[2, "FIELD2"], "Another Valid") + self.assertEqual(result_df.loc[3, "FIELD2"], "") + + def test_clean_na_values_json_list_placeholders(self): + """Test clean_na_values handles JSON list placeholders.""" + # Test with JSON string lists + input_df = pd.DataFrame( + { + "FIELD1": ['["N/A"]', '["Medicare", "N/A"]', '["N/A", "N/A"]'], + "FIELD2": ['["Valid"]', '["N/A"]', '["Medicare"]'], + } + ) + + result_df = clean_na_values(input_df) + + # '["N/A"]' should be removed (only placeholder) + self.assertEqual(result_df.loc[0, "FIELD1"], "") + # '["Medicare", "N/A"]' should be kept (has non-placeholder) + self.assertEqual(result_df.loc[1, "FIELD1"], '["Medicare", "N/A"]') + # '["N/A", "N/A"]' should be removed (all placeholders) + self.assertEqual(result_df.loc[2, "FIELD1"], "") + + self.assertEqual(result_df.loc[0, "FIELD2"], '["Valid"]') + self.assertEqual(result_df.loc[1, "FIELD2"], "") + self.assertEqual(result_df.loc[2, "FIELD2"], '["Medicare"]') + + def test_clean_na_values_empty_list_representations(self): + """Test clean_na_values blanks [] and ['[]'] instead of showing them.""" + input_df = pd.DataFrame( + { + "FIELD1": ["[]", "['[]']", '["[]"]', "Valid"], + "FIELD2": [["[]"], ["Valid"], [], "[]"], + } + ) + + result_df = clean_na_values(input_df) + + self.assertEqual(result_df.loc[0, "FIELD1"], "") + self.assertEqual(result_df.loc[1, "FIELD1"], "") + self.assertEqual(result_df.loc[2, "FIELD1"], "") + self.assertEqual(result_df.loc[3, "FIELD1"], "Valid") + + self.assertEqual(result_df.loc[0, "FIELD2"], "") + self.assertEqual(result_df.loc[1, "FIELD2"], ["Valid"]) + self.assertEqual(result_df.loc[2, "FIELD2"], "") + self.assertEqual(result_df.loc[3, "FIELD2"], "") + + def test_clean_na_values_python_lists(self): + """Test clean_na_values handles Python list types.""" + # Test with Python lists + input_df = pd.DataFrame( + { + "FIELD1": [["N/A"], ["Medicare", "N/A"], ["N/A", "N/A"]], + "FIELD2": [["Valid"], ["N/A"], ["Medicare"]], + } + ) + + result_df = clean_na_values(input_df) + + # ["N/A"] should be removed + self.assertEqual(result_df.loc[0, "FIELD1"], "") + # ["Medicare", "N/A"] should be kept + self.assertEqual(result_df.loc[1, "FIELD1"], ["Medicare", "N/A"]) + # ["N/A", "N/A"] should be removed + self.assertEqual(result_df.loc[2, "FIELD1"], "") + + self.assertEqual(result_df.loc[0, "FIELD2"], ["Valid"]) + self.assertEqual(result_df.loc[1, "FIELD2"], "") + self.assertEqual(result_df.loc[2, "FIELD2"], ["Medicare"]) + + def test_clean_na_values_none_and_nan(self): + """Test clean_na_values handles None and NaN values.""" + import numpy as np + + input_df = pd.DataFrame( + { + "FIELD1": [None, np.nan, pd.NA, "Valid"], + "FIELD2": ["", " ", "Valid", None], + } + ) + + result_df = clean_na_values(input_df) + + # None, NaN, pd.NA, empty strings should be removed (converted to empty strings) + self.assertEqual(result_df.loc[0, "FIELD1"], "") + self.assertEqual(result_df.loc[1, "FIELD1"], "") + # pd.NA should be converted to empty string + result_val = result_df.loc[2, "FIELD1"] + self.assertTrue( + result_val == "" or pd.isna(result_val), + f"Expected empty string or NA, got: {result_val}", + ) + self.assertEqual(result_df.loc[3, "FIELD1"], "Valid") + + self.assertEqual(result_df.loc[0, "FIELD2"], "") + self.assertEqual(result_df.loc[1, "FIELD2"], "") + self.assertEqual(result_df.loc[2, "FIELD2"], "Valid") + self.assertEqual(result_df.loc[3, "FIELD2"], "") + + def test_clean_na_values_empty_dataframe(self): + """Test clean_na_values handles empty DataFrame.""" + empty_df = pd.DataFrame() + result_df = clean_na_values(empty_df) + pd.testing.assert_frame_equal(result_df, empty_df) + + def test_clean_na_values_case_insensitive(self): + """Test clean_na_values is case-insensitive for placeholder matching.""" + input_df = pd.DataFrame( + { + "FIELD1": ["n/a", "N/A", "Na", "unknown", "UNKNOWN", "None", "NULL"], + } + ) + + result_df = clean_na_values(input_df) + + # All should be removed (case-insensitive matching) + for idx in range(len(result_df)): + self.assertEqual(result_df.loc[idx, "FIELD1"], "") + + def test_standard_postprocess_cleans_na_and_normalizes_ind(self): + """Test standard_postprocess removes N/A values and normalizes _IND fields.""" + constants = Constants() + input_df = pd.DataFrame( + { + "FILE_NAME": ["test1.pdf", "test2.pdf"], + "PAYER_NAME": [ + "N/A", + "Valid Value", + ], # Use actual column from FIELD_FORMAT_MAPPING + "AUTO_RENEWAL_IND": ["Y", ""], + "DSH_IND": [None, "N"], + "IME_IND": [" ", "y"], + "NETWORK": ['["N/A"]', '["Medicare"]'], + } + ) + + result_df = standard_postprocess(input_df, constants) + + # N/A should be removed (PAYER_NAME is in FIELD_FORMAT_MAPPING so it will be preserved) + self.assertEqual(result_df.loc[0, "PAYER_NAME"], "") + self.assertEqual(result_df.loc[1, "PAYER_NAME"], "Valid Value") + + # _IND fields should be normalized to Y or N + self.assertEqual(result_df.loc[0, "AUTO_RENEWAL_IND"], "Y") + self.assertEqual(result_df.loc[1, "AUTO_RENEWAL_IND"], "N") # Empty becomes N + + self.assertEqual(result_df.loc[0, "DSH_IND"], "N") # None becomes N + self.assertEqual(result_df.loc[1, "DSH_IND"], "N") + + self.assertEqual(result_df.loc[0, "IME_IND"], "N") # Whitespace becomes N + self.assertEqual(result_df.loc[1, "IME_IND"], "Y") # "y" becomes "Y" + + # JSON list with only N/A should be removed + self.assertEqual(result_df.loc[0, "NETWORK"], "") + # JSON list with valid value should be kept + self.assertEqual(result_df.loc[1, "NETWORK"], '["Medicare"]') + + def test_standard_postprocess_empty_dataframe(self): + """Test standard_postprocess handles empty DataFrame.""" + constants = Constants() + empty_df = pd.DataFrame() + result_df = standard_postprocess(empty_df, constants) + pd.testing.assert_frame_equal(result_df, empty_df) + + def test_standard_postprocess_cleans_na_without_ind_fields(self): + """Test standard_postprocess cleans N/A values even when no _IND fields are present.""" + constants = Constants() + input_df = pd.DataFrame( + { + "FILE_NAME": ["test1.pdf", "test2.pdf"], + "PAYER_NAME": [ + "N/A", + "Valid", + ], # Use actual column from FIELD_FORMAT_MAPPING + "CONTRACT_TITLE": [ + "UNKNOWN", + "Another Valid", + ], # Use actual column from FIELD_FORMAT_MAPPING + } + ) + + result_df = standard_postprocess(input_df, constants) + + # Should still clean N/A values (these columns are in FIELD_FORMAT_MAPPING so they will be preserved) + self.assertEqual(result_df.loc[0, "PAYER_NAME"], "") + self.assertEqual(result_df.loc[1, "PAYER_NAME"], "Valid") + self.assertEqual(result_df.loc[0, "CONTRACT_TITLE"], "") + self.assertEqual(result_df.loc[1, "CONTRACT_TITLE"], "Another Valid") + + +class TestStandardizeReimbMethodAndFeeScheduleUOM(unittest.TestCase): + """Tests for UNIT_OF_MEASURE post-processing when DEFAULT_IND='Y' and flat rate.""" + + def test_cc_default_flat_rate_uom_blank(self): + """CC only: DEFAULT_IND='Y' and Flat Rate -> UNIT_OF_MEASURE blank.""" + df = pd.DataFrame( + { + "DEFAULT_IND": ["Y"], + "AARETE_DERIVED_REIMB_METHOD": ["Flat Rate"], + "UNIT_OF_MEASURE": ["Per Unit"], + } + ) + result = blank_uom_for_default_flat_rate_cc(df) + self.assertEqual(result.loc[0, "UNIT_OF_MEASURE"], "") + + def test_cc_default_flat_rate_case_insensitive(self): + """CC only: DEFAULT_IND='Y' and 'flat rate' (lowercase) -> UNIT_OF_MEASURE blank.""" + df = pd.DataFrame( + { + "DEFAULT_IND": ["Y"], + "AARETE_DERIVED_REIMB_METHOD": ["flat rate"], + "UNIT_OF_MEASURE": ["Per Visit"], + } + ) + result = blank_uom_for_default_flat_rate_cc(df) + self.assertEqual(result.loc[0, "UNIT_OF_MEASURE"], "") + + def test_standardize_does_not_blank_default_flat_rate(self): + """Shared path: default flat rate keeps UOM (blanking is CC-only).""" + constants = Constants() + df = pd.DataFrame( + { + "DEFAULT_IND": ["Y"], + "AARETE_DERIVED_REIMB_METHOD": ["Flat Rate"], + "UNIT_OF_MEASURE": ["Per Unit"], + "REIMB_TERM": [""], + } + ) + result = standardize_reimb_method_and_fee_schedule( + df, constants.VALID_UNIT_OF_MEASURE + ) + self.assertEqual(result.loc[0, "UNIT_OF_MEASURE"], "Per Unit") + + def test_non_default_flat_rate_uom_per_unit(self): + """DEFAULT_IND='N', flat rate, empty UOM -> UNIT_OF_MEASURE becomes Per Unit.""" + constants = Constants() + df = pd.DataFrame( + { + "DEFAULT_IND": ["N"], + "AARETE_DERIVED_REIMB_METHOD": ["Flat Rate"], + "UNIT_OF_MEASURE": [""], + "REIMB_TERM": [""], + } + ) + result = standardize_reimb_method_and_fee_schedule( + df, constants.VALID_UNIT_OF_MEASURE + ) + self.assertEqual(result.loc[0, "UNIT_OF_MEASURE"], "Per Unit") + + def test_default_non_flat_rate_uom_unchanged(self): + """DEFAULT_IND='Y' and Fee Schedule -> UNIT_OF_MEASURE unchanged.""" + constants = Constants() + df = pd.DataFrame( + { + "DEFAULT_IND": ["Y"], + "AARETE_DERIVED_REIMB_METHOD": ["Fee Schedule"], + "UNIT_OF_MEASURE": ["Per Visit"], + "REIMB_TERM": [""], + } + ) + result = standardize_reimb_method_and_fee_schedule( + df, constants.VALID_UNIT_OF_MEASURE + ) + self.assertEqual(result.loc[0, "UNIT_OF_MEASURE"], "Per Visit") + + +class TestOutputFileStructure(unittest.TestCase): + """Test that the new output file structure is generated correctly.""" + + def setUp(self): + """Create temporary directory for outputs.""" + self.temp_dir = tempfile.mkdtemp() + self.run_timestamp = "run_20250106_10-30_test" + self.batch_id = "test_batch" + + def tearDown(self): + """Clean up temporary directory.""" + import shutil + + shutil.rmtree(self.temp_dir, ignore_errors=True) + + @patch("src.utils.io_utils.config") + def test_write_local_cc_results_full_structure(self, mock_config): + """Test that cc_results_full creates correct directory structure.""" + mock_config.BATCH_ID = "test_batch" + mock_config.CONSOLIDATED_OUTPUT_DIRECTORY = self.temp_dir + mock_config.PERFORM_PARENT_CHILD_MAPPING = False + mock_config.MAX_ROWS_PER_SPLIT = 70000 + + from src.utils.io_utils import write_local + + # Add FILE_NAME column required for splitting logic + test_df = pd.DataFrame( + {"FILE_NAME": ["file1.txt", "file2.txt"], "col1": [1, 2], "col2": [3, 4]} + ) + + write_local(test_df, "", self.run_timestamp, "cc_results_full") + + # Check directory structure was created + expected_dir = os.path.join( + self.temp_dir, self.run_timestamp, "full_outputs", "cc_results" + ) + self.assertTrue(os.path.exists(expected_dir)) + + # Check file was created + expected_file = os.path.join(expected_dir, "test_batch-RESULTS-FULL.csv") + self.assertTrue(os.path.exists(expected_file)) + + @patch("src.utils.io_utils.config") + def test_write_local_dashboard_results_full_structure(self, mock_config): + """Test that dashboard_results_full creates correct directory structure.""" + mock_config.BATCH_ID = "test_batch" + mock_config.CONSOLIDATED_OUTPUT_DIRECTORY = self.temp_dir + mock_config.PERFORM_PARENT_CHILD_MAPPING = False + mock_config.MAX_ROWS_PER_SPLIT = 70000 + + from src.utils.io_utils import write_local + + # Add FILE_NAME column required for splitting logic + test_df = pd.DataFrame( + {"FILE_NAME": ["file1.txt", "file2.txt"], "col1": [1, 2], "col2": [3, 4]} + ) + + write_local(test_df, "", self.run_timestamp, "dashboard_results_full") + + # Check directory structure was created + expected_dir = os.path.join( + self.temp_dir, self.run_timestamp, "full_outputs", "dashboard_results" + ) + self.assertTrue(os.path.exists(expected_dir)) + + # Check file was created + expected_file = os.path.join(expected_dir, "test_batch-RESULTS-dashboard.csv") + self.assertTrue(os.path.exists(expected_file)) + + @patch("src.utils.io_utils.config") + def test_write_local_error_in_full_outputs(self, mock_config): + """Test that error files go to full_outputs/ directory.""" + mock_config.BATCH_ID = "test_batch" + mock_config.CONSOLIDATED_OUTPUT_DIRECTORY = self.temp_dir + mock_config.PERFORM_PARENT_CHILD_MAPPING = False + + from src.utils.io_utils import write_local + + test_df = pd.DataFrame({"error": ["Error 1", "Error 2"]}) + + write_local(test_df, "", self.run_timestamp, "error") + + # Check directory structure was created + expected_dir = os.path.join(self.temp_dir, self.run_timestamp, "full_outputs") + self.assertTrue(os.path.exists(expected_dir)) + + # Check file was created + expected_file = os.path.join(expected_dir, "test_batch-ERRORS.csv") + self.assertTrue(os.path.exists(expected_file)) + + @patch("src.utils.io_utils.config") + def test_write_local_qc_qa_cc_full_structure(self, mock_config): + """Test that qc_qa_cc_full creates correct directory structure.""" + mock_config.BATCH_ID = "test_batch" + mock_config.CONSOLIDATED_OUTPUT_DIRECTORY = self.temp_dir + mock_config.PERFORM_PARENT_CHILD_MAPPING = False + mock_config.MAX_ROWS_PER_SPLIT = 70000 + + from src.utils.io_utils import write_local + + # Add FILE_NAME column required for splitting logic + test_df = pd.DataFrame( + {"FILE_NAME": ["file1.txt", "file2.txt"], "col1": [1, 2], "col2": [3, 4]} + ) + + write_local(test_df, "", self.run_timestamp, "qc_qa_cc_full") + + # Check directory structure was created + expected_dir = os.path.join( + self.temp_dir, self.run_timestamp, "automation_qa-qc" + ) + self.assertTrue(os.path.exists(expected_dir)) + + # Check file was created + expected_file = os.path.join(expected_dir, "test_batch-RESULTS-QC-QA-FULL.csv") + self.assertTrue(os.path.exists(expected_file)) + + @patch("src.utils.io_utils.config") + def test_write_local_qc_qa_stats_structure(self, mock_config): + """Test that qc_qa_stats creates correct directory structure.""" + mock_config.BATCH_ID = "test_batch" + mock_config.CONSOLIDATED_OUTPUT_DIRECTORY = self.temp_dir + mock_config.PERFORM_PARENT_CHILD_MAPPING = False + + from src.utils.io_utils import write_local + + test_df = pd.DataFrame({"stat": ["value1", "value2"]}) + + write_local(test_df, "", self.run_timestamp, "qc_qa_stats") + + # Check directory structure was created + expected_dir = os.path.join( + self.temp_dir, self.run_timestamp, "automation_qa-qc" + ) + self.assertTrue(os.path.exists(expected_dir)) + + # Check file was created + expected_file = os.path.join(expected_dir, "test_batch-QC-QA-STATS.csv") + self.assertTrue(os.path.exists(expected_file)) + + @patch("src.utils.io_utils.config") + def test_write_local_parent_child_structure(self, mock_config): + """Test that parent_child creates correct directory structure and file path.""" + mock_config.BATCH_ID = "test_batch" + mock_config.CONSOLIDATED_OUTPUT_DIRECTORY = self.temp_dir + mock_config.PERFORM_PARENT_CHILD_MAPPING = False + + from src.utils.io_utils import write_local + + test_df = pd.DataFrame({"col1": [1, 2]}) + + # Mock to_csv on the DataFrame instance to avoid actual file I/O + test_df.to_csv = MagicMock() + + write_local(test_df, "", self.run_timestamp, "parent_child") + + # Verify to_csv was called on the DataFrame + test_df.to_csv.assert_called_once() + # Verify the call had index=False and quoting=1 + call_kwargs = test_df.to_csv.call_args[1] + self.assertEqual(call_kwargs.get("index"), False) + self.assertEqual(call_kwargs.get("quoting"), 1) + + # Verify the file path contains the expected components + call_args = test_df.to_csv.call_args[0] + file_path = call_args[0] if call_args else None + self.assertIsNotNone(file_path, "to_csv should be called with a file path") + self.assertIn( + "parent-child", + file_path, + f"File path should contain 'parent-child', got: {file_path}", + ) + self.assertIn( + "test_batch-PC.csv", + file_path, + f"File path should contain 'test_batch-PC.csv', got: {file_path}", + ) + + # Verify the directory structure would be correct + expected_dir = os.path.join(self.temp_dir, self.run_timestamp, "parent-child") + self.assertIn( + expected_dir, + file_path, + f"File path should contain expected directory {expected_dir}, got: {file_path}", + ) + + @patch("src.utils.io_utils.config") + @patch("src.utils.io_utils.logging") + def test_write_s3_cc_results_full_path(self, mock_logging, mock_config): + """Test that write_s3 uses correct S3 path for cc_results_full.""" + mock_config.BATCH_ID = "test_batch" + mock_config.S3_OUTPUT_BUCKET = "test-bucket" + mock_config.S3_CLIENT = MagicMock() + mock_config.S3_CLIENT.put_object = MagicMock() + mock_config.PERFORM_PARENT_CHILD_MAPPING = False + mock_config.MAX_ROWS_PER_SPLIT = 70000 + + from src.utils.io_utils import write_s3 + + # Add FILE_NAME column required for splitting logic + test_df = pd.DataFrame( + {"FILE_NAME": ["file1.txt", "file2.txt"], "col1": [1, 2]} + ) + + write_s3(test_df, "", self.run_timestamp, "cc_results_full") + + # Verify S3 put_object was called + mock_config.S3_CLIENT.put_object.assert_called_once() + + # Verify the S3 key/path + call_args = mock_config.S3_CLIENT.put_object.call_args + s3_key = call_args[1]["Key"] + expected_key = f"test_batch/{self.run_timestamp}/full_outputs/cc_results/test_batch-RESULTS-FULL.csv" + self.assertEqual(s3_key, expected_key) + + @patch("src.utils.io_utils.config") + @patch("src.utils.io_utils.logging") + def test_write_s3_dashboard_results_full_path(self, mock_logging, mock_config): + """Test that write_s3 uses correct S3 path for dashboard_results_full.""" + mock_config.BATCH_ID = "test_batch" + mock_config.S3_OUTPUT_BUCKET = "test-bucket" + mock_config.S3_CLIENT = MagicMock() + mock_config.S3_CLIENT.put_object = MagicMock() + mock_config.PERFORM_PARENT_CHILD_MAPPING = False + mock_config.MAX_ROWS_PER_SPLIT = 70000 + + from src.utils.io_utils import write_s3 + + # Add FILE_NAME column required for splitting logic + test_df = pd.DataFrame( + {"FILE_NAME": ["file1.txt", "file2.txt"], "col1": [1, 2]} + ) + + write_s3(test_df, "", self.run_timestamp, "dashboard_results_full") + + # Verify S3 put_object was called + mock_config.S3_CLIENT.put_object.assert_called_once() + + # Verify the S3 key/path + call_args = mock_config.S3_CLIENT.put_object.call_args + s3_key = call_args[1]["Key"] + expected_key = f"test_batch/{self.run_timestamp}/full_outputs/dashboard_results/test_batch-RESULTS-dashboard.csv" + self.assertEqual(s3_key, expected_key) + + @patch("src.utils.io_utils.config") + @patch("src.utils.io_utils.logging") + def test_write_s3_qc_qa_paths(self, mock_logging, mock_config): + """Test that write_s3 uses correct S3 paths for QC/QA outputs.""" + mock_config.BATCH_ID = "test_batch" + mock_config.S3_OUTPUT_BUCKET = "test-bucket" + mock_config.S3_CLIENT = MagicMock() + mock_config.S3_CLIENT.put_object = MagicMock() + mock_config.PERFORM_PARENT_CHILD_MAPPING = False + mock_config.MAX_ROWS_PER_SPLIT = 70000 + + from src.utils.io_utils import write_s3 + + # Add FILE_NAME column required for splitting logic + test_df = pd.DataFrame( + {"FILE_NAME": ["file1.txt", "file2.txt"], "col1": [1, 2]} + ) + + # Test qc_qa_cc_full + write_s3(test_df, "", self.run_timestamp, "qc_qa_cc_full") + call_args = mock_config.S3_CLIENT.put_object.call_args + s3_key = call_args[1]["Key"] + expected_key = f"test_batch/{self.run_timestamp}/automation_qa-qc/test_batch-RESULTS-QC-QA-FULL.csv" + self.assertEqual(s3_key, expected_key) + + # Test qc_qa_stats + mock_config.S3_CLIENT.put_object.reset_mock() + test_df_stats = pd.DataFrame( + {"col1": [1, 2]} + ) # qc_qa_stats doesn't need FILE_NAME + write_s3(test_df_stats, "", self.run_timestamp, "qc_qa_stats") + call_args = mock_config.S3_CLIENT.put_object.call_args + s3_key = call_args[1]["Key"] + expected_key = f"test_batch/{self.run_timestamp}/automation_qa-qc/test_batch-QC-QA-STATS.csv" + self.assertEqual(s3_key, expected_key) + if __name__ == "__main__": unittest.main() diff --git a/src/tests/test_qc_qa_pipeline.py b/src/tests/test_qc_qa_pipeline.py index 5acd397..4200e0f 100644 --- a/src/tests/test_qc_qa_pipeline.py +++ b/src/tests/test_qc_qa_pipeline.py @@ -574,52 +574,151 @@ class TestPipelineFunctions: assert "Manual Review Flag (Blank)" in result.columns # Tests for save_qc_qa_outputs + def _patch_save_qc_qa_common(self, mocker): + """Shared patches for save_qc_qa_outputs tests. + + Returns (mock_excel_writer, mock_stats_df) where mock_stats_df is a + MagicMock whose .to_excel() is a no-op (avoids real file I/O when + pandas calls to_excel on the writer context). + """ + mocker.patch("os.makedirs") + mocker.patch("src.config.BATCH_ID", "test_batch") + mocker.patch("src.config.QC_QA_OUTPUT_DIRECTORY", "/tmp/qc_qa") + + mock_writer_ctx = MagicMock() + mock_excel_writer = mocker.patch("src.qc_qa.pipeline.pd.ExcelWriter") + mock_excel_writer.return_value.__enter__ = MagicMock( + return_value=mock_writer_ctx + ) + mock_excel_writer.return_value.__exit__ = MagicMock(return_value=False) + + # Use a MagicMock DataFrame so .to_excel() is a no-op + mock_stats_df = MagicMock() + return mock_excel_writer, mock_stats_df + def test_save_qc_qa_outputs_local_only(self, sample_dataframe, mocker): """Test saving outputs locally without S3.""" mock_write_local = mocker.patch("src.utils.io_utils.write_local") mock_write_s3 = mocker.patch("src.utils.io_utils.write_s3") + mock_excel_writer, mock_stats_df = self._patch_save_qc_qa_common(mocker) save_qc_qa_outputs( validated_df=sample_dataframe, - stats_df=sample_dataframe, + stats_df=mock_stats_df, run_timestamp="run_20241114_10-00_test", write_to_s3=False, ) - # Verify local writes were called - assert mock_write_local.call_count == 2 - mock_write_local.assert_any_call( + # write_local called once for validated_df only (stats go to Excel) + mock_write_local.assert_called_once_with( sample_dataframe, "", "run_20241114_10-00_test", "qc_qa_validated" ) - mock_write_local.assert_any_call( - sample_dataframe, "", "run_20241114_10-00_test", "qc_qa_stats" - ) - # Verify S3 was NOT called + # Excel writer was invoked for stats + mock_excel_writer.assert_called_once() + + # S3 was NOT called mock_write_s3.assert_not_called() def test_save_qc_qa_outputs_with_s3(self, sample_dataframe, mocker): - """Test saving outputs locally and to S3.""" + """Test saving outputs locally and to S3 (validated_df + Excel upload).""" mock_write_local = mocker.patch("src.utils.io_utils.write_local") mock_write_s3 = mocker.patch("src.utils.io_utils.write_s3") + mock_excel_writer, mock_stats_df = self._patch_save_qc_qa_common(mocker) + mocker.patch("src.config.S3_OUTPUT_BUCKET", "test-bucket") + mock_s3_client = MagicMock() + mocker.patch("src.config.S3_CLIENT", mock_s3_client) + mocker.patch("builtins.open", mocker.mock_open(read_data=b"")) save_qc_qa_outputs( validated_df=sample_dataframe, - stats_df=sample_dataframe, + stats_df=mock_stats_df, run_timestamp="run_20241114_10-00_test", write_to_s3=True, ) - # Verify local writes - assert mock_write_local.call_count == 2 - - # Verify S3 writes were called - assert mock_write_s3.call_count == 2 - mock_write_s3.assert_any_call( + # write_local called once for validated_df + mock_write_local.assert_called_once_with( sample_dataframe, "", "run_20241114_10-00_test", "qc_qa_validated" ) - mock_write_s3.assert_any_call( - sample_dataframe, "", "run_20241114_10-00_test", "qc_qa_stats" + + # write_s3 called once for validated_df + mock_write_s3.assert_called_once_with( + sample_dataframe, "", "run_20241114_10-00_test", "qc_qa_validated" + ) + + # S3 put_object called once for the combined Excel + mock_s3_client.put_object.assert_called_once() + call_kwargs = mock_s3_client.put_object.call_args[1] + assert call_kwargs["Bucket"] == "test-bucket" + assert "QC-QA-STATS.xlsx" in call_kwargs["Key"] + + def test_save_qc_qa_outputs_with_tin_stats(self, sample_dataframe, mocker): + """Test that tin_stats_df and tin_contract_summary_df are written as extra sheets.""" + mocker.patch("src.utils.io_utils.write_local") + mock_excel_writer, mock_stats_df = self._patch_save_qc_qa_common(mocker) + + # Use MagicMocks so to_excel() calls are no-ops + tin_stats = MagicMock() + tin_summary = MagicMock() + + save_qc_qa_outputs( + validated_df=sample_dataframe, + stats_df=mock_stats_df, + run_timestamp="run_20241114_10-00_test", + write_to_s3=False, + tin_stats_df=tin_stats, + tin_contract_summary_df=tin_summary, + ) + + # ExcelWriter created with the expected path + mock_excel_writer.assert_called_once() + excel_path_arg = mock_excel_writer.call_args[0][0] + assert "test_batch-QC-QA-STATS.xlsx" in excel_path_arg + + # to_excel called on all three DataFrames (stats, tin_stats, tin_summary) + mock_stats_df.to_excel.assert_called_once() + tin_stats.to_excel.assert_called_once() + tin_summary.to_excel.assert_called_once() + + def test_save_qc_qa_outputs_without_optional_tin_dfs( + self, sample_dataframe, mocker + ): + """Test that omitting tin_stats_df / tin_contract_summary_df writes only QC-QA-STATS sheet.""" + mocker.patch("src.utils.io_utils.write_local") + mock_excel_writer, mock_stats_df = self._patch_save_qc_qa_common(mocker) + + save_qc_qa_outputs( + validated_df=sample_dataframe, + stats_df=mock_stats_df, + run_timestamp="run_20241114_10-00_test", + write_to_s3=False, + ) + + # ExcelWriter created with the expected path + excel_path_arg = mock_excel_writer.call_args[0][0] + assert "test_batch-QC-QA-STATS.xlsx" in excel_path_arg + + # Only stats_df.to_excel was called (no TIN sheets) + mock_stats_df.to_excel.assert_called_once() + + def test_save_qc_qa_outputs_s3_failure_is_warned(self, sample_dataframe, mocker): + """Test that S3 upload failure logs a warning rather than raising.""" + mocker.patch("src.utils.io_utils.write_local") + mocker.patch("src.utils.io_utils.write_s3") + _, mock_stats_df = self._patch_save_qc_qa_common(mocker) + mocker.patch("src.config.S3_OUTPUT_BUCKET", "test-bucket") + mock_s3_client = MagicMock() + mock_s3_client.put_object.side_effect = Exception("S3 unavailable") + mocker.patch("src.config.S3_CLIENT", mock_s3_client) + mocker.patch("builtins.open", mocker.mock_open(read_data=b"")) + + # Should not raise — failure is caught and logged as warning + save_qc_qa_outputs( + validated_df=sample_dataframe, + stats_df=mock_stats_df, + run_timestamp="run_20241114_10-00_test", + write_to_s3=True, ) # Tests for generate_statistics diff --git a/src/utils/formatting_utils.py b/src/utils/formatting_utils.py index c2ee6fc..e4a030b 100644 --- a/src/utils/formatting_utils.py +++ b/src/utils/formatting_utils.py @@ -81,8 +81,8 @@ def _normalize_to_str(value: Any) -> str: elif len(value) == 1: return str(value[0]) else: - # Join multiple values with a delimiter (pipe for backward compatibility) - return "|".join(str(v) for v in value) + # Multiple values + return str(value) elif isinstance(value, dict): # Convert dict to string representation return json.dumps(value) diff --git a/src/utils/io_utils.py b/src/utils/io_utils.py index 739f0dd..d47bb55 100644 --- a/src/utils/io_utils.py +++ b/src/utils/io_utils.py @@ -338,6 +338,163 @@ def read_xlsb( return df +def split_dataframe_by_filename( + df: pd.DataFrame, max_rows_per_split: int = 70000 +) -> list[pd.DataFrame]: + """ + Split a DataFrame into multiple DataFrames based on filename groups. + + This function ensures that all rows from the same filename stay together. + It sorts filenames by row count (ascending) and creates splits that are as + close to max_rows_per_split as possible without breaking up filename groups. + + Args: + df (pd.DataFrame): The DataFrame to split. Must have a 'FILE_NAME' column. + max_rows_per_split (int): Maximum number of rows per split file. Default: 70000. + + Returns: + list[pd.DataFrame]: List of DataFrames, each representing a split file. + + Raises: + ValueError: If DataFrame doesn't have 'FILE_NAME' column or is empty. + """ + if df.empty: + return [df] + + if "FILE_NAME" not in df.columns: + raise ValueError("DataFrame must have a 'FILE_NAME' column for splitting") + + # Count rows per filename + filename_counts = df["FILE_NAME"].value_counts().sort_values(ascending=True) + filename_order = filename_counts.index.tolist() + + # Create list to store split DataFrames + splits = [] + current_split_rows = [] + current_split_count = 0 + + for filename in filename_order: + filename_df = df[df["FILE_NAME"] == filename] + filename_row_count = len(filename_df) + + # If adding this filename would exceed the limit, start a new split + if ( + current_split_count + filename_row_count > max_rows_per_split + and current_split_rows + ): + # Create and save current split + current_split = pd.concat(current_split_rows, ignore_index=True) + splits.append(current_split) + # Start new split with current filename + current_split_rows = [filename_df] + current_split_count = filename_row_count + else: + # Add filename to current split + current_split_rows.append(filename_df) + current_split_count += filename_row_count + + # Add the last split if it has any rows + if current_split_rows: + final_split = pd.concat(current_split_rows, ignore_index=True) + splits.append(final_split) + + return splits if splits else [df] + + +def _write_local_split_files( + df: pd.DataFrame, + output_dir: str, + full_filename: str, + split_filename_template: str, + return_path_for_pc: bool = False, +) -> Optional[str]: + """ + Helper function to write split files locally. + + Args: + df: DataFrame to write (will be split if needed) + output_dir: Full output directory path + full_filename: Filename for single file (e.g., "{BATCH_ID}-RESULTS-FULL.csv") + split_filename_template: Template for split files (e.g., "{BATCH_ID}-RESULTS-SPLIT{idx}.csv") + return_path_for_pc: Whether to return path for parent-child mapping + + Returns: + Full file path string if return_path_for_pc is True, None otherwise + """ + os.makedirs(output_dir, exist_ok=True) + splits = split_dataframe_by_filename(df, config.MAX_ROWS_PER_SPLIT) + + if len(splits) == 1: + output_path = os.path.join(output_dir, full_filename) + splits[0].to_csv(output_path, index=False, quoting=1) + return output_path if return_path_for_pc else None + else: + # Multiple splits - write numbered files + for idx, split_df in enumerate(splits, start=1): + split_filename = split_filename_template.format(idx=idx) + split_output_path = os.path.join(output_dir, split_filename) + split_df.to_csv(split_output_path, index=False, quoting=1) + # For parent-child mapping, use the first split file + if return_path_for_pc: + first_split_filename = split_filename_template.format(idx=1) + return os.path.join(output_dir, first_split_filename) + return None + + +def _write_s3_split_files( + df: pd.DataFrame, + run_timestamp: str, + base_path: str, + full_filename: str, + split_filename_template: str, + return_path_for_pc: bool = False, +) -> Optional[str]: + """ + Helper function to write split files to S3. + + Args: + df: DataFrame to write (will be split if needed) + run_timestamp: Timestamp string for path + base_path: Base S3 path (e.g., "full_outputs/cc_results") + full_filename: Filename for single file (e.g., "{BATCH_ID}-RESULTS-FULL.csv") + split_filename_template: Template for split files (e.g., "{BATCH_ID}-RESULTS-SPLIT{idx}.csv") + return_path_for_pc: Whether to return path for parent-child mapping + + Returns: + S3 path string if return_path_for_pc is True, None otherwise + """ + splits = split_dataframe_by_filename(df, config.MAX_ROWS_PER_SPLIT) + + if len(splits) == 1: + output_path = f"{config.BATCH_ID}/{run_timestamp}/{base_path}/{full_filename}" + csv_buffer = splits[0].to_csv(index=False).encode() + config.S3_CLIENT.put_object( + Bucket=config.S3_OUTPUT_BUCKET, Key=output_path, Body=csv_buffer + ) + logging.info(f"Saved {base_path.split('/')[-1]} full file: {output_path}") + return output_path if return_path_for_pc else None + else: + # Multiple splits - upload each one + for idx, split_df in enumerate(splits, start=1): + split_filename = split_filename_template.format(idx=idx) + split_output_path = ( + f"{config.BATCH_ID}/{run_timestamp}/{base_path}/{split_filename}" + ) + csv_buffer = split_df.to_csv(index=False).encode() + config.S3_CLIENT.put_object( + Bucket=config.S3_OUTPUT_BUCKET, Key=split_output_path, Body=csv_buffer + ) + logging.info( + f"Saved {base_path.split('/')[-1]} split files ({len(splits)} files)" + ) + if return_path_for_pc: + first_split_filename = split_filename_template.format(idx=1) + return ( + f"{config.BATCH_ID}/{run_timestamp}/{base_path}/{first_split_filename}" + ) + return None + + def write_local( df: pd.DataFrame, filename: str, run_timestamp: str, output_type: str ) -> Optional[str]: @@ -386,18 +543,12 @@ def write_local( config.OUTPUT_DIRECTORY, f"{base_filename}_-RESULTS.csv" ) - return None - elif output_type == "parent_child": - output_dir = os.path.join(config.CONSOLIDATED_OUTPUT_DIRECTORY, run_timestamp) - os.makedirs(output_dir, exist_ok=True) - df.to_csv( - os.path.join(output_dir, f"{config.BATCH_ID}-Parent_Child.csv"), - index=False, - quoting=1, - ) return None elif output_type == "error": - output_dir = os.path.join(config.CONSOLIDATED_OUTPUT_DIRECTORY, run_timestamp) + # Error file goes in full_outputs/ folder + output_dir = os.path.join( + config.CONSOLIDATED_OUTPUT_DIRECTORY, run_timestamp, "full_outputs" + ) os.makedirs(output_dir, exist_ok=True) df.to_csv( os.path.join(output_dir, f"{config.BATCH_ID}-ERRORS.csv"), @@ -407,6 +558,74 @@ def write_local( if config.PERFORM_PARENT_CHILD_MAPPING: return os.path.join(output_dir, f"{config.BATCH_ID}-RESULTS.csv") return None + elif output_type == "cc_results_full": + # Full CC results (may be split into multiple files) + output_dir = os.path.join( + config.CONSOLIDATED_OUTPUT_DIRECTORY, + run_timestamp, + "full_outputs", + "cc_results", + ) + return _write_local_split_files( + df=df, + output_dir=output_dir, + full_filename=f"{config.BATCH_ID}-RESULTS-FULL.csv", + split_filename_template=f"{config.BATCH_ID}-RESULTS-SPLIT{{idx}}.csv", + return_path_for_pc=config.PERFORM_PARENT_CHILD_MAPPING, + ) + elif output_type == "dashboard_results_full": + # Full dashboard results (may be split into multiple files) + output_dir = os.path.join( + config.CONSOLIDATED_OUTPUT_DIRECTORY, + run_timestamp, + "full_outputs", + "dashboard_results", + ) + _write_local_split_files( + df=df, + output_dir=output_dir, + full_filename=f"{config.BATCH_ID}-RESULTS-dashboard.csv", + split_filename_template=f"{config.BATCH_ID}-RESULTS-dashboard-SPLIT{{idx}}.csv", + return_path_for_pc=False, + ) + return None + elif output_type == "qc_qa_cc_full": + # QC/QA CC full results (may be split into multiple files) + output_dir = os.path.join( + config.CONSOLIDATED_OUTPUT_DIRECTORY, run_timestamp, "automation_qa-qc" + ) + _write_local_split_files( + df=df, + output_dir=output_dir, + full_filename=f"{config.BATCH_ID}-RESULTS-QC-QA-FULL.csv", + split_filename_template=f"{config.BATCH_ID}-RESULTS-QC-QA-SPLIT{{idx}}.csv", + return_path_for_pc=False, + ) + return None + elif output_type == "qc_qa_error": + # QC/QA error results + output_dir = os.path.join( + config.CONSOLIDATED_OUTPUT_DIRECTORY, run_timestamp, "automation_qa-qc" + ) + os.makedirs(output_dir, exist_ok=True) + df.to_csv( + os.path.join(output_dir, f"{config.BATCH_ID}-ERRORS-QC-QA.csv"), + index=False, + quoting=1, + ) + return None + elif output_type == "parent_child": + # Parent-child output + output_dir = os.path.join( + config.CONSOLIDATED_OUTPUT_DIRECTORY, run_timestamp, "parent-child" + ) + os.makedirs(output_dir, exist_ok=True) + df.to_csv( + os.path.join(output_dir, f"{config.BATCH_ID}-PC.csv"), + index=False, + quoting=1, + ) + return None elif output_type == "usage": output_dir = os.path.join( config.CONSOLIDATED_OUTPUT_DIRECTORY, run_timestamp, "tracking" @@ -458,8 +677,10 @@ def write_local( ) return None elif output_type == "qc_qa_stats": - # QC/QA statistics output goes to outputs/qc_qa folder - output_dir = os.path.join(config.QC_QA_OUTPUT_DIRECTORY, run_timestamp) + # QC/QA statistics output goes to automation_qa-qc folder + output_dir = os.path.join( + config.CONSOLIDATED_OUTPUT_DIRECTORY, run_timestamp, "automation_qa-qc" + ) os.makedirs(output_dir, exist_ok=True) df.to_csv( os.path.join(output_dir, f"{config.BATCH_ID}-QC-QA-STATS.csv"), @@ -467,9 +688,31 @@ def write_local( quoting=1, ) return None + elif output_type == "qc_qa_tin_stats": + output_dir = os.path.join(config.QC_QA_OUTPUT_DIRECTORY, run_timestamp) + os.makedirs(output_dir, exist_ok=True) + df.to_csv( + os.path.join(output_dir, f"{config.BATCH_ID}-QC-QA-TIN-STATS.csv"), + index=False, + quoting=1, + ) + return None + elif output_type == "individual_cc": + # Individual CC results + base_filename = os.path.splitext(filename)[0].strip() + output_dir = os.path.join( + config.CONSOLIDATED_OUTPUT_DIRECTORY, run_timestamp, "individual" + ) + os.makedirs(output_dir, exist_ok=True) + df.to_csv( + os.path.join(output_dir, f"{base_filename}-RESULTS-CC.csv"), + index=False, + quoting=1, + ) + return None else: logging.error( - f"Unknown output_type: {output_type}. Valid types are: 'final', 'individual', 'error', 'usage', 'usage_summary', 'dtc', 'dtc_summary', 'qc_qa_validated', 'qc_qa_stats'" + f"Unknown output_type: {output_type}. Valid types are: 'final', 'individual', 'error', 'usage', 'usage_summary', 'dtc', 'dtc_summary', 'qc_qa_validated', 'qc_qa_stats', 'cc_results_full', 'dashboard_results_full', 'qc_qa_cc_full', 'qc_qa_error', 'parent_child', 'individual_cc'" ) return None @@ -507,42 +750,150 @@ def write_s3( """ if output_type == "final": output_path = f"{config.BATCH_ID}/{run_timestamp}/{config.BATCH_ID}-RESULTS.csv" + csv_buffer = df.to_csv(index=False).encode() + try: + config.S3_CLIENT.put_object( + Bucket=config.S3_OUTPUT_BUCKET, Key=output_path, Body=csv_buffer + ) + except ClientError as e: + logging.error(f"Error uploading to S3: {e}") + return None elif output_type == "individual": output_path = ( f"{config.BATCH_ID}/{run_timestamp}/individual/{filename}-RESULTS.csv" ) + csv_buffer = df.to_csv(index=False).encode() + config.S3_CLIENT.put_object( + Bucket=config.S3_OUTPUT_BUCKET, Key=output_path, Body=csv_buffer + ) + elif output_type == "individual_cc": + output_path = ( + f"{config.BATCH_ID}/{run_timestamp}/individual/{filename}-RESULTS-CC.csv" + ) + csv_buffer = df.to_csv(index=False).encode() + config.S3_CLIENT.put_object( + Bucket=config.S3_OUTPUT_BUCKET, Key=output_path, Body=csv_buffer + ) elif output_type == "parent_child": output_path = ( - f"{config.BATCH_ID}/{run_timestamp}/parent_child/{filename}-RESULTS.csv" + f"{config.BATCH_ID}/{run_timestamp}/parent-child/{config.BATCH_ID}-PC.csv" + ) + csv_buffer = df.to_csv(index=False).encode() + config.S3_CLIENT.put_object( + Bucket=config.S3_OUTPUT_BUCKET, Key=output_path, Body=csv_buffer ) elif output_type == "error": - output_path = f"{config.BATCH_ID}/{run_timestamp}/{config.BATCH_ID}-ERRORS.csv" + output_path = f"{config.BATCH_ID}/{run_timestamp}/full_outputs/{config.BATCH_ID}-ERRORS.csv" + csv_buffer = df.to_csv(index=False).encode() + try: + config.S3_CLIENT.put_object( + Bucket=config.S3_OUTPUT_BUCKET, Key=output_path, Body=csv_buffer + ) + except ClientError as e: + logging.error(f"Error uploading to S3: {e}") + return None + elif output_type in ("cc_results_full", "dashboard_results_full", "qc_qa_cc_full"): + # Will be handled in try block using helper function + output_path = None + parent_child_path = None if output_type == "cc_results_full" else None + elif output_type == "qc_qa_error": + output_path = f"{config.BATCH_ID}/{run_timestamp}/automation_qa-qc/{config.BATCH_ID}-ERRORS-QC-QA.csv" + csv_buffer = df.to_csv(index=False).encode() + config.S3_CLIENT.put_object( + Bucket=config.S3_OUTPUT_BUCKET, Key=output_path, Body=csv_buffer + ) elif output_type == "usage": output_path = ( f"{config.BATCH_ID}/{run_timestamp}/tracking/{config.BATCH_ID}-USAGE.csv" ) + csv_buffer = df.to_csv(index=False).encode() + config.S3_CLIENT.put_object( + Bucket=config.S3_OUTPUT_BUCKET, Key=output_path, Body=csv_buffer + ) elif output_type == "usage_summary": output_path = f"{config.BATCH_ID}/{run_timestamp}/tracking/{config.BATCH_ID}-USAGE-SUMMARY.csv" + csv_buffer = df.to_csv(index=False).encode() + config.S3_CLIENT.put_object( + Bucket=config.S3_OUTPUT_BUCKET, Key=output_path, Body=csv_buffer + ) elif output_type == "dtc": output_path = f"{config.BATCH_ID}/{run_timestamp}/{config.BATCH_ID}-DTC.csv" + csv_buffer = df.to_csv(index=False).encode() + config.S3_CLIENT.put_object( + Bucket=config.S3_OUTPUT_BUCKET, Key=output_path, Body=csv_buffer + ) elif output_type == "dtc_summary": output_path = ( f"{config.BATCH_ID}/{run_timestamp}/{config.BATCH_ID}-DTC-SUMMARY.csv" ) - elif output_type == "qc_qa_validated": - output_path = f"{config.BATCH_ID}/{run_timestamp}/automation_qa-qc/{config.BATCH_ID}-QC-QA-VALIDATED.csv" - elif output_type == "qc_qa_stats": - output_path = f"{config.BATCH_ID}/{run_timestamp}/automation_qa-qc/{config.BATCH_ID}-QC-QA-STATS.csv" - else: - # Fallback path for unknown output types - output_path = f"{config.BATCH_ID}/{run_timestamp}/{config.BATCH_ID}-unknown-{output_type}.csv" - - csv_buffer = df.to_csv(index=False).encode() - try: + csv_buffer = df.to_csv(index=False).encode() config.S3_CLIENT.put_object( Bucket=config.S3_OUTPUT_BUCKET, Key=output_path, Body=csv_buffer ) - if output_type == "final": + elif output_type == "qc_qa_validated": + output_path = f"{config.BATCH_ID}/{run_timestamp}/automation_qa-qc/{config.BATCH_ID}-QC-QA-VALIDATED.csv" + csv_buffer = df.to_csv(index=False).encode() + config.S3_CLIENT.put_object( + Bucket=config.S3_OUTPUT_BUCKET, Key=output_path, Body=csv_buffer + ) + elif output_type == "qc_qa_stats": + output_path = f"{config.BATCH_ID}/{run_timestamp}/automation_qa-qc/{config.BATCH_ID}-QC-QA-STATS.csv" + csv_buffer = df.to_csv(index=False).encode() + config.S3_CLIENT.put_object( + Bucket=config.S3_OUTPUT_BUCKET, Key=output_path, Body=csv_buffer + ) + elif output_type == "qc_qa_tin_stats": + output_path = f"{config.BATCH_ID}/{run_timestamp}/automation_qa-qc/{config.BATCH_ID}-QC-QA-TIN-STATS.csv" + csv_buffer = df.to_csv(index=False).encode() + config.S3_CLIENT.put_object( + Bucket=config.S3_OUTPUT_BUCKET, Key=output_path, Body=csv_buffer + ) + elif output_type == "individual_cc": + output_path = ( + f"{config.BATCH_ID}/{run_timestamp}/individual/{filename}-RESULTS-CC.csv" + ) + csv_buffer = df.to_csv(index=False).encode() + config.S3_CLIENT.put_object( + Bucket=config.S3_OUTPUT_BUCKET, Key=output_path, Body=csv_buffer + ) + else: + # Fallback path for unknown output types + output_path = f"{config.BATCH_ID}/{run_timestamp}/{config.BATCH_ID}-unknown-{output_type}.csv" + csv_buffer = df.to_csv(index=False).encode() + config.S3_CLIENT.put_object( + Bucket=config.S3_OUTPUT_BUCKET, Key=output_path, Body=csv_buffer + ) + + try: + # Handle split files for cc_results_full, dashboard_results_full, and qc_qa_cc_full + if output_type == "cc_results_full": + parent_child_path = _write_s3_split_files( + df=df, + run_timestamp=run_timestamp, + base_path="full_outputs/cc_results", + full_filename=f"{config.BATCH_ID}-RESULTS-FULL.csv", + split_filename_template=f"{config.BATCH_ID}-RESULTS-SPLIT{{idx}}.csv", + return_path_for_pc=config.PERFORM_PARENT_CHILD_MAPPING, + ) + elif output_type == "dashboard_results_full": + _write_s3_split_files( + df=df, + run_timestamp=run_timestamp, + base_path="full_outputs/dashboard_results", + full_filename=f"{config.BATCH_ID}-RESULTS-dashboard.csv", + split_filename_template=f"{config.BATCH_ID}-RESULTS-dashboard-SPLIT{{idx}}.csv", + return_path_for_pc=False, + ) + elif output_type == "qc_qa_cc_full": + _write_s3_split_files( + df=df, + run_timestamp=run_timestamp, + base_path="automation_qa-qc", + full_filename=f"{config.BATCH_ID}-RESULTS-QC-QA-FULL.csv", + split_filename_template=f"{config.BATCH_ID}-RESULTS-QC-QA-SPLIT{{idx}}.csv", + return_path_for_pc=False, + ) + elif output_type == "final": logging.info(f"Saved batch output file: {output_path}") elif output_type == "individual": logging.info(f"Saved individual output file: {output_path}") @@ -562,11 +913,18 @@ def write_s3( logging.info(f"Saved QC/QA validated data file: {output_path}") elif output_type == "qc_qa_stats": logging.info(f"Saved QC/QA statistics file: {output_path}") + elif output_type == "qc_qa_tin_stats": + logging.info(f"Saved QC/QA TIN statistics file: {output_path}") + elif output_type == "qc_qa_error": + logging.info(f"Saved QC/QA error file: {output_path}") + elif output_type == "individual_cc": + logging.info(f"Saved individual CC file: {output_path}") else: logging.info(f"Saved unknown output type file: {output_path}") - if config.PERFORM_PARENT_CHILD_MAPPING: - return f"s3://{config.S3_OUTPUT_BUCKET}/{output_path}" + if config.PERFORM_PARENT_CHILD_MAPPING and output_type == "cc_results_full": + if "parent_child_path" in locals() and parent_child_path: + return f"s3://{config.S3_OUTPUT_BUCKET}/{parent_child_path}" return None except ClientError as e: @@ -574,6 +932,51 @@ def write_s3( return None +def upload_local_file_to_s3( + local_path: str, + run_timestamp: str, + s3_subpath: str, + content_type: Optional[str] = None, +) -> Optional[str]: + """ + Upload a local file to S3 using the same path structure as other pipeline outputs. + + Args: + local_path: Full path to the local file. + run_timestamp: Run timestamp (e.g. run_20260212_13-53_test_batch). + s3_subpath: Subpath under run_timestamp (e.g. parent-child). + content_type: Optional MIME type. Inferred from extension if not provided. + + Returns: + S3 URI on success, None on failure. + """ + if not os.path.isfile(local_path): + logging.error(f"Cannot upload: file not found: {local_path}") + return None + filename = os.path.basename(local_path) + s3_key = f"{config.BATCH_ID}/{run_timestamp}/{s3_subpath}/{filename}" + if content_type is None and filename.lower().endswith(".xlsx"): + content_type = ( + "application/vnd.openxmlformats-officedocument.spreadsheetml.sheet" + ) + try: + with open(local_path, "rb") as f: + put_kwargs = { + "Bucket": config.S3_OUTPUT_BUCKET, + "Key": s3_key, + "Body": f.read(), + } + if content_type: + put_kwargs["ContentType"] = content_type + config.S3_CLIENT.put_object(**put_kwargs) + s3_uri = f"s3://{config.S3_OUTPUT_BUCKET}/{s3_key}" + logging.info(f"Uploaded file to S3: {s3_uri}") + return s3_uri + except ClientError as e: + logging.error(f"Failed to upload {local_path} to S3: {e}") + return None + + def save_result_to_json(filename, result, json_folder): """Save a result dictionary to an individual JSON file in the specified folder. diff --git a/src/utils/logging_utils.py b/src/utils/logging_utils.py index cb367ba..4f78ee5 100644 --- a/src/utils/logging_utils.py +++ b/src/utils/logging_utils.py @@ -28,7 +28,9 @@ Architecture: """ import contextvars +import io import logging +import sys from pathlib import Path from src import config @@ -96,7 +98,8 @@ class PerFileHandler(logging.Handler): # Get cached handler or create a new one if not exists if safe_filename not in self._file_handlers: # mode="w" to overwrite existing per-file logs for fresh run - file_handler = logging.FileHandler(log_file, mode="w") + # encoding="utf-8" to handle Unicode characters (arrows, special chars) + file_handler = logging.FileHandler(log_file, mode="w", encoding="utf-8") formatter = logging.Formatter( "%(asctime)s - %(name)s - %(levelname)s - %(message)s" ) @@ -193,8 +196,14 @@ def setup_per_file_logging() -> None: ) # Always add console output for monitoring overall progress - console_handler = logging.StreamHandler() - console_handler.setLevel(logging.INFO) + # Wrap stdout with UTF-8 encoding to handle Unicode characters (arrows, etc.) on Windows + utf8_stdout = io.TextIOWrapper( + sys.stdout.buffer, encoding="utf-8", errors="replace" + ) + console_handler = logging.StreamHandler(utf8_stdout) + # Use config.LOG_LEVEL for console output (defaults to INFO) + console_log_level = getattr(logging, config.LOG_LEVEL, logging.INFO) + console_handler.setLevel(console_log_level) console_formatter = logging.Formatter("%(asctime)s - %(levelname)s - %(message)s") console_handler.setFormatter(console_formatter) logger.addHandler(console_handler) @@ -202,9 +211,6 @@ def setup_per_file_logging() -> None: # Prevent propagation to avoid duplicate console output logger.propagate = False - # Prevent propagation to avoid duplicate console output - logger.propagate = False - # Suppress noisy third-party loggers that clutter output noisy_loggers = [ "botocore", diff --git a/src/utils/string_utils.py b/src/utils/string_utils.py index 8515815..1e422c9 100644 --- a/src/utils/string_utils.py +++ b/src/utils/string_utils.py @@ -14,8 +14,79 @@ from src.constants.regex_patterns import ( BACKTICK_PATTERN, PIPE_PATTERN, TRIPLE_BACKTICK_PATTERN, + DBA_PATTERNS, ) +LEGAL_SUFFIXES = { + "inc", + "inc.", + "incorporated", + "corp", + "corporation", + "llc", + "l.l.c", + "ltd", + "limited", + "co", + "company", + "affiliates", +} + +STOPWORDS = {"of", "the", "and", "its"} + +US_STATES = [ + "alabama", + "alaska", + "arizona", + "arkansas", + "california", + "colorado", + "connecticut", + "delaware", + "florida", + "georgia", + "hawaii", + "idaho", + "illinois", + "indiana", + "iowa", + "kansas", + "kentucky", + "louisiana", + "maine", + "maryland", + "massachusetts", + "michigan", + "minnesota", + "mississippi", + "missouri", + "montana", + "nebraska", + "nevada", + "new hampshire", + "new jersey", + "new mexico", + "new york", + "north carolina", + "north dakota", + "ohio", + "oklahoma", + "oregon", + "pennsylvania", + "rhode island", + "south carolina", + "south dakota", + "tennessee", + "texas", + "utah", + "vermont", + "virginia", + "washington", + "west virginia", + "wisconsin", + "wyoming", +] + def extract_text_from_delimiters( raw_output: str, delimiter: Delimiter, match_index: int = -1 @@ -806,3 +877,218 @@ def normalize_to_json_list(val): result = json.dumps(expanded) return result return val + + +def normalize_text(text: str) -> str: + """ + Normalize a text string by applying common preprocessing steps. + + This function: + - Converts all characters to lowercase + - Removes punctuation and special characters + - Replaces multiple whitespace characters with a single space + - Strips leading and trailing whitespace + + Args: + text (str): The input text to normalize. + + Returns: + str: The normalized text. + """ + # Convert the text to lowercase for case-insensitive processing + text = text.lower() + + # Replace all characters that are not letters, numbers, or whitespace with a space + text = re.sub(r"[^\w\s]", " ", text) + + # Replace multiple whitespace characters with a single space + # and remove leading/trailing spaces + text = re.sub(r"\s+", " ", text).strip() + + return text + + +def remove_legal_suffixes(text: str) -> str: + """ + Remove legal entity suffixes from a text string. + + This function splits the input text into tokens (words) and + removes any tokens that match known legal suffixes + (e.g., "llc", "inc", "corp"). + + Args: + text (str): The input text containing a business name. + + Returns: + str: The text with legal suffixes removed. + """ + # Split the text into individual words + tokens = text.split() + + # Filter out any tokens that are legal suffixes + tokens = [t for t in tokens if t not in LEGAL_SUFFIXES] + + # Rejoin the remaining tokens into a single string + return " ".join(tokens) + + +def extract_dba_name(text: str) -> tuple[str | None, str | None]: + """ + Extract a DBA (Doing Business As) name from a text string. + + The function checks the input text against a list of predefined + DBA patterns (e.g., "dba", "doing business as"). If a pattern is found, + it returns the text that appears after the pattern. + + Ex: + Input: NEW YORK STATE CATHOLIC HEALTH PLAN, INC. d/b/a FIDELIS CARE NEW YORK + + Output: FIDELIS CARE NEW YORK + + Args: + text (str): The input text that may contain a DBA reference. + + Returns: + tuple[str | None, str | None]: A tuple containing: + - The text before the DBA pattern (potentially the legal name) + - The text after the DBA pattern (the extracted DBA name) + If no DBA pattern is found, both elements of the tuple will be None. + """ + # Iterate through all known DBA patterns + for pattern in DBA_PATTERNS: + # Split the text using the current pattern (case-insensitive) + match = re.split(pattern, text, flags=re.I) + + # If the pattern was found, the split will produce more than one part + if len(match) > 1: + # Return the text after the matched pattern, trimmed of whitespace + return match[0].strip(), match[-1].strip() + + # Return None if no DBA pattern is found + return None, None + + +def remove_states(text: str) -> str: + """ + Remove U.S. state names or abbreviations from a text string. + + The function iterates through a list of U.S. states and removes + any exact (word-boundary matched) occurrences from the input text. + Extra whitespace created by removals is cleaned up before returning. + + Remove state names when they appear as a regional suffix + (i.e., at the end of the string), but preserve them when + they are leading brand identifiers (e.g., 'Oklahoma Complete Health'). + + Args: + text (str): The input text that may contain U.S. state names. + + Returns: + str: The text with U.S. state references removed. + """ + + tokens = text.split() + if not tokens: + return text + + for state in US_STATES: + state_tokens = state.split() + + # Check if state appears at END of string + if tokens[-len(state_tokens) :] == state_tokens: + + # Do NOT remove if state is the first token + if tokens[0] == state_tokens[0]: + return text + + # Remove state AND trailing "of" if present + new_tokens = tokens[: -len(state_tokens)] + if new_tokens and new_tokens[-1] == "of": + new_tokens = new_tokens[:-1] + return " ".join(new_tokens).strip() + + # Also handle "of " pattern anywhere in the string, as it often indicates a regional suffixe.g., "Health Plan of New York" should become "Health Plan" + # Remove each state using a case-insensitive regex match. It looks for the word "of" followed by the state name. + for state in US_STATES: + text = re.sub(rf"\bof\s+{state}\b", "", text, flags=re.I) + + # Normalize whitespace after removals + return re.sub(r"\s+", " ", text).strip() + + +def detect_state(text: str) -> str | None: + """Return state name if found in text.""" + for state in US_STATES: + if re.search(rf"\b{state}\b", text, re.I): + return state + return None + + +def is_filled(x) -> bool: + """ + Returns True if x contains a meaningful (non-empty) value. + + Treats None, NaN, and strings like "", "[]", "nan" as empty. + Inverse of is_empty for scalar values, with additional handling for "[]". + + Args: + x: The value to check. + + Returns: + bool: True if x is considered filled, False otherwise. + """ + return ( + x is not None + and not (isinstance(x, float) and pd.isna(x)) + and str(x).strip() not in ("", "[]", "nan") + ) + + +def extract_nested_values(val) -> list[str]: + """ + Recursively extract leaf values from arbitrarily nested string representations. + + Handles strings that represent nested lists (e.g. "['a', ['b', 'c']]") by + parsing with ast.literal_eval and recursing into any list items. + + Args: + val: The value to extract from; will be converted to string. + + Returns: + list[str]: Flat list of non-empty leaf string values. + """ + s = str(val).strip() + try: + parsed = ast.literal_eval(s) + except Exception: + return [s] if s else [] + if isinstance(parsed, list): + results = [] + for item in parsed: + results.extend(extract_nested_values(str(item))) + return [r for r in results if r.strip()] + elif isinstance(parsed, str): + return extract_nested_values(parsed) if parsed != s else ([s] if s else []) + else: + return [str(parsed).strip()] + + +def token_signature(text: str) -> tuple: + """ + Generate a normalized token signature from a text string. + + The function removes stopwords, sorts the remaining tokens, + and returns them as an immutable tuple. This signature can be + used for fast comparison or deduplication of similar texts. + + Args: + text (str): The input text to tokenize. + + Returns: + tuple: A sorted tuple of tokens excluding stopwords. + """ + # Split the text into tokens and remove stopwords + tokens = [t for t in text.split() if t not in STOPWORDS] + + # Sort tokens to ensure order-independent comparison + return tuple(sorted(tokens))