Merged in bugfix/parser-downstream-improvements (pull request #875)
Bugfix/parser downstream improvements * Refactor: Implement field-aware JSON parsers with centralized normalization This refactor introduces a robust system for normalizing LLM output based on field format mappings, ensuring consistent data types throughout the pipeline. Key Changes: - Add FIELD_FORMAT_MAPPING constant defining expected formats for all fields - Create format_normalization.py utility for type-aware normalization - Update json_utils.py parsers to accept field_names/field_name parameters - Refactor prompt_templates.py to use parser factories (_create_json_dict_parser, _create_json_list_parser) that bind field metadata for automatic normalization - Update prompt_calls.py to pass field names to parsers, eliminating redundant normalization logic - Remove parse_json_dict_or_list (unused, ambiguous function) - Simplify METHODOLOGY_BREAKOUT and REIMBURSEMENT_PRIMARY to use helper functions - Add comprehensive integration tests verifying normalization works end-to-end Benefits: - Single source of truth for field formats (FIELD_FORMAT_… * refactor: normalize helper prompt outputs at prompt_calls level - Update CARVEOUT_CHECK to use field-aware parser for CARVEOUT_CD normalization - Update LOB_RELATIONSHIP to normalize to string format in prompt_calls.py - Update SPLIT_REIMB_DATES to normalize date values in prompt_calls.py - Remove defensive normalization from one_to_n_funcs.py for LOB relationships - Remove manual normalization from split_reimb_dates() - values now normalized upstream - All helper prompts that populate fields now normalize at prompt_calls.py level - Downstream functions receive correctly formatted values without additional processing * refactor: remove band-aid normalization functions and migrate HSC to field-aware parsers - Update ONE_TO_ONE_SINGLE_FIELD_TEMPLATE to use field-aware parser with field_name parameter - Remove list wrapping logic in hybrid_smart_chunking_funcs (field-aware parser handles normalization) - Remove normalize_one_to_one_field_value and normalize_one_to_one_answers_dict from string_utils.py - Remove all debug print statements from HSC processing - Remove commented-out normalization calls from client-specific files (clover, bcbs_promise) - All normalization now handled exclusively through FIELD_FORMAT_MAPPING via field-aware parsers * Ran Black for formatting * Print Statements removed, more cleaning * Merge branch 'DEV' into bugfix/parser-downstream-improvements * refactor: combine FIELD_FORMAT_MAPPING into investment_columns.py - Merged field_format_mapping.py into investment_columns.py to create single source of truth - FIELD_FORMAT_MAPPING now ordered by COLUMN_ORDER (161 fields) - Added 5 missing fields from COLUMN_ORDER with default format types - Updated all imports across codebase to use investment_columns - Python dict preserves insertion order (3.7+), maintaining COLUMN_ORDER sequence - All tests passing (38 field-aware tests verified) * Deprecate COLUMN_ORDER, rely on Mapping only * Merged DEV into bugfix/parser-downstream-improvements Approved-by: Katon Minhas
This commit is contained in:
Praneel Panchigar
committed by
Katon Minhas
Katon Minhas
parent
4587042c43
commit
9b0a344b13
@@ -806,101 +806,3 @@ def normalize_to_json_list(val):
|
||||
result = json.dumps(expanded)
|
||||
return result
|
||||
return val
|
||||
|
||||
|
||||
def normalize_one_to_one_field_value(field_name: str, value) -> str | list:
|
||||
"""
|
||||
Normalize a 1:1 field value to ensure single-value fields are strings, not lists.
|
||||
|
||||
This function handles the common case where LLM returns a single-value field as
|
||||
a list with one element (e.g., ['The Value']). It extracts the string from such
|
||||
lists while preserving legitimate multi-value lists.
|
||||
|
||||
Args:
|
||||
field_name (str): Name of the field being normalized (for logging/debugging)
|
||||
value: The field value to normalize. Can be:
|
||||
- str: Returned as-is
|
||||
- list with 1 element: Extract the string
|
||||
- list with >1 elements: Returned unchanged (may be legitimate multi-value)
|
||||
- None/empty: Returned as-is
|
||||
|
||||
Returns:
|
||||
str | list: Normalized value:
|
||||
- If input is a list with exactly 1 element: returns that element as string
|
||||
- If input is already a string: returns as-is
|
||||
- If input is a list with >1 elements: returns list unchanged
|
||||
- If input is None or empty: returns as-is
|
||||
|
||||
Examples:
|
||||
>>> normalize_one_to_one_field_value("CONTRACT_TITLE", ["Provider Agreement"])
|
||||
'Provider Agreement'
|
||||
>>> normalize_one_to_one_field_value("PAYER_NAME", "Aetna")
|
||||
'Aetna'
|
||||
>>> normalize_one_to_one_field_value("PROVIDER_NAME", ["Provider A", "Provider B"])
|
||||
['Provider A', 'Provider B'] # Multi-value preserved
|
||||
>>> normalize_one_to_one_field_value("LOB", ["Medicare", "Medicaid"])
|
||||
['Medicare', 'Medicaid'] # Multi-value preserved
|
||||
"""
|
||||
# Handle None and empty values
|
||||
if value is None or (isinstance(value, str) and is_empty(value)):
|
||||
return value
|
||||
|
||||
# If it's already a string, return as-is
|
||||
if isinstance(value, str):
|
||||
return value
|
||||
|
||||
# If it's a list
|
||||
if isinstance(value, list):
|
||||
# List with exactly 1 element: extract the string
|
||||
if len(value) == 1:
|
||||
return str(value[0]) if value[0] is not None else "N/A"
|
||||
# List with >1 elements: preserve (may be legitimate multi-value)
|
||||
# This handles cases like PROVIDER_NAME with multiple providers,
|
||||
# or dynamic fields (LOB, PROGRAM, PRODUCT, NETWORK) when passed to 1:1
|
||||
elif len(value) > 1:
|
||||
return value
|
||||
# Empty list: return "N/A"
|
||||
else:
|
||||
return "N/A"
|
||||
|
||||
# For any other type, convert to string
|
||||
return str(value) if value is not None else "N/A"
|
||||
|
||||
|
||||
def normalize_one_to_one_answers_dict(
|
||||
answers_dict: dict[str, str | list],
|
||||
) -> dict[str, str | list]:
|
||||
"""
|
||||
Normalize all values in a 1:1 answers dictionary.
|
||||
|
||||
Applies normalization to all fields in the dictionary, converting single-element
|
||||
lists to strings while preserving legitimate multi-value lists.
|
||||
|
||||
Args:
|
||||
answers_dict (dict): Dictionary of field names to values (may be strings or lists)
|
||||
|
||||
Returns:
|
||||
dict: Dictionary with normalized values (single-element lists converted to strings)
|
||||
|
||||
Example:
|
||||
>>> answers = {
|
||||
... "CONTRACT_TITLE": ["Provider Agreement"],
|
||||
... "PAYER_NAME": "Aetna",
|
||||
... "PROVIDER_NAME": ["Provider A", "Provider B"]
|
||||
... }
|
||||
>>> normalize_one_to_one_answers_dict(answers)
|
||||
{
|
||||
"CONTRACT_TITLE": "Provider Agreement",
|
||||
"PAYER_NAME": "Aetna",
|
||||
"PROVIDER_NAME": ["Provider A", "Provider B"]
|
||||
}
|
||||
"""
|
||||
DO_NOT_NORMALIZE = ["PROV_INFO_JSON", "PAYER_STATE", "PROVIDER_STATE"]
|
||||
normalized = {}
|
||||
for field_name, value in answers_dict.items():
|
||||
if field_name not in DO_NOT_NORMALIZE:
|
||||
normalized[field_name] = normalize_one_to_one_field_value(field_name, value)
|
||||
else:
|
||||
normalized[field_name] = value
|
||||
|
||||
return normalized
|
||||
|
||||
Reference in New Issue
Block a user