Merged in bugfix/parser-downstream-improvements (pull request #875)

Bugfix/parser downstream improvements

* Refactor: Implement field-aware JSON parsers with centralized normalization

This refactor introduces a robust system for normalizing LLM output based on
field format mappings, ensuring consistent data types throughout the pipeline.

Key Changes:
- Add FIELD_FORMAT_MAPPING constant defining expected formats for all fields
- Create format_normalization.py utility for type-aware normalization
- Update json_utils.py parsers to accept field_names/field_name parameters
- Refactor prompt_templates.py to use parser factories (_create_json_dict_parser,
  _create_json_list_parser) that bind field metadata for automatic normalization
- Update prompt_calls.py to pass field names to parsers, eliminating redundant
  normalization logic
- Remove parse_json_dict_or_list (unused, ambiguous function)
- Simplify METHODOLOGY_BREAKOUT and REIMBURSEMENT_PRIMARY to use helper functions
- Add comprehensive integration tests verifying normalization works end-to-end

Benefits:
- Single source of truth for field formats (FIELD_FORMAT_…
* refactor: normalize helper prompt outputs at prompt_calls level

- Update CARVEOUT_CHECK to use field-aware parser for CARVEOUT_CD normalization
- Update LOB_RELATIONSHIP to normalize to string format in prompt_calls.py
- Update SPLIT_REIMB_DATES to normalize date values in prompt_calls.py
- Remove defensive normalization from one_to_n_funcs.py for LOB relationships
- Remove manual normalization from split_reimb_dates() - values now normalized upstream
- All helper prompts that populate fields now normalize at prompt_calls.py level
- Downstream functions receive correctly formatted values without additional processing

* refactor: remove band-aid normalization functions and migrate HSC to field-aware parsers

- Update ONE_TO_ONE_SINGLE_FIELD_TEMPLATE to use field-aware parser with field_name parameter
- Remove list wrapping logic in hybrid_smart_chunking_funcs (field-aware parser handles normalization)
- Remove normalize_one_to_one_field_value and normalize_one_to_one_answers_dict from string_utils.py
- Remove all debug print statements from HSC processing
- Remove commented-out normalization calls from client-specific files (clover, bcbs_promise)
- All normalization now handled exclusively through FIELD_FORMAT_MAPPING via field-aware parsers

* Ran Black for formatting

* Print Statements removed, more cleaning

* Merge branch 'DEV' into bugfix/parser-downstream-improvements

* refactor: combine FIELD_FORMAT_MAPPING into investment_columns.py

- Merged field_format_mapping.py into investment_columns.py to create single source of truth
- FIELD_FORMAT_MAPPING now ordered by COLUMN_ORDER (161 fields)
- Added 5 missing fields from COLUMN_ORDER with default format types
- Updated all imports across codebase to use investment_columns
- Python dict preserves insertion order (3.7+), maintaining COLUMN_ORDER sequence
- All tests passing (38 field-aware tests verified)

* Deprecate COLUMN_ORDER, rely on Mapping only

* Merged DEV into bugfix/parser-downstream-improvements


Approved-by: Katon Minhas

src/pipelines/shared/postprocessing/aarete_derived.py

@@ -2,9 +2,11 @@ import logging
 import src.config as config
 import src.utils.string_utils as string_utils
 from src.constants.constants import Constants
+from src.constants.investment_columns import FIELD_FORMAT_MAPPING
 from src.crosswalk.crosswalk_builder import CrosswalkBuilder
 from src.prompts.fieldset import FieldSet
 from src.utils.crosswalk_utils import apply_crosswalk
+from src.utils.formatting_utils import normalize_field_value
 
 
 def fill_na_from_field(results_dict, to_field, from_field, crosswalk_path):
@@ -172,16 +174,9 @@ def get_crosswalk_fields(answer_dicts: list, constants: Constants):
     """
     Apply crosswalk mappings to derive target fields from source fields.
 
-    Fields that should be single values (strings) even when crosswalk creates lists:
-    - AARETE_DERIVED_CLAIM_TYPE_CD: Claim type is a single classification value
-    - Other AARETE_DERIVED_* fields that represent single classifications (not multi-value like LOB)
+    Normalizes output based on FIELD_FORMAT_MAPPING to ensure correct format
+    (str vs list[str]) for each target field.
     """
-    # Fields that should always be single values (strings), not lists
-    # These are typically classification/derived fields that represent a single value
-    SINGLE_VALUE_FIELDS = {
-        "AARETE_DERIVED_CLAIM_TYPE_CD",
-        # Add other single-value derived fields here as needed
-    }
 
     crosswalk_fields = FieldSet(file_path=config.FIELD_JSON_PATH, crosswalk=True)
     for to_field in crosswalk_fields.fields:
@@ -233,14 +228,12 @@ def get_crosswalk_fields(answer_dicts: list, constants: Constants):
                                         individual_from_field_value
                                     )
 
-                    # Assign the result: convert to string if it's a single-value field with one element
-                    if (
-                        to_field_name in SINGLE_VALUE_FIELDS
-                        and len(to_field_answer_list) == 1
-                    ):
-                        answer_dict[to_field_name] = to_field_answer_list[0]
-                    else:
-                        answer_dict[to_field_name] = to_field_answer_list
+                    # Normalize the result based on FIELD_FORMAT_MAPPING
+                    format_type = FIELD_FORMAT_MAPPING.get(to_field_name, "list[str]")
+                    normalized_value = normalize_field_value(
+                        to_field_name, to_field_answer_list, format_type
+                    )
+                    answer_dict[to_field_name] = normalized_value
 
                     # DO NOT update from_field_name - preserve its original format
                     # The crosswalk should only create/update the target field (to_field_name),