Merged in bugfix/generic_dynamic_jan26 (pull request #859)

Bugfix/generic dynamic jan26

* Refactor: Extract dynamic primary metrics logic to separate module

- Created src/testbed/dynamic_primary_metrics.py with all dynamic primary field analysis logic
- Moved tally-based analysis functions to new module for better modularity
- Updated testbed_utils.py to import and use functions from dynamic_primary_metrics
- Simplified testbed_metrics_dynamic_only.py to use new module
- Removed unused analyze_dynamic_primary_fields function (400+ lines)
- Added field filtering in match_rows to handle missing columns gracefully
- Minimal changes to testbed_utils.py (~100 lines vs 845 before)

* feat: Add debug print blocks and prompt changes for dynamic primary fields in 1:1 processing

This commit adds prompt changes for when we pass dynamic primary to 1:1  and comprehensive debug logging for dynamic primary fields when they are escalated to 1:1 processing via Hybrid Smart Chunking (HSC) or Full Context methods.

Changes:
- Added debug print blocks in hybrid_smart_chunking_funcs.py to log:
  * Retrieval question, context chunks, and final prompt for HSC processing
  * Raw LLM output (reasoning + answer) for dynamic primary fields
- Added debug print blocks in prompt_calls.py to log:
  * Contract context, final prompt for Full Context processing
  * Raw LLM output (reasoning + answer) for dynamic primary fields
- Added FULL_CONTEXT_DYNAMIC_PRIMARY_INSTRUCTION() to prompt_templates.py:
  * Provides specific guidance for dynamic primary fields in full context
  * Emphasizes extracting values only if they refer to entire contract
  * Includes pipe-delimited formatting instructions
- Updated dynamic_fun…
* Ran black, for formatting

* Refine dynamic primary field prompts and add Duals auto-detection

- Enhanced DYNAMIC_ASSIGNMENT_INSTRUCTION with structural boundary rules
- Added exhibit header and subsection context hierarchy for LOB assignment
- Implemented Duals auto-detection in dynamic_primary discovery (Medicare+Medicaid -> Duals)
- Added Duals value formatting instruction to prevent oversimplification
- Commented out DUAL_LOB_CHECK for performance testing
- Removed verbose debug print blocks from HSC and Full Context processing
- Added focused debug print for dynamic assignment raw LLM output

* Remove debug print blocks from pipeline files

- Removed all debug print blocks from file_processing.py (8 blocks)
- Removed all debug print blocks from dynamic_funcs.py (8 blocks)
- Removed debug print block from prompt_calls.py (dynamic assignment)
- DUAL_LOB_CHECK remains commented as requested
- Total: 188 lines of debug code removed

* Merge remote-tracking branch 'origin/main' into bugfix/generic_dynamic_jan26

* Ran black for CI

* Remove update_lob_for_duals

* Merge branch 'DEV' into bugfix/generic_dynamic_jan26


Approved-by: Katon Minhas

src/pipelines/shared/postprocessing/postprocessing_funcs.py

@@ -526,45 +526,6 @@ def add_aarete_derived_product(df):
     return df
 
 
-def update_lob_for_duals(answer_dicts):
-    final_answer_dicts = []
-    # Iterate through each row in the dataframe
-    for answer_dict in answer_dicts:
-        # Get SERVICE_TERM value
-        service_term = answer_dict.get("SERVICE_TERM", "").lower()
-        # Check if the SERVICE_TERM contains "Medicare" and "Medicaid" - case insensitive
-        # AND check that there's no explicit PROGRAM or PRODUCT that would override this logic
-        if (
-            "medicare" in service_term
-            and "medicaid" in service_term
-            and string_utils.is_empty(answer_dict.get("AARETE_DERIVED_PROGRAM"))
-            and string_utils.is_empty(answer_dict.get("AARETE_DERIVED_PRODUCT"))
-        ):
-            prompt = prompt_templates.DUAL_LOB_CHECK(service_term)
-            logging.debug(f"Prompt for dual LOB check: {prompt}")
-            claude_answer_raw = llm_utils.invoke_claude(
-                prompt,
-                model_id="legacy_sonnet",
-                filename="",
-                max_tokens=200,
-                cache=True,
-                instruction=prompt_templates.DUAL_LOB_CHECK_INSTRUCTION(),
-            )
-            logging.debug(f"Claude response for dual LOB check: {claude_answer_raw}")
-            claude_answer_extracted = string_utils.extract_text_from_delimiters(
-                claude_answer_raw, Delimiter.PIPE
-            )
-            if (
-                claude_answer_extracted.lower() == "neither"
-                or claude_answer_extracted.lower() == "n/a"
-            ):  # If neither, take original AARETE_DERIVED_LOB
-                pass
-            else:
-                answer_dict["AARETE_DERIVED_LOB"] = claude_answer_extracted
-        final_answer_dicts.append(answer_dict)
-    return final_answer_dicts
-
-
 def fill_empty_reimb_pct_rate(df):
     """
     Fill 'REIMB_PCT_RATE' based on reimbursement method: