Merged in feature/1toN-Optimization (pull request #799)

Feature/1toN Optimization * Update config - remove Haiku 3 * Merged in optimization/trigger-cap (pull request #792) Optimization/trigger cap * trigger cap breakout added * Merge remote-tracking branch 'origin/feature/1toN-Optimization' into optimization/trigger-cap * carveout breakout added Approved-by: Katon Minhas * Merged in bugfix/utah_issue_fixes (pull request #791) Bugfix/utah issue fixes to feature/1toN-Optimization * updated valid values for AARETE_DERIVED_REIMB_METHOD * removed example reimbursements * removed reciprocatory agreements section reimbs * Merged feature/1toN-Optimization into bugfix/utah_issue_fixes * update reimb primary Approved-by: Katon Minhas * Update unit tests * Update llm_utils * Merge branch 'feature/deprecate-haiku-3' into feature/1toN-Optimization * fix over-filtering of lesser of * Merged in bugfix/UT-methodology-breakout (pull request #794) Bugfix/UT methodology breakout * updated valid values for AARETE_DERIVED_REIMB_METHOD * removed example reimbursements * prompt update * prompt update * Merged feature/1toN-Optimization into bugfix/nv_issue_fixes * add service term in mb prompts * Merged feature/1toN-Optimization into bugfix/UT-methodology-breakout * print statement removed * Merge branch 'bugfix/UT-methodology-breakout' of https://bitbucket.org/aarete/doczy.ai into bugfix/UT-methodology-breakout * primary prompt update * remove duplicate prompt Approved-by: Katon Minhas * Merge branch 'main' into feature/1toN-Optimization * Merge branch 'main' into feature/1toN-Optimization * Update preprocessing to make the exhibit_chunk_mapping start at first page * Merge row_funcs.py changes from feature/cross-exhibit-dynamic * Merged in bugfix/UT-grouper-issues (pull request #796) Bugfix/UT grouper issues * updated valid values for AARETE_DERIVED_REIMB_METHOD * removed example reimbursements * prompt update * prompt update * Merged feature/1toN-Optimization into bugfix/nv_issue_fixes * add service term in mb prompts * Merged feature/1toN-Optimization into bugfix/UT-methodology-breakout * print statement removed * Merge branch 'bugfix/UT-methodology-breakout' of https://bitbucket.org/aarete/doczy.ai into bugfix/UT-methodology-breakout * primary prompt update * prompt update * Merge remote-tracking branch 'origin/feature/1toN-Optimization' into bugfix/UT-grouper-issues * removed temp changes * removed temp changes * Update reimb primary Approved-by: Katon Minhas * Merged in bugfix/validation_fixes (pull request #795) bugfix/validation_fixes to feature/1toN-Optimization * updated validation of clean claims reimbursement * Merged feature/1toN-Optimization into bugfix/validation_fixes Approved-by: Katon Minhas * Re-add dynamic codes and reimb-info * Re-structure empty reimbursement prompt * Fix lesser of check overfiltering * Merged in bugfix/generic-methodology-breakout (pull request #797) Bugfix/generic methodology breakout * fee schedule fix * carveout+special case prompt changed * fee schedule changes * prompt changes and column order Approved-by: Katon Minhas * pytest fix * remove prints * Update mapping * Update tests * Update test * Address PR review comments: add logging, fix return types, fix variable propagation, and add tests - Add warning log in code_funcs.py try/except block for grouper code parsing failures - Add warning log in hybrid_smart_chunking_funcs.py except block for JSON parsing failures - Add return type annotation to lesser_of_distribution function - Fix first_reimbursement_page variable propagation issue by returning updated value - Add comprehensive tests for simplify_exhibit, chunk_by_exhibit, and lesser_of_distribution functions * Merged in bugfix/prov_info_json (pull request #802) Bugfix/prov info json to feature/1toN-Optimization * prov_info_json added * remove print statements Approved-by: Katon Minhas Approved-by: Siddhant Medar
2025-12-10 20:01:52 +00:00
parent 83fcb104a2
commit 0e2622f156
30 changed files with 1664 additions and 4228 deletions
@@ -1,7 +1,9 @@
 import pytest
-from src.investment.preprocessing_funcs import (clean_law_symbols,
+from src.investment.preprocessing_funcs import (chunk_by_exhibit,
+                                                clean_law_symbols,
                                                filter_quick_review,
-                                                remove_page_indicators)
+                                                remove_page_indicators,
+                                                simplify_exhibit)


 class TestPreprocessingFuncs:
@@ -80,3 +82,143 @@ class TestPreprocessingFuncs:
        contract, quick_review = filter_quick_review(input_dict)
        assert contract == expected_contract
        assert quick_review == expected_quick_review
+
+    # Test cases for chunk_by_exhibit
+    def test_chunk_by_exhibit_basic(self):
+        """Test basic exhibit chunking functionality."""
+        text_dict = {"1": "Page 1", "2": "Page 2", "3": "Page 3", "4": "Page 4"}
+        exhibit_pages = ["1", "3"]
+        
+        result = chunk_by_exhibit(text_dict, exhibit_pages)
+        
+        assert result == {"1": ["1", "2"], "3": ["3", "4"]}
+
+    def test_chunk_by_exhibit_single_exhibit(self):
+        """Test chunking when all pages belong to one exhibit."""
+        text_dict = {"1": "Page 1", "2": "Page 2", "3": "Page 3"}
+        exhibit_pages = ["1"]
+        
+        result = chunk_by_exhibit(text_dict, exhibit_pages)
+        
+        assert result == {"1": ["1", "2", "3"]}
+
+    def test_chunk_by_exhibit_empty_exhibit_pages(self):
+        """Test chunking when exhibit_pages is empty - each page becomes its own exhibit."""
+        text_dict = {"1": "Page 1", "2": "Page 2", "3": "Page 3"}
+        exhibit_pages = []
+        
+        result = chunk_by_exhibit(text_dict, exhibit_pages)
+        
+        assert result == {"1": ["1"], "2": ["2"], "3": ["3"]}
+
+    def test_chunk_by_exhibit_exhibit_at_start(self):
+        """Test chunking when exhibit starts at first page."""
+        text_dict = {"1": "Page 1", "2": "Page 2", "3": "Page 3"}
+        exhibit_pages = ["1", "2"]
+        
+        result = chunk_by_exhibit(text_dict, exhibit_pages)
+        
+        assert result == {"1": ["1"], "2": ["2", "3"]}
+
+    def test_chunk_by_exhibit_multiple_exhibits(self):
+        """Test chunking with multiple exhibits."""
+        text_dict = {"1": "Page 1", "2": "Page 2", "3": "Page 3", "4": "Page 4", "5": "Page 5"}
+        exhibit_pages = ["1", "3", "5"]
+        
+        result = chunk_by_exhibit(text_dict, exhibit_pages)
+        
+        assert result == {"1": ["1", "2"], "3": ["3", "4"], "5": ["5"]}
+
+    # Test cases for simplify_exhibit
+    def test_simplify_exhibit_no_tables(self):
+        """Test simplify_exhibit when there are no tables."""
+        text_dict = {"1": "Page 1 content", "2": "Page 2 content"}
+        exhibit_page_nums = ["1", "2"]
+        current_page_num = "1"
+        
+        result = simplify_exhibit(text_dict, exhibit_page_nums, current_page_num)
+        
+        assert result == "Page 1 content\nPage 2 content"
+
+    def test_simplify_exhibit_with_tables_current_page_preserved(self):
+        """Test that current page tables are preserved while others are simplified."""
+        text_dict = {
+            "1": "Before table-------Table Start--------Table content-------Table End--------After",
+            "2": "Before table-------Table Start--------Table content-------Table End--------After"
+        }
+        exhibit_page_nums = ["1", "2"]
+        current_page_num = "1"
+        
+        result = simplify_exhibit(text_dict, exhibit_page_nums, current_page_num)
+        
+        # Current page (1) should have full table content
+        assert "-------Table Start--------" in result.split("\n")[0]
+        # Other page (2) should have table replaced with placeholder
+        # Note: Table counter is incremented when counting tables on current page, so page 2 gets "Table 2"
+        assert "Table 2" in result.split("\n")[1]
+        assert "-------Table Start--------" not in result.split("\n")[1]
+
+    def test_simplify_exhibit_table_numbering_consistency(self):
+        """Test that table numbering is consistent across pages."""
+        text_dict = {
+            "1": "Text-------Table Start--------Table 1-------Table End--------Text",
+            "2": "Text-------Table Start--------Table 2-------Table End--------Text",
+            "3": "Text-------Table Start--------Table 3-------Table End--------Text"
+        }
+        exhibit_page_nums = ["1", "2", "3"]
+        current_page_num = "2"
+        
+        result = simplify_exhibit(text_dict, exhibit_page_nums, current_page_num)
+        result_pages = result.split("\n")
+        
+        # Page 1 (simplified) should have "Table 1"
+        assert "Table 1" in result_pages[0]
+        # Page 2 (current, preserved) should have full table content
+        assert "-------Table Start--------" in result_pages[1]
+        # Page 3 (simplified) should have "Table 3" 
+        # (counter is 2 after page 1, then incremented to 3 when counting page 2's table)
+        assert "Table 3" in result_pages[2]
+
+    def test_simplify_exhibit_multiple_tables_per_page(self):
+        """Test simplify_exhibit with multiple tables on a page."""
+        text_dict = {
+            "1": "Text-------Table Start--------T1-------Table End--------Text-------Table Start--------T2-------Table End--------",
+            "2": "Text-------Table Start--------T3-------Table End--------"
+        }
+        exhibit_page_nums = ["1", "2"]
+        current_page_num = "2"
+        
+        result = simplify_exhibit(text_dict, exhibit_page_nums, current_page_num)
+        result_pages = result.split("\n")
+        
+        # Page 1 should have both tables replaced: "Table 1" and "Table 2"
+        assert "Table 1" in result_pages[0]
+        assert "Table 2" in result_pages[0]
+        # Page 2 (current) should preserve table content
+        assert "-------Table Start--------" in result_pages[1]
+
+    def test_simplify_exhibit_missing_page_in_text_dict(self):
+        """Test simplify_exhibit when a page number is missing from text_dict."""
+        text_dict = {"1": "Page 1 content"}
+        exhibit_page_nums = ["1", "2"]
+        current_page_num = "1"
+        
+        result = simplify_exhibit(text_dict, exhibit_page_nums, current_page_num)
+        
+        # Should handle missing page gracefully (empty string)
+        assert "Page 1 content" in result
+        assert result.count("\n") == 1  # Two pages separated by newline
+
+    def test_simplify_exhibit_table_without_end_marker(self):
+        """Test simplify_exhibit when table has no end marker."""
+        text_dict = {
+            "1": "Text-------Table Start--------Table content without end",
+            "2": "Page 2 content"
+        }
+        exhibit_page_nums = ["1", "2"]
+        current_page_num = "2"
+        
+        result = simplify_exhibit(text_dict, exhibit_page_nums, current_page_num)
+        
+        # Should still process and replace table, stopping at end of text
+        assert "Table 1" in result or "-------Table Start--------" in result