Merged in bugfix/effective_date_fix (pull request #504)

DRAFT: Bugfix/effective date fix

* Merged in bugfix/extra-fields (pull request #500)

Do not run Exhibit level prompt if there are no exhibit level fields

* Do not run Exhibit level prompt if there are no exhibit level fields

* Update dynamic test

* Remove test from one-to-n test module

* Update test

* Fix unit test


Approved-by: Alex Galarce

* Merged in bugfix/tin-npi-prompt-fixes (pull request #499)

Bugfix/tin npi prompt fixes

* Change delimiter for other provider information from comma to pipe

* filter out invalid providers

* Update extraction and formatting instructions for provider entities in TIN_NPI_TEMPLATE

* Merged main into bugfix/tin-npi-small-bugfixes

* Refactor deduplication logic to use string_utils for checking empty provider fields

* Merge branch 'bugfix/tin-npi-small-bugfixes' of https://bitbucket.org/aarete/doczy.ai into bugfix/tin-npi-small-bugfixes

* Update deduplication logic to exclude providers with 'UNKNOWN' TIN, NPI, or NAME

* Merge remote-tracking branch 'origin/main' into bugfix/tin-npi-small-bugfixes

* Fix deduplication logic to not drop subsequent names

* Fix formatting of provider names in deduplication logic

* add payer name filtering

* reorder execution order to allow to find payer_name first

* Add handling for previously unfound group providers in identify_group_provider function

* Merge remote-trackin…
* Merged in feature/update-testbed-script (pull request #501)

Feature/update testbed script

* Update preprocessing

* Update preprocessing

* remove quit

* Update date for filename

* Merged main into feature/update-testbed-script


Approved-by: Alex Galarce

* Merged in bugfix/smart-chunks (pull request #503)

missing of text in chunks fixed

* missing of text in chunks fixed


Approved-by: Katon Minhas

* Merged in feature/rework-reimb-tin-npi (pull request #502)

Feature/rework reimb tin npi

* use LLM for reimb_[tin/npi/name]

* Merge remote-tracking branch 'origin/main' into feature/rework-reimb-tin-npi

* Merge remote-tracking branch 'origin/main' into feature/rework-reimb-tin-npi

* deduplicate tin/name/npi before adding to prompt

* Pull out prompt and llm_response in methodology_breakout_single_row for easier debugging

* Refactor reimbursement_tin_npi to handle multiple providers and update valid TIN, NPI, and NAME fields accordingly

* Reorder REIMB_PROV_NAME field in investment_prompts.json

* Update REIMB_PROV_NAME prompt to specify valid values for provider names

* Remove valid_values constraint from prov_name to raise accuracy

* Remove debugging print statements from reimbursement_tin_npi and related functions

* Remove debugging print statements from run_provider_info_fields function

* Merge remote-tracking branch 'origin/main' into feature/rework-reimb-tin-npi

* Remove debugging print statem…
* Merged in feature/preformat-single-quotes-input (pull request #506)

Replace double quotes with single quotes in context strings for parsability

* Replace double quotes with single quotes in context strings for parsability

* Merged main into feature/preformat-single-quotes-input


Approved-by: Katon Minhas

* updated sig pages fnxn

* Merge branch 'main' into bugfix/effective_date_fix

* poetry add pymupdf

* created vision funcs

* Merged main into bugfix/effective_date_fix

* fixed pipeline issue

* updated string_utils_test

* updated test values

* updated extract signature page fxn

* updated test values

* updated test values

* Merged main into bugfix/effective_date_fix

* Prompt template in all-caps

* remove debugging prints

* Merged main into bugfix/effective_date_fix

* Change vision to False by default

* Change vision to False by default

* fix tests

* fix tests


Approved-by: Katon Minhas

fieldExtraction/src/utils/string_utils.py

@@ -384,7 +384,6 @@ def get_exhibit_chunk(text_dict: dict,
 def datetime_str():
     return datetime.now().strftime("[%Y-%m-%d %H:%M:%S]")
 
-
 def extract_signature_page(text_dict: dict, filename: str) -> dict:
     """Extracts the signature page(s) from a contract text dictionary and returns it as a dictionary.
     First tries to match the textract marker "this page has N signature(s)", then falls back to looking 
@@ -408,10 +407,57 @@ def extract_signature_page(text_dict: dict, filename: str) -> dict:
         if textract_pattern.search(page_text):
             signature_pages[page_num] = page_text
     
-    # If no pages found with textract pattern, fall back to "signature" keyword
+    # If no pages found with textract pattern, try with  keyword
     if not signature_pages:
-        for page_num, page_text in text_dict.items():
-            if "signature" in page_text.lower():
-                signature_pages[page_num] = page_text
+        page_items = list(text_dict.items())
+        for i, (page_num, page_text) in enumerate(page_items):
+            lower_text = page_text.lower()
+            if "signature page follow" in lower_text or "signature authorization" in lower_text:
+                # curr_page = int(float(page_num))
+                if page_num not in signature_pages:
+                    signature_pages[page_num] = page_text  # current page
+                if i + 1 < len(page_items):
+                    next_page_num, next_page_text = page_items[i + 1]
+                    # next_page = int(float(next_page_num))
+                    if next_page_num not in signature_pages:
+                        signature_pages[next_page_num] = next_page_text  # next page 
+
 
     return signature_pages
+
+
+def extract_effective_date_pages(text_dict: dict, filename: str) -> dict:
+    """
+    Extracts the effective date page(s) from a contract text dictionary and returns it as a dictionary.
+
+    This function identifies pages that either contain the Textract marker 
+    "this page has N signature(s)" or the keyword 'effective date:'. 
+    Additionally, it ensures that the first page (page 1) is always included 
+    in the result, as it often contains effective date.
+
+    Args:
+        text_dict (dict): A dictionary containing text data, organized by pages.
+        filename (str): The name of the file being processed.
+
+    Returns:
+        dict: A dictionary containing the extracted effective date page(s), 
+              where keys are page numbers and values are the corresponding page text.
+    """
+
+    effective_date_pages = {}
+
+    # Regex pattern to match the Textract marker for signature pages which often has effective dates
+    # This pattern accounts for both numeric and textual representations of numbers (one-ten).
+    textract_pattern = re.compile(r"this page has (?:\d+|one|two|three|four|five|six|seven|eight|nine|ten) signature", re.IGNORECASE)
+        
+    # Iterate through each page in the text dictionary.
+    for page_num, page_text in text_dict.items():
+        # page_num = int(float(page_num))
+        # Check if the page contains the Textract marker or the keyword 'effective date:'.
+        if textract_pattern.search(page_text) or 'effective date:' in page_text.lower():
+            effective_date_pages[page_num] = page_text
+        # Always include the first page (page 1) as it often contains effective date.
+        elif page_num == 1:
+            effective_date_pages[page_num] = page_text
+
+    return effective_date_pages