Merged in bugfix/DAIP2-2230-mcs-issues-effective-date (pull request #922)

Bugfix/DAIP2-2230 mcs issues effective date

* Merged DEV into bugfix/DAIP2-2230-mcs-issues-effective-date

* update prompt to not extract dates from exhibits

* resolving dates issue fix

* ignore dates from subsidiary agreements, exhibits, attachements fix

* resolve referenced dates fix

* remove document modifier effective date

* prompt optimization

* right effective date fix

* removed rates effective date

* black format fix

* removed debug statements

* Merged DEV into bugfix/DAIP2-2230-mcs-issues-effective-date

* optimized prompt


Approved-by: Katon Minhas

src/pipelines/shared/preprocessing/hybrid_smart_chunking_funcs.py

@@ -449,6 +449,9 @@ def prompt_hsc_single_field(
         if field.field_name in ["CONTRACT_TITLE", "PAYER_NAME"]:
             # concatenate values from first 3 items in text_dict to provide more context for contract title extraction, as it is often found in the beginning of the contract
             context = "\n\n".join(list(text_dict.values())[:3])
+        if field.field_name == "EFFECTIVE_DT":
+            # for effective date always provide first 3 pages as context as effective date is often found in the beginning of the contract and it is important to provide enough context for LLM to extract it correctly
+            context = "\n\n".join(list(text_dict.values())[:3]) + "\n\n" + context
         prompt, _parser = prompt_templates.ONE_TO_ONE_SINGLE_FIELD_TEMPLATE(
             context, llm_prompt, field_name=field.field_name
         )

src/prompts/investment_prompts.json

@@ -61,7 +61,6 @@
         "field_type": "methodology_breakout",
         "prompt": "If the methodology text indicates that the reimbursement shall be the greater of two or more values, write 'Y' for this field. Otherwise, write 'N'."
     },
-    
     {
         "field_name": "REIMB_DATES",
         "relationship": "one_to_n",
@@ -111,7 +110,7 @@
         "field_type": "dynamic_primary",
         "prompt": "A type of managed care structures that defines what providers a patient can use. Choose Networks ONLY from the following values: {valid_values}. Only return values that are explicitly written.",
         "valid_values": "VALID_NETWORKS",
-        "retrieval_question" : "What Network (HMO, PPO, etc) does this contract apply to?"
+        "retrieval_question": "What Network (HMO, PPO, etc) does this contract apply to?"
     },
     {
         "field_name": "AARETE_DERIVED_NETWORK",
@@ -267,7 +266,6 @@
         "field_type": "smart_chunked",
         "prompt": "Extract the PROVIDER organization name(s) from the retrieved contract text.\n\nThe provider is the healthcare organization entering into the agreement (NOT the payer/insurance company like Highmark, Aetna, UnitedHealthcare, Blue Cross Blue Shield, etc.).\n\nGUIDELINES:\n1. Look for the provider name in phrases like:\n   - 'This Agreement is between [PAYER] and [PROVIDER]'\n   - 'Agreement between [PAYER] and [PROVIDER]'\n   - 'by and between [PAYER] and [PROVIDER]'\n   - '[PROVIDER] (hereinafter referred to as \"Provider\")'\n   - 'Provider: [PROVIDER]'\n2. The provider name often appears in the opening paragraph or preamble of the contract.\n3. Return the EXACT provider name as it appears in the document, including legal suffixes (LLC, Inc., P.C., etc.).\n4. If the provider name includes 'd/b/a' (doing business as), return the FULL name including the d/b/a portion.\n5. If MULTIPLE provider organization names appear in the contract (e.g., multiple entities entering the agreement), return ALL of them as a JSON list.\n   - Example: [\"Provider Group A, LLC\", \"Provider Group B, P.C.\", \"Provider Group C d/b/a City Medical\"]\n6. Do NOT return:\n   - The payer/insurance company name\n   - Individual physician names (unless that is the practice name)\n   - Department names only\n   - Addresses or other metadata\n7. If the provider name cannot be determined from the retrieved text, return 'N/A'.\n\nBriefly explain your reasoning. Return your answer as a valid JSON dictionary with the key \"PROVIDER_NAME\". If multiple provider names are found, the value should be a JSON list. If a single provider name is found, the value should be a string. Example: {\"PROVIDER_NAME\": [\"Provider Group A, LLC\", \"Provider Group B, P.C.\"]} or {\"PROVIDER_NAME\": \"Provider Group A, LLC\"} or {\"PROVIDER_NAME\": \"N/A\"}",
         "retrieval_question": "What is the provider organization name? Agreement between payer and provider. This agreement is made between. By and between. Hereinafter referred to as Provider. Provider name in contract preamble. Provider party to the agreement. Healthcare organization entering agreement."
-
     },
     {
         "field_name": "AARETE_DERIVED_PROVIDER_NAME",
@@ -298,7 +296,7 @@
         "field_name": "EFFECTIVE_DT",
         "relationship": "one_to_one",
         "field_type": "smart_chunked",
-        "prompt": "Analyze the contract and extract the effective date of the CURRENT contract or amendment (not exhibits/reimbursements). The date may not be explicitly labeled.\nRETRIEVAL RULES (in order):\n1. Look for these trigger phrases followed by a date:'Effective Date:', 'is entered into as of', 'effective as of', 'shall begin on', 'Effective Date of Amendment', 'made this', 'made and entered into' - or any phrase semantically indicating when the contract/amendment takes effect, begins, or was created.\n2. Resolve referenced dates: If the effective date is defined by a formula or reference (e.g., \"the earlier of [reference] or [date]\"), locate the actual date(s) the reference points to, substitute them, then apply the formula to determine the final date.\n3. Signature dates: Use ONLY if explicitly marked as the effective date, or if the contract states it is effective as of the earlier of the signature date and another date.\n4. Blank \"Effective Date:\" label: If no explicit date exists but the label appears with a blank field, use a nearby unassigned/free-floating date ONLY IF other nearby dates are explicitly labeled and no clearer date exists elsewhere.\nDATE FORMAT: Return the date EXACTLY as it appears - preserve all spacing, punctuation, and formatting. Do not infer missing components; return incomplete/blank dates as-is. If multiple matches: Prioritize the current contract/amendment over attachments. If still multiple, return the latest.\nReturn \"N/A\" only if no valid date is found.\nOutput: Show reasoning (list all candidate dates, explain each), then return:{\"EFFECTIVE_DT\": \"<exact date string>\"} or {\"EFFECTIVE_DT\": \"N/A\"}",
+        "prompt": "Extract the effective date of the CURRENT contract or amendment only. Ignore effective dates from exhibits, attachments, rate provisions, or any other subsidiary sections. The date may not be explicitly labeled.\n\nSCOPE: Always extract the effective date for the current (primary, topmost) contract or amendment being analyzed — this means the top-level document in the hierarchy, NOT the most recently dated contract or amendment. For amendments, use that amendment's effective date; for base agreements, use the main agreement's effective date.\n\nRETRIEVAL RULES (apply in order):\n1. TRIGGER PHRASES: Search for these phrases followed by a date: 'Effective Date:', 'is entered into as of', 'effective as of', 'shall begin on', 'Effective Date of Amendment', 'made this', 'made and entered into', 'dated' — or any semantically equivalent phrase indicating when the contract/amendment takes effect or was created. If a trigger phrase contains a date (even incomplete or partially blank), use it and do not override it.\n2. FORMULA/REFERENCED DATES: If the effective date is defined by a formula or reference (e.g., 'the earlier of [X] or [Y]', 'dated above'):\n   - Locate and substitute the actual referenced date(s); apply the formula.\n   - Use only explicitly identified referenced dates. If a referenced date is unavailable, compute using only the available ones.\n   - Return only the resolved/computed date — not the full phrase, formula, or any individual input date unless it is itself the computed result.\n   - Do not substitute missing referenced dates with dates from unrelated sections (e.g., signature dates).\n   - If a reference (e.g., 'date above') points to a trigger phrase, use that trigger phrase as-is even if incomplete; do not substitute with dates from other sections.\n3. SIGNATURE DATES: Use only if the contract explicitly states the effective date equals the signature date(s) (e.g., 'last signature date', 'earlier of signature date and [date]'), or if a Rule 2 formula explicitly references signature dates.\n4. BLANK 'EFFECTIVE DATE:' LABEL (fallback — apply only if Rules 1–3 yield no valid date): If the label appears with a blank field, use a nearby free-floating date ONLY IF it is in the same immediate context (same line, sentence, or closely adjacent lines), other nearby dates are explicitly labeled, and no clearer date exists elsewhere. Do not use dates from the Signature section or distant parts of the document.\n\nDATE FORMAT: Return the date exactly as it appears — preserve all spacing, punctuation, and formatting. Do not infer or complete missing components.\n\nMULTIPLE CANDIDATES: Prioritize the effective date of the current contract/amendment; if still multiple, return the latest.\n\nReturn \"N/A\" only if no valid effective date is found.\n\nOutput: Show reasoning (list all candidate dates with explanation), then return:{\"EFFECTIVE_DT\": \"<exact date string>\"} or {\"EFFECTIVE_DT\": \"N/A\"}",
         "keywords": [
             "effective date",
             "effective date:",