Merged in bugfix/reimb-term-issues (pull request #705)

Add time-based rate handling rules and update section labels in prompt templates

* Add time-based rate handling rules and update section labels in prompt templates

* Add mapping for MyCare Ohio in crosswalk_program.json and update state mapping for Ohio in crosswalk_program_lob.json

* Update test parameters and state configuration in testing notebook

* Update exhibit-wide payment constraint prompt to exclude administrative processing language

* Merge remote-tracking branch 'origin/main' into bugfix/reimb-term-issues

* Refine exhibit-wide payment constraint prompt to clarify extraction criteria and examples

* Enhance exhibit-wide payment constraint prompt with guidance for abbreviated terms and clarify acceptable constraints

* Update SERVICE_TERM prompt to specify healthcare services and clarify extraction criteria


Approved-by: Katon Minhas

fieldExtraction/constants/mappings/crosswalk_program.json

@@ -195,7 +195,8 @@
             "Healthy Steps" : "CHIP"
         },
         "OH" : {
-            "Healthy Start" : "CHIP"
+            "Healthy Start" : "CHIP",
+            "MyCare Ohio" : "MYCAREOH"
         },
         "OK" : {
             "SoonerCare" : "SOONER",

fieldExtraction/constants/mappings/crosswalk_program_lob.json

@@ -162,7 +162,7 @@
             
         },
         "OH" : {
-            
+            "MYCAREOH" : "Duals"
         },
         "OK" : {
             "SOONER" : "Medicaid"

fieldExtraction/scripts/investment_field_testing/testing.ipynb

@@ -84,8 +84,15 @@
     "test_params['local_input_dir'] = TEST_DATA_INPUT_DIR\n",
     "test_params['max_workers'] = 1\n",
     "test_params['input_files'] = [ # you can specify specific files to run here\n",
-    "    \"02-0677066-Eyemasters-ICMProviderAgreement_64220_2.txt\", # replace with your test file(s)\n",
-    "]"
+    "    \"Akron General Health System_Eleventh Amendment_20171001.txt\", # replace with your test file(s)\n",
+    "]\n",
+    "\n",
+    "# Specify state(s) for testing, if needed\n",
+    "# Specify them here instead of in `test_params` bc `config` is initialized \n",
+    "# before `test_params` is read in main.py, which means the state-level constants\n",
+    "# won't be set correctly if we put it in `test_params`.\n",
+    "config.STATE = [\"OH\"] # for single states\n",
+    "# config.STATE = [\"OH\", \"CA\"] # for multi-state testing"
    ]
   },
   {

fieldExtraction/src/prompts/investment_prompts.json

@@ -1,9 +1,9 @@
 [
     {
-        "field_name" :"SERVICE_TERM",
-        "relationship" : "one_to_n",
-        "field_type" : "reimbursement_level", 
-        "prompt" : "What is the Service that is being reimbursed? Include the full Service, including any distinguishing context. Include any codes that are part of the service, including revenue codes and procedure codes. Use only exact text from the document. Do not leave this field N/A."
+        "field_name": "SERVICE_TERM",
+        "relationship": "one_to_n",
+        "field_type": "reimbursement_level", 
+        "prompt": "What is the healthcare Service that is being reimbursed? Include the full Service, including any distinguishing context. Include any codes that are part of the service, including revenue codes and procedure codes. Use only exact text from the document.\n\nEXCLUDE the following non-healthcare services:\n- Administrative fees (copying, records requests, processing fees)\n- Operational costs (supplies, equipment rental)\n- Legal/compliance costs (audits, investigations)\n- General business expenses\n- General payment process or claim processing language (e.g. Clean Claims)\n\nOnly extract services that involve direct patient care, medical procedures, or clinical services. If no healthcare service is being reimbursed, return 'N/A'."
     },
     {
         "field_name" : "REIMB_TERM",
@@ -27,7 +27,7 @@
         "field_name": "EXHIBIT_LESSER_OF_STATEMENT",
         "relationship": "one_to_n",
         "field_type": "exhibit_level",
-        "prompt": "Look for exhibit-wide payment constraints that apply universally to ALL services in this exhibit, regardless of service category or line of business.\n\nValid exhibit-wide constraints have language like:\n- 'All services under this exhibit are subject to the lesser of...'\n- 'Except as otherwise provided in this Exhibit, the Allowed Amount for Covered Services shall be the lesser of...'\n- 'Payment for any service in this exhibit shall not exceed...'\n- 'Unless specifically stated otherwise, all reimbursements are limited to the lesser of...'\n\nDO NOT extract constraints that:\n- Apply only to specific lines of business (e.g., 'Commercial HMO services are lesser of...')\n- Apply only to specific service categories (e.g., 'Supplies are lesser of...', 'Medications are lesser of...')\n- Are part of individual service methodologies\n- Are introduced by category-specific headers\n\nIf you find a true exhibit-wide constraint, return ONLY the constraint value (what comes after 'lesser of the exhibit rates or'). \n\nExamples:\n- 'Except as otherwise provided in this Exhibit, payment shall be the lesser of contracted rates or Provider's billed charges' → return 'Provider's billed charges'\n- 'All services are subject to the lesser of fee schedule or 100% of charges' → return '100% of charges'\n\nIf no exhibit-wide constraint exists, return 'N/A'."
+        "prompt": "Look for exhibit-wide payment constraints that apply universally to ALL services in this exhibit, regardless of service category or line of business.\n\nValid exhibit-wide constraints have language like:\n- 'All services under this exhibit are subject to the lesser of...'\n- 'Except as otherwise provided in this Exhibit, the Allowed Amount for Covered Services shall be the lesser of...'\n- 'Payment for any service in this exhibit shall not exceed...'\n- 'At no time shall Plan pay an amount that exceeds...'\n\nDO NOT extract constraints that:\n- Apply only to specific lines of business or service categories\n- Are part of individual service methodologies\n\nIf you find a true exhibit-wide constraint, return the component that represents the actual payment limitation, excluding administrative processing language such as:\n- 'less any applicable co-payments, deductibles and coinsurance'\n- 'minus patient responsibility amounts'\n\n**GUIDANCE FOR ABBREVIATED TERMS:**\n- 'billed' can be expanded to 'Provider's billed charges'\n- 'allowable' typically refers to amounts defined elsewhere in the contract and is not an acceptable exhibit-wide constraint due to its nonspecificity\n- Other generic terms like 'contracted rate' or 'negotiated amount' should be avoided unless they represent specific, quantifiable constraints\n- Specific percentages of external fee schedules (e.g., '105% of Medicare rates') are acceptable constraints\n\nExamples:\n- 'all rates will be reimbursed at the lesser of billed or allowable' → return 'Provider's billed charges'\n- 'Lesser of 105% of Medicare rates or allowable charges' → return '105% of Medicare rates'\n- 'Lesser of Provider's charges or contracted rates' → return 'Provider's billed charges'\n- 'At no time shall payment exceed Provider's billed charges' → return 'Provider's billed charges'\n\nIf no exhibit-wide constraint exists, return 'N/A'."
     },
     {
         "field_name": "RATE_ESCALATOR_STATEMENT",

fieldExtraction/src/prompts/prompt_templates.py

@@ -305,6 +305,8 @@ CRITICAL: Apply these rules to handle content that may appear in different posit
 
 3. **Consistent Application**: Ensure that services appearing in the same contract section receive consistent treatment of global rules, regardless of their position relative to shared text.
 
+4. **EXCEPTION**: When time-period columns contain different rates, this completeness rule does NOT apply. Instead, create separate entries per the time-based handling rules below.
+
 [C. CRITICAL INSTRUCTIONS]
 
 LESSER OF LANGUAGE: When a contract contains "lesser of" statements:
@@ -337,8 +339,15 @@ Follow this sequence:
 3. **Constraint Application**: Apply identified global constraints to each extracted rate
 4. **Consistency Check**: Verify that similar services receive similar constraint treatment
 
+[E. TIME-BASED RATE HANDLING]
+When tables contain time-period columns (Initial Term, Renewal Term, Year 1, Year 2, etc.):
+- **Different rates across periods**: Create separate entries for each time period
+- **Identical rates across periods**: Create single entry with combined time range
+- **Apply global constraints**: Include exhibit-level "lesser of" statements consistently
 
-[E. EXAMPLES]
+Example: Service with "90% CMS (Initial)" and "95% CMS (Renewal)" → Two separate entries with respective rates and time periods.
+
+[F. EXAMPLES]
 [EXAMPLE 1]
 Text: 'Professional Covered Services is the lesser of: (i) Allowable Charges; or (ii) the "Contracted Rate" percentage found in Table 1. Table 1 - OB and Anesthesia Services : 100% of the Payor's Oregon Health Plan DMAP fee schedule. Radiology Services : 110% of Medicare Fee Schedule'
 Answer: "[{{'SERVICE_TERM' : 'OB and Anesthesia Services', 'REIMB_TERM' : 'lesser of (i) Allowable Charges; or (ii) 100% of the Payor's Oregon Health Plan DMAP fee schedule', ...}},
@@ -393,7 +402,7 @@ Answer: "[{{'SERVICE_TERM': 'A6544', 'REIMB_TERM': 'lesser of $25.00 and Provide
 
 Note: Both services receive the same global cap treatment regardless of table position.
 
-[E. ANALYSIS CONTEXT]
+[G. ANALYSIS CONTEXT]
 [Start Context]
 {context.replace('"', "'")}
 [End Context]