Merged in feature/DAIP2-1803-add-aarete-derived-provider-name (pull request #884)

Feature/DAIP2-1803 add aarete derived provider name

* Merged DEV into feature/DAIP2-1562-add-aarete_derived_payer_name

* state logic added in clustering

* removed print statements

* pull request updates

* black format fix

* Merged DEV into feature/DAIP2-1562-add-aarete_derived_payer_name

* added scalability feature and optimization

* black format

* Pull Request Changes

* generalised funcs from aarete derived payer name

* updated remove states and updated docstring for build similarity matrix

* derived payer name functionality added

* config and main file changes

* sync dev into ADD AARETE DERIVED PROVIDER GROUP NAME FULL

* updated derived_provider name

* pipeline fixes

* Merged DEV into feature/DAIP2-1803-add-aarete-derived-provider-name

* config changes

* field name updated to AARETE_DERIVED_PROVIDER_NAME

* Merged DEV into feature/DAIP2-1803-add-aarete-derived-provider-name

* black format fix

* Fix docstring and add empty column when not found

* Rename test file and add missing column test case

- Renamed test_derived_payer_name.py to test_derived_names.py since
  file now covers both payer and provider name functionality
- Added test_derived_provider_names_column_missing to verify empty
  column is added when AARETE_DERIVED_PROVIDER_NAME is not present

* Remove trailing whitespace

* Merge remote-tracking branch 'origin/DEV' into feature/DAIP2-1803-add-aarete-derived-provider-name


Approved-by: Siddhant Medar

src/utils/string_utils.py

@@ -932,7 +932,7 @@ def remove_legal_suffixes(text: str) -> str:
     return " ".join(tokens)
 
 
-def extract_dba_payer_name(text: str) -> tuple[str | None, str | None]:
+def extract_dba_name(text: str) -> tuple[str | None, str | None]:
     """
     Extract a DBA (Doing Business As) name from a text string.
 
@@ -1001,8 +1001,10 @@ def remove_states(text: str) -> str:
             if tokens[0] == state_tokens[0]:
                 return text
 
-            # Remove trailing state tokens
+            # Remove state AND trailing "of" if present
             new_tokens = tokens[: -len(state_tokens)]
+            if new_tokens and new_tokens[-1] == "of":
+                new_tokens = new_tokens[:-1]
             return " ".join(new_tokens).strip()
 
     # Also handle "of <state>" pattern anywhere in the string, as it often indicates a regional suffixe.g., "Health Plan of New York" should become "Health Plan"