Merged in feature/DAIP2-2112-implement-pre-commit-hooks--- (pull request #924)

Feature/DAIP2-2112 implement pre commit hooks

* Add pre-commit hooks for local code quality checks

Adds .pre-commit-config.yaml with file hygiene hooks (trailing-whitespace,
end-of-file-fixer, check-yaml/json, check-merge-conflict, check-added-large-files)
plus local hooks for black and mypy on pre-commit and pytest on pre-push.
Includes setup documentation and adds pre-commit to dev dependencies.
Auto-fixed trailing whitespace and missing newlines caught by the new hooks.

* Add sample pre-commit hook for testing purposes

* Add test file for pre-commit hook validation

* Remove test file for pre-commit hook validation

* Merged DEV into feature/DAIP2-2112-implement-pre-commit-hooks---

* Merged DEV into feature/DAIP2-2112-implement-pre-commit-hooks---

* Move to documentation


Approved-by: Katon Minhas

.pre-commit-config.yaml

@@ -0,0 +1,39 @@
+default_language_version:
+  python: python3.12
+
+default_install_hook_types: [pre-commit, pre-push]
+
+repos:
+  - repo: https://github.com/pre-commit/pre-commit-hooks
+    rev: v4.5.0
+    hooks:
+      - id: trailing-whitespace
+      - id: end-of-file-fixer
+      - id: check-yaml
+      - id: check-json
+      - id: check-merge-conflict
+      - id: check-added-large-files
+        args: ['--maxkb=500']
+
+  - repo: local
+    hooks:
+      - id: black
+        name: black (formatter)
+        entry: uv run black src/
+        language: system
+        pass_filenames: false
+        types: [python]
+
+      - id: mypy
+        name: mypy (type check)
+        entry: uv run mypy src/
+        language: system
+        pass_filenames: false
+        types: [python]
+
+      - id: pytest
+        name: pytest (unit tests)
+        entry: uv run pytest src/tests/
+        language: system
+        pass_filenames: false
+        stages: [pre-push]

documentation/PRE-COMMIT-SETUP.md

@@ -0,0 +1,62 @@
+# Pre-commit Local Setup (uv)
+
+## What is pre-commit?
+
+`pre-commit` runs checks before each commit (for example: formatting and file sanity checks).
+
+This repository uses pre-commit locally only. Bitbucket Pipelines does not run pre-commit.
+
+## Install and set up locally (with uv)
+
+1. Install `uv` (if not installed):
+
+```bash
+# Windows (PowerShell)
+powershell -c "irm https://astral.sh/uv/install.ps1 | iex"
+
+# macOS/Linux
+curl -LsSf https://astral.sh/uv/install.sh | sh
+```
+
+2. Install all dependencies (including pre-commit):
+
+```bash
+uv sync --group dev
+```
+
+3. Install git hooks in this repo:
+
+```bash
+uv run pre-commit install
+```
+
+## How it works
+
+When you run `git commit`, pre-commit automatically runs these checks on your staged files:
+
+| Hook | What it does | Auto-fix? |
+|---|---|---|
+| `trailing-whitespace` | Removes spaces at end of lines | Yes |
+| `end-of-file-fixer` | Ensures files end with a newline | Yes |
+| `check-yaml` | Validates YAML syntax | Manual fix |
+| `check-json` | Validates JSON syntax | Manual fix |
+| `check-merge-conflict` | Detects `<<<<<<<` conflict markers | Manual fix |
+| `check-added-large-files` | Blocks files larger than 500KB | Manual fix |
+
+- **Pass** -> commit goes through
+- **Fail (auto-fix)** -> commit blocked, files fixed, re-stage and commit again
+- **Fail (manual fix)** -> commit blocked, fix the error, re-stage and commit again
+
+## Common commands
+
+Run on all files:
+
+```bash
+uv run pre-commit run --all-files
+```
+
+Run on staged files (happens automatically on commit):
+
+```bash
+uv run pre-commit run
+```

src/pipelines/clients/bcbs_promise/prompts/__init__.py

@@ -1 +0,0 @@
-

src/pipelines/clients/clover/prompts/__init__.py

@@ -1 +0,0 @@
-

src/pipelines/saas/prompts/__init__.py

@@ -1 +0,0 @@
-

src/pipelines/shared/extraction/__init__.py

@@ -1 +0,0 @@
-

src/pipelines/shared/postprocessing/__init__.py

@@ -1 +0,0 @@
-

src/pipelines/shared/preprocessing/__init__.py

@@ -1 +0,0 @@
-

src/pipelines/shared/preprocessing/preprocessing_funcs.py

@@ -589,11 +589,11 @@ def identify_section_boundaries(page_text: str, context_chars: int = 100) -> str
     """
     Identifies potential section/exhibit headers in a contract page and returns
     a compact string with context around each potential boundary.
-    
+
     Args:
         page_text: Text content from a single page
         context_chars: Number of characters to include before/after each header
-    
+
     Returns:
         A string with markers and context for each potential section boundary
     """

src/prompts/prompt_templates.py

@@ -482,7 +482,7 @@ CRITICAL CONSIDERATIONS:
    3. If you find a section marker, that creates a HARD BOUNDARY - subsection context from BEFORE that marker is COMPLETELY INVALIDATED and must be ignored, even if it mentioned the table.
    4. Only consider subsection context that appears AFTER the most recent section marker and BEFORE the structural element.
    5. If no section marker is found, then use the closest subsection context before the structural element.
-   
+
     Section markers (numbered items like "1.", "2.", "3.", "4.") create hard boundaries. Example: If "4. Allowable Charges...for any Medicare inpatient admission" appears before "Table 2", you MUST use ONLY context from section 4. Completely ignore subsection context from earlier sections (like "For Covered Services rendered to Covered Person who are eligible for Medicare and enrolled in a Medicare Plan that may include coverage for both Medicare and Medicaid") even if that earlier text mentioned "Table 2" - the section 4 marker invalidates all earlier context.
 
    When a relevant subsection context is found WITHIN THE SAME structural boundary with extremely clear wording about eligibility or program coverage, it should:
@@ -700,7 +700,7 @@ DECISION LOGIC:
    -Treat it as a separate reimbursement methodology.
    -Do NOT apply it as a constraint to the primary METHODOLOGY.
    -Do NOT merge it into the output.
-   -Only include “lesser of” or “not to exceed” language when it operates as an overarching ceiling that applies to the base METHODOLOGY itself.   
+   -Only include “lesser of” or “not to exceed” language when it operates as an overarching ceiling that applies to the base METHODOLOGY itself.
 
 3. COMBINE & FORMAT
 
@@ -1119,7 +1119,7 @@ Here is the term to analyze:
 {term}
 
 [OUTPUT FORMAT]
-Briefly explain your answer, then put your final answer in JSON dictionary format. 
+Briefly explain your answer, then put your final answer in JSON dictionary format.
 """
 
     # Create parser with field_names bound for format-aware normalization
@@ -1641,7 +1641,7 @@ def ONE_TO_ONE_SINGLE_FIELD_TEMPLATE(
     Returns:
         Tuple of (prompt_string, parser_function) where parser expects JSON dict output.
     """
-    prompt = f"""The following is a contract (or excerpt thereof). 
+    prompt = f"""The following is a contract (or excerpt thereof).
 Answer the following question: {field_prompt}
 
 ## START CONTRACT TEXT ##
@@ -1842,7 +1842,7 @@ def EXTRACT_AMENDMENT_NUM_FROM_FILENAME_INSTRUCTION() -> str:
         Analyze the given filename string to extract the amendment number or identifier of the contract document if it exists.
         Amendment identifiers can be numeric, letters only, or alphanumeric. Follow these guidelines:
         - Numeric: '3', '12', '1' (with leading zeros dropped, e.g., '001' → '1')
-        - Letters only: 'A', 'AB', 'B' 
+        - Letters only: 'A', 'AB', 'B'
         - Alphanumeric: '3A', '2B', '1A2'
         - Roman numerals should be converted to numbers (e.g., 'IV' → '4')
         - Return the amendment identifier exactly as it appears in the filename, applying the above rules.
@@ -2307,13 +2307,13 @@ def EFFECTIVE_DATE_FIX_PROMPT() -> Tuple[str, Callable[[str], dict]]:
     Returns:
         Tuple of (prompt_string, parser_function) where parser expects JSON dict output.
     """
-    prompt = """Extract the effective date of the contract or contract effective date or Effective Date of Agreement or Effective Date of Amendment or the amendment effective date from the given image data. ALSO LOOK FOR dates in phrases that semantically indicate an effective date (may or may not be explictly labelled as effective date), even if they do not exactly match the phrases above.\nExamples include but ARE NOT LIMITED TO:\nPhrases indicating when an contract or amendment begins or takes effect, Phrases specifying the start date of the contract or amendment, Phrases defining when the contract or amendment becomes operational, or Any phrase indicating contract or amendment creation or execution date.\nIt is possible that the contract will specify that the effective date is derived from elsewhere in the contract. Give the date in a JSON FORMAT with AARETE_DERIVED_EFFECTIVE_DATE as the key and effective date value in YYYY/MM/DD format as the value. 
+    prompt = """Extract the effective date of the contract or contract effective date or Effective Date of Agreement or Effective Date of Amendment or the amendment effective date from the given image data. ALSO LOOK FOR dates in phrases that semantically indicate an effective date (may or may not be explictly labelled as effective date), even if they do not exactly match the phrases above.\nExamples include but ARE NOT LIMITED TO:\nPhrases indicating when an contract or amendment begins or takes effect, Phrases specifying the start date of the contract or amendment, Phrases defining when the contract or amendment becomes operational, or Any phrase indicating contract or amendment creation or execution date.\nIt is possible that the contract will specify that the effective date is derived from elsewhere in the contract. Give the date in a JSON FORMAT with AARETE_DERIVED_EFFECTIVE_DATE as the key and effective date value in YYYY/MM/DD format as the value.
     Rules:
     1. Add 2000 to 2-digit years 0-49, 1900 to 50-99
     2. Return N/A if: missing/ambiguous month or year components, or invalid dates.  If just the day is missing, assume it's the first of the month.
     3. For ambiguous input cases, assume American format (MM/DD/YYYY) and apply YYYY/MM/DD formatting
     4. Any day values with spaces or leading zeros (e.g., '0 1' or '01') are interpreted as single-digit days
-    
+
     Return N/A only for date if NO valid effective date is found. give the answer in the format: {{\"AARETE_DERIVED_EFFECTIVE_DT\": \"YYYY/MM/DD\"}}.  nothing else."""
 
     return (prompt, _json_dict_parser)

src/scripts/tin_pull.py

@@ -30,13 +30,13 @@ def find_tin_numbers(text):
 
 def invoke_llm(context, llm_selected=LLM_SELECTED):
     prompt_data = f"""
-    
+
     Human: Use the following pieces of context to provide a concise answer to the questions at the end. You must answer in python list format.
-    
+
     {context}
-    
+
     Question: What is the Group provider's taxpayer identification number stated in the contract? This is a 9 digit long number typically with a hyphen after the first 2 digits. Return only this 9 digit number with a hyphen after the first 2 digits. Do not elaborate or add context.
-    
+
     Assistant: Answer in list format: ["""
 
     if llm_selected == "Claude 3 - Sonnet":

src/tests/test_prompt_calls.py

@@ -825,11 +825,11 @@ class TestPromptCalls(unittest.TestCase):
         """Test that prompts handle LLM reasoning text before JSON."""
         # Simulate LLM returning reasoning text before JSON (common pattern)
         mock_invoke.return_value = """Let me analyze this:
-        
+
         Based on the contract text, I can see:
-        
+
         {"PRODUCT": ["Product1"], "PROGRAM": "Program1"}
-        
+
         This is the final answer."""
 
         field_prompts = {