Merged in feature/lesser-table-caching-refactor-hybrid (pull request #847)

Feature/lesser table caching refactor hybrid

* chore: Remove unused duplicate main.py from shared pipeline

* fix: Correct crosswalk paths in aarete_derived.py

* chore: Remove unused documentation files from fieldExtraction

* docs: Add documentation files to documentation folder

* docs: Update README with uv setup, expanded project structure, and branching conventions

* docs: Add uv installation steps with Ubuntu/WSL emphasis

* Enable prompt caching for all remaining LLM calls

- Add _INSTRUCTION() functions for: EXHIBIT_HEADER, EXHIBIT_LINKAGE,
  EXHIBIT_TITLE_MATCH, DATE_FIX, DERIVED_TERM_DATE, CHECK_PROVIDER_NAME_MATCH,
  SPECIAL_CASE_ASSIGNMENT
- Update all invoke_claude() calls in saas and clover pipelines to use
  cache=True with corresponding _INSTRUCTION() functions
- Add new instructions to get_cacheable_instructions() for cache warming
- Update tests for new instruction functions

Functions now using caching:
- prompt_exhibit_level
- prompt_exhibit_lesser (EXHIBIT_LEVEL_LESSER_OF)
- prompt_fee_schedule_breakout
- prompt_grouper_breakout
- prompt_special_case_assignment
- prompt_exhibit_linkage
- prompt_exhibit_header
- prompt_smart_chunked (ONE_TO_ONE templates)
- prompt_date_fix
- prompt_derived_term_date
- prompt_exhibit_title_match
- provider_name_match_check

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* Reorder

* feat: Add bcbs_promise client pipeline with OFFSET_TERM extraction

- Add new bcbs_promise client with HSC-based OFFSET_TERM field extraction
- Extract full paragraph text of offset/recoupment provisions from contracts
- Derive OFFSET_INDICATOR (Y/N) from OFFSET_TERM presence
- Fix reorder_columns to preserve extra columns not in COLUMN_ORDER
- Update QC/QA output path to outputs/qc_qa/

* fix: Update dev deps and test assertions for QC/QA output path

- Add pytest/pytest-mock to dev dependencies for mypy type checking
- Update test assertions to expect outputs/qc_qa instead of qa_qc_output

* style: Apply black formatting to prompt_templates.py

* Merge main, move scripts

* Archive some scripts

* update py version

* remove .py version file

* Remove ASCII characters

* Restore testbed code

* restore tracking

* Update testbed metrics

* Enable prompt caching for CODE_LAST_CHECK, FILL_BILL_TYPE, DUAL_LOB_CHECK, and GROUPER_BREAKOUT

- Add CODE_LAST_CHECK_INSTRUCTION() for service specificity classification
- Add FILL_BILL_TYPE_INSTRUCTION() for bill type code determination
- Add DUAL_LOB_CHECK_INSTRUCTION() for Medicare/Medicaid classification
- Update code_funcs.py to use caching for CODE_LAST_CHECK, FILL_BILL_TYPE, GROUPER_BREAKOUT
- Update postprocessing_funcs.py to use caching for DUAL_LOB_CHECK
- Add new instructions to get_cacheable_instructions() for cache warming
- Add unit tests for new instruction functions

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* Fix postprocessing_funcs to remove invalid columns

* Merge branch 'main' into feature/lesser-table-caching-refactor-hybrid

* Revert prompt caching changes from aed1b73c

* update formatting

* Update imports


Approved-by: Sha Brown
Approved-by: Praneel Panchigar

.dockerignore

@@ -1 +1,37 @@
-.mypy_cache
+# Git
+.git/
+.gitignore
+
+# Python
+__pycache__/
+*.py[cod]
+*.pyo
+.mypy_cache/
+.pytest_cache/
+
+# IDE
+.idea/
+.cursor/
+.vscode/
+
+# Outputs
+outputs/
+output/
+*.log
+
+# Environment
+.env
+*.env
+myenv/
+
+# Tests
+src/tests/
+
+# Docs
+docs/
+*.md
+
+# Data files (usually loaded from S3)
+*.csv
+*.xlsx
+data/

.gitignore

@@ -1,45 +1,55 @@
-# These are some examples of commonly ignored file patterns.
+# Python
-# You should customize this list as applicable to your project.
-# Learn more about .gitignore:
-#     https://www.atlassian.com/git/tutorials/saving-changes/gitignore
-
-# Node artifact files
-node_modules/
-dist/
-
-# Compiled Java class files
-*.class
-
-# Compiled Python bytecode
 *.py[cod]
+*.pyo
+__pycache__/
+.mypy_cache/
+.pytest_cache/
+myenv/
+*.egg-info/
 
-# Log files
+# IDE
-*.log
-
-# Package files
-*.jar
-*.png
-# Maven
-target/
-dist/
-
-# IDE files
 .idea/
 .cursor/
+.vscode/
 
-# Unit test reports
+# OS
-TEST*.xml
-
-# Generated by MacOS
 .DS_Store
-
-# Generated by Windows
 Thumbs.db
+*:Zone.Identifier
 
-# Applications
+# Logs
+*.log
+
+# Environment
+.env
+*.env
+
+# Data files (general exclusion)
+*.csv
+*.xlsx
+*.xlsb
+*.zip
+*.json
+*.txt
+*.pdf
+*.PDF
+*.TXT
+
+# Allow tracked data files in src/
+!src/constants/**/*.csv
+!src/constants/**/*.json
+!src/prompts/*.json
+!requirements.txt
+
+# Build artifacts
+dist/
+build/
+target/
+*.class
+*.jar
+*.war
 *.app
 *.exe
-*.war
 
 # Large media files
 *.mp4
@@ -49,15 +59,28 @@ Thumbs.db
 *.mov
 *.wmv
 
-# Terraform files
+# Terraform
 .terraform/
-terraform.tfstate.backup
+terraform.tfstate*
-terraform.tfstate
 .terraform.lock.hcl
-build/
+*.tfplan
+*.pem
 textract-pipeline/terraform/builds/
 
-# Data Files
+# Outputs
+output/
+outputs/
+
+# Local/temporary
+subset/
+docs/
+local_scripts/
+untracked_tests/
+new/
+textfiles/
+texts/
+
+# Streamlit (legacy)
 streamlit/history.csv
 streamlit/RESULTS
 streamlit/DB/
@@ -66,47 +89,22 @@ streamlit/SOURCE_DOCUMENTS/
 streamlit/contract_field_values.csv
 streamlit/contract_fields.csv
 streamlit/sample.csv
-streamlit/temp1.csv
+streamlit/temp*.csv
-streamlit/temp2.csv
 streamlit/results.csv
-
-# env
 streamlit/venv
-*.pdf
-*.PDF
-*.TXT
-*.txt
-*:Zone.Identifier
-*.tfplan
-*.pem
-textfiles/
-texts/
-*.csv
-*.xlsx
-*.zip
-*.json
-myenv/
-*.env
-subset/
-output/
-docs/
-local_scripts/
-untracked_tests/
 
-.mypy_cache/
+# Legacy paths
-.pytest_cache/
-__pycache__/
-
-.env
-new/
-
-# Embeddings
 fieldExtraction/embeddings/
 
-# Output reports
+# Node (if any)
-fieldExtraction/output_reports/
+node_modules/
 
-# PRD Documents (local only, living documents)
+# Test reports
+TEST*.xml
+
+# Shell scripts (debugging/utility)
+*.sh
+
+# PRD Documents
 *.prd
 *prd.md
-usage-cost-monitoring-prd.md