BCBS: change strict == "YES" to "YES" in final_answer. The strict
equality was rejecting borderline LLM responses (e.g. "YES, ..."),
dropping row counts from ~7 to 1.
Clover: remove .strip().upper() which crashed with AttributeError
because the parser returns a list, not a string. Every validation
call failed, dropping row counts from 106 to 1.
Both overrides now use the same "YES" in final_answer logic as SaaS.
E2E verified: BCBS 5 rows (within LLM variance of baseline 7),
Clover 106 rows (exact match to baseline).
Delete BCBS/Clover/CHC client file_processing.py files (1090 lines of
96% stale drift with 3 crash points). Recover the 4% real business
logic (BCBS OFFSET_TERM, Clover 12 full_context fields) into the
shared pipeline via config-driven field loading.
Make runner.py the canonical pipeline entry point by merging all
production features from main.py (timing, duplicate detection, aarete
derived fields, TIN statistics, column reordering). Reduce main.py to
a thin wrapper that delegates to runner.main().
Fix the root cause of all client overrides being dead in production:
set_active_client is now called with the correct client name instead
of being hardcoded to "saas". Both resolver shims (file_processing
and prompt_calls) now route correctly.
Includes temporary [DEBUG ROUTING] print statements at 6 routing
decision points for E2E verification. Cherry-pick this commit to
restore debug instrumentation for future testing.
E2E verified: BCBS Promise and Clover both pass with correct routing.
Bugfix/generic issue fixes
* Patch for llm responses
* Merged dev into bugfix/generic-issue-fixes
* Update Exhibit-Level instruction for Claim Type and Bill Type
* Update Service-Level instruction for Claim Type and Bill Type
* Update Bill Type code to prompt for DESC field
* Black format
* Remove test
Approved-by: Siddhant Medar
Bugfix/unify prompt exhibit header
* Remove dead prompt_exhibit_header overrides and legacy get_exhibit_pages
Investigation (AST call tree analysis) proved both were dead code:
1. get_exhibit_pages() in preprocessing_funcs.py had ZERO callers in the
entire codebase. It was superseded by get_exhibit_pages_new() which is
the only live path (called by preprocess.one_to_n_exhibit_chunking).
2. Client prompt_exhibit_header overrides (2-arg) in bcbs_promise and
clover prompt_calls.py were dead for two independent reasons:
a) Their only consumer (get_exhibit_pages) has zero callers
b) The live consumer (get_exhibit_pages_new) calls with 3 args —
if the resolver returned the client 2-arg function, Python would
raise TypeError. The SaaS 3-arg version was always running.
Verification:
- Programmatic AST call tree (local_scripts/call_tree_analysis.py)
confirms zero callers and signature mismatch
- Resolver test confirms all clients resolve to SaaS 3-arg implementation
- Full unit suite: 1077 passed, 0 failures
- E2E: BCBS P…
* Merged dev into bugfix/unify_prompt_exhibit_header
* Add dev dependencies for local code index
Add tree-sitter, lancedb, and tiktoken as dev dependencies for a local
codebase indexing system. The index provides hybrid search (symbol table,
call graph, semantic embeddings, BM25) for token-efficient code exploration.
The index lives in a gitignored .index/ folder — each developer builds
their own locally.
Approved-by: Katon Minhas
Feature/pre doczy status column
* pre-doczy changes
* format
* Merged dev into feature/pre-doczy-status-column
* function moved to utils
Approved-by: Katon Minhas
Bugfix/implicit cpt update
* DAIP2-2310: enrich service term before implicit code extraction
- Add SERVICE_ENRICHMENT prompt + enrich_service_for_codes() that runs
after explicit extraction and before the implicit pipeline, stripping
provider/facility/contractual noise while preserving billing codes.
- Tighten CODE_IMPLICIT_ARBITRATION instructions to require strict
synonymy and prefer no_match over best-effort guesses.
- Remove single-candidate auto-accept bypass so all implicit candidates
go through LLM arbitration.
- Update unit tests for new arbitration path and enrichment mock.
* DAIP2-2310: refine service enrichment and track implicit mapping source
- Service enrichment prompt: bare 'Inpatient'/'Outpatient' place-of-service
rows now collapse to 'Covered Services' to prevent spurious Level 1 matches.
When paired with a real procedure (e.g. 'Outpatient Surgery'), the place
modifier is preserved.
- code_implicit_rag: record per-match mapping origin (CPT_LEVEL1,
HCPCS_LEVEL1, CPT_LEVEL2, HCPCS_LEVEL2) on the code answer dict for
debugging which mapping produced each implicit match.
- constants: add TODO next to HCPCS_LEVEL1_MAPPING load flagging the
upcoming DME code range change.
* Merged dev into DAIP2-2310-implicit-cpt-update
* DAIP2-2310: fix 3-digit revenue code extraction and enrichment exclusion rule
Two related fixes surfaced during short-circuit analysis of the NICU rows:
1. 3-digit revenue codes were being dropped by validate_explicit_codes,
causing the explicit path to return empty and the row to fall through
to implicit RAG (which would then semantic-match against CPT/HCPCS
and produce 'Specific - No Match'). Root causes:
- _revenue_code_format_valid required exactly 4 digits, rejecting the
common contract form like 'Revenue Code 173'.
- REV_MAPPING keys are loaded as ints by pandas (leading zero stripped
from all-numeric CSV columns), so even after accepting 3-digit form
the mapping lookup missed.
Fixes in src/codes/code_funcs.py:
- _revenue_code_format_valid now accepts 3 or 4 digits.
- New _normalize_revenue_code helper returns canonical 4-digit form.
- validate_explicit_codes stores the normalized form in REVENUE_CD.
- _has_unmatched_codes normalizes before membership chec…
* Merged dev into bugfix/implicit-cpt-update
* DAIP2-2310: accept revenue code wildcards and ranges; add unit tests
Extends the earlier 3-digit revenue code fix to cover two more forms the
CODE_EXPLICIT prompt instructs the LLM to return:
1. X-suffix wildcards (e.g. '17X', '173X') - standard UB-04 shorthand
meaning any digit across a revenue family. The prompt explicitly says
'may end in X' but the validator was rejecting them, causing the same
silent drop + implicit-RAG fall-through as the 3-digit bug.
2. Ranges (e.g. '173-174', '0170-0179') - the CODE_EXPLICIT prompt
instructs ranges in 'LOW-HIGH' form. Procedure codes support ranges
but revenue codes did not. The Gulf Coast 'Rev Codes 173-174' rows
were landing on Specific - No Match because of this.
New helpers in src/codes/code_funcs.py:
- _revenue_wildcard_format_valid / _revenue_range_format_valid /
_revenue_code_any_format_valid: targeted format checks.
- _normalize_revenue_any: canonical form for any valid kind
(single code -> '0173', wildcard -> '017X', range -> '0173-0…
* DAIP2-2310: prevent enrichment from hallucinating or dropping descriptors
Two related prompt changes to SERVICE_ENRICHMENT_INSTRUCTION based on
review feedback:
1. Add a 'no information not in input' rule. The enrichment is a
faithful rewrite that strips noise — not a knowledge-augmented
expansion. The LLM must NOT look up what a code means and inject
the description, infer category names from codes, or translate
medical shorthand into clinical terms not present in the input.
2. Add a 'preserve descriptive labels' rule. Things like 'Level 1',
'Level 2-5', 'Tier A', 'Class III', 'low complexity' etc. are
meaningful service descriptors written by the contract author and
must remain in the output even when codes are also present. The
prior ER few-shot example showed the LLM stripping 'Level 1'..
'Level 5' labels — that's information loss.
Updated few-shot examples to match the new rules:
- Emergency Room example output now preserves 'Level 1 - 99281,
Level 2 - 99282, ...' verbati…
Approved-by: Katon Minhas
DAIP2-2346 update client prompt calls
* DAIP2-2346: retire accidental client prompt_calls drift
Delete 14 client prompt_calls overrides in bcbs_promise and clover that were
stale forks of old SaaS implementations. Shared callers all expect SaaS return
shapes; client shims were either redundant with parser-side normalization or
outright broken dead code (prompt_dynamic_assignment unpack crash,
prompt_lesser_of_check double-parse). Resolver shim at
src/pipelines/shared/prompts/prompt_calls.py now falls through to SaaS.
Preserved with annotations:
- validate_reimbursements_for_llm: real business rule (strict YES equality)
- prompt_exhibit_header: required by 2-arg call site at preprocessing_funcs.py:192
* DAIP2-2346: also retire prompt_exhibit_level and prompt_dynamic overrides
Follow-up to the Bucket A cleanup. Both client overrides only drifted by
omitting the optional field_names kwarg on prompt_templates.EXHIBIT_LEVEL.
SaaS passes field_names and gets the format-aware parser
(_create_json_dict_parser); falling through to SaaS is a strict improvement
in normalization. Client files now contain only the two truly
client-specific overrides.
* Changes to gitignore
* Merged dev into DAIP2-2346-update-client-prompt-calls
Approved-by: Katon Minhas
Bugfix/DAIP2-2287 payer state fix
* updated payer name prompt
* Merged dev into bugfix/DAIP2-2287-payer-state-fix
* Merged dev into bugfix/DAIP2-2287-payer-state-fix
* added client state field
Approved-by: Katon Minhas
Feature/dynamic codes final
* Move Claim Type to dynamic code group
* Update CLaim Type and Bill TYpe prompts
* Prompt optimization
* Prompt tuning
* streamline
* Format
* Merged dev into feature/dynamic-codes
* test claim and bill type
* Merged dev into DAIP2-2186-service-claim-type-testing
* Merged dev into feature/dynamic-codes
* prompt updated for claim type
* Merge branch 'feature/dynamic-codes' into DAIP2-2186-service-claim-type-testing
* Merged dev into DAIP2-2186-service-claim-type-testing
* Remove print
* Switch to map
* Merged dev into feature/dynamic-codes-final
* prompt inconsistency fixed
Approved-by: Praneel Panchigar
Approved-by: Siddhant Medar
Feature/DAIP2-1615 refactor maximize code reusability
* added reusability
* client resolver added
* Merged DEV into feature/DAIP2-1615-refactor-maximize-code-reusability
* Refactor client prompt reuse and harden resolver test coverage.
Remove duplicated client prompt functions so unresolved names fall back to SaaS, add resolver contract tests with runtime-context isolation, and clean resolver-path test imports to prevent cross-test client leakage.
* Merge origin/dev into feature/DAIP2-1615-refactor-maximize-code-reusability.
Bring in latest vendor pipeline framework changes from dev and resolve runner routing conflict by preserving vendor processor dispatch while keeping shared function-level client fallback behavior.
* Merged dev into feature/DAIP2-1615-refactor-maximize-code-reusability
* Remove tracked drift review report from documentation.
Keep drift/alignment notes as local-only artifacts under ignored docs/ and stop tracking documentation/function_level_override_drift_review.md in the branch.
* Merged dev into feature/DAIP2-1615-refactor-maximize-code-reusability
Approved-by: Katon Minhas
Feature/DAIP2-2301 flag and remove identical contracts
* initiated flaging duplicate contracts
* Merged dev into feature/DAIP2-2301-flag-and-remove-identical-contracts
* Merged dev into feature/DAIP2-2301-flag-and-remove-identical-contracts
* updated duplicate detection
* pipeline fixes
* remove print statements
* Merged dev into feature/DAIP2-2301-flag-and-remove-identical-contracts
Approved-by: Katon Minhas
Feature/DAIP2-1575 set up dev prod uat branches
* Add branch promotion gates and release pipelines
* Add resolve_source_branch.sh script and update .gitignore; fix pipeline step name casing 'dev'
* Add AI code review step and refactor branch resolution in pipelines
* Merged dev into feature/DAIP2-1575-set-up-dev-prod-uat-branches-
* Merged dev into feature/DAIP2-1575-set-up-dev-prod-uat-branches-
* Merged dev into feature/DAIP2-1575-set-up-dev-prod-uat-branches-
Approved-by: Katon Minhas
removed 2 signature related fields
* removed 2 signature related fields
* Merged dev into DAIP2-1595-optimize-signature-fields
* Merged dev into DAIP2-1595-optimize-signature-fields
Approved-by: Katon Minhas
Bugfix/DAIP2-2164 reimbursement primary issue fixes
* sample calculations reimbursements removed
* split service term added
* Merged DEV into bugfix/DAIP2-2164-reimbursement-primary-issue-fixes
* updated prompt and unit test added
* added prompt for codes listed below
* Merged DEV into bugfix/DAIP2-2164-reimbursement-primary-issue-fixes
* Output format at end
* Genericize prompt
* Merged DEV into bugfix/DAIP2-2164-reimbursement-primary-issue-fixes
* context missing issue fixed
* updated service term split prompt for program varints
* Merged DEV into bugfix/DAIP2-2164-reimbursement-primary-issue-fixes
* Merged DEV into bugfix/DAIP2-2164-reimbursement-primary-issue-fixes
* Merged DEV into bugfix/DAIP2-2164-reimbursement-primary-issue-fixes
* Merged DEV into bugfix/DAIP2-2164-reimbursement-primary-issue-fixes
* updated split service term prompt
* optimized and minimized split service term prompt
* updated service term split prompt
* Merged DEV into bugfix/DAIP2-2164-reimbursement-primary-issue-fixes
* Merged dev into bugfix/DAIP2-2164-reimbursement-primary-issue-fixes
* Updated Prompt
* grouped instructions
* removed print statements
Approved-by: Katon Minhas
Feature/vendor generic
* initial commit for generic vendor logic
* updated generic field list
* combined special fields
* joint fields template
* payment term and amount extraction
* model update
* model update
* added constants
* care-source fields added
* care-source constants
* fields update for bcbs-az
* prompt update for term type
* prompt update for payment type
* post process additions for generic
* Merge remote-tracking branch 'origin/DEV' into feature/vendor-generic
* Merged DEV into feature/vendor-generic
* Merged DEV into feature/vendor-generic
* Black format
* Merge remote-tracking branch 'origin/DEV' into feature/vendor-generic
* Merge branch 'DEV' into feature/vendor-generic
* PR comments
* deleted redundant file
* Merged dev into feature/vendor-generic
Approved-by: Katon Minhas
Bugfix/DAIP2-2230 mcs issues effective date new
* updated scope and fixes
* optimized prompt
* shortened prompt
* remove redundant prompt line
* Merged DEV into bugfix/DAIP2-2230-mcs-issues-effective-date-new
Approved-by: Katon Minhas
Feature/DAIP2-2112 implement pre commit hooks
* Add pre-commit hooks for local code quality checks
Adds .pre-commit-config.yaml with file hygiene hooks (trailing-whitespace,
end-of-file-fixer, check-yaml/json, check-merge-conflict, check-added-large-files)
plus local hooks for black and mypy on pre-commit and pytest on pre-push.
Includes setup documentation and adds pre-commit to dev dependencies.
Auto-fixed trailing whitespace and missing newlines caught by the new hooks.
* Add sample pre-commit hook for testing purposes
* Add test file for pre-commit hook validation
* Remove test file for pre-commit hook validation
* Merged DEV into feature/DAIP2-2112-implement-pre-commit-hooks---
* Merged DEV into feature/DAIP2-2112-implement-pre-commit-hooks---
* Move to documentation
Approved-by: Katon Minhas
DAIP2-2178 dynamic primary issue fixes LOBs incorrectly added
* LOB addition from product and program fixed
* Revert "LOB addition from product and program fixed"
This reverts commit fb0cb48a2e74eff6ab66340d07bcf080eb92708c.
* PROGRAM prompt updated
Approved-by: Katon Minhas
DEV
* Apply formatting fix
* Merged in DAIP2-1898-one-to-one-issues-filename-tin-auto-renewal-and-contract-title (pull request #888)
DAIP2-1898 one to one issues filename tin auto renewal and contract title
* contract title fixes
* Add instruction to strip hyphens from TIN and NPI
* prompt update
* Merge branch 'bugfix/auto-renewal-term' into bugfix/DAIP2-1699-issue-tracker-fixes-one-to-one-dates-prov-info
* removed print statements
* pipeline error fixed
* signed_ind logic changed
* signatory_ind logic updated
* prompt update
* prompt update
* pipeline error fixed
* Merge branch 'DEV' into bugfix/DAIP2-1699-issue-tracker-fixes-one-to-one-dates-prov-info
* Merge branch 'DEV' into bugfix/DAIP2-1699-issue-tracker-fixes-one-to-one-dates-prov-info
* requested changes applied
* test case added
* prompt update
* contract title prompt update
* prompt update
* Merge branch 'DEV' into DAIP2-1898-one-to-one-issues-filename-tin-auto-renewal-and-contract-title
Approved-by: Siddhant Medar
* Merged in feature/DAIP2-1803-add-aarete-derived-provider-name (pull request #884)
Feature/DAIP2-1803 add aarete derived provider name
* Merged DEV into feature/DAIP2-1562-add-aarete_derived_payer_name
* state logic added in clustering
* removed print statements
* pull request updates
* black format fix
* Merged DEV into feature/DAIP2-1562-add-aarete_derived_payer_name
* added scalability feature and optimization
* black format
* Pull Request Changes
* generalised funcs from aarete derived payer name
* updated remove states and updated docstring for build similarity matrix
* derived payer name functionality added
* config and main file changes
* sync dev into ADD AARETE DERIVED PROVIDER GROUP NAME FULL
* updated derived_provider name
* pipeline fixes
* Merged DEV into feature/DAIP2-1803-add-aarete-derived-provider-name
* config changes
* field name updated to AARETE_DERIVED_PROVIDER_NAME
* Merged DEV into feature/DAIP2-1803-add-aarete-derived-provider-name
* black format fix
* Fix docstr…
* Merged in improve-logging (pull request #886)
Improve logging
* Add timing blocks for comprehensive pipeline logging
- Add timing_utils.timed_block() to key extraction functions in one_to_n_funcs.py:
- exhibit_level sub-functions (prompt_exhibit_level, dynamic_primary, etc.)
- reimbursement_level extraction and cleaning
- breakout functions (methodology_breakout, special_case_breakout)
- carveout_and_special_case parallel processing
- one_to_n_cleaning sub-steps (crosswalk, lob_relationship, split_reimb_dates)
- Add timing to dynamic_funcs.py:
- dynamic_assignment parallel processing
- Add timing to postprocess.py:
- standard_postprocess, contract_config_postprocess, dashboard_postprocess
- Add timing to preprocess.py:
- exhibit_chunking sub-steps (get_exhibit_pages, link_exhibit_pages, chunk_by_exhibit)
- Add info-level logging statements for completed operations with row/item counts
* Standardize logging levels and remove deprecated code
DAIP2-1798: Ensure logging statements are at …
* Merged in bugfix/DAIP2-1913-lesser-of-indicator-issues (pull request #889)
Bugfix/DAIP2-1913 lesser of indicator issues
* contract title fixes
* Add instruction to strip hyphens from TIN and NPI
* prompt update
* Merge branch 'bugfix/auto-renewal-term' into bugfix/DAIP2-1699-issue-tracker-fixes-one-to-one-dates-prov-info
* removed print statements
* pipeline error fixed
* signed_ind logic changed
* signatory_ind logic updated
* prompt update
* prompt update
* pipeline error fixed
* Merge branch 'DEV' into bugfix/DAIP2-1699-issue-tracker-fixes-one-to-one-dates-prov-info
* Merge branch 'DEV' into bugfix/DAIP2-1699-issue-tracker-fixes-one-to-one-dates-prov-info
* requested changes applied
* test case added
* prompt update
* contract title prompt update
* prompt update
* Merge branch 'DEV' into DAIP2-1898-one-to-one-issues-filename-tin-auto-renewal-and-contract-title
* handle nested cases of lesser_of and greater_of
* Merge branch 'DEV' into bugfix/DAIP2-1913-lesser-of-indicator-issues
* pi…
* Merged in bugfix/reimb_primary (pull request #890)
Update reimbursement prompts for consistent PMPM handling, time-unit dedup, and validation improvements
* Update reimbursement prompts for consistent PMPM handling, time-unit dedup, and validation improvements
- REIMBURSEMENT_PRIMARY: append unit of measure (PMPM, per visit, etc.) from table headers into REIMB_TERM
- METHODOLOGY_BREAKOUT: collapse mathematically equivalent time-unit rates into single entry
- VALIDATE_REIMBURSEMENTS: add umbrella terms as valid service terms, accept CPT/HCPCS percentage as concrete rate, narrow COB disqualifier to primary subject only
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Approved-by: Siddhant Medar
* Merged in feature/update-postprocessing-columns (pull request #893)
move column order to final step
* move column order to final step
* Black
Approved-by: Siddhant Medar
* Merged in bugfix/update-postprocessing-paths (pull request #896)
Update postprocessing paths
* Update postprocessing paths
Approved-by: Siddhant Medar
* Merged in feature/TIN_stats_update (pull request #898)
Feature/TIN stats update
* Tin_stats_report
* black formatting
* minor changes
* black format applied
* Addressed the katons comments
* Merged DEV into feature/TIN_stats_update
* black reformat
Approved-by: Katon Minhas
* Merged main into DEV
* Merged in feature/exhibit-smart-chunking (pull request #883)
Feature/exhibit smart chunking
* exhibit processing per page
* header dict deduplication
* dedup prompt refinment
* refinment for header extraction proecess
* Merge remote-tracking branch 'origin/DEV' into feature/exhibit-smart-chunking
* merge updates
* minor fix
* prompt fix for reimb type
* COB defenition for clear understanding
* black formatting
* remove quit statement
* pipiline test
* pipeline test
* black formatting
* Merge remote-tracking branch 'origin/DEV' into feature/exhibit-smart-chunking
* black formating
* Merge remote-tracking branch 'origin/DEV' into feature/exhibit-smart-chunking
* typo
* PR comment fixes
* exhibit funcs refactored
* black formatting
* Refactor exhibit chunking config into dedicated class
Created ExhibitChunkingConfig class to centralize exhibit smart chunking
configuration parameters (DEFAULT_SUBCHUNK_SIZE, MIN_PARENT_CHUNK_SIZE,
CHUNK_RELEVANCE_THRESHOLD). This improves code organization …
* Merged in bugfix/DAIP2-1870-dynamic-issues (pull request #892)
Bugfix/DAIP2-1870 dynamic issues
* updated list format appending
* fill bill type from claim type cd
* Fixed remaining format fixes
* Merged DEV into bugfix/DAIP2-1870-dynamic-issues
* changed crosswalk mapping from PRODUCT to AARETE_DERIVED_PRODUCT
* Merged DEV into bugfix/DAIP2-1870-dynamic-issues
* updating LOB field values as well from PRODUCT AND PROGRAM
* updated apply crosswalk for reverse mapping
* Merged DEV into bugfix/DAIP2-1870-dynamic-issues
* added QHP and BHP
* removed lob reverse mapping
* Merged DEV into bugfix/DAIP2-1870-dynamic-issues
* Remove Bill Type/Claim Type fill
* Merged DEV into bugfix/DAIP2-1870-dynamic-issues
* Black format
* Merged DEV into bugfix/DAIP2-1870-dynamic-issues
Approved-by: Katon Minhas
* Merged in bugfix/prov_info_json_fixes (pull request #899)
Bugfix/prov info json fixes
* fix: robust PROV_INFO_JSON sanitization and TIN backfill logic
json_utils:
- Add sanitize_prov_info_json with layered parsing (JSON, literal_eval,
empty-value-after-colon fix, best-effort dict extraction).
- Add _normalize_prov_entries and _prov_value_to_str for uniform
str-valued output; flatten list values, strip TIN hyphens.
- format_prov_info_json now delegates to sanitize_prov_info_json.
postprocessing_funcs:
- Add fill_prov_info_tin_from_filename_tin for TIN backfill.
- Add validate_and_reformat_date (pipe-wrapped, datetime strings).
- Add format_as_json_list (pipe-delimited, comma-separated, quote
stripping).
postprocess:
- Integrate new postprocessing helpers into pipeline flow.
postprocess_existing_output:
- Support CSV and Excel input, configurable paths, fillna for CSV.
tests:
- Add test_json_parsers.py for PROV_INFO_JSON parsing coverage.
- Add test_postprocess.py for date/list formatting and defau…
* Merged in bugfix/reimb_primary_issues (pull request #900)
Bugfix/reimb primary issues
* Update lesser of distribution prompt to preserve reimb term langauge
* Added prompt change to extract correct service when there are multiple sub-rates; Updated validate reimb term to prevent filtering out erm with valid rate
* Make reimb primary prompt change
* Revert "Make reimb primary prompt change"
This reverts commit 55f7d73b9cb523097c96ea50b6614c1db53b46d2.
Approved-by: Katon Minhas
* Merged in dtc_report (pull request #902)
Dtc report
* dtc_report_added
* lint format fixed
* Changed regex location call
Approved-by: Katon Minhas
* Merged in feature/historical-cost-analysis (pull request #901)
Feature/historical cost analysis
* Initial commit
* Finalize cost_analysis
* Black format
* Merged DEV into feature/historical-cost-analysis
* Refactor
* Black
* Merged DEV into feature/historical-cost-analysis
Approved-by: Siddhant Medar
* Merged in feature/ai-pr-review-agent (pull request #906)
Add AI code review agent to pull request pipeline
* Add AI code review agent to pull request pipeline
Adds a pull-requests pipeline that runs CI checks (lint, type check,
unit tests) followed by an AI code review step. The AI review uses
OIDC to authenticate with AWS Bedrock - no hardcoded AWS keys. It
clones the code-review-agent repo and posts a review comment on the PR.
* Fix AI code review step: install git in python:3.12-slim image
* Trigger pipeline
* Trigger pipeline
* Trigger pipeline
* Trigger pipeline
* Trigger pipeline
* Trigger pipeline
* Trigger pipeline
* Remove debug token decode line
* Update guide: all checklist items complete, agent working
* Remove guide file from repo
* Trigger pipeline
Approved-by: Katon Minhas
* Merged in DAIP2-1980-aarete-derived-effective-dt-issues (pull request #904)
DAIP2-1980 aarete derived effective dt issues
* fixed earlier of a and b
* prompt update
* prompt shortened
* test changes reverted
* Merge branch 'DEV' into DAIP2-1980-aarete-derived-effective-dt-issues
* test changes reverted
* Merged DEV into DAIP2-1980-aarete-derived-effective-dt-issues
* Fix effective date prompt
* Replace em-dashes with readable version
Approved-by: Katon Minhas
* Merged in feature/DAIP2-2023-eliminate-full-context-processing (pull request #905)
Feature/DAIP2-2023 eliminate full context processing
* testing full context fields
* remove full context processing
* merge Dev with DAIP2-2-23
* full context removal in client codes
* AARETE_DERIVED_PROVIDER_NAME field changes
* Merged DEV into feature/DAIP2-2023-eliminate-full-context-processing
* optimized provider name
* black format fix
* contract title fixes
* PAYER NAME AUTO RENEWAL IND fixes
* Merged DEV into feature/DAIP2-2023-eliminate-full-context-processing
* Merge branch 'DEV' into feature/DAIP2-2023-eliminate-full-context-processing
* Remove prints
Approved-by: Katon Minhas
* Merged in feature/code_optimization (pull request #903)
Feature/code optimization
* Ran Black
* made a small change in code_last_check, fixed so it returns string and not single char
* Made changes to make sure that default_ind postprocess only happens to the cc output and not dashboard
* Merge bugfix/default_ind_postprocess into feature/code_optimization
- Parent-child: child_rank column init and cols_to_keep filter
- Postprocess: default_ind only on cc output
- code_last_check: parser str/list return handling + exception logging
- Prompt: FIELD ASSIGNMENT for code extraction (no CRITICAL prefix)
* More prompt changes and debugging print statements
* Added system to group same service term + bill type cd + claim type cd and then parallelize the code extraction (will improve consistency)
* Revert one-time PROV_INFO_JSON ad hoc logic; retain json_utils and output format
- Remove temporary postprocess step that filled empty TIN in PROV_INFO_JSON from
FILENAME_TIN (fill_prov_info_tin_from_filename_ti…
* Merged in DAIP2-2121-prov-info-fields-issue-fixes (pull request #908)
empty prov_info_json fixed
* empty prov_info_json fixed
* Merged DEV into DAIP2-2121-prov-info-fields-issue-fixes
* format of PROV_OTHER_TIN fixed
* Revert "format of PROV_OTHER_TIN fixed"
This reverts commit 580847de03265520b1b4fad473bbf38ef1573753.
* formatting and deduplication
* pipeline error fixed
* test cases added
* pipeline error fixed
Approved-by: Katon Minhas
* Merged in feature/context-caching (pull request #909)
Feature/context caching
* Initial commit - context caching for DYNAMIC_PRIMARY
* implement context caching for all relevant prompts
* Remove option to not context cache
* IndentationError fixed
* Merge branch 'DEV' into feature/context-caching
* Merge and format
* Move documentation
* Merged DEV into feature/context-caching
* Update unit tests
* Merged DEV into feature/context-caching
* Update signatures
* Fix test coverage gap
Approved-by: Praneel Panchigar
Approved-by: Karan Desai
* Merged in bugfix/DAIP2-2138-reimbursement-primary-testing (pull request #910)
Bugfix/DAIP2-2138 reimbursement primary testing
* updated table instructions for reimb term extraction
* updated split tables to include header rows
* issue fix
* Merged DEV into bugfix/DAIP2-2138-reimbursement-primary-testing
* black format fix
Approved-by: Katon Minhas
* Merged in feature/doczy_reports (pull request #911)
Feature/doczy reports
* Added post_doczy_reports
* black format fix
* Merged DEV into feature/doczy_reports
* fixed code review
Approved-by: Katon Minhas
* Merged in feature/median-cost-analysis (pull request #912)
Add median
* Add median
Approved-by: Siddhant Medar
Approved-by: Siddhant Medar
Bugfix/DAIP2-2138 reimbursement primary testing
* updated table instructions for reimb term extraction
* updated split tables to include header rows
* issue fix
* Merged DEV into bugfix/DAIP2-2138-reimbursement-primary-testing
* black format fix
Approved-by: Katon Minhas
Feature/context caching
* Initial commit - context caching for DYNAMIC_PRIMARY
* implement context caching for all relevant prompts
* Remove option to not context cache
* IndentationError fixed
* Merge branch 'DEV' into feature/context-caching
* Merge and format
* Move documentation
* Merged DEV into feature/context-caching
* Update unit tests
* Merged DEV into feature/context-caching
* Update signatures
* Fix test coverage gap
Approved-by: Praneel Panchigar
Approved-by: Karan Desai
Feature/code optimization
* Ran Black
* made a small change in code_last_check, fixed so it returns string and not single char
* Made changes to make sure that default_ind postprocess only happens to the cc output and not dashboard
* Merge bugfix/default_ind_postprocess into feature/code_optimization
- Parent-child: child_rank column init and cols_to_keep filter
- Postprocess: default_ind only on cc output
- code_last_check: parser str/list return handling + exception logging
- Prompt: FIELD ASSIGNMENT for code extraction (no CRITICAL prefix)
* More prompt changes and debugging print statements
* Added system to group same service term + bill type cd + claim type cd and then parallelize the code extraction (will improve consistency)
* Revert one-time PROV_INFO_JSON ad hoc logic; retain json_utils and output format
- Remove temporary postprocess step that filled empty TIN in PROV_INFO_JSON from
FILENAME_TIN (fill_prov_info_tin_from_filename_tin call in postprocess.py).
This behavior is intended to move upstream per JIRA (tin_npi_funcs / extraction).
- Revert postprocess_existing_output.py to generic config: empty INPUT_DIR,
INPUT_FILENAME, OUTPUT_CSV. Remove one-time hyphen-strip for TIN/NPI in
PROV_INFO_JSON and remove _strip_hyphens_from_prov_info_json_cell.
- Keep json_utils PROV_INFO_JSON helpers (parse_prov_info_json_cell,
serialize_prov_info_json, format_prov_info_json) and (str,str) serialization
in contract_config_postprocess. Keep fill_prov_info_tin_from_filename_tin and
related helpers in postprocessing_funcs for potential upstream reuse.
- Tests in test_json_parsers.py updated for (str,str) PROV_INFO_JSON output.
* Add arbitration-based implicit code flow and prefer Level 2 RAG (Fix 5d, 4c+7)
Implicit flow no longer uses first-match-wins. We build a case from all
applicable strategies and run a single arbitration step to choose the
best candidate.
- build_implicit_candidates: Run Category (when explicit had Category),
Special, and RAG; return list of {source, code_answer_dict}. RAG
result uses preferred level when both return (Fix 5d).
- code_implicit_rag (Fix 5d): Run both Level 1 and Level 2, collect
results; when both return non-empty, prefer Level 2 over Level 1.
- code_implicit_arbitration: 0 candidates -> None; 1 candidate -> return
it with CODE_METHODOLOGY = 'Implicit - Arbitration (Source)' (no LLM);
2+ candidates -> one CODE_IMPLICIT_ARBITRATION LLM call (chosen_index or
no_match). If no candidates or no_match, code_last_check runs as before.
- prompt_templates: Add CODE_IMPLICIT_ARBITRATION and
CODE_IMPLICIT_ARBITRATION_INSTRUCTION (cached). Use
_create_json_dict_parser(field_names=['chose…
* fix: test_code_breakout avoid LLM call and flaky order on CI
- Add BILL_TYPE_CD_DESC to test data so pre-pass skips fill_bill_type (no Bedrock on CI)
- Use side_effect function keyed by SERVICE_TERM for deterministic result under ThreadPoolExecutor
* ran black
* feat: hybrid code validation with unmatched tagging and code quality fixes
Validation and unmatched tagging:
- Add format-only filtering for explicit codes (drop wrong-format, preserve valid)
- Add unified _has_unmatched_codes() used by both explicit and implicit paths
- Tag CODE_METHODOLOGY with "- Unmatched" when codes are format-valid but
not explicit keys in our mappings (codes and range keys count as mapped)
- Remove retry logic and "Explicit - Validation Failed" in favor of single
LLM call with Unmatched tagging
- Include all revenue level mappings in valid revenue set
Code quality and bug fixes:
- Fix fill_bill_type += string bug (was splitting "11X" into chars)
- Fix bare except clauses in code_explicit, code_category,
code_implicit_special, fill_bill_type (now except Exception as e)
- Replace eval() with json.loads() in fill_grouper_cd_desc
- Implement INVALID_SERVICE short-circuit in RAG (Fix 5a)
- Cache valid procedure/revenue code sets per Constants instance
- Store mappings in level_dicts…
* chore: remove debug print statements from code extraction pipeline
Remove all [DEBUG_CODE_EXTRACTION] print statements added during
development of the hybrid validation and implicit arbitration flow.
* refactor: robust PROV_INFO_JSON sanitization with multi-format parsing
Replace the simple format_prov_info_json with a layered
sanitize_prov_info_json that handles all known malformed variants:
single-quoted dicts, empty-value-after-colon patterns, list-typed
field values, and hyphenated TINs. Extract _normalize_prov_entries
and _prov_value_to_str helpers for uniform str-valued output.
format_prov_info_json now delegates to sanitize_prov_info_json.
* merge: sync feature/code_optimization with DEV
Resolve conflict in prompt_templates.py: keep detailed CRITICAL field
assignment instructions from feature branch; accept new
AARETE_DERIVED_PROVIDER_NAME prompt function from DEV.
* Minor change to explict prompt
* fix: resolve mypy errors and remove stale pipe-delim splits in RAG
- Narrow grouper_cd type to str|None with explicit None check to
satisfy mypy in fill_grouper_cd_desc.
- Remove dead pipe-delim split logic from CPT and HCPCS matching
blocks (no pipes in those mappings).
- Retain pipe split for revenue block only (rev_level1.csv keys are
pipe-delimited, e.g. "0810|0811|0812").
* Code extraction: format-only explicit, keep unmapped tag, RAG Level 1/2 as separate candidates
- Explicit: accept all format-valid codes (single + range); do not drop for mapping.
Validity check only appends "- Unmatched" to methodology when code not in mappings.
- Add _procedure_code_or_range_format_valid; use in validate_explicit_codes and _has_unmatched_codes.
- RAG: return list of Level 1 and Level 2 candidates (distinct source keys); remove Level 2 preference.
- build_implicit_candidates: consume RAG list and append each as separate candidate.
- CODE_EXPLICIT: clarify range extraction when service term describes a range.
- CODE_IMPLICIT_ARBITRATION: choose most appropriate candidate(s), avoid overly broad.
- Tests updated for new RAG return shape and validate_explicit_codes behavior.
* Resolved Merge Conflicts
* Fixed failing tests
* Remove unused functions
* Remove unused functions
* Revert "Remove unused functions"
This reverts commit 39954db6a421e55501282d4e6a270c88624049fe.
* remove unused
* Merged DEV into feature/code_optimization
* Merged DEV into feature/code_optimization
Approved-by: Katon Minhas
Feature/DAIP2-2023 eliminate full context processing
* testing full context fields
* remove full context processing
* merge Dev with DAIP2-2-23
* full context removal in client codes
* AARETE_DERIVED_PROVIDER_NAME field changes
* Merged DEV into feature/DAIP2-2023-eliminate-full-context-processing
* optimized provider name
* black format fix
* contract title fixes
* PAYER NAME AUTO RENEWAL IND fixes
* Merged DEV into feature/DAIP2-2023-eliminate-full-context-processing
* Merge branch 'DEV' into feature/DAIP2-2023-eliminate-full-context-processing
* Remove prints
Approved-by: Katon Minhas
DAIP2-1980 aarete derived effective dt issues
* fixed earlier of a and b
* prompt update
* prompt shortened
* test changes reverted
* Merge branch 'DEV' into DAIP2-1980-aarete-derived-effective-dt-issues
* test changes reverted
* Merged DEV into DAIP2-1980-aarete-derived-effective-dt-issues
* Fix effective date prompt
* Replace em-dashes with readable version
Approved-by: Katon Minhas
Add AI code review agent to pull request pipeline
* Add AI code review agent to pull request pipeline
Adds a pull-requests pipeline that runs CI checks (lint, type check,
unit tests) followed by an AI code review step. The AI review uses
OIDC to authenticate with AWS Bedrock - no hardcoded AWS keys. It
clones the code-review-agent repo and posts a review comment on the PR.
* Fix AI code review step: install git in python:3.12-slim image
* Trigger pipeline
* Trigger pipeline
* Trigger pipeline
* Trigger pipeline
* Trigger pipeline
* Trigger pipeline
* Trigger pipeline
* Remove debug token decode line
* Update guide: all checklist items complete, agent working
* Remove guide file from repo
* Trigger pipeline
Approved-by: Katon Minhas
Feature/historical cost analysis
* Initial commit
* Finalize cost_analysis
* Black format
* Merged DEV into feature/historical-cost-analysis
* Refactor
* Black
* Merged DEV into feature/historical-cost-analysis
Approved-by: Siddhant Medar
Bugfix/reimb primary issues
* Update lesser of distribution prompt to preserve reimb term langauge
* Added prompt change to extract correct service when there are multiple sub-rates; Updated validate reimb term to prevent filtering out erm with valid rate
* Make reimb primary prompt change
* Revert "Make reimb primary prompt change"
This reverts commit 55f7d73b9cb523097c96ea50b6614c1db53b46d2.
Approved-by: Katon Minhas
Bugfix/prov info json fixes
* fix: robust PROV_INFO_JSON sanitization and TIN backfill logic
json_utils:
- Add sanitize_prov_info_json with layered parsing (JSON, literal_eval,
empty-value-after-colon fix, best-effort dict extraction).
- Add _normalize_prov_entries and _prov_value_to_str for uniform
str-valued output; flatten list values, strip TIN hyphens.
- format_prov_info_json now delegates to sanitize_prov_info_json.
postprocessing_funcs:
- Add fill_prov_info_tin_from_filename_tin for TIN backfill.
- Add validate_and_reformat_date (pipe-wrapped, datetime strings).
- Add format_as_json_list (pipe-delimited, comma-separated, quote
stripping).
postprocess:
- Integrate new postprocessing helpers into pipeline flow.
postprocess_existing_output:
- Support CSV and Excel input, configurable paths, fillna for CSV.
tests:
- Add test_json_parsers.py for PROV_INFO_JSON parsing coverage.
- Add test_postprocess.py for date/list formatting and default_ind.
* Merge branch 'DAIP2-1947-tin-and-prov-info-json-issues' into bugfix/prov_info_json_fixes
* Merged DEV into bugfix/prov_info_json_fixes
* Strip out unused functionality
* Update filename_tin functionality
* Add docstring
* Update filename_tin cleaning in PROV_INFO_JSON
* test prep
* black format
* missing function added
* Merge branch 'DEV' into bugfix/prov_info_json_fixes
* Black format
* Strip unused functions
* Strip unused code
* update unit tests
* Merge branch 'DEV' into bugfix/prov_info_json_fixes
* Update test
* Simplify process
* handle list of group names
* Resolve run_provider_info_field call
* Merge branch 'DEV' into bugfix/prov_info_json_fixes
* Correct type hints
* Fix unit tests
* Fix unit test
* Remove redundant postprocessing_funcs
Approved-by: Katon Minhas
Bugfix/DAIP2-1870 dynamic issues
* updated list format appending
* fill bill type from claim type cd
* Fixed remaining format fixes
* Merged DEV into bugfix/DAIP2-1870-dynamic-issues
* changed crosswalk mapping from PRODUCT to AARETE_DERIVED_PRODUCT
* Merged DEV into bugfix/DAIP2-1870-dynamic-issues
* updating LOB field values as well from PRODUCT AND PROGRAM
* updated apply crosswalk for reverse mapping
* Merged DEV into bugfix/DAIP2-1870-dynamic-issues
* added QHP and BHP
* removed lob reverse mapping
* Merged DEV into bugfix/DAIP2-1870-dynamic-issues
* Remove Bill Type/Claim Type fill
* Merged DEV into bugfix/DAIP2-1870-dynamic-issues
* Black format
* Merged DEV into bugfix/DAIP2-1870-dynamic-issues
Approved-by: Katon Minhas
Feature/exhibit smart chunking
* exhibit processing per page
* header dict deduplication
* dedup prompt refinment
* refinment for header extraction proecess
* Merge remote-tracking branch 'origin/DEV' into feature/exhibit-smart-chunking
* merge updates
* minor fix
* prompt fix for reimb type
* COB defenition for clear understanding
* black formatting
* remove quit statement
* pipiline test
* pipeline test
* black formatting
* Merge remote-tracking branch 'origin/DEV' into feature/exhibit-smart-chunking
* black formating
* Merge remote-tracking branch 'origin/DEV' into feature/exhibit-smart-chunking
* typo
* PR comment fixes
* exhibit funcs refactored
* black formatting
* Refactor exhibit chunking config into dedicated class
Created ExhibitChunkingConfig class to centralize exhibit smart chunking
configuration parameters (DEFAULT_SUBCHUNK_SIZE, MIN_PARENT_CHUNK_SIZE,
CHUNK_RELEVANCE_THRESHOLD). This improves code organization by consolidating
related constants and makes configuration more maintainable.
Changes:
- Created ExhibitChunkingConfig class with ESC_CONFIG instance
- Moved CHUNK_RELEVANCE_THRESHOLD from config.py to ExhibitChunkingConfig
- Updated all constant references to use ESC_CONFIG prefix
- Added missing EXHIBIT_HEADER_MARKERS parameter documentation
* Fix logging levels and refactor imports for exhibit chunking
- Upgrade logging from WARNING to ERROR for embedding and semantic search failures
- Remove unused constant imports from exhibit_funcs.py
- Update test imports to use ESC_CONFIG pattern for configuration constants
- Add warning when no exhibit headers found during deduplication
- Expand mypy type checking by removing s3_utilities from exclude list
* Merged DEV into feature/exhibit-smart-chunking
* Merged DEV into feature/exhibit-smart-chunking
Approved-by: Siddhant Medar
Praneel Panchigar