Commit Graph

1824 Commits

Author SHA1 Message Date
Mayank Aamseek 2813d3ceb8 Merged in bugfix/DAIP2-2679-fix-nebraska-issues (pull request #995)
incorrect inclusion of CPT4_PROC_CD fixed

* incorrect inclusion of CPT4_PROC_CD fixed


Approved-by: Katon Minhas
2026-05-07 19:54:18 +00:00
Katon Minhas c6df22c1a3 Merged in bugfix/black-format (pull request #994)
Black format for pipeline pass

* Black format for pipeline pass
2026-05-05 21:10:10 +00:00
Praneel Panchigar bff8e103b0 Merged in bugfix/molina_ut_dynamic_primary (pull request #991)
Bugfix/molina ut dynamic primary

* Prompt changes for dynamic primary

* Route cover-sheet-only files to ERRORS.csv instead of leaking phantom rows

When every page of a contract was filtered out as a cover sheet / quick-review
form, process_file silently returned a FILE_NAME-only DataFrame. Because the
runner routes by checking for an "error" column, that file landed in
RESULTS.csv as a near-empty row and no ERRORS.csv was generated for the run.

- saas/file_processing.py: raise ValueError when text_dict is empty after
  cover-sheet filtering, so safe_process_file produces a proper error row.
- runner.py: add _is_phantom_result defense-in-depth — promote any result
  with no extracted fields beyond FILE_NAME to error_results with
  error_type=PhantomSuccess.

* Merged dev into bugfix/molina_ut_dynamic_primary

* Tighten PRODUCT prompt: restrict to valid_values, prune LOB/PROGRAM examples

* Merge branch 'bugfix/molina_ut_dynamic_primary' of bitbucket.org:aarete/doczy.ai into bugfix/molina_ut_dynamic_primary

* Add PROVIDER_NAME and DISCOUNT_TERM to FIELD_FORMAT_MAPPING

* Revert "Route cover-sheet-only files to ERRORS.csv instead of leaking phantom rows"

This reverts commit de79a1391511acaf78c92a6bee53697f419d3996.

* Drop internal-only columns from final output

Four columns were leaking into RESULTS.csv that have no entry in
FIELD_FORMAT_MAPPING and shouldn't ship to production:
  - DOCUMENT_TYPE: from the document classification stage, not a contract field
  - REIMB_PAGE / EXHIBIT_TEXT: internal extraction-trace columns
  - CODE_MAPPING_SOURCE: debug column

Drop them in standard_postprocess just before reorder_columns. Done after
attach_sid_column so AARETE_DERIVED_SID (which reads DOCUMENT_TYPE) still
gets populated correctly. Uses errors='ignore' so the drop is a no-op
when a column isn't present.

* Strict-filter columns to FIELD_FORMAT_MAPPING and tighten PRODUCT prompt

* Merged dev into bugfix/molina_ut_dynamic_primary

* Dedupe values within LOB/PROGRAM/PRODUCT/NETWORK fields

* Move hard-codes to within function


Approved-by: Katon Minhas
2026-05-01 21:27:39 +00:00
Rahul Ailaboina ef44beffeb Merged in hotfix/postprocess_csv (pull request #990)
Hotfix/postprocess csv

* date issue_fix

* black_format

* Merged dev into hotfix/postprocess_csv


Approved-by: Katon Minhas
2026-05-01 13:54:22 +00:00
Karan Desai b70759fda0 Merged in hotfix/exhibit-header-in-tables (pull request #987)
Hotfix/exhibit header in tables

* Merged in feature/FixplaceholderIssue (pull request #977)

Remove curly braces from echo statements in dev->stg

* Remove curly brances from echo statements in dev->stg

* Removed curly braces in echo statements in feature-> dev gate


Approved-by: Sujit Deokar

* Merged in feature/standardized-services (pull request #958)

Feature/standardized services

* service term standardization

* prompt update

* prompt updates for standardization

* only service standardization

* new file

* add supporting files and test scripts for standardization work

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* remove old files

* Merge remote-tracking branch 'origin/dev' into feature/standardized-services

* final fixes

* Merged dev into feature/standardized-services

* additional features

* removed  unwanted files

* remove unwanted files

* Merge branch 'dev' into feature/standardized-services

* Merge remote-tracking branch 'origin/dev' into feature/standardized-services

* reversed  vendor changes

* Merged dev into feature/standardized-services

* addressed PR comments

* Merge branch 'dev' into feature/standardized-services

* Merged dev into feature/standardized-services

* addressed 3 remaining comments inPR

* black formatting

* incorporated feedback from AI c…
* prompt added for amendment intent

* amendment intent language

* update field name

* amendment intent tag

* prompt update

* Merge remote-tracking branch 'origin/dev' into feature/active-rates

* Merge remote-tracking branch 'origin/dev' into feature/active-rates

* remove docusign lines

* add unit tests for clean_header_footer docusign/deleted_lines changes

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* black forrmatting

* Merge branch 'feature/active-rates' into temp-branch

* initial commit

* exhibit header in table

* Merge branch 'temp-branch' into hotfix/exhibit-header-in-tables

* exhibit header within table

* Merge branch 'dev' into hotfix/exhibit-header-in-tables

* deduplicaton prompt update

* foramatting

* removed amend intent code

* Merged dev into hotfix/exhibit-header-in-tables

* added keywords to constants

* changes reverted

* formatting


Approved-by: Katon Minhas
2026-04-30 17:59:09 +00:00
Rahul Ailaboina 0c3636f13b Merged in hotfix/fileextension_issue (pull request #989)
Hotfix/fileextension issue

* fixed strip_ext issue

* black format


Approved-by: Katon Minhas
2026-04-30 01:15:33 +00:00
Praneel Panchigar f1df468587 Merged in bugfix/exhibit-smart-chunking-cost-improvements (pull request #985)
Bugfix/exhibit smart chunking cost improvements

* Add opt-in instrumentation for per-call token and row-count tracing

Introduce src/utils/instrumentation.py (thread-safe CSV logger) and
src/utils/instrumentation_context.py (ContextVar scope plus
submit_with_context / map_with_context helpers for propagating context
into ThreadPoolExecutor workers).

Emit events at every Bedrock call in llm_utils.invoke_claude, including
in-memory claude_cache hits, with full input/output/cache-read/cache-write
token breakdown. Emit row-count events at each row-mutating stage in the
one-to-N pipeline (clean_reimbursement_primary,
filter_services_without_reimbursements, methodology_breakout,
split_service_terms, carveout, dynamic_code_assignment,
lesser_of_distribution, dynamic_assignment) and chunking / retrieval
events in exhibit smart chunking (chunking_done, retrieval_done) plus
exhibit lifecycle events (exhibit_start, exhibit_gate_skip,
stage_transition).

All hooks are no-ops unless DOCZY_INSTRUMENTATION=1; production defaults
unchanged. Runner and inspector scripts in …
* Extend instrumentation: segment, cache_miss_reason, retry/error events, runtime hookup, analyzer

Schema expansion and new event types:
- Add segment column to every llm_call / llm_call_inmem_hit, resolved from
  USAGE_LABEL_TO_SEGMENT (authoritative) with scope fallback. Mapping built
  from grep + smoke-run ground truth; earlier guessed entries removed.
- Add cache_miss_reason classifier: hit / first_call / ttl_expired /
  under_min_tokens / silent_miss / not_attempted. Uses a per-process
  _cache_key_seen dict guarded by its own lock.
- Emit llm_retry on each retry attempt (attempt, error_class, error_msg,
  backoff_sec) and llm_error on exhausted retries in ec2_claude_3_and_up
  (rotation and non-rotation branches) plus local_claude_3_and_up.
- Add six new CSV columns: segment, cache_miss_reason, attempt,
  error_class, error_msg, backoff_sec. Total schema now 34 columns.

Defensive kwarg hygiene:
- _ctx_minus_explicit_keys filter on all emit sites to prevent
  segment / filename kwarg collisions between …
* Merged dev into bugfix/exhibit-smart-chunking-cost-improvements

* Merged dev into bugfix/exhibit-smart-chunking-cost-improvements

* Make instrumentation tracking on by default

Flip the gate: instrumentation is now enabled unless DOCZY_INSTRUMENTATION
is explicitly set to a falsy value (0/false/no/off). Replaces the prior
opt-in behaviour where it was off unless DOCZY_INSTRUMENTATION=1 was set.

* Fix missing WRITE_TO_S3 patch in upload_instrumentation_csv test

* Merged dev into bugfix/exhibit-smart-chunking-cost-improvements


Approved-by: Katon Minhas
2026-04-28 15:58:21 +00:00
Mayank Aamseek 7a39dabcb7 Merged in bugfix/DAIP2-2524-carveout-code-optimization (pull request #980)
CARVEOUT_CD issue fixed

* CARVEOUT_CD issue fixed

* pipeline error fixed

* Merged dev into bugfix/DAIP2-2524-carveout-code-optimization

* Merged dev into bugfix/DAIP2-2524-carveout-code-optimization

* Merged dev into bugfix/DAIP2-2524-carveout-code-optimization

* trigger cap issue fixed

* trigger cap prompt updated

* Merged dev into bugfix/DAIP2-2524-carveout-code-optimization


Approved-by: Katon Minhas
2026-04-27 21:56:27 +00:00
Rahul Ailaboina f9464fc8c6 Merged in hotfix/provider_name_group_fix (pull request #988)
Hotfix/provider name group fix

* Fixes done in GROUP column

* black format fix


Approved-by: Katon Minhas
2026-04-27 21:55:21 +00:00
Rahul Ailaboina e4650064aa Merged in feature/PC_logic_cleanup_output (pull request #984)
Feature/PC logic cleanup output

* Few tweaks PC_logics

* Fixed orphan_ranking

* black formatting

* Changes on output field and ranking method

* updated few hotfixes

* black format fix


Approved-by: Katon Minhas
2026-04-27 15:04:07 +00:00
Karan Desai f351ced7ea Merged in bugfix/filter-docusign-lines (pull request #983)
remove docusign lines

* remove docusign lines

* add unit tests for clean_header_footer docusign/deleted_lines changes

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* black forrmatting


Approved-by: Katon Minhas
2026-04-27 15:01:45 +00:00
Faizan Mohiuddin f01fa147f8 Merged in feature/DAIP2-pacificsource-reimbursements-issues (pull request #982)
Feature/DAIP2 pacificsource reimbursements issues

* Tighten PREMIUM_TERM and DISCOUNT_TERM classifier prompts

CARVEOUT_CHECK was misrouting table rate rows into special-case fields,
dropping them from the reimbursement output:

- "110% of CMS allowed" (base fee-schedule rates) was being classified as
  PREMIUM_TERM because the prompt treated "above 100% of reference" as an
  implicit premium. Seen on PacificSource Medicare_Attachment_A1 and A2
  Facility contracts where Inpatient/Outpatient rows were missing (2556)
  or silently fell back to 100% fee-schedule (2557).

- Per-service discount rates like "Progressive Lenses: 15% discount",
  "Contact Lenses: 2% discount", "Frame: 20% discount" were being
  classified as DISCOUNT_TERM because the prompt only required the word
  "discount" to appear. Seen on PacificSource Commercial_Attachment_A2
  and A5 Professional contracts (2558, 2559).

Prompts now require the literal keyword AND explicitly exclude the
common false-positive patterns, keeping per-service rate rows in the
reimbursement output.

* Stop splitting multi-page exhibits on repeated page footers

Two bugs were collaborating to split a single Attachment into several
Exhibit objects, causing intra-exhibit lesser-of search to miss notes
living on a later page of the same exhibit:

1. EXHIBIT_HEADER extraction picked up page-footer lines as if they
   were new section headers. On PacificSource contracts, pages end with
   a short-form repeat like "Attachment A Oregon Health & Science
   University" next to the page number and contract date, which the
   ATTACHMENT-prefix regex and the LLM both accepted as a header.

2. EXHIBIT_HEADER_DEDUP parsing required the LLM output to be wrapped
   in a strict |pipes| JSON block. When the LLM prefixed the response
   with prose ("# Analysis ..."), parsing threw ValueError and the
   code fell back to the original, un-deduped dict — defeating dedup
   entirely.

Combined, these produced 3 Exhibit objects for Commercial_Attachment_A2
(page 1 header + two entries on page 2: repeated header + footer),
leaving …
* Merged dev into feature/DAIP2-pacificsource-reimbursements-issues


Approved-by: Katon Minhas
2026-04-23 16:19:24 +00:00
Praneel Panchigar da37f694ec Merged in bugfix/prov_info_json_fixes (pull request #981)
fix: fall back to PROVIDER_NAME in PROV_INFO_JSON when no TIN/NPI extracted

* fix: fall back to PROVIDER_NAME in PROV_INFO_JSON when no TIN/NPI extracted

When get_prov_info_json short-circuits due to no TIN/NPI regex matches,
PROV_INFO_JSON was left as [] even when PROVIDER_NAME was successfully
extracted via the one-to-one pipeline. This caused inconsistent output
across contracts with the same provider — some files produced a NAME-only
entry (via a false-positive regex hit triggering the LLM), others produced [].

Reconcile at add_group_and_other, the first point where both extraction
streams' results are available. When PROV_INFO_JSON is empty but
PROVIDER_NAME is known, synthesize a NAME-only entry with IS_GROUP:"Y" and
populate PROV_GROUP_NAME_FULL directly — skipping the provider_name_match_check
LLM call since the match is tautological by construction.

Adds 5 unit tests covering the str, list, already-populated, empty-name,
and all-empty-list cases.


Approved-by: Katon Minhas
2026-04-23 15:09:51 +00:00
Aditi Agarwal 3ccd901743 Merged in feature/fixPlaceholder2 (pull request #979)
Updated feature> dev gate to print variables of echo statements

* Updated feature> dev gate to print variables of echo statement

* Reverted placeholder changes


Approved-by: Sujit Deokar
2026-04-22 10:06:08 +00:00
Karan Desai 44cecb5d00 Merged in feature/standardized-services (pull request #958)
Feature/standardized services

* service term standardization

* prompt update

* prompt updates for standardization

* only service standardization

* new file

* add supporting files and test scripts for standardization work

Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>

* remove old files

* Merge remote-tracking branch 'origin/dev' into feature/standardized-services

* final fixes

* Merged dev into feature/standardized-services

* additional features

* removed  unwanted files

* remove unwanted files

* Merge branch 'dev' into feature/standardized-services

* Merge remote-tracking branch 'origin/dev' into feature/standardized-services

* reversed  vendor changes

* Merged dev into feature/standardized-services

* addressed PR comments

* Merge branch 'dev' into feature/standardized-services

* Merged dev into feature/standardized-services

* addressed 3 remaining comments inPR

* black formatting

* incorporated feedback from AI code reviewer

* reused existing  functionality


Approved-by: Katon Minhas
2026-04-20 21:26:47 +00:00
Aditi Agarwal a0effa87f2 Merged in feature/FixplaceholderIssue (pull request #977)
Remove curly braces from echo statements in dev->stg

* Remove curly brances from echo statements in dev->stg

* Removed curly braces in echo statements in feature-> dev gate


Approved-by: Sujit Deokar
2026-04-20 11:53:09 +00:00
Rahul Ailaboina ac857b9953 Merged in bugfix/post_doczy_report_lob (pull request #973)
Post_doczy_report_lob_fix

* Post_doczy_report_lob_fix


Approved-by: Katon Minhas
2026-04-17 20:44:34 +00:00
Rahul Ailaboina 73f345da86 Merged in feature/Pre_doczy_report (pull request #968)
pre_doczy_ru_base_prefix_changes

* pre_doczy_ru_base_prefix_changes

* Merged dev into feature/Pre_doczy_report

* Merged dev into feature/Pre_doczy_report


Approved-by: Katon Minhas
2026-04-17 16:32:49 +00:00
Sujit Deokar 38ae8c64ba Merged in feature/upDatepipelinesFile (pull request #971)
Feature/upDatepipelinesFile

* Added logic to capture failure status for invalid branch names

* Merged dev into feature/upDatepipelinesFile

* Bugfix the print logs were not able to populate the placeholders. These are now fix to use the sh file

* Merge branch 'dev' into feature/upDatepipelinesFile
2026-04-17 14:17:38 +00:00
Sujit Deokar 1527fbc740 Merged in feature/upDatepipelinesFile (pull request #963)
Added logic to capture failure status for invalid branch names

* Added logic to capture failure status for invalid branch names

* Merged dev into feature/upDatepipelinesFile
2026-04-17 12:15:30 +00:00
Praneel Panchigar a936e0ce05 Merged in bugfix/retire_stale_client_file_processing (pull request #961)
Return None for dashboard output when dashboard postprocessing is off

* Return None for dashboard output when dashboard postprocessing is off

FINAL_RESULT_DF_DASHBOARD was initialized as an empty DataFrame even
when RUN_DASHBOARD_POSTPROCESSING was False, causing downstream code
to needlessly process it (reorder_columns, etc). Now returns None
when dashboard is not requested, matching the postprocess() contract.

* Merged dev into bugfix/retire_stale_client_file_processing

* Merge dev (with revert) into feature branch

* Re-apply retire stale client file_processing changes

Revert of the revert (4f53b528) to restore the original changes
from the feature branch for proper PR review.

* Merge branch 'bugfix/retire_stale_client_file_processing' of bitbucket.org:aarete/doczy.ai into bugfix/retire_stale_client_file_processing


Approved-by: Katon Minhas
2026-04-16 21:43:00 +00:00
ppanchigar 4f53b5286e Revert premature merge of bugfix/retire_stale_client_file_processing
PR #959 was merged into dev without approval. This reverts commits
5e143c10, 63f32c41, 849aa626, and 927abcae to restore dev to its
pre-merge state. The changes will be re-submitted via a new PR
after proper review.
2026-04-16 15:49:44 -05:00
Rahul Ailaboina 9c07257f76 Merged in bugfix/post-doczy-excel-output-types (pull request #960)
Bugfix/post doczy excel output types

* Issue fix for output generation

* lint format fix

* Merged dev into bugfix/post-doczy-excel-output-types


Approved-by: Siddhant Medar
2026-04-16 16:13:40 +00:00
ppanchigar 50773656e9 Merge remote-tracking branch 'origin/dev' into bugfix/retire_stale_client_file_processing 2026-04-16 09:16:35 -05:00
Aditi Agarwal 8fe29d35c2 Merged in feature/upDatepipelinesFile (pull request #956)
Update bitbucket pipelines

Approved-by: Sujit Deokar
2026-04-16 12:09:12 +00:00
Sujit Deokar 7f3879470a Merged dev into feature/upDatepipelinesFile 2026-04-16 12:07:54 +00:00
sujit deokar 72ccdcd341 harden pipeline: improve security for OIDC token handling and add version component validation 2026-04-16 11:49:33 +00:00
sujit deokar fda9c2b243 enhance pipeline: add validation for release tag format and improve AI code review steps
remove duplicate lint, type and unit test validation from PR pipeline.
2026-04-16 10:44:00 +00:00
sujit deokar d9acfc7cc7 Based on code review feedback, made the changes. 2026-04-16 08:55:49 +00:00
ppanchigar 927abcae16 Fix reorder_columns dropping client-specific fields from output
reorder_columns() was only keeping columns listed in FIELD_FORMAT_MAPPING,
silently discarding any extra columns like BCBS OFFSET_TERM/OFFSET_INDICATOR
and Clover's 12 full_context fields. The function's docstring documented
appending extra columns but the implementation was missing that step.
2026-04-15 16:04:40 -05:00
Praneel Panchigar 9474860e37 Merged dev into bugfix/retire_stale_client_file_processing 2026-04-15 20:06:06 +00:00
Katon Minhas b9ecdf8dfc Merged in bugfix/empty-doc (pull request #957)
Safe exit for empty docs

* Safe exit for empty docs


Approved-by: Siddhant Medar
2026-04-15 19:29:14 +00:00
Siddhant Medar e94c5e0100 apply two real items from second PR review round
1. Tag format validation in release-prod (MEDIUM, defensive).
   Before parsing major/minor/patch from the latest tag, assert it
   matches ^v[0-9]+\.[0-9]+\.[0-9]+$. Empirically tested: catches
   v1, v1.2, v1.2.3.4, v1.0.0-rc1, v1.2.3-beta+build, and still
   accepts v0.0.0 (the first-release fallback).

   Without this, bash arithmetic silently mangles non-semver tags
   into wrong results via its 'treat empty/non-numeric as 0' rule.
   Worst case: tag 'v1' → cut -d. -f2/-f3 both return '1' →
   minor bump produces v1.1.2 instead of v1.0.1. No visible error.

2. Rewrite the AI review comment to match actual behavior.
   The old comment said 'advisory, not a gate' but also admitted
   git clone / apt-get are hard failures, contradicting itself.
   The new comment makes the runtime vs. infrastructure distinction
   explicit: runtime failures (agent crash, STS errors, Python
   exceptions) become warnings via || echo; infrastructure failures
   (missing token, clone failure, apt-get failure) stay hard.

   This is a doc fix, not a code change. Deliberately preserving
   the hard-fail behavior on config errors because silent
   degradation of AI review is the worst outcome — feature
   disappears from CI with no signal.

Explicitly rejected from the second review (empirically verified):
- Gate logic 'fragile' claim: tested in bash, correct for all
  our hardcoded ALLOWED values.
- ROLLBACK_TAG -z check 'passes empty': tested, -z correctly
  catches empty strings. Reviewer has the semantics wrong.
- chmod 600 + set +x: our comment already documents chmod 600
  as cosmetic/scanner-silencing; set +x wouldn't help because
  Bitbucket's line-echo is runner-level, not bash set -x.
- cd/python refactor (previous round): already rebutted in
  commit 4836e36f via empirical bash AND-OR list tests.
- Option B for git clone wrapping: silent degradation of
  config errors is worse than the current loud-fail behavior.
2026-04-15 13:48:17 -05:00
Siddhant Medar 4836e36f65 harden pipeline per PR review: locked lockfile, token permissions, image comment
1. Replace 'uv sync --frozen' with 'uv sync --locked'. --frozen silently
   uses a stale lockfile if pyproject.toml is updated without regenerating
   uv.lock, masking missed dependencies. --locked fails loudly with a clear
   error when the lockfile is out of sync. Verified empirically with uv
   0.9.14: --frozen exits 0 on mismatch, --locked exits 1.

2. Add 'chmod 600' on the OIDC token file after writing it. The container
   is already single-user root so this is defensive/cosmetic, but it
   silences security scanners and signals intent.

3. Add a comment explaining why gate steps use atlassian/default-image:4
   instead of python:3.12.7 (gates run bash only, no Python toolchain
   needed — lighter image, faster pull).

Explicitly rejected from the PR review:
- Token-masking sed mitigation (delimiter collision with / in real
  Bitbucket clone tokens; cmd | sed || exit 1 swallows git failures
  without set -o pipefail). Bitbucket's built-in Secured variable
  masking is the correct mitigation and is already documented as a
  setup requirement.
- SSH key alternative for git clone (architecturally worse — more
  secrets to manage; HTTPS+Secured is the Bitbucket-recommended pattern).
- cd/python refactor ('fragile logic bug'). Verified empirically that
  'cd X && python Y || echo Z' with set -e correctly catches both
  cd and python failures via the ||. The step exits 0 as intended
  by the 'advisory, not a gate' design. The suggested refactor would
  introduce a hard-fail regression.
2026-04-15 09:01:35 -05:00
Siddhant Medar 2a0617cd79 fix: escape colon in WARNING message to prevent YAML mapping parse
The unquoted ': ' (colon + space) in 'WARNING: AI review step failed'
was being interpreted as a YAML mapping key-value separator, causing
the entire script item to be parsed as a dict instead of a command
string. Bitbucket then rejected it with 'Missing or empty command
string' error at pull-requests > feature/* > 2 > step > script > 9.

Replaced the colon with a hyphen. Validated with yaml.safe_load that
all 10 script items in the ai-code-review step now parse as strings.
2026-04-14 16:43:59 -05:00
Siddhant Medar cbe941a7d2 fix: move indented comment to correct indentation in ai-code-review
Bitbucket's YAML parser was interpreting the deeper-indented comment
after the printf line as a phantom empty list item, causing a
'Missing or empty command string' error at script item 9.
2026-04-14 16:31:18 -05:00
Siddhant Medar f3a36eea1e fix: harden bitbucket-pipelines.yml with security and correctness improvements 2026-04-14 16:24:40 -05:00
ppanchigar 849aa626dc Fix client validate_reimbursements_for_llm to match SaaS behavior
BCBS: change strict == "YES" to "YES" in final_answer. The strict
equality was rejecting borderline LLM responses (e.g. "YES, ..."),
dropping row counts from ~7 to 1.

Clover: remove .strip().upper() which crashed with AttributeError
because the parser returns a list, not a string. Every validation
call failed, dropping row counts from 106 to 1.

Both overrides now use the same "YES" in final_answer logic as SaaS.
E2E verified: BCBS 5 rows (within LLM variance of baseline 7),
Clover 106 rows (exact match to baseline).
2026-04-14 12:58:30 -05:00
ppanchigar 63f32c418c Remove debug routing print statements
Strip all [DEBUG ROUTING] print statements added for E2E verification.
Cherry-pick the previous commit (5e143c10) to restore them for testing.
2026-04-14 10:42:10 -05:00
ppanchigar 5e143c10f0 Retire stale client file_processing forks, unify pipeline entry point
Delete BCBS/Clover/CHC client file_processing.py files (1090 lines of
96% stale drift with 3 crash points). Recover the 4% real business
logic (BCBS OFFSET_TERM, Clover 12 full_context fields) into the
shared pipeline via config-driven field loading.

Make runner.py the canonical pipeline entry point by merging all
production features from main.py (timing, duplicate detection, aarete
derived fields, TIN statistics, column reordering). Reduce main.py to
a thin wrapper that delegates to runner.main().

Fix the root cause of all client overrides being dead in production:
set_active_client is now called with the correct client name instead
of being hardcoded to "saas". Both resolver shims (file_processing
and prompt_calls) now route correctly.

Includes temporary [DEBUG ROUTING] print statements at 6 routing
decision points for E2E verification. Cherry-pick this commit to
restore debug instrumentation for future testing.

E2E verified: BCBS Promise and Clover both pass with correct routing.
2026-04-14 10:39:51 -05:00
Aditi Agarwal fb1266ba33 Merged dev into feature/upDatepipelinesFile 2026-04-14 15:22:43 +00:00
Aditi Agarwal 6058ab0b7f Update bitbucket pipelines 2026-04-14 20:51:06 +05:30
Katon Minhas 96967d14f3 Merged in bugfix/generic-issue-fixes (pull request #953)
Bugfix/generic issue fixes

* Patch for llm responses

* Merged dev into bugfix/generic-issue-fixes

* Update Exhibit-Level instruction for Claim Type and Bill Type

* Update Service-Level instruction for Claim Type and Bill Type

* Update Bill Type code to prompt for DESC field

* Black format

* Remove test


Approved-by: Siddhant Medar
2026-04-13 13:14:37 +00:00
Siddhant Medar 8a872cdc67 Merged in bugfix/unify_prompt_exhibit_header (pull request #952)
Bugfix/unify prompt exhibit header

* Remove dead prompt_exhibit_header overrides and legacy get_exhibit_pages

Investigation (AST call tree analysis) proved both were dead code:

1. get_exhibit_pages() in preprocessing_funcs.py had ZERO callers in the
   entire codebase. It was superseded by get_exhibit_pages_new() which is
   the only live path (called by preprocess.one_to_n_exhibit_chunking).

2. Client prompt_exhibit_header overrides (2-arg) in bcbs_promise and
   clover prompt_calls.py were dead for two independent reasons:
   a) Their only consumer (get_exhibit_pages) has zero callers
   b) The live consumer (get_exhibit_pages_new) calls with 3 args —
      if the resolver returned the client 2-arg function, Python would
      raise TypeError. The SaaS 3-arg version was always running.

Verification:
- Programmatic AST call tree (local_scripts/call_tree_analysis.py)
  confirms zero callers and signature mismatch
- Resolver test confirms all clients resolve to SaaS 3-arg implementation
- Full unit suite: 1077 passed, 0 failures
- E2E: BCBS P…
* Merged dev into bugfix/unify_prompt_exhibit_header

* Add dev dependencies for local code index

Add tree-sitter, lancedb, and tiktoken as dev dependencies for a local
codebase indexing system. The index provides hybrid search (symbol table,
call graph, semantic embeddings, BM25) for token-efficient code exploration.

The index lives in a gitignored .index/ folder — each developer builds
their own locally.


Approved-by: Katon Minhas
2026-04-10 16:26:26 +00:00
Katon Minhas 8732ab136c Merged in bugfix/black-format (pull request #951)
Black format

* Black format
2026-04-09 18:13:30 +00:00
Rahul Ailaboina 52563b9359 Merged in feature/pre-doczy-status-column (pull request #950)
Feature/pre doczy status column

* pre-doczy changes

* format

* Merged dev into feature/pre-doczy-status-column

* function moved to utils


Approved-by: Katon Minhas
2026-04-09 18:10:25 +00:00
Praneel Panchigar 37e1e9c228 Merged in bugfix/implicit-cpt-update (pull request #949)
Bugfix/implicit cpt update

* DAIP2-2310: enrich service term before implicit code extraction

- Add SERVICE_ENRICHMENT prompt + enrich_service_for_codes() that runs
  after explicit extraction and before the implicit pipeline, stripping
  provider/facility/contractual noise while preserving billing codes.
- Tighten CODE_IMPLICIT_ARBITRATION instructions to require strict
  synonymy and prefer no_match over best-effort guesses.
- Remove single-candidate auto-accept bypass so all implicit candidates
  go through LLM arbitration.
- Update unit tests for new arbitration path and enrichment mock.

* DAIP2-2310: refine service enrichment and track implicit mapping source

- Service enrichment prompt: bare 'Inpatient'/'Outpatient' place-of-service
  rows now collapse to 'Covered Services' to prevent spurious Level 1 matches.
  When paired with a real procedure (e.g. 'Outpatient Surgery'), the place
  modifier is preserved.
- code_implicit_rag: record per-match mapping origin (CPT_LEVEL1,
  HCPCS_LEVEL1, CPT_LEVEL2, HCPCS_LEVEL2) on the code answer dict for
  debugging which mapping produced each implicit match.
- constants: add TODO next to HCPCS_LEVEL1_MAPPING load flagging the
  upcoming DME code range change.

* Merged dev into DAIP2-2310-implicit-cpt-update

* DAIP2-2310: fix 3-digit revenue code extraction and enrichment exclusion rule

Two related fixes surfaced during short-circuit analysis of the NICU rows:

1. 3-digit revenue codes were being dropped by validate_explicit_codes,
   causing the explicit path to return empty and the row to fall through
   to implicit RAG (which would then semantic-match against CPT/HCPCS
   and produce 'Specific - No Match'). Root causes:
   - _revenue_code_format_valid required exactly 4 digits, rejecting the
     common contract form like 'Revenue Code 173'.
   - REV_MAPPING keys are loaded as ints by pandas (leading zero stripped
     from all-numeric CSV columns), so even after accepting 3-digit form
     the mapping lookup missed.

   Fixes in src/codes/code_funcs.py:
   - _revenue_code_format_valid now accepts 3 or 4 digits.
   - New _normalize_revenue_code helper returns canonical 4-digit form.
   - validate_explicit_codes stores the normalized form in REVENUE_CD.
   - _has_unmatched_codes normalizes before membership chec…
* Merged dev into bugfix/implicit-cpt-update

* DAIP2-2310: accept revenue code wildcards and ranges; add unit tests

Extends the earlier 3-digit revenue code fix to cover two more forms the
CODE_EXPLICIT prompt instructs the LLM to return:

1. X-suffix wildcards (e.g. '17X', '173X') - standard UB-04 shorthand
   meaning any digit across a revenue family. The prompt explicitly says
   'may end in X' but the validator was rejecting them, causing the same
   silent drop + implicit-RAG fall-through as the 3-digit bug.

2. Ranges (e.g. '173-174', '0170-0179') - the CODE_EXPLICIT prompt
   instructs ranges in 'LOW-HIGH' form. Procedure codes support ranges
   but revenue codes did not. The Gulf Coast 'Rev Codes 173-174' rows
   were landing on Specific - No Match because of this.

New helpers in src/codes/code_funcs.py:
- _revenue_wildcard_format_valid / _revenue_range_format_valid /
  _revenue_code_any_format_valid: targeted format checks.
- _normalize_revenue_any: canonical form for any valid kind
  (single code -> '0173', wildcard -> '017X', range -> '0173-0…
* DAIP2-2310: prevent enrichment from hallucinating or dropping descriptors

Two related prompt changes to SERVICE_ENRICHMENT_INSTRUCTION based on
review feedback:

1. Add a 'no information not in input' rule. The enrichment is a
   faithful rewrite that strips noise — not a knowledge-augmented
   expansion. The LLM must NOT look up what a code means and inject
   the description, infer category names from codes, or translate
   medical shorthand into clinical terms not present in the input.

2. Add a 'preserve descriptive labels' rule. Things like 'Level 1',
   'Level 2-5', 'Tier A', 'Class III', 'low complexity' etc. are
   meaningful service descriptors written by the contract author and
   must remain in the output even when codes are also present. The
   prior ER few-shot example showed the LLM stripping 'Level 1'..
   'Level 5' labels — that's information loss.

Updated few-shot examples to match the new rules:
- Emergency Room example output now preserves 'Level 1 - 99281,
  Level 2 - 99282, ...' verbati…

Approved-by: Katon Minhas
2026-04-08 21:49:12 +00:00
Praneel Panchigar 8c9aa1d46d Merged in DAIP2-2346-update-client-prompt-calls (pull request #948)
DAIP2-2346 update client prompt calls

* DAIP2-2346: retire accidental client prompt_calls drift

Delete 14 client prompt_calls overrides in bcbs_promise and clover that were
stale forks of old SaaS implementations. Shared callers all expect SaaS return
shapes; client shims were either redundant with parser-side normalization or
outright broken dead code (prompt_dynamic_assignment unpack crash,
prompt_lesser_of_check double-parse). Resolver shim at
src/pipelines/shared/prompts/prompt_calls.py now falls through to SaaS.

Preserved with annotations:
- validate_reimbursements_for_llm: real business rule (strict YES equality)
- prompt_exhibit_header: required by 2-arg call site at preprocessing_funcs.py:192

* DAIP2-2346: also retire prompt_exhibit_level and prompt_dynamic overrides

Follow-up to the Bucket A cleanup. Both client overrides only drifted by
omitting the optional field_names kwarg on prompt_templates.EXHIBIT_LEVEL.
SaaS passes field_names and gets the format-aware parser
(_create_json_dict_parser); falling through to SaaS is a strict improvement
in normalization. Client files now contain only the two truly
client-specific overrides.

* Changes to gitignore

* Merged dev into DAIP2-2346-update-client-prompt-calls


Approved-by: Katon Minhas
2026-04-08 19:29:48 +00:00
Venkatakrishna Reddy Avula 781e90085d Merged in bugfix/DAIP2-2287-payer-state-fix (pull request #946)
Bugfix/DAIP2-2287 payer state fix

* updated payer name prompt

* Merged dev into bugfix/DAIP2-2287-payer-state-fix

* Merged dev into bugfix/DAIP2-2287-payer-state-fix

* added client state field


Approved-by: Katon Minhas
2026-04-07 18:35:13 +00:00
Katon Minhas 491abd31c2 Merged in feature/dynamic-codes-final (pull request #942)
Feature/dynamic codes final

* Move Claim Type to dynamic code group

* Update CLaim Type and Bill TYpe prompts

* Prompt optimization

* Prompt tuning

* streamline

* Format

* Merged dev into feature/dynamic-codes

* test claim and bill type

* Merged dev into DAIP2-2186-service-claim-type-testing

* Merged dev into feature/dynamic-codes

* prompt updated for claim type

* Merge branch 'feature/dynamic-codes' into DAIP2-2186-service-claim-type-testing

* Merged dev into DAIP2-2186-service-claim-type-testing

* Remove print

* Switch to map

* Merged dev into feature/dynamic-codes-final

* prompt inconsistency fixed


Approved-by: Praneel Panchigar
Approved-by: Siddhant Medar
2026-04-07 18:33:28 +00:00