Feature/PC logic cleanup output
* Few tweaks PC_logics
* Fixed orphan_ranking
* black formatting
* Changes on output field and ranking method
* updated few hotfixes
* black format fix
Approved-by: Katon Minhas
Feature/DAIP2 pacificsource reimbursements issues
* Tighten PREMIUM_TERM and DISCOUNT_TERM classifier prompts
CARVEOUT_CHECK was misrouting table rate rows into special-case fields,
dropping them from the reimbursement output:
- "110% of CMS allowed" (base fee-schedule rates) was being classified as
PREMIUM_TERM because the prompt treated "above 100% of reference" as an
implicit premium. Seen on PacificSource Medicare_Attachment_A1 and A2
Facility contracts where Inpatient/Outpatient rows were missing (2556)
or silently fell back to 100% fee-schedule (2557).
- Per-service discount rates like "Progressive Lenses: 15% discount",
"Contact Lenses: 2% discount", "Frame: 20% discount" were being
classified as DISCOUNT_TERM because the prompt only required the word
"discount" to appear. Seen on PacificSource Commercial_Attachment_A2
and A5 Professional contracts (2558, 2559).
Prompts now require the literal keyword AND explicitly exclude the
common false-positive patterns, keeping per-service rate rows in the
reimbursement output.
* Stop splitting multi-page exhibits on repeated page footers
Two bugs were collaborating to split a single Attachment into several
Exhibit objects, causing intra-exhibit lesser-of search to miss notes
living on a later page of the same exhibit:
1. EXHIBIT_HEADER extraction picked up page-footer lines as if they
were new section headers. On PacificSource contracts, pages end with
a short-form repeat like "Attachment A Oregon Health & Science
University" next to the page number and contract date, which the
ATTACHMENT-prefix regex and the LLM both accepted as a header.
2. EXHIBIT_HEADER_DEDUP parsing required the LLM output to be wrapped
in a strict |pipes| JSON block. When the LLM prefixed the response
with prose ("# Analysis ..."), parsing threw ValueError and the
code fell back to the original, un-deduped dict — defeating dedup
entirely.
Combined, these produced 3 Exhibit objects for Commercial_Attachment_A2
(page 1 header + two entries on page 2: repeated header + footer),
leaving …
* Merged dev into feature/DAIP2-pacificsource-reimbursements-issues
Approved-by: Katon Minhas
fix: fall back to PROVIDER_NAME in PROV_INFO_JSON when no TIN/NPI extracted
* fix: fall back to PROVIDER_NAME in PROV_INFO_JSON when no TIN/NPI extracted
When get_prov_info_json short-circuits due to no TIN/NPI regex matches,
PROV_INFO_JSON was left as [] even when PROVIDER_NAME was successfully
extracted via the one-to-one pipeline. This caused inconsistent output
across contracts with the same provider — some files produced a NAME-only
entry (via a false-positive regex hit triggering the LLM), others produced [].
Reconcile at add_group_and_other, the first point where both extraction
streams' results are available. When PROV_INFO_JSON is empty but
PROVIDER_NAME is known, synthesize a NAME-only entry with IS_GROUP:"Y" and
populate PROV_GROUP_NAME_FULL directly — skipping the provider_name_match_check
LLM call since the match is tautological by construction.
Adds 5 unit tests covering the str, list, already-populated, empty-name,
and all-empty-list cases.
Approved-by: Katon Minhas
Updated feature> dev gate to print variables of echo statements
* Updated feature> dev gate to print variables of echo statement
* Reverted placeholder changes
Approved-by: Sujit Deokar
Feature/standardized services
* service term standardization
* prompt update
* prompt updates for standardization
* only service standardization
* new file
* add supporting files and test scripts for standardization work
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* remove old files
* Merge remote-tracking branch 'origin/dev' into feature/standardized-services
* final fixes
* Merged dev into feature/standardized-services
* additional features
* removed unwanted files
* remove unwanted files
* Merge branch 'dev' into feature/standardized-services
* Merge remote-tracking branch 'origin/dev' into feature/standardized-services
* reversed vendor changes
* Merged dev into feature/standardized-services
* addressed PR comments
* Merge branch 'dev' into feature/standardized-services
* Merged dev into feature/standardized-services
* addressed 3 remaining comments inPR
* black formatting
* incorporated feedback from AI code reviewer
* reused existing functionality
Approved-by: Katon Minhas
Remove curly braces from echo statements in dev->stg
* Remove curly brances from echo statements in dev->stg
* Removed curly braces in echo statements in feature-> dev gate
Approved-by: Sujit Deokar
pre_doczy_ru_base_prefix_changes
* pre_doczy_ru_base_prefix_changes
* Merged dev into feature/Pre_doczy_report
* Merged dev into feature/Pre_doczy_report
Approved-by: Katon Minhas
Feature/upDatepipelinesFile
* Added logic to capture failure status for invalid branch names
* Merged dev into feature/upDatepipelinesFile
* Bugfix the print logs were not able to populate the placeholders. These are now fix to use the sh file
* Merge branch 'dev' into feature/upDatepipelinesFile
Added logic to capture failure status for invalid branch names
* Added logic to capture failure status for invalid branch names
* Merged dev into feature/upDatepipelinesFile
Return None for dashboard output when dashboard postprocessing is off
* Return None for dashboard output when dashboard postprocessing is off
FINAL_RESULT_DF_DASHBOARD was initialized as an empty DataFrame even
when RUN_DASHBOARD_POSTPROCESSING was False, causing downstream code
to needlessly process it (reorder_columns, etc). Now returns None
when dashboard is not requested, matching the postprocess() contract.
* Merged dev into bugfix/retire_stale_client_file_processing
* Merge dev (with revert) into feature branch
* Re-apply retire stale client file_processing changes
Revert of the revert (4f53b528) to restore the original changes
from the feature branch for proper PR review.
* Merge branch 'bugfix/retire_stale_client_file_processing' of bitbucket.org:aarete/doczy.ai into bugfix/retire_stale_client_file_processing
Approved-by: Katon Minhas
PR #959 was merged into dev without approval. This reverts commits
5e143c10, 63f32c41, 849aa626, and 927abcae to restore dev to its
pre-merge state. The changes will be re-submitted via a new PR
after proper review.
reorder_columns() was only keeping columns listed in FIELD_FORMAT_MAPPING,
silently discarding any extra columns like BCBS OFFSET_TERM/OFFSET_INDICATOR
and Clover's 12 full_context fields. The function's docstring documented
appending extra columns but the implementation was missing that step.
1. Tag format validation in release-prod (MEDIUM, defensive).
Before parsing major/minor/patch from the latest tag, assert it
matches ^v[0-9]+\.[0-9]+\.[0-9]+$. Empirically tested: catches
v1, v1.2, v1.2.3.4, v1.0.0-rc1, v1.2.3-beta+build, and still
accepts v0.0.0 (the first-release fallback).
Without this, bash arithmetic silently mangles non-semver tags
into wrong results via its 'treat empty/non-numeric as 0' rule.
Worst case: tag 'v1' → cut -d. -f2/-f3 both return '1' →
minor bump produces v1.1.2 instead of v1.0.1. No visible error.
2. Rewrite the AI review comment to match actual behavior.
The old comment said 'advisory, not a gate' but also admitted
git clone / apt-get are hard failures, contradicting itself.
The new comment makes the runtime vs. infrastructure distinction
explicit: runtime failures (agent crash, STS errors, Python
exceptions) become warnings via || echo; infrastructure failures
(missing token, clone failure, apt-get failure) stay hard.
This is a doc fix, not a code change. Deliberately preserving
the hard-fail behavior on config errors because silent
degradation of AI review is the worst outcome — feature
disappears from CI with no signal.
Explicitly rejected from the second review (empirically verified):
- Gate logic 'fragile' claim: tested in bash, correct for all
our hardcoded ALLOWED values.
- ROLLBACK_TAG -z check 'passes empty': tested, -z correctly
catches empty strings. Reviewer has the semantics wrong.
- chmod 600 + set +x: our comment already documents chmod 600
as cosmetic/scanner-silencing; set +x wouldn't help because
Bitbucket's line-echo is runner-level, not bash set -x.
- cd/python refactor (previous round): already rebutted in
commit 4836e36f via empirical bash AND-OR list tests.
- Option B for git clone wrapping: silent degradation of
config errors is worse than the current loud-fail behavior.
1. Replace 'uv sync --frozen' with 'uv sync --locked'. --frozen silently
uses a stale lockfile if pyproject.toml is updated without regenerating
uv.lock, masking missed dependencies. --locked fails loudly with a clear
error when the lockfile is out of sync. Verified empirically with uv
0.9.14: --frozen exits 0 on mismatch, --locked exits 1.
2. Add 'chmod 600' on the OIDC token file after writing it. The container
is already single-user root so this is defensive/cosmetic, but it
silences security scanners and signals intent.
3. Add a comment explaining why gate steps use atlassian/default-image:4
instead of python:3.12.7 (gates run bash only, no Python toolchain
needed — lighter image, faster pull).
Explicitly rejected from the PR review:
- Token-masking sed mitigation (delimiter collision with / in real
Bitbucket clone tokens; cmd | sed || exit 1 swallows git failures
without set -o pipefail). Bitbucket's built-in Secured variable
masking is the correct mitigation and is already documented as a
setup requirement.
- SSH key alternative for git clone (architecturally worse — more
secrets to manage; HTTPS+Secured is the Bitbucket-recommended pattern).
- cd/python refactor ('fragile logic bug'). Verified empirically that
'cd X && python Y || echo Z' with set -e correctly catches both
cd and python failures via the ||. The step exits 0 as intended
by the 'advisory, not a gate' design. The suggested refactor would
introduce a hard-fail regression.
The unquoted ': ' (colon + space) in 'WARNING: AI review step failed'
was being interpreted as a YAML mapping key-value separator, causing
the entire script item to be parsed as a dict instead of a command
string. Bitbucket then rejected it with 'Missing or empty command
string' error at pull-requests > feature/* > 2 > step > script > 9.
Replaced the colon with a hyphen. Validated with yaml.safe_load that
all 10 script items in the ai-code-review step now parse as strings.
Bitbucket's YAML parser was interpreting the deeper-indented comment
after the printf line as a phantom empty list item, causing a
'Missing or empty command string' error at script item 9.
BCBS: change strict == "YES" to "YES" in final_answer. The strict
equality was rejecting borderline LLM responses (e.g. "YES, ..."),
dropping row counts from ~7 to 1.
Clover: remove .strip().upper() which crashed with AttributeError
because the parser returns a list, not a string. Every validation
call failed, dropping row counts from 106 to 1.
Both overrides now use the same "YES" in final_answer logic as SaaS.
E2E verified: BCBS 5 rows (within LLM variance of baseline 7),
Clover 106 rows (exact match to baseline).
Delete BCBS/Clover/CHC client file_processing.py files (1090 lines of
96% stale drift with 3 crash points). Recover the 4% real business
logic (BCBS OFFSET_TERM, Clover 12 full_context fields) into the
shared pipeline via config-driven field loading.
Make runner.py the canonical pipeline entry point by merging all
production features from main.py (timing, duplicate detection, aarete
derived fields, TIN statistics, column reordering). Reduce main.py to
a thin wrapper that delegates to runner.main().
Fix the root cause of all client overrides being dead in production:
set_active_client is now called with the correct client name instead
of being hardcoded to "saas". Both resolver shims (file_processing
and prompt_calls) now route correctly.
Includes temporary [DEBUG ROUTING] print statements at 6 routing
decision points for E2E verification. Cherry-pick this commit to
restore debug instrumentation for future testing.
E2E verified: BCBS Promise and Clover both pass with correct routing.
Bugfix/generic issue fixes
* Patch for llm responses
* Merged dev into bugfix/generic-issue-fixes
* Update Exhibit-Level instruction for Claim Type and Bill Type
* Update Service-Level instruction for Claim Type and Bill Type
* Update Bill Type code to prompt for DESC field
* Black format
* Remove test
Approved-by: Siddhant Medar
Bugfix/unify prompt exhibit header
* Remove dead prompt_exhibit_header overrides and legacy get_exhibit_pages
Investigation (AST call tree analysis) proved both were dead code:
1. get_exhibit_pages() in preprocessing_funcs.py had ZERO callers in the
entire codebase. It was superseded by get_exhibit_pages_new() which is
the only live path (called by preprocess.one_to_n_exhibit_chunking).
2. Client prompt_exhibit_header overrides (2-arg) in bcbs_promise and
clover prompt_calls.py were dead for two independent reasons:
a) Their only consumer (get_exhibit_pages) has zero callers
b) The live consumer (get_exhibit_pages_new) calls with 3 args —
if the resolver returned the client 2-arg function, Python would
raise TypeError. The SaaS 3-arg version was always running.
Verification:
- Programmatic AST call tree (local_scripts/call_tree_analysis.py)
confirms zero callers and signature mismatch
- Resolver test confirms all clients resolve to SaaS 3-arg implementation
- Full unit suite: 1077 passed, 0 failures
- E2E: BCBS P…
* Merged dev into bugfix/unify_prompt_exhibit_header
* Add dev dependencies for local code index
Add tree-sitter, lancedb, and tiktoken as dev dependencies for a local
codebase indexing system. The index provides hybrid search (symbol table,
call graph, semantic embeddings, BM25) for token-efficient code exploration.
The index lives in a gitignored .index/ folder — each developer builds
their own locally.
Approved-by: Katon Minhas
Feature/pre doczy status column
* pre-doczy changes
* format
* Merged dev into feature/pre-doczy-status-column
* function moved to utils
Approved-by: Katon Minhas
Bugfix/implicit cpt update
* DAIP2-2310: enrich service term before implicit code extraction
- Add SERVICE_ENRICHMENT prompt + enrich_service_for_codes() that runs
after explicit extraction and before the implicit pipeline, stripping
provider/facility/contractual noise while preserving billing codes.
- Tighten CODE_IMPLICIT_ARBITRATION instructions to require strict
synonymy and prefer no_match over best-effort guesses.
- Remove single-candidate auto-accept bypass so all implicit candidates
go through LLM arbitration.
- Update unit tests for new arbitration path and enrichment mock.
* DAIP2-2310: refine service enrichment and track implicit mapping source
- Service enrichment prompt: bare 'Inpatient'/'Outpatient' place-of-service
rows now collapse to 'Covered Services' to prevent spurious Level 1 matches.
When paired with a real procedure (e.g. 'Outpatient Surgery'), the place
modifier is preserved.
- code_implicit_rag: record per-match mapping origin (CPT_LEVEL1,
HCPCS_LEVEL1, CPT_LEVEL2, HCPCS_LEVEL2) on the code answer dict for
debugging which mapping produced each implicit match.
- constants: add TODO next to HCPCS_LEVEL1_MAPPING load flagging the
upcoming DME code range change.
* Merged dev into DAIP2-2310-implicit-cpt-update
* DAIP2-2310: fix 3-digit revenue code extraction and enrichment exclusion rule
Two related fixes surfaced during short-circuit analysis of the NICU rows:
1. 3-digit revenue codes were being dropped by validate_explicit_codes,
causing the explicit path to return empty and the row to fall through
to implicit RAG (which would then semantic-match against CPT/HCPCS
and produce 'Specific - No Match'). Root causes:
- _revenue_code_format_valid required exactly 4 digits, rejecting the
common contract form like 'Revenue Code 173'.
- REV_MAPPING keys are loaded as ints by pandas (leading zero stripped
from all-numeric CSV columns), so even after accepting 3-digit form
the mapping lookup missed.
Fixes in src/codes/code_funcs.py:
- _revenue_code_format_valid now accepts 3 or 4 digits.
- New _normalize_revenue_code helper returns canonical 4-digit form.
- validate_explicit_codes stores the normalized form in REVENUE_CD.
- _has_unmatched_codes normalizes before membership chec…
* Merged dev into bugfix/implicit-cpt-update
* DAIP2-2310: accept revenue code wildcards and ranges; add unit tests
Extends the earlier 3-digit revenue code fix to cover two more forms the
CODE_EXPLICIT prompt instructs the LLM to return:
1. X-suffix wildcards (e.g. '17X', '173X') - standard UB-04 shorthand
meaning any digit across a revenue family. The prompt explicitly says
'may end in X' but the validator was rejecting them, causing the same
silent drop + implicit-RAG fall-through as the 3-digit bug.
2. Ranges (e.g. '173-174', '0170-0179') - the CODE_EXPLICIT prompt
instructs ranges in 'LOW-HIGH' form. Procedure codes support ranges
but revenue codes did not. The Gulf Coast 'Rev Codes 173-174' rows
were landing on Specific - No Match because of this.
New helpers in src/codes/code_funcs.py:
- _revenue_wildcard_format_valid / _revenue_range_format_valid /
_revenue_code_any_format_valid: targeted format checks.
- _normalize_revenue_any: canonical form for any valid kind
(single code -> '0173', wildcard -> '017X', range -> '0173-0…
* DAIP2-2310: prevent enrichment from hallucinating or dropping descriptors
Two related prompt changes to SERVICE_ENRICHMENT_INSTRUCTION based on
review feedback:
1. Add a 'no information not in input' rule. The enrichment is a
faithful rewrite that strips noise — not a knowledge-augmented
expansion. The LLM must NOT look up what a code means and inject
the description, infer category names from codes, or translate
medical shorthand into clinical terms not present in the input.
2. Add a 'preserve descriptive labels' rule. Things like 'Level 1',
'Level 2-5', 'Tier A', 'Class III', 'low complexity' etc. are
meaningful service descriptors written by the contract author and
must remain in the output even when codes are also present. The
prior ER few-shot example showed the LLM stripping 'Level 1'..
'Level 5' labels — that's information loss.
Updated few-shot examples to match the new rules:
- Emergency Room example output now preserves 'Level 1 - 99281,
Level 2 - 99282, ...' verbati…
Approved-by: Katon Minhas
DAIP2-2346 update client prompt calls
* DAIP2-2346: retire accidental client prompt_calls drift
Delete 14 client prompt_calls overrides in bcbs_promise and clover that were
stale forks of old SaaS implementations. Shared callers all expect SaaS return
shapes; client shims were either redundant with parser-side normalization or
outright broken dead code (prompt_dynamic_assignment unpack crash,
prompt_lesser_of_check double-parse). Resolver shim at
src/pipelines/shared/prompts/prompt_calls.py now falls through to SaaS.
Preserved with annotations:
- validate_reimbursements_for_llm: real business rule (strict YES equality)
- prompt_exhibit_header: required by 2-arg call site at preprocessing_funcs.py:192
* DAIP2-2346: also retire prompt_exhibit_level and prompt_dynamic overrides
Follow-up to the Bucket A cleanup. Both client overrides only drifted by
omitting the optional field_names kwarg on prompt_templates.EXHIBIT_LEVEL.
SaaS passes field_names and gets the format-aware parser
(_create_json_dict_parser); falling through to SaaS is a strict improvement
in normalization. Client files now contain only the two truly
client-specific overrides.
* Changes to gitignore
* Merged dev into DAIP2-2346-update-client-prompt-calls
Approved-by: Katon Minhas
Bugfix/DAIP2-2287 payer state fix
* updated payer name prompt
* Merged dev into bugfix/DAIP2-2287-payer-state-fix
* Merged dev into bugfix/DAIP2-2287-payer-state-fix
* added client state field
Approved-by: Katon Minhas
Feature/dynamic codes final
* Move Claim Type to dynamic code group
* Update CLaim Type and Bill TYpe prompts
* Prompt optimization
* Prompt tuning
* streamline
* Format
* Merged dev into feature/dynamic-codes
* test claim and bill type
* Merged dev into DAIP2-2186-service-claim-type-testing
* Merged dev into feature/dynamic-codes
* prompt updated for claim type
* Merge branch 'feature/dynamic-codes' into DAIP2-2186-service-claim-type-testing
* Merged dev into DAIP2-2186-service-claim-type-testing
* Remove print
* Switch to map
* Merged dev into feature/dynamic-codes-final
* prompt inconsistency fixed
Approved-by: Praneel Panchigar
Approved-by: Siddhant Medar
Feature/DAIP2-1615 refactor maximize code reusability
* added reusability
* client resolver added
* Merged DEV into feature/DAIP2-1615-refactor-maximize-code-reusability
* Refactor client prompt reuse and harden resolver test coverage.
Remove duplicated client prompt functions so unresolved names fall back to SaaS, add resolver contract tests with runtime-context isolation, and clean resolver-path test imports to prevent cross-test client leakage.
* Merge origin/dev into feature/DAIP2-1615-refactor-maximize-code-reusability.
Bring in latest vendor pipeline framework changes from dev and resolve runner routing conflict by preserving vendor processor dispatch while keeping shared function-level client fallback behavior.
* Merged dev into feature/DAIP2-1615-refactor-maximize-code-reusability
* Remove tracked drift review report from documentation.
Keep drift/alignment notes as local-only artifacts under ignored docs/ and stop tracking documentation/function_level_override_drift_review.md in the branch.
* Merged dev into feature/DAIP2-1615-refactor-maximize-code-reusability
Approved-by: Katon Minhas
Feature/DAIP2-2301 flag and remove identical contracts
* initiated flaging duplicate contracts
* Merged dev into feature/DAIP2-2301-flag-and-remove-identical-contracts
* Merged dev into feature/DAIP2-2301-flag-and-remove-identical-contracts
* updated duplicate detection
* pipeline fixes
* remove print statements
* Merged dev into feature/DAIP2-2301-flag-and-remove-identical-contracts
Approved-by: Katon Minhas
Feature/DAIP2-1575 set up dev prod uat branches
* Add branch promotion gates and release pipelines
* Add resolve_source_branch.sh script and update .gitignore; fix pipeline step name casing 'dev'
* Add AI code review step and refactor branch resolution in pipelines
* Merged dev into feature/DAIP2-1575-set-up-dev-prod-uat-branches-
* Merged dev into feature/DAIP2-1575-set-up-dev-prod-uat-branches-
* Merged dev into feature/DAIP2-1575-set-up-dev-prod-uat-branches-
Approved-by: Katon Minhas
removed 2 signature related fields
* removed 2 signature related fields
* Merged dev into DAIP2-1595-optimize-signature-fields
* Merged dev into DAIP2-1595-optimize-signature-fields
Approved-by: Katon Minhas
Bugfix/DAIP2-2164 reimbursement primary issue fixes
* sample calculations reimbursements removed
* split service term added
* Merged DEV into bugfix/DAIP2-2164-reimbursement-primary-issue-fixes
* updated prompt and unit test added
* added prompt for codes listed below
* Merged DEV into bugfix/DAIP2-2164-reimbursement-primary-issue-fixes
* Output format at end
* Genericize prompt
* Merged DEV into bugfix/DAIP2-2164-reimbursement-primary-issue-fixes
* context missing issue fixed
* updated service term split prompt for program varints
* Merged DEV into bugfix/DAIP2-2164-reimbursement-primary-issue-fixes
* Merged DEV into bugfix/DAIP2-2164-reimbursement-primary-issue-fixes
* Merged DEV into bugfix/DAIP2-2164-reimbursement-primary-issue-fixes
* Merged DEV into bugfix/DAIP2-2164-reimbursement-primary-issue-fixes
* updated split service term prompt
* optimized and minimized split service term prompt
* updated service term split prompt
* Merged DEV into bugfix/DAIP2-2164-reimbursement-primary-issue-fixes
* Merged dev into bugfix/DAIP2-2164-reimbursement-primary-issue-fixes
* Updated Prompt
* grouped instructions
* removed print statements
Approved-by: Katon Minhas
Feature/vendor generic
* initial commit for generic vendor logic
* updated generic field list
* combined special fields
* joint fields template
* payment term and amount extraction
* model update
* model update
* added constants
* care-source fields added
* care-source constants
* fields update for bcbs-az
* prompt update for term type
* prompt update for payment type
* post process additions for generic
* Merge remote-tracking branch 'origin/DEV' into feature/vendor-generic
* Merged DEV into feature/vendor-generic
* Merged DEV into feature/vendor-generic
* Black format
* Merge remote-tracking branch 'origin/DEV' into feature/vendor-generic
* Merge branch 'DEV' into feature/vendor-generic
* PR comments
* deleted redundant file
* Merged dev into feature/vendor-generic
Approved-by: Katon Minhas
Bugfix/DAIP2-2230 mcs issues effective date new
* updated scope and fixes
* optimized prompt
* shortened prompt
* remove redundant prompt line
* Merged DEV into bugfix/DAIP2-2230-mcs-issues-effective-date-new
Approved-by: Katon Minhas
Rahul Ailaboina