Fix KeyError in compute_flagged when a *_CONF column has no value sibling
* Fix KeyError in compute_flagged when a *_CONF column has no value sibling
Production hit a hard crash at the end of every run:
KeyError: "['DYNAMIC_PRIMARY_ENTITIES'] not in index"
src/qc_qa/confidence/summary.py:143
compute_flagged was iterating over *_CONF columns and unconditionally
indexing the dataframe with both the FILE_NAME column and the stripped
value column. That assumed every <FIELD>_CONF column has a sibling
<FIELD> value column in final_df. That isn't always true: dynamic-primary
features carry only the _CONF side (their value side is dropped by
reorder_columns since it isn't in FIELD_FORMAT_MAPPING but its _CONF
suffix matches the explicit _CONF carve-out). When the model produced a
below-threshold score for one of these and the value column was absent,
pandas .loc raised KeyError and the runner crashed.
Fix:
- Build the .loc column list defensively: drop any name that isn't
actually present in final_df.
- Make the per-row dict construction robust to either FILE_NAME or
…
Approved-by: Katon Minhas
Feature/one to one confidence scoring
* T1 plumbing: capture per-field confidence + retrieved-chunk metadata for 1:1 HSC fields
Prep work for the 1:1 confidence-scoring stage. No scoring logic yet — this
just collects the inputs the next ticket (rule-based scorer) will consume.
- ONE_TO_ONE_SINGLE_FIELD_TEMPLATE: ask the LLM for confidence (0.0-1.0),
verdict (correct/uncertain/not_found), and supporting_snippet alongside
the field value. Existing field parser passes the extra keys through
unchanged.
- prompt_hsc_single_field: now returns a 4-tuple (name, value, field,
metadata) where metadata holds the confidence/verdict/snippet plus a
lightweight summary of which chunks the LLM saw (count + ids).
_extract_hsc_metadata is defensive: clamps out-of-range confidences,
defaults a missing/garbage verdict, caps the snippet at 500 chars,
returns _empty_hsc_metadata() on every bail-out path.
- run_hybrid_smart_chunked_fields: optional return_metadata flag. Default
return shape unchanged (dict of values) for backwards compatibility;
wh…
* DAIP2-2692: field-type-aware rule scorer for 1:1 confidence
Replaces the T1 placeholder _CONF (just the LLM-stated confidence) with
a rule-based score per field. No new LLM calls -- pure local computation.
New module: src/qc_qa/confidence/
- field_types.py : FieldType enum + per-field mapping + weight presets
(date / money / tin / npi / code / name / boolean /
free_text). New 1:1 fields default to FREE_TEXT.
- signals.py : exact / token_overlap / fuzzy / number_match / regex /
grounding / na_check. Each signal returns float in
[0,1] or None when not applicable.
- scorer.py : score_field(field_name, value, metadata, contract_text,
field_prompt) -> float. Runs the signals that have
non-zero weight for the field's type, renormalizes
over the signals that actually fired, blends lightly
with the LLM-stated confidence (15%…
* DAIP2-2694: integrate calibration table into 1:1 confidence scoring
Adds a per-field-type calibration step that maps raw confidence scores
to empirically-calibrated values via a linearly-interpolated curve.
Identity by default, so the stage is a no-op until the curve is
backfilled from QC data; once backfilled, a "0.X confidence" output
reflects roughly X% empirical accuracy.
New files:
- src/qc_qa/confidence/calibration.py
* loads + caches the curve from JSON
* linear interpolation between anchor points
* defensive: bad inputs / missing file / malformed table all fall
back to identity (no scoring regression on failure)
- src/qc_qa/confidence/calibration_table.json
* identity baseline shipped as the starting point
* one curve per FieldType plus a 'default' fallback
* documents the regeneration entry point for the QC-data backfill
.gitignore:
* carve-out !src/qc_qa/confidence/*.json so the table actually ships
Scorer wiring:
* apply calibration AFTER the r…
* DAIP2-2695: field-level confidence distribution stats + flagged-row report
At end of every run, emit two CSVs under tracking/:
- <BATCH_ID>-CONFIDENCE-SUMMARY.csv : per-field distribution stats
columns: field, n, mean, p10, p50, p90, n_below_threshold,
threshold, pct_below_threshold
sorted by pct_below_threshold descending so the worst fields are
at the top of the file.
- <BATCH_ID>-CONFIDENCE-FLAGGED.csv : the reviewer hit list
columns: FILE_NAME, field, confidence, value, threshold
sorted by confidence ascending.
Both are derived from the per-row <FIELD>_CONF columns the rule scorer
populates. Empty inputs (no _CONF columns yet, or no rows) skip the
write rather than emitting empty files.
Threshold:
* config.CONFIDENCE_THRESHOLD (CLI arg confidence_threshold=, default 0.6)
New module:
- src/qc_qa/confidence/summary.py
compute_summary(final_df, threshold) -> per-field stats DataFrame
compute_flagged(final_df, threshold) -> below-threshold r…
* Skip _CONF columns in standard_postprocess value-shape normalizers
The per-column loop in standard_postprocess substring-matches column
names ("_IND" in col, "TIN" in col, "_DT" in col, etc.) to decide which
shape normalizer to apply. After DAIP2-2692 introduced <FIELD>_CONF
confidence-score columns, those columns started colliding with the
match conditions:
- AUTO_RENEWAL_IND_CONF matched "_IND in col" -> normalize_indicator_field
coerced the 1.0 float to the string "N", destroying the score.
- *_DT_CONF / *_TIN_CONF etc. were also at risk via the same pattern.
Fix: short-circuit the loop with `if col.endswith("_CONF"): continue`
so confidence floats are never passed through the value-shape
normalizers, regardless of what string happens to appear in the
underlying field name.
Verified on the 1:1 smoke test (conf-1to1-smoke2): AUTO_RENEWAL_IND_CONF
now lands as 1.0 in RESULTS-FULL.csv and CONFIDENCE-SUMMARY.csv shows
all 10 HSC fields with proper float distributions.
* Merge remote-tracking branch 'origin/dev' into feature/one-to-one-confidence-scoring
* Black formatting: split generator in reorder_columns _CONF carve-out
Black wanted the multi-line generator expression formatted with one
clause per line. Behaviour unchanged.
* Fix na_check so amendment docs differentiate from base agreements
Three small bugs were stacking up to make CONTRACT_AMENDMENT_NUM flag every
file that returned a missing value, regardless of whether the document
actually was an amendment.
1) is_na() did not treat pandas/float NaN as N/A. When the LLM returned
no value, the cell came through as a float NaN, str()'d to "nan", and
the N/A token set ({"", "n/a", "na", "none", "null", "not found"})
did not match. na_check therefore never fired and the other signals
ran on the literal string "nan", producing a flat low score driven
only by the LLM's self-reported confidence. Now also catches float
NaN via value != value.
2) signal_na_check used three coarse buckets (0.30/0.50/0.85) and a
per-keyword saturation of 2 mentions. Both produced the same bucket
for docs that mention a concept 2x in boilerplate and docs whose
entire topic is that concept. Switched to a continuous score with
a saturation floor of ~30 mentions, so 8 mentions…
* Merged dev into feature/one-to-one-confidence-scoring
Approved-by: Katon Minhas
Feature/active rates
* initial commit
* Merged in feature/FixplaceholderIssue (pull request #977)
Remove curly braces from echo statements in dev->stg
* Remove curly brances from echo statements in dev->stg
* Removed curly braces in echo statements in feature-> dev gate
Approved-by: Sujit Deokar
* Merged in feature/standardized-services (pull request #958)
Feature/standardized services
* service term standardization
* prompt update
* prompt updates for standardization
* only service standardization
* new file
* add supporting files and test scripts for standardization work
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* remove old files
* Merge remote-tracking branch 'origin/dev' into feature/standardized-services
* final fixes
* Merged dev into feature/standardized-services
* additional features
* removed unwanted files
* remove unwanted files
* Merge branch 'dev' into feature/standardized-services
* Merge remote-tracking branch 'origin/dev' into feature/standardized-services
* reversed vendor changes
* Merged dev into feature/standardized-services
* addressed PR comments
* Merge branch 'dev' into feature/standardized-services
* Merged dev into feature/standardized-services
* addressed 3 remaining comments inPR
* black formatting
* incorporated feedback from AI c…
* prompt added for amendment intent
* amendment intent language
* update field name
* amendment intent tag
* prompt update
* Merge remote-tracking branch 'origin/dev' into feature/active-rates
* Merge remote-tracking branch 'origin/dev' into feature/active-rates
* Merge branch 'dev' into feature/active-rates
* exhibit standardization updates
* use AARETE_DERIVED_EXHIBIT_TITLE for intent
* active rates stuff
* Merge branch 'dev' into feature/active-rates
* prompt update
* amendment intent types
* active rates logic update
* unit tests
* Merged dev into feature/active-rates
* logic updates
* Merge remote-tracking branch 'origin/dev' into feature/active-rates
* Merge remote-tracking branch 'origin/dev' into feature/active-rates
* Merged dev into feature/active-rates
Approved-by: Katon Minhas
Feature/document index
* Add Document Index preprocessing — Layers 1, 2, and 3 wiring
Parse the Textract-emitted Document Index block at the top of each contract
with a single cached LLM call (prompt_document_index) instead of one per-page
call per page. Layer 2 verifies parsed entries via literal string match and
structural regex sweep, escalating suspect pages back to the existing per-page
path. Layer 1+2 failure triggers a full fallback to today's per-page flow.
New symbols:
- preprocessing_funcs.extract_document_index_block — regex slice of index prefix
- preprocessing_funcs.verify_index_against_pages — structural verifier (plain dict return)
- prompt_templates.DOCUMENT_INDEX_INSTRUCTION / DOCUMENT_INDEX — cached prompt pair
- prompt_calls.prompt_document_index — LLM wrapper (usage_label DOCUMENT_INDEX_PARSE)
- config: DOCUMENT_INDEX_PARSE_ENABLED and three threshold flags
- instrumentation: DOCUMENT_INDEX_PARSE mapped to preprocessing segment
one_to_n_exhibit_chunking now accepts optional contract_text; file_processing.py
pas…
* Add exhibit-level relevance filtering and deprecate pipe-delimited output
Step 5: extend the Document Index LLM call to emit reimbursement_relevant per
exhibit (no extra cost), then filter exhibit_header_dict before downstream
one-to-n extraction. Safety net A (keyword override) and safety net B (never
drop all exhibits) prevent false negatives. Controlled by
DOCUMENT_INDEX_RELEVANCE_FILTER_ENABLED flag.
Remove all pipe-delimited output from prompt_templates.py:
EXHIBIT_HEADER_INSTRUCTION, EXHIBIT_HEADER, EXHIBIT_HEADER_DEDUP_INSTRUCTION,
and EXHIBIT_HEADER_DEDUP now use JSON_DICT_FORMAT_INSTRUCTIONS and the
standard "return your final answer as a valid JSON dictionary" closing pattern.
DOCUMENT_INDEX_INSTRUCTION already used the JSON standard; consistent across
all preprocessing prompts.
* Add Tier 1 and Tier 2 unit tests for Document Index feature
Tier 1 — src/tests/test_document_index.py (31 tests):
- TestExtractDocumentIndexBlock: regex extraction (6 tests)
- TestDocumentIndexPromptTemplates: instruction shape, parser, cache
registration (11 tests)
- TestPromptDocumentIndex: mocked LLM wrapper — usage_label, cache=True,
fallback on malformed/non-list/empty responses (6 tests)
- TestVerifyIndexAgainstPages: literal match pass/fail, coverage sweep
escalation, page drift, high-failure-ratio fallback, behavioural pin
that Layer 2 never adds headers from regex (8 tests)
Tier 2 — src/tests/test_preprocessing_funcs.py (6 new tests):
- TestOneToNExhibitChunkingHybrid: Layer 1 success path, escalation routing,
global fallback, feature-flag-off, no contract_text, no index block
* Fix relevance overwrite bug and remove unused config flag
Bug: relevance_lookup was built from all parsed entries, so a landmark or
noise entry on the same page as an exhibit_start could overwrite the
exhibit_start's reimbursement_relevant=True flag. Fixed by filtering the
lookup to exhibit_start entries only.
Remove DOCUMENT_INDEX_COVERAGE_THRESHOLD from config.py — the flag was
defined but never referenced in the verifier or orchestrator. Removing
it avoids misleading operators expecting coverage-threshold fallback.
Docstring correction: extract_document_index_block docstring claimed
BOM-tolerance; removed that claim since Textract-generated .txt files
don't carry BOMs and the implementation doesn't strip .
Add TestRelevanceFilterLogic (3 tests): landmark-overwrite regression,
irrelevant exhibit exclusion, and safety-net-B preservation.
* Add Tier 3 offline study script for Document Index
Runs Layer 1+2 (extract, LLM parse, structural verify) against every .txt
in an input dir without the full pipeline. Emits a per-file CSV with 17
metrics columns (coverage_score, classification distribution, verification
failures, escalation count, fallback flag, runtime) and a summary markdown.
Marked DELETABLE post-rollout — intended for threshold tuning before Tier 4
E2E runs, not for production.
Usage:
uv run python -m src.testbed.document_index_offline
uv run python -m src.testbed.document_index_offline --input-dir data/pacific-source
uv run python -m src.testbed.document_index_offline --max-files 10
* Fix relevance overwrite bug and remove unused config flag
Remove coverage-sweep escalation — high-trust architecture
The Document Index path is now authoritative when it passes quality
thresholds. The per-page escalation of individual uncovered pages is
removed; the only fallback is the whole-document global fallback (empty
parse, page drift, low exhibit density, high failure ratio). This
eliminates the 8–50 false-positive escalations per file observed in the
Tier 3 offline study (continuation pages containing sub-headings were
being sent back to the per-page LLM unnecessarily).
verify_index_against_pages now returns 3 keys instead of 4:
verified_dict, metrics, global_fallback (escalation_pages removed)
one_to_n_exhibit_chunking success path no longer runs get_exhibit_pages_new
or prompt_header_deduplication — it returns verified_dict directly after
the relevance filter.
Add TestVerifyIndexAgainstPagesV2 (40 tests) written contract-first by the
testing agent: return-shape invariants, all 4 fallback tr…
* Trim TestVerifyIndexAgainstPagesV2 to 10 targeted boundary tests
* Remove exhibit-level relevance filter (Step 5)
The filter classified exhibits as reimbursement-relevant or not based
purely on header text from the Document Index table of contents — never
reading the exhibit content. This is unreliable: "AMENDMENT NO. 3" or
"GENERAL TERMS" could contain critical rate changes; the header alone
cannot distinguish them from truly irrelevant exhibits.
Gate 4 (has_reimbursements check after Step 1) already handles this
correctly by reading actual content. The header-based pre-filter added
correctness risk without proportional benefit.
Removed: DOCUMENT_INDEX_RELEVANCE_FILTER_ENABLED config flag, the Step 5
filter block in one_to_n_exhibit_chunking, reimbursement_relevant from
DOCUMENT_INDEX_INSTRUCTION output schema (back to 3 keys: page, header,
classification), TestRelevanceFilterLogic test class. Plan document
updated with reasoning.
Also fix document_index_offline.py to use split_text + lightweight page
wrapper instead of split_text_with_pages, avoiding expensive table
sp…
* Enrich comparison script with page spans and DI parent mapping
* Merged dev into feature/document-index
* Restructure Document Index quality gates: fuzzy match + failure-ratio + tail-coverage
Verifier semantics changed to lead with the index and only fall back on real
quality signals:
- Fuzzy header match: literal-with-whitespace match first, then rapidfuzz
partial_ratio (threshold 85) on the top of the page. Handles dash variants
and minor formatting differences.
- Failure-ratio gate: >10% of exhibit_start entries failing the fuzzy match
triggers fallback (parse is untrustworthy).
- Tail-coverage gate: replaces the old 'first exhibit must start at page 1'
rule, which fired falsely on contracts where page 1 is preamble. New rule
measures the trailing span: tail_pages = total_pages - max_exhibit_start.
For docs >15 pages, fallback when tail > max(15, 30%) of the document — the
signal that the LLM stopped parsing partway through the index.
- Thresholds are hardcoded module constants in preprocessing_funcs.py
(_DI_FUZZY_MATCH_THRESHOLD, _DI_MAX_FAILURE_RATIO, _DI_TAIL_COVERAGE_*) —
not config fla…
* Move Document Index testbed scripts to local_scripts (untracked)
The DI offline study and comparison tools are throwaway local utilities, not
production code. Moving them to local_scripts/ (already gitignored) keeps
src/testbed/ clean and removes them from the repo.
* Document Index: add SECTION as exhibit_start keyword, disambiguate sub-clauses
Empirical signal from a 100-file generic-corpus comparison run showed 37
legacy-only exhibit boundaries on three SECTION-style contracts (Gulf Coast
Division, UTMB Amendment 10, Rehab Designs) that DI was missing because
SECTION wasn't listed alongside EXHIBIT/ATTACHMENT/ARTICLE/etc. Also added
a counter-example to the landmark bullet so numeric sub-clauses like
'2.3 Confidentiality' or '10.4 Billing Procedures' stay out of exhibit_start.
Two unit tests pin the new behavior: SECTION appears in the exhibit_start
paragraph, and the numeric-sub-clause disambiguation appears in landmark.
* Trim Document Index tests: drop low-value prompt-content pinning, consolidate
41 tests for a ~200-line feature was over-investment. Cut to 25 by:
- Dropping prompt-string-content tests that pinned copy rather than behavior
(will-break-on-tuning, value-add zero) — non-empty-string, required-section
headers, noise-example pinning, SECTION/landmark prompt-content checks
- Removing trivial happy-path tests for DOCUMENT_INDEX factory (returns tuple,
embeds block, parser parses valid JSON) that duplicate parser-level coverage
in test_json_parsers.py
- Trimming TestPromptDocumentIndex to the three meaningful contracts:
usage_label, cache=True, malformed-JSON tolerance
- Folding TestVerifyIndexAgainstPagesBoundaries into the main verify class
and consolidating four 'doesn't crash' tests into a single sweep test
Net: 41 → 25 tests. Test density now matches comparable small features in
the codebase (test_clean_header_footer 13 tests, test_chc_extraction 6).
* Document Index: rewrite verified headers with page-text via single LLM call
The Document Index returns thin TOC entries ("EXHIBIT A - FEE SCHEDULE");
downstream code (chunk-boundary matching, dynamic-primary LLM input,
EXHIBIT_TITLE column, active-rates exhibit-title standardization) is
calibrated for the rich page-text version that the legacy per-page path
produced (multi-line headers with provider names, effective dates, etc.).
After DI parse + verification, bundle the full text of each verified page
under per-page boundary markers and run a single LLM call to extract the
page-formatted headers. Two-headers-on-one-page is supported by allowing
list values per boundary — that case is the original reason DI is the
preferred path. On any LLM/parse failure the original verified_dict is
preserved so the pipeline never breaks.
Cost stays bounded by count of header-bearing pages (typically 5-15 per
contract), independent of exhibit length. A 60-page contract with 8
exhibits costs 2 LLM calls (DI parse + bundle …
* Merge remote-tracking branch 'origin/dev' into feature/document-index
* Document Index: recover orphan-anchor pages dropped by TOC
When OCR strips the "ATTACHMENT X" / "Exhibit N" label from a Document Index
line, leaving only the secondary title (e.g. "MEDICARE ADVANTAGE COMPENSATION"
where "ATTACHMENT B" was expected), the DI LLM classifies the orphan secondary
as `landmark`, the page never enters verified_dict, and the exhibit is silently
dropped - even though the actual page begins with the anchor label.
Add a regex-based recovery pass that runs between verify_index_against_pages
and replace_index_headers_with_page_text. Scans every unverified page for a
strict whole-line header anchor, then applies five guards before promoting:
1. Strict whole-line regex (optional " - Secondary Title" suffix)
2. Length cap (anchors are short; <=80 chars)
3. Running-header filter (line on >=2 pages AND >40% of pages)
4. List detection (>=3 distinct anchors on one page -> body-text list, skip)
5. Continuation guard (same anchor as previous page -> multi-page exhibit, skip)
No new …
* Document Index: strip running-header bleed from page-text rewriter
The verified-pages header LLM was concatenating document running headers
(e.g. 'PROVIDER SERVICES AGREEMENT' repeated on every page) into the
rewritten exhibit title, producing entries like
'PROVIDER SERVICES AGREEMENTExhibit 1Services & Compensation'.
Lift the running-header set computation to the DI orchestrator so a single
set is shared between orphan-anchor recovery (Fix A) and the page-text
rewriter (Fix B). The rewriter now strips lines matching the set from each
page before bundling, so the LLM never sees the bleed and can't include it.
Verified on the 6 affected Clover contracts: DAVITA p7, Amedisys p34, HMH
p3, Inspira p4/p5, PHOEBE p17, CedarBridge p6 — all now produce clean
anchor + newline + secondary-title titles.
* Merge remote-tracking branch 'origin/dev' into feature/document-index
Approved-by: Karan Desai
Approved-by: Faizan Mohiuddin
Feature/DAIP2-2314 DAIP2 1687 hybrid
* remove -files from s3 prefix requirements
* Resolve input paths
* fix: VendorProcessor.process_file returns (df, None) tuple
runner.safe_process_file unpacks the result as (cc_df, dashboard_df), so
returning a single DataFrame caused every vendor/generic file to fail with
"too many values to unpack (expected 2)" — Python iterates DataFrame columns
during unpacking. Vendor pipelines have no dashboard variant; second slot is
None and the existing `dashboard_result is not None` guard in runner.py
already handles it.
Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>
* DAIP2-2314 + DAIP2-1687: pad DYNAMIC_PRIMARY + DYNAMIC_PRIMARY_ENTITY_CLASSIFICATION over 1024-token cache floor
- Pad DYNAMIC_PRIMARY_INSTRUCTION with three new sections: [SCOPE BOUNDARIES], [SOURCE TEXT INTERPRETATION], [REASONING DISCIPLINE], plus a [WORKED EXAMPLES] block. Estimated tokens: 447 -> 1117 (Sonnet 4.5 1024-min, +93 margin). All additions reinforce existing rules (sibling-field separation, alias mapping, pricing-vs-LOB distinction, contrastive-clause exclusion, exhibit-header binding) — no new directives that could bias extraction.
- Pad DYNAMIC_PRIMARY_ENTITY_CLASSIFICATION_INSTRUCTION with a [FINAL CHECKLIST BEFORE OUTPUT] block. Estimated tokens: 956 -> 1101 (Sonnet 4.5 1024-min, +77 margin). Reinforces the existing 4-step anti-duplication protocol and JSON shape requirements.
- Register both new entries in cache_registry: DYNAMIC_PRIMARY_ENTITY_CLASSIFICATION as INSTRUCTION_PLUS_CONTEXT (caches at warm-up), DYNAMIC_PRIMARY_ENTITIES as CONTEXT (instruction is intentionally short; CONTEXT c…
* black format fix
* Merged dev into feature/DAIP2-2314-DAIP2-1687-hybrid
* fixed raw lob values in base lob field mapping and composite entities fix
* black format fix
* fixed LOB Program output issues
* issue fixes
* remove debugging code
* Updated prompts
* updated additional instructions
* Update Program-->LOB
* LLM-based AD-Program/Product mapping to LOB even when there is a crosswalk
* black format fix
* Merged dev into feature/DAIP2-2314-DAIP2-1687-hybrid
* added logging in prompt call tracking
* added updated logging in prompt call tracking
* aaded min cache token per usage label
* added cache registry for dynamic primary mapping prompt calls
* reolved mapping prompts ambiguities
* black format fix
* Phase 2 modifications added
* reverted phase 2 modifications
Approved-by: Katon Minhas
Bugfix/molina ut dynamic primary
* Prompt changes for dynamic primary
* Route cover-sheet-only files to ERRORS.csv instead of leaking phantom rows
When every page of a contract was filtered out as a cover sheet / quick-review
form, process_file silently returned a FILE_NAME-only DataFrame. Because the
runner routes by checking for an "error" column, that file landed in
RESULTS.csv as a near-empty row and no ERRORS.csv was generated for the run.
- saas/file_processing.py: raise ValueError when text_dict is empty after
cover-sheet filtering, so safe_process_file produces a proper error row.
- runner.py: add _is_phantom_result defense-in-depth — promote any result
with no extracted fields beyond FILE_NAME to error_results with
error_type=PhantomSuccess.
* Merged dev into bugfix/molina_ut_dynamic_primary
* Tighten PRODUCT prompt: restrict to valid_values, prune LOB/PROGRAM examples
* Merge branch 'bugfix/molina_ut_dynamic_primary' of bitbucket.org:aarete/doczy.ai into bugfix/molina_ut_dynamic_primary
* Add PROVIDER_NAME and DISCOUNT_TERM to FIELD_FORMAT_MAPPING
* Revert "Route cover-sheet-only files to ERRORS.csv instead of leaking phantom rows"
This reverts commit de79a1391511acaf78c92a6bee53697f419d3996.
* Drop internal-only columns from final output
Four columns were leaking into RESULTS.csv that have no entry in
FIELD_FORMAT_MAPPING and shouldn't ship to production:
- DOCUMENT_TYPE: from the document classification stage, not a contract field
- REIMB_PAGE / EXHIBIT_TEXT: internal extraction-trace columns
- CODE_MAPPING_SOURCE: debug column
Drop them in standard_postprocess just before reorder_columns. Done after
attach_sid_column so AARETE_DERIVED_SID (which reads DOCUMENT_TYPE) still
gets populated correctly. Uses errors='ignore' so the drop is a no-op
when a column isn't present.
* Strict-filter columns to FIELD_FORMAT_MAPPING and tighten PRODUCT prompt
* Merged dev into bugfix/molina_ut_dynamic_primary
* Dedupe values within LOB/PROGRAM/PRODUCT/NETWORK fields
* Move hard-codes to within function
Approved-by: Katon Minhas
Bugfix/exhibit smart chunking cost improvements
* Add opt-in instrumentation for per-call token and row-count tracing
Introduce src/utils/instrumentation.py (thread-safe CSV logger) and
src/utils/instrumentation_context.py (ContextVar scope plus
submit_with_context / map_with_context helpers for propagating context
into ThreadPoolExecutor workers).
Emit events at every Bedrock call in llm_utils.invoke_claude, including
in-memory claude_cache hits, with full input/output/cache-read/cache-write
token breakdown. Emit row-count events at each row-mutating stage in the
one-to-N pipeline (clean_reimbursement_primary,
filter_services_without_reimbursements, methodology_breakout,
split_service_terms, carveout, dynamic_code_assignment,
lesser_of_distribution, dynamic_assignment) and chunking / retrieval
events in exhibit smart chunking (chunking_done, retrieval_done) plus
exhibit lifecycle events (exhibit_start, exhibit_gate_skip,
stage_transition).
All hooks are no-ops unless DOCZY_INSTRUMENTATION=1; production defaults
unchanged. Runner and inspector scripts in …
* Extend instrumentation: segment, cache_miss_reason, retry/error events, runtime hookup, analyzer
Schema expansion and new event types:
- Add segment column to every llm_call / llm_call_inmem_hit, resolved from
USAGE_LABEL_TO_SEGMENT (authoritative) with scope fallback. Mapping built
from grep + smoke-run ground truth; earlier guessed entries removed.
- Add cache_miss_reason classifier: hit / first_call / ttl_expired /
under_min_tokens / silent_miss / not_attempted. Uses a per-process
_cache_key_seen dict guarded by its own lock.
- Emit llm_retry on each retry attempt (attempt, error_class, error_msg,
backoff_sec) and llm_error on exhausted retries in ec2_claude_3_and_up
(rotation and non-rotation branches) plus local_claude_3_and_up.
- Add six new CSV columns: segment, cache_miss_reason, attempt,
error_class, error_msg, backoff_sec. Total schema now 34 columns.
Defensive kwarg hygiene:
- _ctx_minus_explicit_keys filter on all emit sites to prevent
segment / filename kwarg collisions between …
* Merged dev into bugfix/exhibit-smart-chunking-cost-improvements
* Merged dev into bugfix/exhibit-smart-chunking-cost-improvements
* Make instrumentation tracking on by default
Flip the gate: instrumentation is now enabled unless DOCZY_INSTRUMENTATION
is explicitly set to a falsy value (0/false/no/off). Replaces the prior
opt-in behaviour where it was off unless DOCZY_INSTRUMENTATION=1 was set.
* Fix missing WRITE_TO_S3 patch in upload_instrumentation_csv test
* Merged dev into bugfix/exhibit-smart-chunking-cost-improvements
Approved-by: Katon Minhas
CARVEOUT_CD issue fixed
* CARVEOUT_CD issue fixed
* pipeline error fixed
* Merged dev into bugfix/DAIP2-2524-carveout-code-optimization
* Merged dev into bugfix/DAIP2-2524-carveout-code-optimization
* Merged dev into bugfix/DAIP2-2524-carveout-code-optimization
* trigger cap issue fixed
* trigger cap prompt updated
* Merged dev into bugfix/DAIP2-2524-carveout-code-optimization
Approved-by: Katon Minhas
Feature/PC logic cleanup output
* Few tweaks PC_logics
* Fixed orphan_ranking
* black formatting
* Changes on output field and ranking method
* updated few hotfixes
* black format fix
Approved-by: Katon Minhas
Feature/DAIP2 pacificsource reimbursements issues
* Tighten PREMIUM_TERM and DISCOUNT_TERM classifier prompts
CARVEOUT_CHECK was misrouting table rate rows into special-case fields,
dropping them from the reimbursement output:
- "110% of CMS allowed" (base fee-schedule rates) was being classified as
PREMIUM_TERM because the prompt treated "above 100% of reference" as an
implicit premium. Seen on PacificSource Medicare_Attachment_A1 and A2
Facility contracts where Inpatient/Outpatient rows were missing (2556)
or silently fell back to 100% fee-schedule (2557).
- Per-service discount rates like "Progressive Lenses: 15% discount",
"Contact Lenses: 2% discount", "Frame: 20% discount" were being
classified as DISCOUNT_TERM because the prompt only required the word
"discount" to appear. Seen on PacificSource Commercial_Attachment_A2
and A5 Professional contracts (2558, 2559).
Prompts now require the literal keyword AND explicitly exclude the
common false-positive patterns, keeping per-service rate rows in the
reimbursement output.
* Stop splitting multi-page exhibits on repeated page footers
Two bugs were collaborating to split a single Attachment into several
Exhibit objects, causing intra-exhibit lesser-of search to miss notes
living on a later page of the same exhibit:
1. EXHIBIT_HEADER extraction picked up page-footer lines as if they
were new section headers. On PacificSource contracts, pages end with
a short-form repeat like "Attachment A Oregon Health & Science
University" next to the page number and contract date, which the
ATTACHMENT-prefix regex and the LLM both accepted as a header.
2. EXHIBIT_HEADER_DEDUP parsing required the LLM output to be wrapped
in a strict |pipes| JSON block. When the LLM prefixed the response
with prose ("# Analysis ..."), parsing threw ValueError and the
code fell back to the original, un-deduped dict — defeating dedup
entirely.
Combined, these produced 3 Exhibit objects for Commercial_Attachment_A2
(page 1 header + two entries on page 2: repeated header + footer),
leaving …
* Merged dev into feature/DAIP2-pacificsource-reimbursements-issues
Approved-by: Katon Minhas
fix: fall back to PROVIDER_NAME in PROV_INFO_JSON when no TIN/NPI extracted
* fix: fall back to PROVIDER_NAME in PROV_INFO_JSON when no TIN/NPI extracted
When get_prov_info_json short-circuits due to no TIN/NPI regex matches,
PROV_INFO_JSON was left as [] even when PROVIDER_NAME was successfully
extracted via the one-to-one pipeline. This caused inconsistent output
across contracts with the same provider — some files produced a NAME-only
entry (via a false-positive regex hit triggering the LLM), others produced [].
Reconcile at add_group_and_other, the first point where both extraction
streams' results are available. When PROV_INFO_JSON is empty but
PROVIDER_NAME is known, synthesize a NAME-only entry with IS_GROUP:"Y" and
populate PROV_GROUP_NAME_FULL directly — skipping the provider_name_match_check
LLM call since the match is tautological by construction.
Adds 5 unit tests covering the str, list, already-populated, empty-name,
and all-empty-list cases.
Approved-by: Katon Minhas
Updated feature> dev gate to print variables of echo statements
* Updated feature> dev gate to print variables of echo statement
* Reverted placeholder changes
Approved-by: Sujit Deokar
Feature/standardized services
* service term standardization
* prompt update
* prompt updates for standardization
* only service standardization
* new file
* add supporting files and test scripts for standardization work
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
* remove old files
* Merge remote-tracking branch 'origin/dev' into feature/standardized-services
* final fixes
* Merged dev into feature/standardized-services
* additional features
* removed unwanted files
* remove unwanted files
* Merge branch 'dev' into feature/standardized-services
* Merge remote-tracking branch 'origin/dev' into feature/standardized-services
* reversed vendor changes
* Merged dev into feature/standardized-services
* addressed PR comments
* Merge branch 'dev' into feature/standardized-services
* Merged dev into feature/standardized-services
* addressed 3 remaining comments inPR
* black formatting
* incorporated feedback from AI code reviewer
* reused existing functionality
Approved-by: Katon Minhas
Remove curly braces from echo statements in dev->stg
* Remove curly brances from echo statements in dev->stg
* Removed curly braces in echo statements in feature-> dev gate
Approved-by: Sujit Deokar
pre_doczy_ru_base_prefix_changes
* pre_doczy_ru_base_prefix_changes
* Merged dev into feature/Pre_doczy_report
* Merged dev into feature/Pre_doczy_report
Approved-by: Katon Minhas
Feature/upDatepipelinesFile
* Added logic to capture failure status for invalid branch names
* Merged dev into feature/upDatepipelinesFile
* Bugfix the print logs were not able to populate the placeholders. These are now fix to use the sh file
* Merge branch 'dev' into feature/upDatepipelinesFile
Added logic to capture failure status for invalid branch names
* Added logic to capture failure status for invalid branch names
* Merged dev into feature/upDatepipelinesFile
Return None for dashboard output when dashboard postprocessing is off
* Return None for dashboard output when dashboard postprocessing is off
FINAL_RESULT_DF_DASHBOARD was initialized as an empty DataFrame even
when RUN_DASHBOARD_POSTPROCESSING was False, causing downstream code
to needlessly process it (reorder_columns, etc). Now returns None
when dashboard is not requested, matching the postprocess() contract.
* Merged dev into bugfix/retire_stale_client_file_processing
* Merge dev (with revert) into feature branch
* Re-apply retire stale client file_processing changes
Revert of the revert (4f53b528) to restore the original changes
from the feature branch for proper PR review.
* Merge branch 'bugfix/retire_stale_client_file_processing' of bitbucket.org:aarete/doczy.ai into bugfix/retire_stale_client_file_processing
Approved-by: Katon Minhas
PR #959 was merged into dev without approval. This reverts commits
5e143c10, 63f32c41, 849aa626, and 927abcae to restore dev to its
pre-merge state. The changes will be re-submitted via a new PR
after proper review.
reorder_columns() was only keeping columns listed in FIELD_FORMAT_MAPPING,
silently discarding any extra columns like BCBS OFFSET_TERM/OFFSET_INDICATOR
and Clover's 12 full_context fields. The function's docstring documented
appending extra columns but the implementation was missing that step.
1. Tag format validation in release-prod (MEDIUM, defensive).
Before parsing major/minor/patch from the latest tag, assert it
matches ^v[0-9]+\.[0-9]+\.[0-9]+$. Empirically tested: catches
v1, v1.2, v1.2.3.4, v1.0.0-rc1, v1.2.3-beta+build, and still
accepts v0.0.0 (the first-release fallback).
Without this, bash arithmetic silently mangles non-semver tags
into wrong results via its 'treat empty/non-numeric as 0' rule.
Worst case: tag 'v1' → cut -d. -f2/-f3 both return '1' →
minor bump produces v1.1.2 instead of v1.0.1. No visible error.
2. Rewrite the AI review comment to match actual behavior.
The old comment said 'advisory, not a gate' but also admitted
git clone / apt-get are hard failures, contradicting itself.
The new comment makes the runtime vs. infrastructure distinction
explicit: runtime failures (agent crash, STS errors, Python
exceptions) become warnings via || echo; infrastructure failures
(missing token, clone failure, apt-get failure) stay hard.
This is a doc fix, not a code change. Deliberately preserving
the hard-fail behavior on config errors because silent
degradation of AI review is the worst outcome — feature
disappears from CI with no signal.
Explicitly rejected from the second review (empirically verified):
- Gate logic 'fragile' claim: tested in bash, correct for all
our hardcoded ALLOWED values.
- ROLLBACK_TAG -z check 'passes empty': tested, -z correctly
catches empty strings. Reviewer has the semantics wrong.
- chmod 600 + set +x: our comment already documents chmod 600
as cosmetic/scanner-silencing; set +x wouldn't help because
Bitbucket's line-echo is runner-level, not bash set -x.
- cd/python refactor (previous round): already rebutted in
commit 4836e36f via empirical bash AND-OR list tests.
- Option B for git clone wrapping: silent degradation of
config errors is worse than the current loud-fail behavior.
1. Replace 'uv sync --frozen' with 'uv sync --locked'. --frozen silently
uses a stale lockfile if pyproject.toml is updated without regenerating
uv.lock, masking missed dependencies. --locked fails loudly with a clear
error when the lockfile is out of sync. Verified empirically with uv
0.9.14: --frozen exits 0 on mismatch, --locked exits 1.
2. Add 'chmod 600' on the OIDC token file after writing it. The container
is already single-user root so this is defensive/cosmetic, but it
silences security scanners and signals intent.
3. Add a comment explaining why gate steps use atlassian/default-image:4
instead of python:3.12.7 (gates run bash only, no Python toolchain
needed — lighter image, faster pull).
Explicitly rejected from the PR review:
- Token-masking sed mitigation (delimiter collision with / in real
Bitbucket clone tokens; cmd | sed || exit 1 swallows git failures
without set -o pipefail). Bitbucket's built-in Secured variable
masking is the correct mitigation and is already documented as a
setup requirement.
- SSH key alternative for git clone (architecturally worse — more
secrets to manage; HTTPS+Secured is the Bitbucket-recommended pattern).
- cd/python refactor ('fragile logic bug'). Verified empirically that
'cd X && python Y || echo Z' with set -e correctly catches both
cd and python failures via the ||. The step exits 0 as intended
by the 'advisory, not a gate' design. The suggested refactor would
introduce a hard-fail regression.
The unquoted ': ' (colon + space) in 'WARNING: AI review step failed'
was being interpreted as a YAML mapping key-value separator, causing
the entire script item to be parsed as a dict instead of a command
string. Bitbucket then rejected it with 'Missing or empty command
string' error at pull-requests > feature/* > 2 > step > script > 9.
Replaced the colon with a hyphen. Validated with yaml.safe_load that
all 10 script items in the ai-code-review step now parse as strings.
Bitbucket's YAML parser was interpreting the deeper-indented comment
after the printf line as a phantom empty list item, causing a
'Missing or empty command string' error at script item 9.
BCBS: change strict == "YES" to "YES" in final_answer. The strict
equality was rejecting borderline LLM responses (e.g. "YES, ..."),
dropping row counts from ~7 to 1.
Clover: remove .strip().upper() which crashed with AttributeError
because the parser returns a list, not a string. Every validation
call failed, dropping row counts from 106 to 1.
Both overrides now use the same "YES" in final_answer logic as SaaS.
E2E verified: BCBS 5 rows (within LLM variance of baseline 7),
Clover 106 rows (exact match to baseline).
Delete BCBS/Clover/CHC client file_processing.py files (1090 lines of
96% stale drift with 3 crash points). Recover the 4% real business
logic (BCBS OFFSET_TERM, Clover 12 full_context fields) into the
shared pipeline via config-driven field loading.
Make runner.py the canonical pipeline entry point by merging all
production features from main.py (timing, duplicate detection, aarete
derived fields, TIN statistics, column reordering). Reduce main.py to
a thin wrapper that delegates to runner.main().
Fix the root cause of all client overrides being dead in production:
set_active_client is now called with the correct client name instead
of being hardcoded to "saas". Both resolver shims (file_processing
and prompt_calls) now route correctly.
Includes temporary [DEBUG ROUTING] print statements at 6 routing
decision points for E2E verification. Cherry-pick this commit to
restore debug instrumentation for future testing.
E2E verified: BCBS Promise and Clover both pass with correct routing.
Bugfix/generic issue fixes
* Patch for llm responses
* Merged dev into bugfix/generic-issue-fixes
* Update Exhibit-Level instruction for Claim Type and Bill Type
* Update Service-Level instruction for Claim Type and Bill Type
* Update Bill Type code to prompt for DESC field
* Black format
* Remove test
Approved-by: Siddhant Medar
Faizan Mohiuddin