Faizan Mohiuddin
339cfc74fc
Feature/one to one confidence scoring
* T1 plumbing: capture per-field confidence + retrieved-chunk metadata for 1:1 HSC fields
Prep work for the 1:1 confidence-scoring stage. No scoring logic yet — this
just collects the inputs the next ticket (rule-based scorer) will consume.
- ONE_TO_ONE_SINGLE_FIELD_TEMPLATE: ask the LLM for confidence (0.0-1.0),
verdict (correct/uncertain/not_found), and supporting_snippet alongside
the field value. Existing field parser passes the extra keys through
unchanged.
- prompt_hsc_single_field: now returns a 4-tuple (name, value, field,
metadata) where metadata holds the confidence/verdict/snippet plus a
lightweight summary of which chunks the LLM saw (count + ids).
_extract_hsc_metadata is defensive: clamps out-of-range confidences,
defaults a missing/garbage verdict, caps the snippet at 500 chars,
returns _empty_hsc_metadata() on every bail-out path.
- run_hybrid_smart_chunked_fields: optional return_metadata flag. Default
return shape unchanged (dict of values) for backwards compatibility;
wh…
* DAIP2-2692: field-type-aware rule scorer for 1:1 confidence
Replaces the T1 placeholder _CONF (just the LLM-stated confidence) with
a rule-based score per field. No new LLM calls -- pure local computation.
New module: src/qc_qa/confidence/
- field_types.py : FieldType enum + per-field mapping + weight presets
(date / money / tin / npi / code / name / boolean /
free_text). New 1:1 fields default to FREE_TEXT.
- signals.py : exact / token_overlap / fuzzy / number_match / regex /
grounding / na_check. Each signal returns float in
[0,1] or None when not applicable.
- scorer.py : score_field(field_name, value, metadata, contract_text,
field_prompt) -> float. Runs the signals that have
non-zero weight for the field's type, renormalizes
over the signals that actually fired, blends lightly
with the LLM-stated confidence (15%…
* DAIP2-2694: integrate calibration table into 1:1 confidence scoring
Adds a per-field-type calibration step that maps raw confidence scores
to empirically-calibrated values via a linearly-interpolated curve.
Identity by default, so the stage is a no-op until the curve is
backfilled from QC data; once backfilled, a "0.X confidence" output
reflects roughly X% empirical accuracy.
New files:
- src/qc_qa/confidence/calibration.py
* loads + caches the curve from JSON
* linear interpolation between anchor points
* defensive: bad inputs / missing file / malformed table all fall
back to identity (no scoring regression on failure)
- src/qc_qa/confidence/calibration_table.json
* identity baseline shipped as the starting point
* one curve per FieldType plus a 'default' fallback
* documents the regeneration entry point for the QC-data backfill
.gitignore:
* carve-out !src/qc_qa/confidence/*.json so the table actually ships
Scorer wiring:
* apply calibration AFTER the r…
* DAIP2-2695: field-level confidence distribution stats + flagged-row report
At end of every run, emit two CSVs under tracking/:
- <BATCH_ID>-CONFIDENCE-SUMMARY.csv : per-field distribution stats
columns: field, n, mean, p10, p50, p90, n_below_threshold,
threshold, pct_below_threshold
sorted by pct_below_threshold descending so the worst fields are
at the top of the file.
- <BATCH_ID>-CONFIDENCE-FLAGGED.csv : the reviewer hit list
columns: FILE_NAME, field, confidence, value, threshold
sorted by confidence ascending.
Both are derived from the per-row <FIELD>_CONF columns the rule scorer
populates. Empty inputs (no _CONF columns yet, or no rows) skip the
write rather than emitting empty files.
Threshold:
* config.CONFIDENCE_THRESHOLD (CLI arg confidence_threshold=, default 0.6)
New module:
- src/qc_qa/confidence/summary.py
compute_summary(final_df, threshold) -> per-field stats DataFrame
compute_flagged(final_df, threshold) -> below-threshold r…
* Skip _CONF columns in standard_postprocess value-shape normalizers
The per-column loop in standard_postprocess substring-matches column
names ("_IND" in col, "TIN" in col, "_DT" in col, etc.) to decide which
shape normalizer to apply. After DAIP2-2692 introduced <FIELD>_CONF
confidence-score columns, those columns started colliding with the
match conditions:
- AUTO_RENEWAL_IND_CONF matched "_IND in col" -> normalize_indicator_field
coerced the 1.0 float to the string "N", destroying the score.
- *_DT_CONF / *_TIN_CONF etc. were also at risk via the same pattern.
Fix: short-circuit the loop with `if col.endswith("_CONF"): continue`
so confidence floats are never passed through the value-shape
normalizers, regardless of what string happens to appear in the
underlying field name.
Verified on the 1:1 smoke test (conf-1to1-smoke2): AUTO_RENEWAL_IND_CONF
now lands as 1.0 in RESULTS-FULL.csv and CONFIDENCE-SUMMARY.csv shows
all 10 HSC fields with proper float distributions.
* Merge remote-tracking branch 'origin/dev' into feature/one-to-one-confidence-scoring
* Black formatting: split generator in reorder_columns _CONF carve-out
Black wanted the multi-line generator expression formatted with one
clause per line. Behaviour unchanged.
* Fix na_check so amendment docs differentiate from base agreements
Three small bugs were stacking up to make CONTRACT_AMENDMENT_NUM flag every
file that returned a missing value, regardless of whether the document
actually was an amendment.
1) is_na() did not treat pandas/float NaN as N/A. When the LLM returned
no value, the cell came through as a float NaN, str()'d to "nan", and
the N/A token set ({"", "n/a", "na", "none", "null", "not found"})
did not match. na_check therefore never fired and the other signals
ran on the literal string "nan", producing a flat low score driven
only by the LLM's self-reported confidence. Now also catches float
NaN via value != value.
2) signal_na_check used three coarse buckets (0.30/0.50/0.85) and a
per-keyword saturation of 2 mentions. Both produced the same bucket
for docs that mention a concept 2x in boilerplate and docs whose
entire topic is that concept. Switched to a continuous score with
a saturation floor of ~30 mentions, so 8 mentions…
* Merged dev into feature/one-to-one-confidence-scoring
Approved-by: Katon Minhas
126 lines
1.4 KiB
Plaintext
126 lines
1.4 KiB
Plaintext
# Python
|
|
*.py[cod]
|
|
*.pyo
|
|
__pycache__/
|
|
.mypy_cache/
|
|
.pytest_cache/
|
|
myenv/
|
|
*.egg-info/
|
|
|
|
# IDE
|
|
.idea/
|
|
.cursor/
|
|
.vscode/
|
|
.claude/
|
|
CLAUDE.md
|
|
|
|
# Claude (local agent notes)
|
|
.claude/
|
|
CLAUDE.md
|
|
|
|
# OS
|
|
.DS_Store
|
|
Thumbs.db
|
|
*:Zone.Identifier
|
|
|
|
# Logs
|
|
*.log
|
|
|
|
# Environment
|
|
.env
|
|
*.env
|
|
|
|
# Data files (general exclusion)
|
|
*.csv
|
|
*.xlsx
|
|
*.xlsb
|
|
*.zip
|
|
*.json
|
|
*.txt
|
|
*.pdf
|
|
*.PDF
|
|
*.TXT
|
|
|
|
# Allow tracked data files in src/
|
|
!src/constants/**/*.csv
|
|
!src/constants/**/*.json
|
|
!src/prompts/*.json
|
|
!src/qc_qa/confidence/*.json
|
|
!requirements.txt
|
|
|
|
# Build artifacts
|
|
dist/
|
|
build/
|
|
target/
|
|
*.class
|
|
*.jar
|
|
*.war
|
|
*.app
|
|
*.exe
|
|
|
|
# Large media files
|
|
*.mp4
|
|
*.tiff
|
|
*.avi
|
|
*.flv
|
|
*.mov
|
|
*.wmv
|
|
|
|
# Terraform
|
|
.terraform/
|
|
terraform.tfstate*
|
|
.terraform.lock.hcl
|
|
*.tfplan
|
|
*.pem
|
|
textract-pipeline/terraform/builds/
|
|
|
|
# Outputs
|
|
output/
|
|
outputs/
|
|
|
|
# Local/temporary
|
|
subset/
|
|
docs/
|
|
local_scripts/
|
|
untracked_tests/
|
|
new/
|
|
textfiles/
|
|
texts/
|
|
|
|
# Streamlit (legacy)
|
|
streamlit/history.csv
|
|
streamlit/RESULTS
|
|
streamlit/DB/
|
|
streamlit/RAW_DOCUMENTS/
|
|
streamlit/SOURCE_DOCUMENTS/
|
|
streamlit/contract_field_values.csv
|
|
streamlit/contract_fields.csv
|
|
streamlit/sample.csv
|
|
streamlit/temp*.csv
|
|
streamlit/results.csv
|
|
streamlit/venv
|
|
|
|
# Legacy paths
|
|
fieldExtraction/embeddings/
|
|
|
|
# Node (if any)
|
|
node_modules/
|
|
|
|
# Test reports
|
|
TEST*.xml
|
|
|
|
# Shell scripts (debugging/utility)
|
|
*.sh
|
|
!resolve_source_branch.sh
|
|
|
|
# PRD Documents
|
|
*.prd
|
|
*prd.md
|
|
.git.instructions.md
|
|
|
|
# Local code index
|
|
.claude/index/
|
|
|
|
# Local output / offline study artifacts
|
|
out/
|