863c9db9a1649cd8bb6fc42eb19c0edcf9216030
5 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Praneel Panchigar
|
863c9db9a1 |
Merged in feature/code_optimization (pull request #903)
Feature/code optimization
* Ran Black
* made a small change in code_last_check, fixed so it returns string and not single char
* Made changes to make sure that default_ind postprocess only happens to the cc output and not dashboard
* Merge bugfix/default_ind_postprocess into feature/code_optimization
- Parent-child: child_rank column init and cols_to_keep filter
- Postprocess: default_ind only on cc output
- code_last_check: parser str/list return handling + exception logging
- Prompt: FIELD ASSIGNMENT for code extraction (no CRITICAL prefix)
* More prompt changes and debugging print statements
* Added system to group same service term + bill type cd + claim type cd and then parallelize the code extraction (will improve consistency)
* Revert one-time PROV_INFO_JSON ad hoc logic; retain json_utils and output format
- Remove temporary postprocess step that filled empty TIN in PROV_INFO_JSON from
FILENAME_TIN (fill_prov_info_tin_from_filename_tin call in postprocess.py).
This behavior is intended to move upstream per JIRA (tin_npi_funcs / extraction).
- Revert postprocess_existing_output.py to generic config: empty INPUT_DIR,
INPUT_FILENAME, OUTPUT_CSV. Remove one-time hyphen-strip for TIN/NPI in
PROV_INFO_JSON and remove _strip_hyphens_from_prov_info_json_cell.
- Keep json_utils PROV_INFO_JSON helpers (parse_prov_info_json_cell,
serialize_prov_info_json, format_prov_info_json) and (str,str) serialization
in contract_config_postprocess. Keep fill_prov_info_tin_from_filename_tin and
related helpers in postprocessing_funcs for potential upstream reuse.
- Tests in test_json_parsers.py updated for (str,str) PROV_INFO_JSON output.
* Add arbitration-based implicit code flow and prefer Level 2 RAG (Fix 5d, 4c+7)
Implicit flow no longer uses first-match-wins. We build a case from all
applicable strategies and run a single arbitration step to choose the
best candidate.
- build_implicit_candidates: Run Category (when explicit had Category),
Special, and RAG; return list of {source, code_answer_dict}. RAG
result uses preferred level when both return (Fix 5d).
- code_implicit_rag (Fix 5d): Run both Level 1 and Level 2, collect
results; when both return non-empty, prefer Level 2 over Level 1.
- code_implicit_arbitration: 0 candidates -> None; 1 candidate -> return
it with CODE_METHODOLOGY = 'Implicit - Arbitration (Source)' (no LLM);
2+ candidates -> one CODE_IMPLICIT_ARBITRATION LLM call (chosen_index or
no_match). If no candidates or no_match, code_last_check runs as before.
- prompt_templates: Add CODE_IMPLICIT_ARBITRATION and
CODE_IMPLICIT_ARBITRATION_INSTRUCTION (cached). Use
_create_json_dict_parser(field_names=['chose…
* fix: test_code_breakout avoid LLM call and flaky order on CI
- Add BILL_TYPE_CD_DESC to test data so pre-pass skips fill_bill_type (no Bedrock on CI)
- Use side_effect function keyed by SERVICE_TERM for deterministic result under ThreadPoolExecutor
* ran black
* feat: hybrid code validation with unmatched tagging and code quality fixes
Validation and unmatched tagging:
- Add format-only filtering for explicit codes (drop wrong-format, preserve valid)
- Add unified _has_unmatched_codes() used by both explicit and implicit paths
- Tag CODE_METHODOLOGY with "- Unmatched" when codes are format-valid but
not explicit keys in our mappings (codes and range keys count as mapped)
- Remove retry logic and "Explicit - Validation Failed" in favor of single
LLM call with Unmatched tagging
- Include all revenue level mappings in valid revenue set
Code quality and bug fixes:
- Fix fill_bill_type += string bug (was splitting "11X" into chars)
- Fix bare except clauses in code_explicit, code_category,
code_implicit_special, fill_bill_type (now except Exception as e)
- Replace eval() with json.loads() in fill_grouper_cd_desc
- Implement INVALID_SERVICE short-circuit in RAG (Fix 5a)
- Cache valid procedure/revenue code sets per Constants instance
- Store mappings in level_dicts…
* chore: remove debug print statements from code extraction pipeline
Remove all [DEBUG_CODE_EXTRACTION] print statements added during
development of the hybrid validation and implicit arbitration flow.
* refactor: robust PROV_INFO_JSON sanitization with multi-format parsing
Replace the simple format_prov_info_json with a layered
sanitize_prov_info_json that handles all known malformed variants:
single-quoted dicts, empty-value-after-colon patterns, list-typed
field values, and hyphenated TINs. Extract _normalize_prov_entries
and _prov_value_to_str helpers for uniform str-valued output.
format_prov_info_json now delegates to sanitize_prov_info_json.
* merge: sync feature/code_optimization with DEV
Resolve conflict in prompt_templates.py: keep detailed CRITICAL field
assignment instructions from feature branch; accept new
AARETE_DERIVED_PROVIDER_NAME prompt function from DEV.
* Minor change to explict prompt
* fix: resolve mypy errors and remove stale pipe-delim splits in RAG
- Narrow grouper_cd type to str|None with explicit None check to
satisfy mypy in fill_grouper_cd_desc.
- Remove dead pipe-delim split logic from CPT and HCPCS matching
blocks (no pipes in those mappings).
- Retain pipe split for revenue block only (rev_level1.csv keys are
pipe-delimited, e.g. "0810|0811|0812").
* Code extraction: format-only explicit, keep unmapped tag, RAG Level 1/2 as separate candidates
- Explicit: accept all format-valid codes (single + range); do not drop for mapping.
Validity check only appends "- Unmatched" to methodology when code not in mappings.
- Add _procedure_code_or_range_format_valid; use in validate_explicit_codes and _has_unmatched_codes.
- RAG: return list of Level 1 and Level 2 candidates (distinct source keys); remove Level 2 preference.
- build_implicit_candidates: consume RAG list and append each as separate candidate.
- CODE_EXPLICIT: clarify range extraction when service term describes a range.
- CODE_IMPLICIT_ARBITRATION: choose most appropriate candidate(s), avoid overly broad.
- Tests updated for new RAG return shape and validate_explicit_codes behavior.
* Resolved Merge Conflicts
* Fixed failing tests
* Remove unused functions
* Remove unused functions
* Revert "Remove unused functions"
This reverts commit 39954db6a421e55501282d4e6a270c88624049fe.
* remove unused
* Merged DEV into feature/code_optimization
* Merged DEV into feature/code_optimization
Approved-by: Katon Minhas
|
||
|
Venkatakrishna Avula
|
6744c57f95 |
Merged in bugfix/DAIP2-1870-dynamic-issues (pull request #892)
Bugfix/DAIP2-1870 dynamic issues * updated list format appending * fill bill type from claim type cd * Fixed remaining format fixes * Merged DEV into bugfix/DAIP2-1870-dynamic-issues * changed crosswalk mapping from PRODUCT to AARETE_DERIVED_PRODUCT * Merged DEV into bugfix/DAIP2-1870-dynamic-issues * updating LOB field values as well from PRODUCT AND PROGRAM * updated apply crosswalk for reverse mapping * Merged DEV into bugfix/DAIP2-1870-dynamic-issues * added QHP and BHP * removed lob reverse mapping * Merged DEV into bugfix/DAIP2-1870-dynamic-issues * Remove Bill Type/Claim Type fill * Merged DEV into bugfix/DAIP2-1870-dynamic-issues * Black format * Merged DEV into bugfix/DAIP2-1870-dynamic-issues Approved-by: Katon Minhas |
||
|
ppanchigar
|
9e4b88395b | Pipe line general fixes, fixed tests, ran black, and fixed mypy type check issues, still have a list of lists issue with prov other name full | ||
|
Katon Minhas
|
e8e8f1ba5a | Strip out dynamic postprocessing | ||
|
Katon Minhas
|
afb6d5185d |
Merged in feature/lesser-table-caching-refactor-hybrid (pull request #847)
Feature/lesser table caching refactor hybrid * chore: Remove unused duplicate main.py from shared pipeline * fix: Correct crosswalk paths in aarete_derived.py * chore: Remove unused documentation files from fieldExtraction * docs: Add documentation files to documentation folder * docs: Update README with uv setup, expanded project structure, and branching conventions * docs: Add uv installation steps with Ubuntu/WSL emphasis * Enable prompt caching for all remaining LLM calls - Add _INSTRUCTION() functions for: EXHIBIT_HEADER, EXHIBIT_LINKAGE, EXHIBIT_TITLE_MATCH, DATE_FIX, DERIVED_TERM_DATE, CHECK_PROVIDER_NAME_MATCH, SPECIAL_CASE_ASSIGNMENT - Update all invoke_claude() calls in saas and clover pipelines to use cache=True with corresponding _INSTRUCTION() functions - Add new instructions to get_cacheable_instructions() for cache warming - Update tests for new instruction functions Functions now using caching: - prompt_exhibit_level - prompt_exhibit_lesser (EXHIBIT_LEVEL_LESSER_OF) - prompt_fee_schedule_breakout - prompt_grouper_breakout - prompt_special_case_assignment - prompt_exhibit_linkage - prompt_exhibit_header - prompt_smart_chunked (ONE_TO_ONE templates) - prompt_date_fix - prompt_derived_term_date - prompt_exhibit_title_match - provider_name_match_check 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * Reorder * feat: Add bcbs_promise client pipeline with OFFSET_TERM extraction - Add new bcbs_promise client with HSC-based OFFSET_TERM field extraction - Extract full paragraph text of offset/recoupment provisions from contracts - Derive OFFSET_INDICATOR (Y/N) from OFFSET_TERM presence - Fix reorder_columns to preserve extra columns not in COLUMN_ORDER - Update QC/QA output path to outputs/qc_qa/ * fix: Update dev deps and test assertions for QC/QA output path - Add pytest/pytest-mock to dev dependencies for mypy type checking - Update test assertions to expect outputs/qc_qa instead of qa_qc_output * style: Apply black formatting to prompt_templates.py * Merge main, move scripts * Archive some scripts * update py version * remove .py version file * Remove ASCII characters * Restore testbed code * restore tracking * Update testbed metrics * Enable prompt caching for CODE_LAST_CHECK, FILL_BILL_TYPE, DUAL_LOB_CHECK, and GROUPER_BREAKOUT - Add CODE_LAST_CHECK_INSTRUCTION() for service specificity classification - Add FILL_BILL_TYPE_INSTRUCTION() for bill type code determination - Add DUAL_LOB_CHECK_INSTRUCTION() for Medicare/Medicaid classification - Update code_funcs.py to use caching for CODE_LAST_CHECK, FILL_BILL_TYPE, GROUPER_BREAKOUT - Update postprocessing_funcs.py to use caching for DUAL_LOB_CHECK - Add new instructions to get_cacheable_instructions() for cache warming - Add unit tests for new instruction functions 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * Fix postprocessing_funcs to remove invalid columns * Merge branch 'main' into feature/lesser-table-caching-refactor-hybrid * Revert prompt caching changes from aed1b73c * update formatting * Update imports Approved-by: Sha Brown Approved-by: Praneel Panchigar |