main
33 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Katon Minhas
|
623799f2a6 |
Merged in stg (pull request #1003)
Merged in dev (pull request #1001) * Merged in feature/fixPlaceholder2 (pull request #979) Updated feature> dev gate to print variables of echo statements * Updated feature> dev gate to print variables of echo statement * Reverted placeholder changes Approved-by: Sujit Deokar * Merged in bugfix/prov_info_json_fixes (pull request #981) fix: fall back to PROVIDER_NAME in PROV_INFO_JSON when no TIN/NPI extracted * fix: fall back to PROVIDER_NAME in PROV_INFO_JSON when no TIN/NPI extracted When get_prov_info_json short-circuits due to no TIN/NPI regex matches, PROV_INFO_JSON was left as [] even when PROVIDER_NAME was successfully extracted via the one-to-one pipeline. This caused inconsistent output across contracts with the same provider — some files produced a NAME-only entry (via a false-positive regex hit triggering the LLM), others produced []. Reconcile at add_group_and_other, the first point where both extraction streams' results are available. When PROV_INFO_JSON is empty but PROVIDER_NAME is known, synthesize a NAME-only entry with IS_GROUP:"Y" and populate PROV_GROUP_NAME_FULL directly — skipping the provider_name_match_check LLM call since the match is tautological by construction. Adds 5 unit tests covering the str, list, already-populated, empty-name, and all-empty-lis… * Merged in feature/DAIP2-pacificsource-reimbursements-issues (pull request #982) Feature/DAIP2 pacificsource reimbursements issues * Tighten PREMIUM_TERM and DISCOUNT_TERM classifier prompts CARVEOUT_CHECK was misrouting table rate rows into special-case fields, dropping them from the reimbursement output: - "110% of CMS allowed" (base fee-schedule rates) was being classified as PREMIUM_TERM because the prompt treated "above 100% of reference" as an implicit premium. Seen on PacificSource Medicare_Attachment_A1 and A2 Facility contracts where Inpatient/Outpatient rows were missing (2556) or silently fell back to 100% fee-schedule (2557). - Per-service discount rates like "Progressive Lenses: 15% discount", "Contact Lenses: 2% discount", "Frame: 20% discount" were being classified as DISCOUNT_TERM because the prompt only required the word "discount" to appear. Seen on PacificSource Commercial_Attachment_A2 and A5 Professional contracts (2558, 2559). Prompts now require the literal keyword … * Merged in bugfix/filter-docusign-lines (pull request #983) remove docusign lines * remove docusign lines * add unit tests for clean_header_footer docusign/deleted_lines changes Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * black forrmatting Approved-by: Katon Minhas * Merged in feature/PC_logic_cleanup_output (pull request #984) Feature/PC logic cleanup output * Few tweaks PC_logics * Fixed orphan_ranking * black formatting * Changes on output field and ranking method * updated few hotfixes * black format fix Approved-by: Katon Minhas * Merged in hotfix/provider_name_group_fix (pull request #988) Hotfix/provider name group fix * Fixes done in GROUP column * black format fix Approved-by: Katon Minhas * Merged in bugfix/DAIP2-2524-carveout-code-optimization (pull request #980) CARVEOUT_CD issue fixed * CARVEOUT_CD issue fixed * pipeline error fixed * Merged dev into bugfix/DAIP2-2524-carveout-code-optimization * Merged dev into bugfix/DAIP2-2524-carveout-code-optimization * Merged dev into bugfix/DAIP2-2524-carveout-code-optimization * trigger cap issue fixed * trigger cap prompt updated * Merged dev into bugfix/DAIP2-2524-carveout-code-optimization Approved-by: Katon Minhas * Merged in bugfix/exhibit-smart-chunking-cost-improvements (pull request #985) Bugfix/exhibit smart chunking cost improvements * Add opt-in instrumentation for per-call token and row-count tracing Introduce src/utils/instrumentation.py (thread-safe CSV logger) and src/utils/instrumentation_context.py (ContextVar scope plus submit_with_context / map_with_context helpers for propagating context into ThreadPoolExecutor workers). Emit events at every Bedrock call in llm_utils.invoke_claude, including in-memory claude_cache hits, with full input/output/cache-read/cache-write token breakdown. Emit row-count events at each row-mutating stage in the one-to-N pipeline (clean_reimbursement_primary, filter_services_without_reimbursements, methodology_breakout, split_service_terms, carveout, dynamic_code_assignment, lesser_of_distribution, dynamic_assignment) and chunking / retrieval events in exhibit smart chunking (chunking_done, retrieval_done) plus exhibit lifecycle events (exhibit_start, exhibit_gate_skip, stage_… * Merged in hotfix/fileextension_issue (pull request #989) Hotfix/fileextension issue * fixed strip_ext issue * black format Approved-by: Katon Minhas * Merged in hotfix/exhibit-header-in-tables (pull request #987) Hotfix/exhibit header in tables * Merged in feature/FixplaceholderIssue (pull request #977) Remove curly braces from echo statements in dev->stg * Remove curly brances from echo statements in dev->stg * Removed curly braces in echo statements in feature-> dev gate Approved-by: Sujit Deokar * Merged in feature/standardized-services (pull request #958) Feature/standardized services * service term standardization * prompt update * prompt updates for standardization * only service standardization * new file * add supporting files and test scripts for standardization work Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * remove old files * Merge remote-tracking branch 'origin/dev' into feature/standardized-services * final fixes * Merged dev into feature/standardized-services * additional features * removed unwanted files * remove unwanted files * Merge branch 'dev' into feature/standardized-services * Merge remote-tr… * Merged in hotfix/postprocess_csv (pull request #990) Hotfix/postprocess csv * date issue_fix * black_format * Merged dev into hotfix/postprocess_csv Approved-by: Katon Minhas * Merged in bugfix/molina_ut_dynamic_primary (pull request #991) Bugfix/molina ut dynamic primary * Prompt changes for dynamic primary * Route cover-sheet-only files to ERRORS.csv instead of leaking phantom rows When every page of a contract was filtered out as a cover sheet / quick-review form, process_file silently returned a FILE_NAME-only DataFrame. Because the runner routes by checking for an "error" column, that file landed in RESULTS.csv as a near-empty row and no ERRORS.csv was generated for the run. - saas/file_processing.py: raise ValueError when text_dict is empty after cover-sheet filtering, so safe_process_file produces a proper error row. - runner.py: add _is_phantom_result defense-in-depth — promote any result with no extracted fields beyond FILE_NAME to error_results with error_type=PhantomSuccess. * Merged dev into bugfix/molina_ut_dynamic_primary * Tighten PRODUCT prompt: restrict to valid_values, prune LOB/PROGRAM examples * Merge branch 'bugfix/molina_ut_dynamic_primary' of bit… * Merged in bugfix/black-format (pull request #994) Black format for pipeline pass * Black format for pipeline pass * Merged in bugfix/DAIP2-2679-fix-nebraska-issues (pull request #995) incorrect inclusion of CPT4_PROC_CD fixed * incorrect inclusion of CPT4_PROC_CD fixed Approved-by: Katon Minhas * Merged in feature/DAIP2-2314-DAIP2-1687-hybrid (pull request #993) Feature/DAIP2-2314 DAIP2 1687 hybrid * remove -files from s3 prefix requirements * Resolve input paths * fix: VendorProcessor.process_file returns (df, None) tuple runner.safe_process_file unpacks the result as (cc_df, dashboard_df), so returning a single DataFrame caused every vendor/generic file to fail with "too many values to unpack (expected 2)" — Python iterates DataFrame columns during unpacking. Vendor pipelines have no dashboard variant; second slot is None and the existing `dashboard_result is not None` guard in runner.py already handles it. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com> * DAIP2-2314 + DAIP2-1687: pad DYNAMIC_PRIMARY + DYNAMIC_PRIMARY_ENTITY_CLASSIFICATION over 1024-token cache floor - Pad DYNAMIC_PRIMARY_INSTRUCTION with three new sections: [SCOPE BOUNDARIES], [SOURCE TEXT INTERPRETATION], [REASONING DISCIPLINE], plus a [WORKED EXAMPLES] block. Estimated tokens: 447 -> 1117 (Sonnet 4.5 … * Merged in bugfix/postprocessing_date_fix (pull request #996) Date formatting changes * Date formatting changes * Merged dev into bugfix/postprocessing_date_fix Approved-by: Katon Minhas * Merged in bugfix/parent-child-rank-orphan-uniqueness (pull request #999) PC_logic bugfix * PC_logic bugfix Approved-by: Katon Minhas * Merged in feature/document-index (pull request #1005) Feature/document index * Add Document Index preprocessing — Layers 1, 2, and 3 wiring Parse the Textract-emitted Document Index block at the top of each contract with a single cached LLM call (prompt_document_index) instead of one per-page call per page. Layer 2 verifies parsed entries via literal string match and structural regex sweep, escalating suspect pages back to the existing per-page path. Layer 1+2 failure triggers a full fallback to today's per-page flow. New symbols: - preprocessing_funcs.extract_document_index_block — regex slice of index prefix - preprocessing_funcs.verify_index_against_pages — structural verifier (plain dict return) - prompt_templates.DOCUMENT_INDEX_INSTRUCTION / DOCUMENT_INDEX — cached prompt pair - prompt_calls.prompt_document_index — LLM wrapper (usage_label DOCUMENT_INDEX_PARSE) - config: DOCUMENT_INDEX_PARSE_ENABLED and three threshold flags - instrumentation: DOCUMENT_INDEX_PARSE mapped to preprocessing segment one… * Merged in feature/active-rates (pull request #1004) Feature/active rates * initial commit * Merged in feature/FixplaceholderIssue (pull request #977) Remove curly braces from echo statements in dev->stg * Remove curly brances from echo statements in dev->stg * Removed curly braces in echo statements in feature-> dev gate Approved-by: Sujit Deokar * Merged in feature/standardized-services (pull request #958) Feature/standardized services * service term standardization * prompt update * prompt updates for standardization * only service standardization * new file * add supporting files and test scripts for standardization work Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com> * remove old files * Merge remote-tracking branch 'origin/dev' into feature/standardized-services * final fixes * Merged dev into feature/standardized-services * additional features * removed unwanted files * remove unwanted files * Merge branch 'dev' into feature/standardized-services * Merge remote-track… * Merged in feature/one-to-one-confidence-scoring (pull request #1002) Feature/one to one confidence scoring * T1 plumbing: capture per-field confidence + retrieved-chunk metadata for 1:1 HSC fields Prep work for the 1:1 confidence-scoring stage. No scoring logic yet — this just collects the inputs the next ticket (rule-based scorer) will consume. - ONE_TO_ONE_SINGLE_FIELD_TEMPLATE: ask the LLM for confidence (0.0-1.0), verdict (correct/uncertain/not_found), and supporting_snippet alongside the field value. Existing field parser passes the extra keys through unchanged. - prompt_hsc_single_field: now returns a 4-tuple (name, value, field, metadata) where metadata holds the confidence/verdict/snippet plus a lightweight summary of which chunks the LLM saw (count + ids). _extract_hsc_metadata is defensive: clamps out-of-range confidences, defaults a missing/garbage verdict, caps the snippet at 500 chars, returns _empty_hsc_metadata() on every bail-out path. - run_hybrid_smart_chunked_fields: opt… * Merged in bugfix/confidence-flagged-missing-field-col (pull request #1006) Fix KeyError in compute_flagged when a *_CONF column has no value sibling * Fix KeyError in compute_flagged when a *_CONF column has no value sibling Production hit a hard crash at the end of every run: KeyError: "['DYNAMIC_PRIMARY_ENTITIES'] not in index" src/qc_qa/confidence/summary.py:143 compute_flagged was iterating over *_CONF columns and unconditionally indexing the dataframe with both the FILE_NAME column and the stripped value column. That assumed every <FIELD>_CONF column has a sibling <FIELD> value column in final_df. That isn't always true: dynamic-primary features carry only the _CONF side (their value side is dropped by reorder_columns since it isn't in FIELD_FORMAT_MAPPING but its _CONF suffix matches the explicit _CONF carve-out). When the model produced a below-threshold score for one of these and the value column was absent, pandas .loc raised KeyError and the runner crashed. Fix: - Build the .loc colu… * Merged in feature/active-rates (pull request #1007) Feature/active rates * amendment intent tag * prompt update * Merge remote-tracking branch 'origin/dev' into feature/active-rates * Merge remote-tracking branch 'origin/dev' into feature/active-rates * Merge branch 'dev' into feature/active-rates * exhibit standardization updates * use AARETE_DERIVED_EXHIBIT_TITLE for intent * active rates stuff * Merge branch 'dev' into feature/active-rates * prompt update * amendment intent types * active rates logic update * unit tests * Merged dev into feature/active-rates * logic updates * Merge remote-tracking branch 'origin/dev' into feature/active-rates * Merge remote-tracking branch 'origin/dev' into feature/active-rates * updated instrumentation cost logs * cost loging * caching updates * Merge remote-tracking branch 'origin/dev' into feature/active-rates * Merge remote-tracking branch 'origin/dev' into feature/active-rates * logging fix * null check issue fixes * caching fix Approved-by… * Merged in bugfix/stg-to-main-prep (pull request #1012) Bugfix/stg to main prep * Merged in dev (pull request #1001) Dev * Revert premature merge of bugfix/retire_stale_client_file_processing PR #959 was merged into dev without approval. This reverts commits |
||
|
Katon Minhas
|
8afe64f60f |
Merged in DEV (pull request #895)
DEV * Merged in bugfix/parser-downstream-improvements (pull request #875) Bugfix/parser downstream improvements * Refactor: Implement field-aware JSON parsers with centralized normalization This refactor introduces a robust system for normalizing LLM output based on field format mappings, ensuring consistent data types throughout the pipeline. Key Changes: - Add FIELD_FORMAT_MAPPING constant defining expected formats for all fields - Create format_normalization.py utility for type-aware normalization - Update json_utils.py parsers to accept field_names/field_name parameters - Refactor prompt_templates.py to use parser factories (_create_json_dict_parser, _create_json_list_parser) that bind field metadata for automatic normalization - Update prompt_calls.py to pass field names to parsers, eliminating redundant normalization logic - Remove parse_json_dict_or_list (unused, ambiguous function) - Simplify METHODOLOGY_BREAKOUT and REIMBURSEMENT_PRIMARY to use helper functions - Add comprehensive integration tests… * Update other lists * Black format * Remove prints * Update qa_qc to account for lists in is_empty check * Resolve issues * Black format * Merged main into DEV * Merged in feature/claim-type-only-runner (pull request #876) Feature/claim type only runner * Add specific_fields config for running extraction on field groups ## What Changed 4 files modified: 1. src/config.py - Added configuration for field-specific extraction: - SPECIFIC_FIELDS arg (default: 'all') - pass field group name or comma-separated field names - FIELD_GROUPS dict - predefined groups: claim_type, dates, provider - get_specific_fields_list() - resolves config to actual field list 2. src/prompts/fieldset.py - Added filter_by_names() method to FieldSet class to filter fields by a list of names 3. src/pipelines/shared/extraction/one_to_n_funcs.py - Updated exhibit_level() to accept specific_fields parameter and skip prompts for fields not in the list 4. src/pipelines/saas/file_processing.py - Passes specific_fields through the call chain to both one_to_n and one_to_one extraction ## How It Works When specific_fields is set to something other than 'all': 1. Config resolves the field l… * Merged in bugfix/generic_lesser_of (pull request #880) Bugfix/generic lesser of * Updated lesser of ditribution and lesser of check prompt * Remove excessive logging * Updated prompt for combining and formatting lesser of statement and reimb term * Code clean upt * Removed unneeded instruction on output format * style: format prompt_calls.py with black Approved-by: Katon Minhas * Merged in feature/new_output_format (pull request #879) Feature/new output format * bugfix * Merge branch 'DEV' into Optimize/DAIP2-1474-restructure-postprocess * requested postprocessing changes * prompt changes reverted * fix pipeline issues * fix pipeline issues * fix pipeline issues * fixed formatting * fixed formatting * Merge branch 'DEV' into Optimize/DAIP2-1474-restructure-postprocess * Merge branch 'DEV' into Optimize/DAIP2-1474-restructure-postprocess * save dashboard and cc output separately * save dashboard output in s3 * pipeline error fixed * json list through postprocessing * Merged DEV into Optimize/DAIP2-1474-restructure-postprocess * Merge remote-tracking branch 'origin/Optimize/DAIP2-1474-restructure-postprocess' into feature/new_output_format * Restructure output file organization and add standard field sanitization Output Structure Changes: - Reorganize output files into hierarchical directory structure: - full_outputs/cc_results/ for consolidated CC results - full… * Merged in bugfix/dynamic_issues_feb12 (pull request #882) Bugfix/dynamic issues feb12 * prompt changes reverted * fix pipeline issues * fix pipeline issues * fix pipeline issues * fixed formatting * fixed formatting * Merge branch 'DEV' into Optimize/DAIP2-1474-restructure-postprocess * Merge branch 'DEV' into Optimize/DAIP2-1474-restructure-postprocess * save dashboard and cc output separately * save dashboard output in s3 * pipeline error fixed * json list through postprocessing * Merged DEV into Optimize/DAIP2-1474-restructure-postprocess * Merge remote-tracking branch 'origin/Optimize/DAIP2-1474-restructure-postprocess' into feature/new_output_format * Restructure output file organization and add standard field sanitization Output Structure Changes: - Reorganize output files into hierarchical directory structure: - full_outputs/cc_results/ for consolidated CC results - full_outputs/dashboard_results/ for consolidated dashboard results - full_outputs/ for error files - automation_q… * Merged in feature/DAIP2-1562-add-aarete_derived_payer_name (pull request #867) Feature/DAIP2-1562 add aarete derived payer name * black format * Merged DEV into feature/DAIP2-1562-add-aarete_derived_payer_name * llm_choose_derived_payer_name function added * threshold updated * Merge branch 'DEV' into feature/DAIP2-1562-add-aarete_derived_payer_name * aarete_derived_payer_name column added * prompt structure updated * state_flag added * pipeline error fixed * pipeline error fixed * Merged DEV into feature/DAIP2-1562-add-aarete_derived_payer_name * remove debug print statement for similarity matrix in clustering function * Merged DEV into feature/DAIP2-1562-add-aarete_derived_payer_name * state logic added in clustering * removed print statements * pull request updates * black format fix * Merged DEV into feature/DAIP2-1562-add-aarete_derived_payer_name * added scalability feature and optimization * black format * Pull Request Changes * updated config parameters * updated main * no pay… * Merged in bugfix/DAIP2-1701-issue-tracker-fixes-methodology-breakout (pull request #885) Bugfix/DAIP2-1701 issue tracker fixes methodology breakout * testing fee schedules issues * logging debug added for lesser of * Merged DEV into bugfix/DAIP2-1701-issue-tracker-fixes-methodology-breakout * special case term fix * updated lesser of distribution prompt * Merged DEV into bugfix/DAIP2-1701-issue-tracker-fixes-methodology-breakout * refactor: use f-string for special case term concatenation Approved-by: Siddhant Medar * Merged in bugfix/DAIP2-1699-issue-tracker-fixes-one-to-one-dates-prov-info (pull request #881) bugfix/DAIP2-1699-issue-tracker-fixes-one-to-one-dates-prov-info * contract title fixes * Add instruction to strip hyphens from TIN and NPI * prompt update * Merge branch 'bugfix/auto-renewal-term' into bugfix/DAIP2-1699-issue-tracker-fixes-one-to-one-dates-prov-info * removed print statements * pipeline error fixed * signed_ind logic changed * signatory_ind logic updated * prompt update * prompt update * pipeline error fixed * Merge branch 'DEV' into bugfix/DAIP2-1699-issue-tracker-fixes-one-to-one-dates-prov-info * Merge branch 'DEV' into bugfix/DAIP2-1699-issue-tracker-fixes-one-to-one-dates-prov-info * requested changes applied * test case added * prompt update * contract title prompt update * latest prompt added Approved-by: Siddhant Medar * Merged in bugfix/default_ind_postprocess (pull request #887) Bugfix/default ind postprocess * Add logic to standardize UNIT_OF_MEASURE for flat-rate reimbursement methods - Implemented functionality in `standardize_reimb_method_and_fee_schedule` to set UNIT_OF_MEASURE to blank for rows where DEFAULT_IND is 'Y' and AARETE_DERIVED_REIMB_METHOD is 'flat rate'. - Added unit tests to verify behavior for various scenarios, including case insensitivity and non-default conditions. - Ensured that UNIT_OF_MEASURE remains unchanged for non-flat rate methods. * Enhance child rank handling and ensure column consistency in parent-child mapping - Added initialization for the `child_rank` column in both parents and children DataFrames to prevent KeyError during concatenation when no children exist. - Updated `cols_to_keep` in `parent_child_mapping` to filter out columns not present in `pc_df`, ensuring robustness in data processing. * Ran Black * made a small change in code_last_check, fixed so it returns string and n… * Apply formatting fix * Merged in DAIP2-1898-one-to-one-issues-filename-tin-auto-renewal-and-contract-title (pull request #888) DAIP2-1898 one to one issues filename tin auto renewal and contract title * contract title fixes * Add instruction to strip hyphens from TIN and NPI * prompt update * Merge branch 'bugfix/auto-renewal-term' into bugfix/DAIP2-1699-issue-tracker-fixes-one-to-one-dates-prov-info * removed print statements * pipeline error fixed * signed_ind logic changed * signatory_ind logic updated * prompt update * prompt update * pipeline error fixed * Merge branch 'DEV' into bugfix/DAIP2-1699-issue-tracker-fixes-one-to-one-dates-prov-info * Merge branch 'DEV' into bugfix/DAIP2-1699-issue-tracker-fixes-one-to-one-dates-prov-info * requested changes applied * test case added * prompt update * contract title prompt update * prompt update * Merge branch 'DEV' into DAIP2-1898-one-to-one-issues-filename-tin-auto-renewal-and-contract-title Approved-by: Siddhant Medar * Merged in feature/DAIP2-1803-add-aarete-derived-provider-name (pull request #884) Feature/DAIP2-1803 add aarete derived provider name * Merged DEV into feature/DAIP2-1562-add-aarete_derived_payer_name * state logic added in clustering * removed print statements * pull request updates * black format fix * Merged DEV into feature/DAIP2-1562-add-aarete_derived_payer_name * added scalability feature and optimization * black format * Pull Request Changes * generalised funcs from aarete derived payer name * updated remove states and updated docstring for build similarity matrix * derived payer name functionality added * config and main file changes * sync dev into ADD AARETE DERIVED PROVIDER GROUP NAME FULL * updated derived_provider name * pipeline fixes * Merged DEV into feature/DAIP2-1803-add-aarete-derived-provider-name * config changes * field name updated to AARETE_DERIVED_PROVIDER_NAME * Merged DEV into feature/DAIP2-1803-add-aarete-derived-provider-name * black format fix * Fix docstr… * Merged in improve-logging (pull request #886) Improve logging * Add timing blocks for comprehensive pipeline logging - Add timing_utils.timed_block() to key extraction functions in one_to_n_funcs.py: - exhibit_level sub-functions (prompt_exhibit_level, dynamic_primary, etc.) - reimbursement_level extraction and cleaning - breakout functions (methodology_breakout, special_case_breakout) - carveout_and_special_case parallel processing - one_to_n_cleaning sub-steps (crosswalk, lob_relationship, split_reimb_dates) - Add timing to dynamic_funcs.py: - dynamic_assignment parallel processing - Add timing to postprocess.py: - standard_postprocess, contract_config_postprocess, dashboard_postprocess - Add timing to preprocess.py: - exhibit_chunking sub-steps (get_exhibit_pages, link_exhibit_pages, chunk_by_exhibit) - Add info-level logging statements for completed operations with row/item counts * Standardize logging levels and remove deprecated code DAIP2-1798: Ensure logging statements are at … * Merged in bugfix/DAIP2-1913-lesser-of-indicator-issues (pull request #889) Bugfix/DAIP2-1913 lesser of indicator issues * contract title fixes * Add instruction to strip hyphens from TIN and NPI * prompt update * Merge branch 'bugfix/auto-renewal-term' into bugfix/DAIP2-1699-issue-tracker-fixes-one-to-one-dates-prov-info * removed print statements * pipeline error fixed * signed_ind logic changed * signatory_ind logic updated * prompt update * prompt update * pipeline error fixed * Merge branch 'DEV' into bugfix/DAIP2-1699-issue-tracker-fixes-one-to-one-dates-prov-info * Merge branch 'DEV' into bugfix/DAIP2-1699-issue-tracker-fixes-one-to-one-dates-prov-info * requested changes applied * test case added * prompt update * contract title prompt update * prompt update * Merge branch 'DEV' into DAIP2-1898-one-to-one-issues-filename-tin-auto-renewal-and-contract-title * handle nested cases of lesser_of and greater_of * Merge branch 'DEV' into bugfix/DAIP2-1913-lesser-of-indicator-issues * pi… * Merged in bugfix/reimb_primary (pull request #890) Update reimbursement prompts for consistent PMPM handling, time-unit dedup, and validation improvements * Update reimbursement prompts for consistent PMPM handling, time-unit dedup, and validation improvements - REIMBURSEMENT_PRIMARY: append unit of measure (PMPM, per visit, etc.) from table headers into REIMB_TERM - METHODOLOGY_BREAKOUT: collapse mathematically equivalent time-unit rates into single entry - VALIDATE_REIMBURSEMENTS: add umbrella terms as valid service terms, accept CPT/HCPCS percentage as concrete rate, narrow COB disqualifier to primary subject only Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com> Approved-by: Siddhant Medar * Merged in feature/update-postprocessing-columns (pull request #893) move column order to final step * move column order to final step * Black Approved-by: Siddhant Medar * Merged in bugfix/update-postprocessing-paths (pull request #896) Update postprocessing paths * Update postprocessing paths Approved-by: Siddhant Medar * Merged in feature/TIN_stats_update (pull request #898) Feature/TIN stats update * Tin_stats_report * black formatting * minor changes * black format applied * Addressed the katons comments * Merged DEV into feature/TIN_stats_update * black reformat Approved-by: Katon Minhas |
||
|
Katon Minhas
|
afb6d5185d |
Merged in feature/lesser-table-caching-refactor-hybrid (pull request #847)
Feature/lesser table caching refactor hybrid * chore: Remove unused duplicate main.py from shared pipeline * fix: Correct crosswalk paths in aarete_derived.py * chore: Remove unused documentation files from fieldExtraction * docs: Add documentation files to documentation folder * docs: Update README with uv setup, expanded project structure, and branching conventions * docs: Add uv installation steps with Ubuntu/WSL emphasis * Enable prompt caching for all remaining LLM calls - Add _INSTRUCTION() functions for: EXHIBIT_HEADER, EXHIBIT_LINKAGE, EXHIBIT_TITLE_MATCH, DATE_FIX, DERIVED_TERM_DATE, CHECK_PROVIDER_NAME_MATCH, SPECIAL_CASE_ASSIGNMENT - Update all invoke_claude() calls in saas and clover pipelines to use cache=True with corresponding _INSTRUCTION() functions - Add new instructions to get_cacheable_instructions() for cache warming - Update tests for new instruction functions Functions now using caching: - prompt_exhibit_level - prompt_exhibit_lesser (EXHIBIT_LEVEL_LESSER_OF) - prompt_fee_schedule_breakout - prompt_grouper_breakout - prompt_special_case_assignment - prompt_exhibit_linkage - prompt_exhibit_header - prompt_smart_chunked (ONE_TO_ONE templates) - prompt_date_fix - prompt_derived_term_date - prompt_exhibit_title_match - provider_name_match_check 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * Reorder * feat: Add bcbs_promise client pipeline with OFFSET_TERM extraction - Add new bcbs_promise client with HSC-based OFFSET_TERM field extraction - Extract full paragraph text of offset/recoupment provisions from contracts - Derive OFFSET_INDICATOR (Y/N) from OFFSET_TERM presence - Fix reorder_columns to preserve extra columns not in COLUMN_ORDER - Update QC/QA output path to outputs/qc_qa/ * fix: Update dev deps and test assertions for QC/QA output path - Add pytest/pytest-mock to dev dependencies for mypy type checking - Update test assertions to expect outputs/qc_qa instead of qa_qc_output * style: Apply black formatting to prompt_templates.py * Merge main, move scripts * Archive some scripts * update py version * remove .py version file * Remove ASCII characters * Restore testbed code * restore tracking * Update testbed metrics * Enable prompt caching for CODE_LAST_CHECK, FILL_BILL_TYPE, DUAL_LOB_CHECK, and GROUPER_BREAKOUT - Add CODE_LAST_CHECK_INSTRUCTION() for service specificity classification - Add FILL_BILL_TYPE_INSTRUCTION() for bill type code determination - Add DUAL_LOB_CHECK_INSTRUCTION() for Medicare/Medicaid classification - Update code_funcs.py to use caching for CODE_LAST_CHECK, FILL_BILL_TYPE, GROUPER_BREAKOUT - Update postprocessing_funcs.py to use caching for DUAL_LOB_CHECK - Add new instructions to get_cacheable_instructions() for cache warming - Add unit tests for new instruction functions 🤖 Generated with [Claude Code](https://claude.com/claude-code) Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com> * Fix postprocessing_funcs to remove invalid columns * Merge branch 'main' into feature/lesser-table-caching-refactor-hybrid * Revert prompt caching changes from aed1b73c * update formatting * Update imports Approved-by: Sha Brown Approved-by: Praneel Panchigar |
||
|
Faizan Mohiuddin
|
8c9060e425 |
Merged in feature/multithreading (pull request #828)
Feature/multithreading * fixes * Merge main into feature/multithreading Resolved conflicts: - Kept timing instrumentation in file_processing.py - Kept new 3-step Exhibit-based approach for one-to-n processing - Maintained parallelization improvements (20 workers) Changes include: - Timing utils integration for performance monitoring - Increased max_workers from 5 to 20 across all components - Parallelized code_breakout and grouper_breakout - Fixed tin_npi_funcs function call parameters * Fix max_workers error for empty documents - Add check to skip parallel processing when no pages exist - Use min(len(all_page_tasks), 20) to prevent max_workers=0 - Handles edge case of documents with no exhibits or pages * Fix max_workers=0 errors in one_to_n_funcs - Add checks before all ThreadPoolExecutor creations - Prevents errors when processing empty lists: - carveout_and_special_case - breakout - special_case_breakout - filter_services_without_reimbursements - run_lob_relationship - Ensures executor only created when there are items to process * Reduce code processing parallelism to prevent API throttling Lower max_workers from 20 to 10 for code_breakout and grouper_breakout to prevent overwhelming Bedrock API with concurrent requests * fixed * Merged main into feature/multithreading * Move documentation into folder * Fix logging statements * Merge branch 'main' into feature/multithreading * Refactor for clarity * Update previous exhibit passing logic * properly simplify exhibits * Merged main into feature/multithreading * update conditional for None * exhibit multithreading changed * exhibit multithreading changed * Merge remote-tracking branch 'origin/main' into feature/multithreading * synced with main * Parallelize dynamic assignment, refactor HSC field worker, add timing/exhibit unit tests, tidy imports/ignore helpers * analyze_regression.py edited online with Bitbucket * count_pages.py edited online with Bitbucket * compare_regressed_with_baseline.py edited online with Bitbucket * simple_testbed_compare.py edited online with Bitbucket * run_testbed_metrics_regressed.py edited online with Bitbucket Approved-by: Katon Minhas |
||
|
Praneel Panchigar
|
d3bfec0b6b |
Merged in bugfix/dynamic_issuefixes-clean (pull request #813)
Fix LOB_PROGRAM_RELATIONSHIP extraction and preserve debugging code * Fix LOB_PROGRAM_RELATIONSHIP extraction and preserve debugging code - Add format requirements to LOB_RELATIONSHIP_INSTRUCTION for proper caching - Strengthen LOB_RELATIONSHIP prompt with explicit pipe delimiter requirements - Preserve all DEBUG blocks and print statements in dynamic_funcs.py and file_processing.py - Update crosswalk mappings and gitignore - Ensure format instructions are cached at API level to prevent missing pipe delimiters * Merge main into bugfix/dynamic_issuefixes-clean and remove all debug print statements * Remove redundant one_to_one_fields initialization to match main Approved-by: Katon Minhas |
||
|
Praneel Panchigar
|
7056c1c687 |
Merged in feature/automated-cost-logging-csv (pull request #768)
Feature/automated cost logging csv
* feat: Add usage and cost tracking with CSV export
- Add usage_tracking.py module for thread-safe token and cost tracking
- Extract actual tokens from Bedrock API responses (replaces word count estimation)
- Integrate usage tracking into llm_utils.py for all LLM invocations
- Add CSV export functionality (per-file/per-model and batch summary)
- Integrate CSV export into main.py for local runs
- Update .gitignore to exclude PRD
Phase 1: Data collection and local CSV export implemented
- Tracks all LLM calls including utility functions (date_fix, derive_term_date)
- Calculates costs per model using centralized cost constants
- Generates two CSV files: USAGE.csv and USAGE-SUMMARY.csv
- Exports to same directory as final results when write_to_s3=False
* - Removed some temporary logging code
- Removed unused imports (uuid, datetime) from llm_utils.py
- Added cache token columns (cache_creation_tokens, cache_read_tokens) to CSV exports
- Added tracking for cache warming
- Updated usage_tracking.py to track cache tokens separately in data structure
All token tracking functionality remains intact.
* Split input tokens into fresh and cache in summary section
- Add total_fresh_input_tokens and total_cache_tokens to GLOBAL_USAGE
- Update summary CSV to include split token columns for better tractability
- Add averages for fresh_input_tokens and cache_tokens
- Maintain total_input_tokens = fresh_input_tokens + cache_tokens relationship
- All calculations verified and match per-file totals correctly
* upload files functionality
* Add comprehensive unit tests for usage_tracking module
- Implement Phase 1-4 tests covering all 10 functions in usage_tracking.py
- Add 98 test cases with parametrized tests for comprehensive coverage
- Fix extract_usage_from_response to handle None input gracefully
- Add python-dotenv dependency to pyproject.toml
- Tests cover: core functions, data access, export functions, and utilities
- All tests passing (98/98)
* Add S3 upload tests for main.py and fix f-string syntax error
- Add tests/test_investment_main_s3.py with 3 test cases:
* test_main_writes_final_and_error_to_s3: validates both final and error DataFrames uploaded when processing has errors
* test_main_writes_only_final_when_no_errors: validates only final DataFrame uploaded when all files succeed
* test_main_writes_usage_when_present: validates usage tracking files uploaded when usage data exists
- Tests use monkeypatching and module stubs to isolate S3 write logic without external dependencies
- Fix syntax error in src/codes/code_funcs.py: nested f-string quotes (level_dict['level_suffix'])
- All tests passing
* Add comprehensive test for usage_tracking.get_usage_dataframes S3 uploads
- Add test_usage_dataframes_written_to_s3 to validate per_file_usage_df and batch_summary_df
- Test verifies both dataframes from usage_tracking.get_usage_dataframes() are written to S3
- Validates 'usage' suffix writes per_file_usage_df with correct schema (file_name, tokens, cost)
- Validates 'usage_summary' suffix writes batch_summary_df with correct schema (totals, averages, region_mode)
- Uses pd.testing.assert_frame_equal to ensure exact dataframes are uploaded
- Fix module caching issue between tests by clearing sys.modules cache
- All 4 tests passing
* Add cleanup fixture to prevent test pollution
- Add autouse pytest fixture to clean up stubbed modules after each test
- Remove manual cache clearing from individual tests (now handled by fixture)
- Prevents our module stubs from affecting other test files in CI/CD pipeline
- Fixes Bitbucket pipeline failures where other tests couldn't find llm_utils attributes
* Refactor tests: add comprehensive io_utils coverage; disable usage_tracking suite
- Added 22 focused tests for write_local and write_s3 plus existing IO behaviors
- Introduced FakeS3Client to avoid boto3 network/credential dependency
- Added preserve_config fixture (no monkeypatch) to isolate config side effects
- Marked test_usage_tracking.py skipped per new consolidation approach
- Verified 26 tests (io_utils + investment_main) all pass locally without AWS exceptions
* Fix S3_CLIENT mocking: use mocker.patch instead of direct assignment
- Changed preserve_config fixture to NOT save/restore S3_CLIENT
- Updated all 5 write_s3 tests to use mocker.patch('src.config.S3_CLIENT', fake)
- This prevents pollution of config.S3_CLIENT across test modules
- Fixes NoCredentialsError in other tests that were importing config after our tests set FakeS3Client
- All 22 io_utils tests pass + 4 investment_main_s3 tests pass
- test_llm_utils::test_invoke_claude_local_mode now passes
* Add tests for write_s3 function
* Resolve merge conflict: keep mocker-based write_s3 tests
* Remove rogue skip mark from test_usage_tracking.py
The skip mark was incorrectly added by remote branch claiming tests were
migrated to io_utils. However, test_io_utils.py only tests io_utils functions
(read/write operations), not usage_tracking functions.
Restore the comprehensive usage_tracking tests from commit 26a556f0 which
includes 98 test cases covering all 10 functions in usage_tracking.py.
* Fix test pollution: add config cleanup fixture and restore RUN_MODE
- Add autouse fixture in test_io_utils.py to restore config values after each test
- Fix test_llm_utils.py to restore RUN_MODE after test_invoke_claude_local_mode
- Prevents config patches from leaking into subsequent tests causing NoCredentialsError
* Remove redundant test_investment_main_s3.py
- File was testing same write_s3 functionality already covered in test_io_utils.py
- Was causing module pollution by stubbing src.utils.llm_utils
- Caused test failures in other test files due to module cache issues
- Integration testing was minimal and mocked write_s3 anyway
* Merged main into feature/automated-cost-logging-csv
Approved-by: Karan Desai
Approved-by: Katon Minhas
|
||
|
Alex Galarce
|
55fa4f2e79 |
Merged in feature/reimb-primary-improvements (pull request #524)
Reimbursement Exhibit Prefiltering * Add IDENTIFY_REIMBURSEMENT_EXHIBITS function to analyze exhibit headers for reimbursement information * Add Zone.Identifier to .gitignore to prevent Windows zone information files from being tracked * Add functionality to identify reimbursement exhibits * Add functionality to identify reimbursement exhibits in run_one_to_n_prompts * Output YAML parsing * Refactor identify_reimbursement_exhibits to improve YAML parsing and error handling * New reimbursement exhibit logic * Update IDENTIFY_REIMBURSEMENT_EXHIBITS_PROMPT to return PAGE NUMBERS instead of exhibit headers * Fix identify_reimbursement_exhibits to return page numbers as strings after YAML parsing * Remove debugging print statement for reimbursement exhibits in run_one_to_n_prompts * Remove commented debugging print statements in identify_reimbursement_exhibits * Increase read_timeout in EC2 configuration from 2000 to 4000 * Merge remote-tracking branch 'origin/main' into feature/reimb-primary-improvements * Remove debugging print statements in get_reimbursement_primary function * mypy error * Fix return type in identify_reimbursement_exhibits to ensure keys are returned as a list * Merge remote-tracking branch 'origin/main' into feature/reimb-primary-improvements Approved-by: Katon Minhas |
||
|
VenkataKrishna Reddy Avula
|
576ea95b6d |
Merged in code-breakout-mismaps (pull request #419)
DAIP2-112 - fixed get code descriptions * indirect proc codes added * bug fix * bug fix * Merge branch 'main' into proc-code-indirect * description mapping handling X's and 0's * bug fix * Merge remote-tracking branch 'origin/proc-code-indirect' into code-breakout-mismaps * testing mismaps * revenue codes mapping for int values * added function to find ranges * removed test files * Merged main into code-breakout-mismaps * Merge branch 'main' into code-breakout-mismaps pulled latest changes from main * added unit test for find containing ranges * modified find_containing_ranges_test file * Merged main into code-breakout-mismaps * added embeddings to git ignore * updated find_containing_ranges_test.py Approved-by: Alex Galarce Approved-by: Katon Minhas |
||
|
Alex Galarce
|
48aa1d836b |
Merged in feature/load_crosswalks (pull request #342)
Feature/load crosswalks * allow .json in fieldExtraction/crosswalk * clean up gitignores, add rule to allow .jsons in crosswalk, create first crosswalk * create mappings directory * add to_csv functionality for MCS to check results * Merged main into feature/load_crosswalks Approved-by: Katon Minhas |
||
|
Alex Galarce
|
56e3fdd1f8 |
Merged in bugfix/all_cnc_hotfixes (pull request #302)
DRAFT: Bugfix/all cnc hotfixes * Pipeline fix - health plan state * Update for prompt vs non-prompt fixes * Revert utils.find_regex_matches() * Remove prints * Non-prompt run * Health Plan State, IRS, NPI, LOB, Lesser, Prov 2, Default, Rate Standard, Contract Effective Date, Term Group * Merged in bugfix/agreement_name (pull request #294) contract title fixed * contract title fixed * Merged bugfix/all_cnc_hotfixes into bugfix/agreement_name Approved-by: Katon Minhas * All hotfixes added * Fixed utils * fix for clean_contract_effective_date * Merged in default-fix (pull request #296) moved default funcs to the top, dropped intermediate columns * moved default funcs to the top, dropped intermediate columns Approved-by: Katon Minhas * quick fix to dropping extra columns * Merged in bugfix/effective_date_meridian (pull request #297) fixed effective date for meridian contracts * fixed effective date for meridian contracts Approved-by: Katon Minhas * Default rate and Lesser fix * Merged in hotfix/irs_npi_updated (pull request #298) updated irs and npi hotfix funcs * updated irs and npi hotfix funcs * Merged bugfix/all_cnc_hotfixes into hotfix/irs_npi_updated Approved-by: Katon Minhas * non-functional commit - default only * debug commit - premerge * Batch 1 Dup File Reconciliation * Default rate fix * Full CNC 1A Run Config * Standard column refactor * Working stitching for 1A * Merged in Alex-Galarce/cnc_hotfix_effective_date_utilspy-edited-1732659983687 (pull request #301) Remove "license" from effective date smart chunking * Remove "license" from effective date smart chunking * Merged bugfix/all_cnc_hotfixes into Alex-Galarce/cnc_hotfix_effective_date_utilspy-edited-1732659983687 Approved-by: Katon Minhas * Merged in hotfix/irs_batch2_feedback (pull request #300) IRS fix for feedback on batch2 * IRS fix for feedback on batch2 * Merged bugfix/all_cnc_hotfixes into hotfix/irs_batch2_feedback Approved-by: Katon Minhas * Merged main into bugfix/all_cnc_hotfixes Approved-by: Katon Minhas |
||
|
Michael McGuinness
|
5928ee0116 |
Merged in feature/add-dotenv (pull request #266)
Feature/add dotenv * transferred sensitive information to .env file from src/config.py * replace model IDs, AWS account info * Merge remote-tracking branch 'origin/main' into feature/add-dotenv * add explanatory comment * Merge remote-tracking branch 'origin/main' into feature/add-dotenv * remove tests that were just for local Approved-by: Katon Minhas |
||
|
Karan Desai
|
1098be5cf5 |
Merged in feature_ac_chunking_clean (pull request #265)
grouped keywords and IRS regex chunking * abandon old branch and rebuild * removed print for field names, prompts * pushing as part of leftovers * IRS regex chunking * Merged in feature/chunk_term_clean (pull request #262) chunk_log_added * chunk_log_added * Merge remote-tracking branch 'remotes/origin/feature_ac_chunking_clean' into feature/chunk_term_clean Approved-by: Alex Galarce * added function for regex based IRS chunking * shifted regex_match_chunk function to utils * added helper functions for regex based chunking * replaced tin_regex function to ac_funcs * removed prompt for irs_others * added regex chunking for irs * removed irs group from smart chunking * testing functionality for regex based irs fields * updated return N/A condition * Merged main into feature_ac_chunking_clean * file_processing.py edited online with Bitbucket * added back conditional prompts * minor fixed for PR * remove install types * mering clean branch * with passed test casses * added flag for regex based tin execution Approved-by: Michael McGuinness |
||
|
Katon Minhas
|
bb279b45fd |
Merged in cnc_fl_main (pull request #258)
Cnc fl main * prov type moved * comment out invalid prov2 bc it's overly restrictive * Update prompt template to ensure N/A rather than blanks * Update AC multi-prompt template. Update prompts to ensure all keys are present and N/As are filled * Update AC postprocessing * Update clean_tin() to add N/A for values other than 10 digits * update tin postprocessing * Update npi postprocessing * Add 'Filename:' to the start of all filenames * Update group TIN/NPI postprocessing * Update parent agreement code * fix health plan state mapping * Update B postprocessing * Updated indicators, re-ordering * Update postprocessing - all columns included and reordered * Ensure Pages is included in AC-only output * Remove excess print statements * remove example_test to pass pipeline * Fix parent agreement code for AC * Updated postprocessing for NPI/TIN other * Update prompts for N explicitely on AC prompt-based indicators * Updated main and utils for more intuitive filtering * Updated file_counting to match new main process * update test * Add example_test Approved-by: Michael McGuinness |
||
|
Michael McGuinness
|
ee3e3eb538 |
Merged in feature/mergeIntoMain (pull request #249)
Merge Prep * mergePRep Approved-by: Katon Minhas |
||
|
Michael McGuinness
|
7cb0b485ad |
Merged in feature/unittestframe (pull request #240)
Unit Testing Framework * emptyframe * fixstatic * save * exampletest * rmreqs * rmUto Approved-by: Alex Galarce |
||
|
Michael McGuinness
|
168d197349 |
Merged in feature/lightStaticTypes (pull request #234)
Add static type check * artifacts * dockerfileforall * examplemypystep * reqs * baseMain * oneperservice * somebasicchanges * deps * removedeps * removeerrs * singlecommand * lessargs * 312base * versionanderr * pythonversion * ithinkfinalerrs * numpyv * versionsandconditional * changesets * tryagain * lint * numpyagain * init * clearstreamlit * forcestreamlit Approved-by: Umang Mistry |
||
|
Michael McGuinness
|
02169d8a5e | add reqs | ||
|
FaizanM2000
|
81f34edc8d | added new postprocessing and bottom up and top down merging. also carveouts | ||
|
FaizanM2000
|
4478955e24 | testing top down intial test complete | ||
|
Grzegorz Huber
|
2919c6ef60 | [DOC-535] Snowsql test | ||
|
Pankaj Katariya
|
d9f217a983 |
Merged in dev_pkatariya (pull request #71)
Dev pkatariya Approved-by: Umang Mistry |
||
|
Pankaj Katariya
|
68b6bf9968 | Added git ignore | ||
|
Pankaj Katariya
|
d618250b65 |
Merged in dev_pkatariya (pull request #65)
Dev pkatariya Approved-by: Umang Mistry |
||
|
Mayank Aamseek
|
d447006548 | post processing for contract_num added | ||
|
Pankaj Katariya
|
d0036c8c54 | Resolved .gitignore conflict | ||
|
Pankaj Katariya
|
76ccb574d6 | Added terraform files to ignore | ||
|
Mayank Aamseek
|
62750d9e57 | getting results.csv | ||
|
Mayank Aamseek
|
a9740ab34a | used BUSINESS_CONFIG table 2 | ||
|
Mayank Aamseek
|
959432f37a | field values updated | ||
|
Ubuntu
|
6bc4e625a8 | debugging added | ||
|
Mayank Aamseek
|
8f55d8f113 | data files removed | ||
|
Mayank Aamseek
|
6f856e1f96 | save results | ||
|
Justin Wolfington
|
ec65796f94 | Initial commit |