Feature/DAIP2-2301 flag and remove identical contracts
* initiated flaging duplicate contracts
* Merged dev into feature/DAIP2-2301-flag-and-remove-identical-contracts
* Merged dev into feature/DAIP2-2301-flag-and-remove-identical-contracts
* updated duplicate detection
* pipeline fixes
* remove print statements
* Merged dev into feature/DAIP2-2301-flag-and-remove-identical-contracts
Approved-by: Katon Minhas
removed 2 signature related fields
* removed 2 signature related fields
* Merged dev into DAIP2-1595-optimize-signature-fields
* Merged dev into DAIP2-1595-optimize-signature-fields
Approved-by: Katon Minhas
Bugfix/DAIP2-2164 reimbursement primary issue fixes
* sample calculations reimbursements removed
* split service term added
* Merged DEV into bugfix/DAIP2-2164-reimbursement-primary-issue-fixes
* updated prompt and unit test added
* added prompt for codes listed below
* Merged DEV into bugfix/DAIP2-2164-reimbursement-primary-issue-fixes
* Output format at end
* Genericize prompt
* Merged DEV into bugfix/DAIP2-2164-reimbursement-primary-issue-fixes
* context missing issue fixed
* updated service term split prompt for program varints
* Merged DEV into bugfix/DAIP2-2164-reimbursement-primary-issue-fixes
* Merged DEV into bugfix/DAIP2-2164-reimbursement-primary-issue-fixes
* Merged DEV into bugfix/DAIP2-2164-reimbursement-primary-issue-fixes
* Merged DEV into bugfix/DAIP2-2164-reimbursement-primary-issue-fixes
* updated split service term prompt
* optimized and minimized split service term prompt
* updated service term split prompt
* Merged DEV into bugfix/DAIP2-2164-reimbursement-primary-issue-fixes
* Merged dev into bugfix/DAIP2-2164-reimbursement-primary-issue-fixes
* Updated Prompt
* grouped instructions
* removed print statements
Approved-by: Katon Minhas
Feature/vendor generic
* initial commit for generic vendor logic
* updated generic field list
* combined special fields
* joint fields template
* payment term and amount extraction
* model update
* model update
* added constants
* care-source fields added
* care-source constants
* fields update for bcbs-az
* prompt update for term type
* prompt update for payment type
* post process additions for generic
* Merge remote-tracking branch 'origin/DEV' into feature/vendor-generic
* Merged DEV into feature/vendor-generic
* Merged DEV into feature/vendor-generic
* Black format
* Merge remote-tracking branch 'origin/DEV' into feature/vendor-generic
* Merge branch 'DEV' into feature/vendor-generic
* PR comments
* deleted redundant file
* Merged dev into feature/vendor-generic
Approved-by: Katon Minhas
Bugfix/DAIP2-2230 mcs issues effective date new
* updated scope and fixes
* optimized prompt
* shortened prompt
* remove redundant prompt line
* Merged DEV into bugfix/DAIP2-2230-mcs-issues-effective-date-new
Approved-by: Katon Minhas
Feature/DAIP2-2112 implement pre commit hooks
* Add pre-commit hooks for local code quality checks
Adds .pre-commit-config.yaml with file hygiene hooks (trailing-whitespace,
end-of-file-fixer, check-yaml/json, check-merge-conflict, check-added-large-files)
plus local hooks for black and mypy on pre-commit and pytest on pre-push.
Includes setup documentation and adds pre-commit to dev dependencies.
Auto-fixed trailing whitespace and missing newlines caught by the new hooks.
* Add sample pre-commit hook for testing purposes
* Add test file for pre-commit hook validation
* Remove test file for pre-commit hook validation
* Merged DEV into feature/DAIP2-2112-implement-pre-commit-hooks---
* Merged DEV into feature/DAIP2-2112-implement-pre-commit-hooks---
* Move to documentation
Approved-by: Katon Minhas
DAIP2-2178 dynamic primary issue fixes LOBs incorrectly added
* LOB addition from product and program fixed
* Revert "LOB addition from product and program fixed"
This reverts commit fb0cb48a2e74eff6ab66340d07bcf080eb92708c.
* PROGRAM prompt updated
Approved-by: Katon Minhas
Bugfix/DAIP2-2138 reimbursement primary testing
* updated table instructions for reimb term extraction
* updated split tables to include header rows
* issue fix
* Merged DEV into bugfix/DAIP2-2138-reimbursement-primary-testing
* black format fix
Approved-by: Katon Minhas
Feature/context caching
* Initial commit - context caching for DYNAMIC_PRIMARY
* implement context caching for all relevant prompts
* Remove option to not context cache
* IndentationError fixed
* Merge branch 'DEV' into feature/context-caching
* Merge and format
* Move documentation
* Merged DEV into feature/context-caching
* Update unit tests
* Merged DEV into feature/context-caching
* Update signatures
* Fix test coverage gap
Approved-by: Praneel Panchigar
Approved-by: Karan Desai
Feature/code optimization
* Ran Black
* made a small change in code_last_check, fixed so it returns string and not single char
* Made changes to make sure that default_ind postprocess only happens to the cc output and not dashboard
* Merge bugfix/default_ind_postprocess into feature/code_optimization
- Parent-child: child_rank column init and cols_to_keep filter
- Postprocess: default_ind only on cc output
- code_last_check: parser str/list return handling + exception logging
- Prompt: FIELD ASSIGNMENT for code extraction (no CRITICAL prefix)
* More prompt changes and debugging print statements
* Added system to group same service term + bill type cd + claim type cd and then parallelize the code extraction (will improve consistency)
* Revert one-time PROV_INFO_JSON ad hoc logic; retain json_utils and output format
- Remove temporary postprocess step that filled empty TIN in PROV_INFO_JSON from
FILENAME_TIN (fill_prov_info_tin_from_filename_tin call in postprocess.py).
This behavior is intended to move upstream per JIRA (tin_npi_funcs / extraction).
- Revert postprocess_existing_output.py to generic config: empty INPUT_DIR,
INPUT_FILENAME, OUTPUT_CSV. Remove one-time hyphen-strip for TIN/NPI in
PROV_INFO_JSON and remove _strip_hyphens_from_prov_info_json_cell.
- Keep json_utils PROV_INFO_JSON helpers (parse_prov_info_json_cell,
serialize_prov_info_json, format_prov_info_json) and (str,str) serialization
in contract_config_postprocess. Keep fill_prov_info_tin_from_filename_tin and
related helpers in postprocessing_funcs for potential upstream reuse.
- Tests in test_json_parsers.py updated for (str,str) PROV_INFO_JSON output.
* Add arbitration-based implicit code flow and prefer Level 2 RAG (Fix 5d, 4c+7)
Implicit flow no longer uses first-match-wins. We build a case from all
applicable strategies and run a single arbitration step to choose the
best candidate.
- build_implicit_candidates: Run Category (when explicit had Category),
Special, and RAG; return list of {source, code_answer_dict}. RAG
result uses preferred level when both return (Fix 5d).
- code_implicit_rag (Fix 5d): Run both Level 1 and Level 2, collect
results; when both return non-empty, prefer Level 2 over Level 1.
- code_implicit_arbitration: 0 candidates -> None; 1 candidate -> return
it with CODE_METHODOLOGY = 'Implicit - Arbitration (Source)' (no LLM);
2+ candidates -> one CODE_IMPLICIT_ARBITRATION LLM call (chosen_index or
no_match). If no candidates or no_match, code_last_check runs as before.
- prompt_templates: Add CODE_IMPLICIT_ARBITRATION and
CODE_IMPLICIT_ARBITRATION_INSTRUCTION (cached). Use
_create_json_dict_parser(field_names=['chose…
* fix: test_code_breakout avoid LLM call and flaky order on CI
- Add BILL_TYPE_CD_DESC to test data so pre-pass skips fill_bill_type (no Bedrock on CI)
- Use side_effect function keyed by SERVICE_TERM for deterministic result under ThreadPoolExecutor
* ran black
* feat: hybrid code validation with unmatched tagging and code quality fixes
Validation and unmatched tagging:
- Add format-only filtering for explicit codes (drop wrong-format, preserve valid)
- Add unified _has_unmatched_codes() used by both explicit and implicit paths
- Tag CODE_METHODOLOGY with "- Unmatched" when codes are format-valid but
not explicit keys in our mappings (codes and range keys count as mapped)
- Remove retry logic and "Explicit - Validation Failed" in favor of single
LLM call with Unmatched tagging
- Include all revenue level mappings in valid revenue set
Code quality and bug fixes:
- Fix fill_bill_type += string bug (was splitting "11X" into chars)
- Fix bare except clauses in code_explicit, code_category,
code_implicit_special, fill_bill_type (now except Exception as e)
- Replace eval() with json.loads() in fill_grouper_cd_desc
- Implement INVALID_SERVICE short-circuit in RAG (Fix 5a)
- Cache valid procedure/revenue code sets per Constants instance
- Store mappings in level_dicts…
* chore: remove debug print statements from code extraction pipeline
Remove all [DEBUG_CODE_EXTRACTION] print statements added during
development of the hybrid validation and implicit arbitration flow.
* refactor: robust PROV_INFO_JSON sanitization with multi-format parsing
Replace the simple format_prov_info_json with a layered
sanitize_prov_info_json that handles all known malformed variants:
single-quoted dicts, empty-value-after-colon patterns, list-typed
field values, and hyphenated TINs. Extract _normalize_prov_entries
and _prov_value_to_str helpers for uniform str-valued output.
format_prov_info_json now delegates to sanitize_prov_info_json.
* merge: sync feature/code_optimization with DEV
Resolve conflict in prompt_templates.py: keep detailed CRITICAL field
assignment instructions from feature branch; accept new
AARETE_DERIVED_PROVIDER_NAME prompt function from DEV.
* Minor change to explict prompt
* fix: resolve mypy errors and remove stale pipe-delim splits in RAG
- Narrow grouper_cd type to str|None with explicit None check to
satisfy mypy in fill_grouper_cd_desc.
- Remove dead pipe-delim split logic from CPT and HCPCS matching
blocks (no pipes in those mappings).
- Retain pipe split for revenue block only (rev_level1.csv keys are
pipe-delimited, e.g. "0810|0811|0812").
* Code extraction: format-only explicit, keep unmapped tag, RAG Level 1/2 as separate candidates
- Explicit: accept all format-valid codes (single + range); do not drop for mapping.
Validity check only appends "- Unmatched" to methodology when code not in mappings.
- Add _procedure_code_or_range_format_valid; use in validate_explicit_codes and _has_unmatched_codes.
- RAG: return list of Level 1 and Level 2 candidates (distinct source keys); remove Level 2 preference.
- build_implicit_candidates: consume RAG list and append each as separate candidate.
- CODE_EXPLICIT: clarify range extraction when service term describes a range.
- CODE_IMPLICIT_ARBITRATION: choose most appropriate candidate(s), avoid overly broad.
- Tests updated for new RAG return shape and validate_explicit_codes behavior.
* Resolved Merge Conflicts
* Fixed failing tests
* Remove unused functions
* Remove unused functions
* Revert "Remove unused functions"
This reverts commit 39954db6a421e55501282d4e6a270c88624049fe.
* remove unused
* Merged DEV into feature/code_optimization
* Merged DEV into feature/code_optimization
Approved-by: Katon Minhas
Feature/DAIP2-2023 eliminate full context processing
* testing full context fields
* remove full context processing
* merge Dev with DAIP2-2-23
* full context removal in client codes
* AARETE_DERIVED_PROVIDER_NAME field changes
* Merged DEV into feature/DAIP2-2023-eliminate-full-context-processing
* optimized provider name
* black format fix
* contract title fixes
* PAYER NAME AUTO RENEWAL IND fixes
* Merged DEV into feature/DAIP2-2023-eliminate-full-context-processing
* Merge branch 'DEV' into feature/DAIP2-2023-eliminate-full-context-processing
* Remove prints
Approved-by: Katon Minhas
DAIP2-1980 aarete derived effective dt issues
* fixed earlier of a and b
* prompt update
* prompt shortened
* test changes reverted
* Merge branch 'DEV' into DAIP2-1980-aarete-derived-effective-dt-issues
* test changes reverted
* Merged DEV into DAIP2-1980-aarete-derived-effective-dt-issues
* Fix effective date prompt
* Replace em-dashes with readable version
Approved-by: Katon Minhas
Feature/historical cost analysis
* Initial commit
* Finalize cost_analysis
* Black format
* Merged DEV into feature/historical-cost-analysis
* Refactor
* Black
* Merged DEV into feature/historical-cost-analysis
Approved-by: Siddhant Medar
Bugfix/reimb primary issues
* Update lesser of distribution prompt to preserve reimb term langauge
* Added prompt change to extract correct service when there are multiple sub-rates; Updated validate reimb term to prevent filtering out erm with valid rate
* Make reimb primary prompt change
* Revert "Make reimb primary prompt change"
This reverts commit 55f7d73b9cb523097c96ea50b6614c1db53b46d2.
Approved-by: Katon Minhas
Bugfix/prov info json fixes
* fix: robust PROV_INFO_JSON sanitization and TIN backfill logic
json_utils:
- Add sanitize_prov_info_json with layered parsing (JSON, literal_eval,
empty-value-after-colon fix, best-effort dict extraction).
- Add _normalize_prov_entries and _prov_value_to_str for uniform
str-valued output; flatten list values, strip TIN hyphens.
- format_prov_info_json now delegates to sanitize_prov_info_json.
postprocessing_funcs:
- Add fill_prov_info_tin_from_filename_tin for TIN backfill.
- Add validate_and_reformat_date (pipe-wrapped, datetime strings).
- Add format_as_json_list (pipe-delimited, comma-separated, quote
stripping).
postprocess:
- Integrate new postprocessing helpers into pipeline flow.
postprocess_existing_output:
- Support CSV and Excel input, configurable paths, fillna for CSV.
tests:
- Add test_json_parsers.py for PROV_INFO_JSON parsing coverage.
- Add test_postprocess.py for date/list formatting and default_ind.
* Merge branch 'DAIP2-1947-tin-and-prov-info-json-issues' into bugfix/prov_info_json_fixes
* Merged DEV into bugfix/prov_info_json_fixes
* Strip out unused functionality
* Update filename_tin functionality
* Add docstring
* Update filename_tin cleaning in PROV_INFO_JSON
* test prep
* black format
* missing function added
* Merge branch 'DEV' into bugfix/prov_info_json_fixes
* Black format
* Strip unused functions
* Strip unused code
* update unit tests
* Merge branch 'DEV' into bugfix/prov_info_json_fixes
* Update test
* Simplify process
* handle list of group names
* Resolve run_provider_info_field call
* Merge branch 'DEV' into bugfix/prov_info_json_fixes
* Correct type hints
* Fix unit tests
* Fix unit test
* Remove redundant postprocessing_funcs
Approved-by: Katon Minhas
Bugfix/DAIP2-1870 dynamic issues
* updated list format appending
* fill bill type from claim type cd
* Fixed remaining format fixes
* Merged DEV into bugfix/DAIP2-1870-dynamic-issues
* changed crosswalk mapping from PRODUCT to AARETE_DERIVED_PRODUCT
* Merged DEV into bugfix/DAIP2-1870-dynamic-issues
* updating LOB field values as well from PRODUCT AND PROGRAM
* updated apply crosswalk for reverse mapping
* Merged DEV into bugfix/DAIP2-1870-dynamic-issues
* added QHP and BHP
* removed lob reverse mapping
* Merged DEV into bugfix/DAIP2-1870-dynamic-issues
* Remove Bill Type/Claim Type fill
* Merged DEV into bugfix/DAIP2-1870-dynamic-issues
* Black format
* Merged DEV into bugfix/DAIP2-1870-dynamic-issues
Approved-by: Katon Minhas
Feature/exhibit smart chunking
* exhibit processing per page
* header dict deduplication
* dedup prompt refinment
* refinment for header extraction proecess
* Merge remote-tracking branch 'origin/DEV' into feature/exhibit-smart-chunking
* merge updates
* minor fix
* prompt fix for reimb type
* COB defenition for clear understanding
* black formatting
* remove quit statement
* pipiline test
* pipeline test
* black formatting
* Merge remote-tracking branch 'origin/DEV' into feature/exhibit-smart-chunking
* black formating
* Merge remote-tracking branch 'origin/DEV' into feature/exhibit-smart-chunking
* typo
* PR comment fixes
* exhibit funcs refactored
* black formatting
* Refactor exhibit chunking config into dedicated class
Created ExhibitChunkingConfig class to centralize exhibit smart chunking
configuration parameters (DEFAULT_SUBCHUNK_SIZE, MIN_PARENT_CHUNK_SIZE,
CHUNK_RELEVANCE_THRESHOLD). This improves code organization by consolidating
related constants and makes configuration more maintainable.
Changes:
- Created ExhibitChunkingConfig class with ESC_CONFIG instance
- Moved CHUNK_RELEVANCE_THRESHOLD from config.py to ExhibitChunkingConfig
- Updated all constant references to use ESC_CONFIG prefix
- Added missing EXHIBIT_HEADER_MARKERS parameter documentation
* Fix logging levels and refactor imports for exhibit chunking
- Upgrade logging from WARNING to ERROR for embedding and semantic search failures
- Remove unused constant imports from exhibit_funcs.py
- Update test imports to use ESC_CONFIG pattern for configuration constants
- Add warning when no exhibit headers found during deduplication
- Expand mypy type checking by removing s3_utilities from exclude list
* Merged DEV into feature/exhibit-smart-chunking
* Merged DEV into feature/exhibit-smart-chunking
Approved-by: Siddhant Medar
Feature/TIN stats update
* Tin_stats_report
* black formatting
* minor changes
* black format applied
* Addressed the katons comments
* Merged DEV into feature/TIN_stats_update
* black reformat
Approved-by: Katon Minhas
Update reimbursement prompts for consistent PMPM handling, time-unit dedup, and validation improvements
* Update reimbursement prompts for consistent PMPM handling, time-unit dedup, and validation improvements
- REIMBURSEMENT_PRIMARY: append unit of measure (PMPM, per visit, etc.) from table headers into REIMB_TERM
- METHODOLOGY_BREAKOUT: collapse mathematically equivalent time-unit rates into single entry
- VALIDATE_REIMBURSEMENTS: add umbrella terms as valid service terms, accept CPT/HCPCS percentage as concrete rate, narrow COB disqualifier to primary subject only
Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>
Approved-by: Siddhant Medar
Feature/DAIP2-1803 add aarete derived provider name
* Merged DEV into feature/DAIP2-1562-add-aarete_derived_payer_name
* state logic added in clustering
* removed print statements
* pull request updates
* black format fix
* Merged DEV into feature/DAIP2-1562-add-aarete_derived_payer_name
* added scalability feature and optimization
* black format
* Pull Request Changes
* generalised funcs from aarete derived payer name
* updated remove states and updated docstring for build similarity matrix
* derived payer name functionality added
* config and main file changes
* sync dev into ADD AARETE DERIVED PROVIDER GROUP NAME FULL
* updated derived_provider name
* pipeline fixes
* Merged DEV into feature/DAIP2-1803-add-aarete-derived-provider-name
* config changes
* field name updated to AARETE_DERIVED_PROVIDER_NAME
* Merged DEV into feature/DAIP2-1803-add-aarete-derived-provider-name
* black format fix
* Fix docstring and add empty column when not found
* Rename test file and add missing column test case
- Renamed test_derived_payer_name.py to test_derived_names.py since
file now covers both payer and provider name functionality
- Added test_derived_provider_names_column_missing to verify empty
column is added when AARETE_DERIVED_PROVIDER_NAME is not present
* Remove trailing whitespace
* Merge remote-tracking branch 'origin/DEV' into feature/DAIP2-1803-add-aarete-derived-provider-name
Approved-by: Siddhant Medar
Bugfix/default ind postprocess
* Add logic to standardize UNIT_OF_MEASURE for flat-rate reimbursement methods
- Implemented functionality in `standardize_reimb_method_and_fee_schedule` to set UNIT_OF_MEASURE to blank for rows where DEFAULT_IND is 'Y' and AARETE_DERIVED_REIMB_METHOD is 'flat rate'.
- Added unit tests to verify behavior for various scenarios, including case insensitivity and non-default conditions.
- Ensured that UNIT_OF_MEASURE remains unchanged for non-flat rate methods.
* Enhance child rank handling and ensure column consistency in parent-child mapping
- Added initialization for the `child_rank` column in both parents and children DataFrames to prevent KeyError during concatenation when no children exist.
- Updated `cols_to_keep` in `parent_child_mapping` to filter out columns not present in `pc_df`, ensuring robustness in data processing.
* Ran Black
* made a small change in code_last_check, fixed so it returns string and not single char
* Made changes to make sure that default_ind postprocess only happens to the cc output and not dashboard
* Merged DEV into bugfix/default_ind_postprocess
* Restore deleted AARETE_DERIVED_PAYER_NAME functions
Functions were removed during previous commit. Restored from DEV to fix
AttributeError in prompt_calls.py.
* Simplify code_last_check return logic and add error logging
- Simplified return to single line with fallback
- Added error logging for failed LLM response parsing
Approved-by: Siddhant Medar
Bugfix/DAIP2-1701 issue tracker fixes methodology breakout
* testing fee schedules issues
* logging debug added for lesser of
* Merged DEV into bugfix/DAIP2-1701-issue-tracker-fixes-methodology-breakout
* special case term fix
* updated lesser of distribution prompt
* Merged DEV into bugfix/DAIP2-1701-issue-tracker-fixes-methodology-breakout
* refactor: use f-string for special case term concatenation
Approved-by: Siddhant Medar
Feature/DAIP2-1562 add aarete derived payer name
* black format
* Merged DEV into feature/DAIP2-1562-add-aarete_derived_payer_name
* llm_choose_derived_payer_name function added
* threshold updated
* Merge branch 'DEV' into feature/DAIP2-1562-add-aarete_derived_payer_name
* aarete_derived_payer_name column added
* prompt structure updated
* state_flag added
* pipeline error fixed
* pipeline error fixed
* Merged DEV into feature/DAIP2-1562-add-aarete_derived_payer_name
* remove debug print statement for similarity matrix in clustering function
* Merged DEV into feature/DAIP2-1562-add-aarete_derived_payer_name
* state logic added in clustering
* removed print statements
* pull request updates
* black format fix
* Merged DEV into feature/DAIP2-1562-add-aarete_derived_payer_name
* added scalability feature and optimization
* black format
* Pull Request Changes
* updated config parameters
* updated main
* no payer name column changes
* Business team feedback: always run derived payer name with state_flag=True
- Add empty DataFrame guard and proper docstring
- Clean up not needed flags
Approved-by: Siddhant Medar
Bugfix/dynamic issues feb12
* prompt changes reverted
* fix pipeline issues
* fix pipeline issues
* fix pipeline issues
* fixed formatting
* fixed formatting
* Merge branch 'DEV' into Optimize/DAIP2-1474-restructure-postprocess
* Merge branch 'DEV' into Optimize/DAIP2-1474-restructure-postprocess
* save dashboard and cc output separately
* save dashboard output in s3
* pipeline error fixed
* json list through postprocessing
* Merged DEV into Optimize/DAIP2-1474-restructure-postprocess
* Merge remote-tracking branch 'origin/Optimize/DAIP2-1474-restructure-postprocess' into feature/new_output_format
* Restructure output file organization and add standard field sanitization
Output Structure Changes:
- Reorganize output files into hierarchical directory structure:
- full_outputs/cc_results/ for consolidated CC results
- full_outputs/dashboard_results/ for consolidated dashboard results
- full_outputs/ for error files
- automation_qa-qc/ for QC/QA validated results and statistics
- parent-child/ for parent-child mapping outputs
- tracking/ for usage and cost tracking data
- individual/ for per-file CC results (dashboard individual files removed)
- Update file naming conventions to match new structure
- Remove QC/QA processing for error files (error files are saved without validation)
Post-Processing Changes:
- Add standard N/A value cleaning: remove placeholder values (N/A, UNKNOWN, etc.)
when they are the only value in a cell (applies before CC/dashboard split)
- Normalize all _IND fields to contain only 'Y' or 'N' values (no blanks)
- Ensure standard cleaning runs before splitting into CC …
* Ran Black for CI
* Refactor file splitting logic and add comprehensive tests
- Refactored splitting logic in io_utils.py:
- Consolidated repeated splitting code into two focused helper functions:
- _write_local_split_files() for local file writing with splitting
- _write_s3_split_files() for S3 file writing with splitting
- Both helpers use shared split_dataframe_by_filename() function
- Added MAX_ROWS_PER_SPLIT configuration (default: 70000) in config.py
- Added comprehensive test coverage for splitting logic:
- Tests for split_dataframe_by_filename() with various scenarios
- Tests for local and S3 write operations with single and multiple splits
- Tests for cc_results_full, dashboard_results_full, and qc_qa_cc_full output types
- Fixed existing test failures (write_s3 error handling, path assertions)
- Improved code maintainability and readability
* Fix failing tests in test_postprocess.py
- Updated standard_postprocess tests to use actual columns from FIELD_FORMAT_MAPPING
(PAYER_NAME, CONTRACT_TITLE) instead of custom test columns that get dropped
- Added FILE_NAME column to all file structure test DataFrames (required for splitting logic)
- Added MAX_ROWS_PER_SPLIT mock configuration for splitting tests
- Fixed patch decorators for S3 tests to properly mock logging
All 41 tests now passing.
* Black for CI
* Blank [] and ['[]'] in output instead of displaying them
- Add placeholder patterns in clean_na_values for [], ['[]'], ["[]"]
- Update format_as_json_list to return blank for empty lists instead of []
- Filter out empty-list placeholder items from list values in format_as_json_list
- Add tests for clean_na_values empty list handling and format_as_json_list
Co-authored-by: Cursor <cursoragent@cursor.com>
* Merged DEV into feature/new_output_format
* feat: dynamic primary debug improvements and LOB partial 1:1 escalation
- Add per-exhibit debug summary for dynamic primary discovery (page, values, raw LLM)
- Pass exhibit_page to dynamic_primary for debug; mark debug-only params for removal
- Fix 1:1 escalation skip: use base_field for PROGRAM/PRODUCT/NETWORK when LOB detected
- Pass LOB to 1:1 when empty in any row (partial detection from stripped headers)
- Add debug exhibit text preview; mark check_and_combine_exhibit_inheritance for deletion
- prompt_dynamic_primary returns (answer, raw); remove per-field prints
Co-authored-by: Cursor <cursoragent@cursor.com>
* feat: pass PROGRAM and PRODUCT to 1:1 when partial LOB
When LOB is empty in some rows (partial detection), pass LOB, PROGRAM, and
PRODUCT to 1:1 for contract-level extraction. Merge fills only empty cells
so 1:N values are preserved. Skip PROGRAM/PRODUCT/NETWORK only when full LOB.
Co-authored-by: Cursor <cursoragent@cursor.com>
* Removed debug print blocks
* Merge origin/DEV into bugfix/dynamic_issues_feb12
Resolved conflicts:
- runner.py: Use RUN_DASHBOARD_POSTPROCESSING for conditional dashboard; parent-child output_dir and S3 upload
- main.py: Use RUN_DASHBOARD_POSTPROCESSING for conditional dashboard
- postprocess.py: Optional dashboard postprocessing when RUN_DASHBOARD_POSTPROCESSING is True
Co-authored-by: Cursor <cursoragent@cursor.com>
Approved-by: Katon Minhas
Feature/new output format
* bugfix
* Merge branch 'DEV' into Optimize/DAIP2-1474-restructure-postprocess
* requested postprocessing changes
* prompt changes reverted
* fix pipeline issues
* fix pipeline issues
* fix pipeline issues
* fixed formatting
* fixed formatting
* Merge branch 'DEV' into Optimize/DAIP2-1474-restructure-postprocess
* Merge branch 'DEV' into Optimize/DAIP2-1474-restructure-postprocess
* save dashboard and cc output separately
* save dashboard output in s3
* pipeline error fixed
* json list through postprocessing
* Merged DEV into Optimize/DAIP2-1474-restructure-postprocess
* Merge remote-tracking branch 'origin/Optimize/DAIP2-1474-restructure-postprocess' into feature/new_output_format
* Restructure output file organization and add standard field sanitization
Output Structure Changes:
- Reorganize output files into hierarchical directory structure:
- full_outputs/cc_results/ for consolidated CC results
- full_outputs/dashboard_results/ for consolidated dashboard results
- full_outputs/ for error files
- automation_qa-qc/ for QC/QA validated results and statistics
- parent-child/ for parent-child mapping outputs
- tracking/ for usage and cost tracking data
- individual/ for per-file CC results (dashboard individual files removed)
- Update file naming conventions to match new structure
- Remove QC/QA processing for error files (error files are saved without validation)
Post-Processing Changes:
- Add standard N/A value cleaning: remove placeholder values (N/A, UNKNOWN, etc.)
when they are the only value in a cell (applies before CC/dashboard split)
- Normalize all _IND fields to contain only 'Y' or 'N' values (no blanks)
- Ensure standard cleaning runs before splitting into CC …
* Ran Black for CI
* Refactor file splitting logic and add comprehensive tests
- Refactored splitting logic in io_utils.py:
- Consolidated repeated splitting code into two focused helper functions:
- _write_local_split_files() for local file writing with splitting
- _write_s3_split_files() for S3 file writing with splitting
- Both helpers use shared split_dataframe_by_filename() function
- Added MAX_ROWS_PER_SPLIT configuration (default: 70000) in config.py
- Added comprehensive test coverage for splitting logic:
- Tests for split_dataframe_by_filename() with various scenarios
- Tests for local and S3 write operations with single and multiple splits
- Tests for cc_results_full, dashboard_results_full, and qc_qa_cc_full output types
- Fixed existing test failures (write_s3 error handling, path assertions)
- Improved code maintainability and readability
* Fix failing tests in test_postprocess.py
- Updated standard_postprocess tests to use actual columns from FIELD_FORMAT_MAPPING
(PAYER_NAME, CONTRACT_TITLE) instead of custom test columns that get dropped
- Added FILE_NAME column to all file structure test DataFrames (required for splitting logic)
- Added MAX_ROWS_PER_SPLIT mock configuration for splitting tests
- Fixed patch decorators for S3 tests to properly mock logging
All 41 tests now passing.
* Black for CI
* Blank [] and ['[]'] in output instead of displaying them
- Add placeholder patterns in clean_na_values for [], ['[]'], ["[]"]
- Update format_as_json_list to return blank for empty lists instead of []
- Filter out empty-list placeholder items from list values in format_as_json_list
- Add tests for clean_na_values empty list handling and format_as_json_list
Co-authored-by: Cursor <cursoragent@cursor.com>
* Merged DEV into feature/new_output_format
* Pipeline config, parent-child, dashboard, and runner fixes
- Parent-child: enable by default, write to run directory, S3 upload via io_utils
- Dashboard: optional (CC only by default), run_dashboard=True to enable
- Add io_utils.upload_local_file_to_s3 for centralized file uploads
- Parent-child returns (row_count, local_path); pipeline handles S3 upload
- Add TODO in config for WRITE_PC_TO_S3 removal after approval
- Fix indentation errors in runner.py
Co-authored-by: Cursor <cursoragent@cursor.com>
Approved-by: Katon Minhas
Bugfix/generic lesser of
* Updated lesser of ditribution and lesser of check prompt
* Remove excessive logging
* Updated prompt for combining and formatting lesser of statement and reimb term
* Code clean upt
* Removed unneeded instruction on output format
* style: format prompt_calls.py with black
Approved-by: Katon Minhas
Feature/claim type only runner
* Add specific_fields config for running extraction on field groups
## What Changed
4 files modified:
1. src/config.py - Added configuration for field-specific extraction:
- SPECIFIC_FIELDS arg (default: 'all') - pass field group name or comma-separated field names
- FIELD_GROUPS dict - predefined groups: claim_type, dates, provider
- get_specific_fields_list() - resolves config to actual field list
2. src/prompts/fieldset.py - Added filter_by_names() method to FieldSet class to filter fields by a list of names
3. src/pipelines/shared/extraction/one_to_n_funcs.py - Updated exhibit_level() to accept specific_fields parameter and skip prompts for fields not in the list
4. src/pipelines/saas/file_processing.py - Passes specific_fields through the call chain to both one_to_n and one_to_one extraction
## How It Works
When specific_fields is set to something other than 'all':
1. Config resolves the field list (either from FIELD_GROUPS dict or comma-separated names)
2. Before running extraction prompt…
* Add specific_fields filtering for claim_type only runs and fix UTF-8 logging
* Merge remote-tracking branch 'origin/DEV' into feature/claim-type-only-runner
* Fix black formatting in one_to_n_funcs.py
* Fix black formatting
* Fix CLAIM_TYPE_CD and AARETE_DERIVED_CLAIM_TYPE_CD to be single values
- Handle pipe-delimited strings in crosswalk by splitting them
- For CLAIM_TYPE_CD, take only first value in crosswalk since claim type should be singular
- Add single_value_fields handling in normalize_field_value for CLAIM_TYPE_CD fields
- Ensures AARETE_DERIVED_CLAIM_TYPE_CD is always a single M or H value
* Merged in feature/fix-claim-type-mapping (pull request #878)
Feature/fix claim type mapping
* Fix CLAIM_TYPE_CD and AARETE_DERIVED_CLAIM_TYPE_CD to be single values
- Handle pipe-delimited strings in crosswalk by splitting them
- For CLAIM_TYPE_CD, take only first value in crosswalk since claim type should be singular
- Add single_value_fields handling in normalize_field_value for CLAIM_TYPE_CD fields
- Ensures AARETE_DERIVED_CLAIM_TYPE_CD is always a single M or H value
* Update prompt template
* Merged feature/claim-type-only-runner into feature/fix-claim-type-mapping
Approved-by: Faizan Mohiuddin
* Address PR review comments
- Remove single_value_fields special handling from formatting_utils.py
- Remove CLAIM_TYPE_CD only section and pipe-delimited handling from aarete_derived.py
- Reference config.FIELD_GROUPS for provider_fields and date_fields in file_processing.py
* Fix CLAIM_TYPE_CD to AARETE_DERIVED_CLAIM_TYPE_CD mapping for claim_type only runs
- Apply crosswalk mapping in skip_reimbursement_processing block so derived
fields are correctly mapped even when running claim_type only extraction
- Fix _normalize_to_str to return first value when list has multiple elements
instead of returning the list itself (violated str return type)
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
* normalize list to str-list
Approved-by: Katon Minhas
Venkatakrishna Reddy Avula