Merged in DEV (pull request #895)

DEV

* Merged in bugfix/parser-downstream-improvements (pull request #875)

Bugfix/parser downstream improvements

* Refactor: Implement field-aware JSON parsers with centralized normalization

This refactor introduces a robust system for normalizing LLM output based on
field format mappings, ensuring consistent data types throughout the pipeline.

Key Changes:
- Add FIELD_FORMAT_MAPPING constant defining expected formats for all fields
- Create format_normalization.py utility for type-aware normalization
- Update json_utils.py parsers to accept field_names/field_name parameters
- Refactor prompt_templates.py to use parser factories (_create_json_dict_parser,
  _create_json_list_parser) that bind field metadata for automatic normalization
- Update prompt_calls.py to pass field names to parsers, eliminating redundant
  normalization logic
- Remove parse_json_dict_or_list (unused, ambiguous function)
- Simplify METHODOLOGY_BREAKOUT and REIMBURSEMENT_PRIMARY to use helper functions
- Add comprehensive integration tests…
* Update other lists

* Black format

* Remove prints

* Update qa_qc to account for lists in is_empty check

* Resolve issues

* Black format

* Merged main into DEV

* Merged in feature/claim-type-only-runner (pull request #876)

Feature/claim type only runner

* Add specific_fields config for running extraction on field groups

## What Changed

4 files modified:

1. src/config.py - Added configuration for field-specific extraction:
   - SPECIFIC_FIELDS arg (default: 'all') - pass field group name or comma-separated field names
   - FIELD_GROUPS dict - predefined groups: claim_type, dates, provider
   - get_specific_fields_list() - resolves config to actual field list

2. src/prompts/fieldset.py - Added filter_by_names() method to FieldSet class to filter fields by a list of names

3. src/pipelines/shared/extraction/one_to_n_funcs.py - Updated exhibit_level() to accept specific_fields parameter and skip prompts for fields not in the list

4. src/pipelines/saas/file_processing.py - Passes specific_fields through the call chain to both one_to_n and one_to_one extraction

## How It Works

When specific_fields is set to something other than 'all':
1. Config resolves the field l…
* Merged in bugfix/generic_lesser_of (pull request #880)

Bugfix/generic lesser of

* Updated lesser of ditribution and lesser of check prompt

* Remove excessive logging

* Updated prompt for combining and formatting lesser of statement and reimb term

* Code clean upt

* Removed unneeded instruction on output format

* style: format prompt_calls.py with black


Approved-by: Katon Minhas

* Merged in feature/new_output_format (pull request #879)

Feature/new output format

* bugfix

* Merge branch 'DEV' into Optimize/DAIP2-1474-restructure-postprocess

* requested postprocessing changes

* prompt changes reverted

* fix pipeline issues

* fix pipeline issues

* fix pipeline issues

* fixed formatting

* fixed formatting

* Merge branch 'DEV' into Optimize/DAIP2-1474-restructure-postprocess

* Merge branch 'DEV' into Optimize/DAIP2-1474-restructure-postprocess

* save dashboard and cc output separately

* save dashboard output in s3

* pipeline error fixed

* json list through postprocessing

* Merged DEV into Optimize/DAIP2-1474-restructure-postprocess

* Merge remote-tracking branch 'origin/Optimize/DAIP2-1474-restructure-postprocess' into feature/new_output_format

* Restructure output file organization and add standard field sanitization

Output Structure Changes:
- Reorganize output files into hierarchical directory structure:
  - full_outputs/cc_results/ for consolidated CC results
  - full…
* Merged in bugfix/dynamic_issues_feb12 (pull request #882)

Bugfix/dynamic issues feb12

* prompt changes reverted

* fix pipeline issues

* fix pipeline issues

* fix pipeline issues

* fixed formatting

* fixed formatting

* Merge branch 'DEV' into Optimize/DAIP2-1474-restructure-postprocess

* Merge branch 'DEV' into Optimize/DAIP2-1474-restructure-postprocess

* save dashboard and cc output separately

* save dashboard output in s3

* pipeline error fixed

* json list through postprocessing

* Merged DEV into Optimize/DAIP2-1474-restructure-postprocess

* Merge remote-tracking branch 'origin/Optimize/DAIP2-1474-restructure-postprocess' into feature/new_output_format

* Restructure output file organization and add standard field sanitization

Output Structure Changes:
- Reorganize output files into hierarchical directory structure:
  - full_outputs/cc_results/ for consolidated CC results
  - full_outputs/dashboard_results/ for consolidated dashboard results
  - full_outputs/ for error files
  - automation_q…
* Merged in feature/DAIP2-1562-add-aarete_derived_payer_name (pull request #867)

Feature/DAIP2-1562 add aarete derived payer name

* black format

* Merged DEV into feature/DAIP2-1562-add-aarete_derived_payer_name

* llm_choose_derived_payer_name function added

* threshold updated

* Merge branch 'DEV' into feature/DAIP2-1562-add-aarete_derived_payer_name

* aarete_derived_payer_name column added

* prompt structure updated

* state_flag added

* pipeline error fixed

* pipeline error fixed

* Merged DEV into feature/DAIP2-1562-add-aarete_derived_payer_name

* remove debug print statement for similarity matrix in clustering function

* Merged DEV into feature/DAIP2-1562-add-aarete_derived_payer_name

* state logic added in clustering

* removed print statements

* pull request updates

* black format fix

* Merged DEV into feature/DAIP2-1562-add-aarete_derived_payer_name

* added scalability feature and optimization

* black format

* Pull Request Changes

* updated config parameters

* updated main

* no pay…
* Merged in bugfix/DAIP2-1701-issue-tracker-fixes-methodology-breakout (pull request #885)

Bugfix/DAIP2-1701 issue tracker fixes methodology breakout

* testing fee schedules issues

* logging debug added for lesser of

* Merged DEV into bugfix/DAIP2-1701-issue-tracker-fixes-methodology-breakout

* special case term fix

* updated lesser of distribution prompt

* Merged DEV into bugfix/DAIP2-1701-issue-tracker-fixes-methodology-breakout

* refactor: use f-string for special case term concatenation


Approved-by: Siddhant Medar

* Merged in bugfix/DAIP2-1699-issue-tracker-fixes-one-to-one-dates-prov-info (pull request #881)

bugfix/DAIP2-1699-issue-tracker-fixes-one-to-one-dates-prov-info

* contract title fixes

* Add instruction to strip hyphens from TIN and NPI

* prompt update

* Merge branch 'bugfix/auto-renewal-term' into bugfix/DAIP2-1699-issue-tracker-fixes-one-to-one-dates-prov-info

* removed print statements

* pipeline error fixed

* signed_ind logic changed

* signatory_ind logic updated

* prompt update

* prompt update

* pipeline error fixed

* Merge branch 'DEV' into bugfix/DAIP2-1699-issue-tracker-fixes-one-to-one-dates-prov-info

* Merge branch 'DEV' into bugfix/DAIP2-1699-issue-tracker-fixes-one-to-one-dates-prov-info

* requested changes applied

* test case added

* prompt update

* contract title prompt update

* latest prompt added


Approved-by: Siddhant Medar

* Merged in bugfix/default_ind_postprocess (pull request #887)

Bugfix/default ind postprocess

* Add logic to standardize UNIT_OF_MEASURE for flat-rate reimbursement methods

- Implemented functionality in `standardize_reimb_method_and_fee_schedule` to set UNIT_OF_MEASURE to blank for rows where DEFAULT_IND is 'Y' and AARETE_DERIVED_REIMB_METHOD is 'flat rate'.
- Added unit tests to verify behavior for various scenarios, including case insensitivity and non-default conditions.
- Ensured that UNIT_OF_MEASURE remains unchanged for non-flat rate methods.

* Enhance child rank handling and ensure column consistency in parent-child mapping

- Added initialization for the `child_rank` column in both parents and children DataFrames to prevent KeyError during concatenation when no children exist.
- Updated `cols_to_keep` in `parent_child_mapping` to filter out columns not present in `pc_df`, ensuring robustness in data processing.

* Ran Black

* made a small change in code_last_check, fixed so it returns string and n…
* Apply formatting fix

* Merged in DAIP2-1898-one-to-one-issues-filename-tin-auto-renewal-and-contract-title (pull request #888)

DAIP2-1898 one to one issues filename tin auto renewal and contract title

* contract title fixes

* Add instruction to strip hyphens from TIN and NPI

* prompt update

* Merge branch 'bugfix/auto-renewal-term' into bugfix/DAIP2-1699-issue-tracker-fixes-one-to-one-dates-prov-info

* removed print statements

* pipeline error fixed

* signed_ind logic changed

* signatory_ind logic updated

* prompt update

* prompt update

* pipeline error fixed

* Merge branch 'DEV' into bugfix/DAIP2-1699-issue-tracker-fixes-one-to-one-dates-prov-info

* Merge branch 'DEV' into bugfix/DAIP2-1699-issue-tracker-fixes-one-to-one-dates-prov-info

* requested changes applied

* test case added

* prompt update

* contract title prompt update

* prompt update

* Merge branch 'DEV' into DAIP2-1898-one-to-one-issues-filename-tin-auto-renewal-and-contract-title


Approved-by: Siddhant Medar

* Merged in feature/DAIP2-1803-add-aarete-derived-provider-name (pull request #884)

Feature/DAIP2-1803 add aarete derived provider name

* Merged DEV into feature/DAIP2-1562-add-aarete_derived_payer_name

* state logic added in clustering

* removed print statements

* pull request updates

* black format fix

* Merged DEV into feature/DAIP2-1562-add-aarete_derived_payer_name

* added scalability feature and optimization

* black format

* Pull Request Changes

* generalised funcs from aarete derived payer name

* updated remove states and updated docstring for build similarity matrix

* derived payer name functionality added

* config and main file changes

* sync dev into ADD AARETE DERIVED PROVIDER GROUP NAME FULL

* updated derived_provider name

* pipeline fixes

* Merged DEV into feature/DAIP2-1803-add-aarete-derived-provider-name

* config changes

* field name updated to AARETE_DERIVED_PROVIDER_NAME

* Merged DEV into feature/DAIP2-1803-add-aarete-derived-provider-name

* black format fix

* Fix docstr…
* Merged in improve-logging (pull request #886)

Improve logging

* Add timing blocks for comprehensive pipeline logging

- Add timing_utils.timed_block() to key extraction functions in one_to_n_funcs.py:
  - exhibit_level sub-functions (prompt_exhibit_level, dynamic_primary, etc.)
  - reimbursement_level extraction and cleaning
  - breakout functions (methodology_breakout, special_case_breakout)
  - carveout_and_special_case parallel processing
  - one_to_n_cleaning sub-steps (crosswalk, lob_relationship, split_reimb_dates)

- Add timing to dynamic_funcs.py:
  - dynamic_assignment parallel processing

- Add timing to postprocess.py:
  - standard_postprocess, contract_config_postprocess, dashboard_postprocess

- Add timing to preprocess.py:
  - exhibit_chunking sub-steps (get_exhibit_pages, link_exhibit_pages, chunk_by_exhibit)

- Add info-level logging statements for completed operations with row/item counts

* Standardize logging levels and remove deprecated code

DAIP2-1798: Ensure logging statements are at …
* Merged in bugfix/DAIP2-1913-lesser-of-indicator-issues (pull request #889)

Bugfix/DAIP2-1913 lesser of indicator issues

* contract title fixes

* Add instruction to strip hyphens from TIN and NPI

* prompt update

* Merge branch 'bugfix/auto-renewal-term' into bugfix/DAIP2-1699-issue-tracker-fixes-one-to-one-dates-prov-info

* removed print statements

* pipeline error fixed

* signed_ind logic changed

* signatory_ind logic updated

* prompt update

* prompt update

* pipeline error fixed

* Merge branch 'DEV' into bugfix/DAIP2-1699-issue-tracker-fixes-one-to-one-dates-prov-info

* Merge branch 'DEV' into bugfix/DAIP2-1699-issue-tracker-fixes-one-to-one-dates-prov-info

* requested changes applied

* test case added

* prompt update

* contract title prompt update

* prompt update

* Merge branch 'DEV' into DAIP2-1898-one-to-one-issues-filename-tin-auto-renewal-and-contract-title

* handle nested cases of lesser_of and greater_of

* Merge branch 'DEV' into bugfix/DAIP2-1913-lesser-of-indicator-issues

* pi…
* Merged in bugfix/reimb_primary (pull request #890)

Update reimbursement prompts for consistent PMPM handling, time-unit dedup, and validation improvements

* Update reimbursement prompts for consistent PMPM handling, time-unit dedup, and validation improvements

- REIMBURSEMENT_PRIMARY: append unit of measure (PMPM, per visit, etc.) from table headers into REIMB_TERM
- METHODOLOGY_BREAKOUT: collapse mathematically equivalent time-unit rates into single entry
- VALIDATE_REIMBURSEMENTS: add umbrella terms as valid service terms, accept CPT/HCPCS percentage as concrete rate, narrow COB disqualifier to primary subject only

Co-Authored-By: Claude Opus 4.6 <noreply@anthropic.com>


Approved-by: Siddhant Medar

* Merged in feature/update-postprocessing-columns (pull request #893)

move column order to final step

* move column order to final step

* Black


Approved-by: Siddhant Medar

* Merged in bugfix/update-postprocessing-paths (pull request #896)

Update postprocessing paths

* Update postprocessing paths


Approved-by: Siddhant Medar

* Merged in feature/TIN_stats_update (pull request #898)

Feature/TIN stats update

* Tin_stats_report

* black formatting

* minor changes

* black format applied

* Addressed the katons comments

* Merged DEV into feature/TIN_stats_update

* black reformat


Approved-by: Katon Minhas

.gitignore

@@ -108,3 +108,4 @@ TEST*.xml
 # PRD Documents
 *.prd
 *prd.md
+.git.instructions.md