1837 Commits

Author SHA1 Message Date
Katon Minhas 791558808a Merged main into DEV 2026-02-02 20:52:22 +00:00
Sha Brown 6efb281d4b Merged in bugfix/release_dockerfile (pull request #862)
Updated dockerfile to use uv instead of poetry

* Updated dockerfile to use uv instead of poetry


Approved-by: Katon Minhas
2026-02-02 20:48:22 +00:00
Katon Minhas b87766c1be Merge branch 'main' into DEV 2026-02-02 13:29:17 -05:00
Katon Minhas 02d10e711e Merged in hotfix/revert-standardize-list-formats (pull request #861)
Revert "Merged in feature/standardize-list-formats (pull request #844)"

* Revert "Merged in feature/standardize-list-formats (pull request #844)"

This reverts commit faf707265d.
2026-02-02 18:27:37 +00:00
VenkataKrishna Reddy Avula 0efb0e5369 Merged in feature/DAIP2-1529-capture-letters-in-amendment-num (pull request #860)
Feature/DAIP2-1529 capture letters in amendment num

* updated CONTRACT_AMENDMENT_NUM Prompt

* updated EXTRACT_AMENDMENT_NUM_FROM_FILENAME_INSTRUCTION

* updated prompt

* commented amendment number fields from numeric fields

* Merged main into feature/DAIP2-1529-capture-letters-in-amendment-num


Approved-by: Katon Minhas
2026-02-02 14:07:46 +00:00
VenkataKrishna Reddy Avula faf707265d Merged in feature/standardize-list-formats (pull request #844)
Feature/standardize list formats

* updated unit test for code funcs

* pipeline fixes

* Merge branch 'main' into feature/standardize-list-formats

* black format

* removed duplicates

* Merge remote-tracking branch 'origin/main' into feature/standardize-list-formats

* updated prov other tin npi names to list

* prompt calls code made common for cleints and saas

* final format fixes

* removed pipes

* crosswalk fixes

* Merged main into feature/standardize-list-formats

* dubug dynamic fields

* dynamic format fixes

* dyanmic fields fixes

* removed pipes from the whole code

* fixed provider TIN NPI Name format issues

* bill code issue fix

* pipeline error fixes

* Merge branch 'feature/standardize-list-formats' of https://bitbucket.org/aarete/doczy.ai into feature/standardize-list-formats

* Merged main into feature/standardize-list-formats

* Black format

* pipeline errors fixed

* pipeline fixes page funcs and tinnpi funcs

* qa_qc_utils fix


Approved-by: Katon Minhas
2026-02-02 13:59:44 +00:00
Sha Brown 1bc9e6aaf3 Merged in bugfix/generic_dynamic_jan26 (pull request #859)
Bugfix/generic dynamic jan26

* Refactor: Extract dynamic primary metrics logic to separate module

- Created src/testbed/dynamic_primary_metrics.py with all dynamic primary field analysis logic
- Moved tally-based analysis functions to new module for better modularity
- Updated testbed_utils.py to import and use functions from dynamic_primary_metrics
- Simplified testbed_metrics_dynamic_only.py to use new module
- Removed unused analyze_dynamic_primary_fields function (400+ lines)
- Added field filtering in match_rows to handle missing columns gracefully
- Minimal changes to testbed_utils.py (~100 lines vs 845 before)

* feat: Add debug print blocks and prompt changes for dynamic primary fields in 1:1 processing

This commit adds prompt changes for when we pass dynamic primary to 1:1  and comprehensive debug logging for dynamic primary fields when they are escalated to 1:1 processing via Hybrid Smart Chunking (HSC) or Full Context methods.

Changes:
- Added debug print blocks in hybrid_smart_chunking_funcs.py to log:
  * Retrieval question, context chunks, and final prompt for HSC processing
  * Raw LLM output (reasoning + answer) for dynamic primary fields
- Added debug print blocks in prompt_calls.py to log:
  * Contract context, final prompt for Full Context processing
  * Raw LLM output (reasoning + answer) for dynamic primary fields
- Added FULL_CONTEXT_DYNAMIC_PRIMARY_INSTRUCTION() to prompt_templates.py:
  * Provides specific guidance for dynamic primary fields in full context
  * Emphasizes extracting values only if they refer to entire contract
  * Includes pipe-delimited formatting instructions
- Updated dynamic_fun…
* Ran black, for formatting

* Refine dynamic primary field prompts and add Duals auto-detection

- Enhanced DYNAMIC_ASSIGNMENT_INSTRUCTION with structural boundary rules
- Added exhibit header and subsection context hierarchy for LOB assignment
- Implemented Duals auto-detection in dynamic_primary discovery (Medicare+Medicaid -> Duals)
- Added Duals value formatting instruction to prevent oversimplification
- Commented out DUAL_LOB_CHECK for performance testing
- Removed verbose debug print blocks from HSC and Full Context processing
- Added focused debug print for dynamic assignment raw LLM output

* Remove debug print blocks from pipeline files

- Removed all debug print blocks from file_processing.py (8 blocks)
- Removed all debug print blocks from dynamic_funcs.py (8 blocks)
- Removed debug print block from prompt_calls.py (dynamic assignment)
- DUAL_LOB_CHECK remains commented as requested
- Total: 188 lines of debug code removed

* Merge remote-tracking branch 'origin/main' into bugfix/generic_dynamic_jan26

* Ran black for CI

* Remove update_lob_for_duals

* Merge branch 'DEV' into bugfix/generic_dynamic_jan26


Approved-by: Katon Minhas
2026-01-30 23:43:48 +00:00
Sha Brown 89515ca5cc Merged in bugfix/row_count (pull request #858)
Bugfix/row count

* Updated lesser of statements filter; Updated lesser of prompt related to GLOBAL and STANDALONE

* Updated validate reimbursement prompt

* Update lesser of check prompt related to GLOBAL assignment

* Restructured VALIDATE_REIMBURSEMENTS_PROMPT

* Updated reimbursement primary; validate reimbursement prompt

* Updated reimb primary prompt and validate reimb prompt

* Code cleaned up. Updated unittest cases

* Updated validate reimbursement prompt

* Added logging

* Update on validate reimbursement prompt; Code clean up

* Fix Black version conflict in dependencies

* Add Black back to dependencies

* Apply black formatting and update uv.lock

* Resolve conflict

* Merge branch 'DEV' into bugfix/row_count


Approved-by: Katon Minhas
2026-01-30 23:28:48 +00:00
Faizan Mohiuddin 90bed0d5f6 Merged in feature/update-caching (pull request #856)
Feature/update caching

* Enable prompt caching for CODE_LAST_CHECK, FILL_BILL_TYPE, DUAL_LOB_CHECK, and GROUPER_BREAKOUT

- Add CODE_LAST_CHECK_INSTRUCTION() for service specificity classification
- Add FILL_BILL_TYPE_INSTRUCTION() for bill type code determination
- Add DUAL_LOB_CHECK_INSTRUCTION() for Medicare/Medicaid classification
- Update code_funcs.py to use caching for CODE_LAST_CHECK, FILL_BILL_TYPE, GROUPER_BREAKOUT
- Update postprocessing_funcs.py to use caching for DUAL_LOB_CHECK
- Add new instructions to get_cacheable_instructions() for cache warming
- Add unit tests for new instruction functions

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* Fix postprocessing_funcs to remove invalid columns

* Merge branch 'main' into feature/lesser-table-caching-refactor-hybrid

* Apply black formatting to 11 files

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* Fix import paths in testbed files

* Merge main into feature/update-caching

* Add SPLIT_REIMB_DATES_INSTRUCTION for prompt caching

* Merged main into feature/update-caching

* Merge remote-tracking branch 'origin/main' into feature/update-caching

* Merge branch 'feature/update-caching' of https://bitbucket.org/aarete/doczy.ai into feature/update-caching

* Add CODE_* instruction functions for prompt caching

* Merged main into feature/update-caching

* Merged main into feature/update-caching

* Maximize prompt caching by moving static content to instructions

- Refactored 12+ prompt functions to move static rules/examples to _INSTRUCTION()
- VALIDATE_REIMBURSEMENTS: Moved 45+ lines of validation rules to instruction
- DATE_FIX: Moved conversion rules and examples to instruction
- EXHIBIT_LEVEL, EXHIBIT_TITLE_MATCH, EXHIBIT_HEADER, EXHIBIT_LINKAGE: Moved rules
- FEE_SCHEDULE_BREAKOUT, GROUPER_BREAKOUT: Moved extraction rules
- CHECK_PROVIDER_NAME_MATCH: Moved matching rules to instruction
- LOB_RELATIONSHIP: Consolidated split instructions
- DERIVED_TERM_DATE: Moved calculation logic to instruction
- DYNAMIC_PRIMARY: Created separate HEADER and TEXT instructions
- Created new REIMB_DATES_ASSIGNMENT_INSTRUCTION with all date rules
- Updated get_cacheable_instructions() with new instruction functions
- Updated all callers in saas, clover, bcbs_promise pipelines

This enables ~90% cost savings via Claude API prompt caching by ensuring
static instruction text is cached and only dynamic content varies.

* Remove Dynamic Primary Header

* Standardize prompt caching across all LLM prompts for ~90% cost savings

Move static content to _INSTRUCTION() functions for caching:
- METHODOLOGY_BREAKOUT: 8 field definitions with resolved valid_values
- FEE_SCHEDULE_BREAKOUT: 3 field definitions
- GROUPER_BREAKOUT: 11 field definitions
- OUTLIER_BREAKOUT: 8 field definitions
- CODE_EXPLICIT: 8 field definitions
- EXHIBIT_HEADER: static header markers list
- CARVEOUT_CHECK: carveout and special case definitions
- SPECIAL_CASE_ASSIGNMENT: task rules and output format

Update callers to use cache=True, instruction, and usage_label params.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* Apply black formatting to prompt_templates.py

* Update unit tests for prompt caching changes

- Fix function signatures in tests (METHODOLOGY_BREAKOUT, FEE_SCHEDULE_BREAKOUT, GROUPER_BREAKOUT, CODE_EXPLICIT)
- Add tests for CARVEOUT_CHECK and EXHIBIT_HEADER prompts
- Add TestInstructionFieldContent class to verify instruction functions contain field definitions
- Add tests for CARVEOUT_CHECK_INSTRUCTION and CODE_EXPLICIT_INSTRUCTION

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* Refactor instructions to load fields from investment_prompts.json and Constants

Address PR review feedback:
- Add _get_constants() helper for lazy Constants instantiation
- Add _get_fields_text() helper to load fields from investment_prompts.json
- METHODOLOGY_BREAKOUT_INSTRUCTION: Load fields via FieldSet
- FEE_SCHEDULE_BREAKOUT_INSTRUCTION: Load fields via FieldSet
- GROUPER_BREAKOUT_INSTRUCTION: Load fields via FieldSet
- OUTLIER_BREAKOUT_INSTRUCTION: Load fields via FieldSet
- EXHIBIT_HEADER_INSTRUCTION: Load markers from Constants.EXHIBIT_HEADER_MARKERS
- CARVEOUT_CHECK_INSTRUCTION: Load carveouts from Constants.VALID_CARVEOUTS,
  special cases from investment_prompts.json
- CODE_EXPLICIT_INSTRUCTION: Keep hardcoded (no code_explicit field_type in JSON)

Update tests to match actual field names from investment_prompts.json.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* Merge main into feature/update-caching

* Remove deprecated dynamic_primary_header

* Update CODE_EXPLICIT_INSTRUCTION to load fields from investment_prompts.json

Use field_type=code_primary_breakout to load code fields dynamically.

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* Merged main into feature/update-caching

* Merge main into feature/update-caching

* Merge branch 'feature/update-caching' of https://bitbucket.org/aarete/doczy.ai into feature/update-caching


Approved-by: Katon Minhas
2026-01-30 23:23:37 +00:00
VenkataKrishna Reddy Avula ff8ec57ed0 Merged in bugfix/daip2-1440-contract-amendment-num (pull request #849)
Bugfix/daip2 1440 contract amendment num to main

* update full context additional instruction for amendment num

* uv formatting done

* remove print statements

* Merged main into bugfix/daip2-1440-contract-amendment-num

* added filename amendment num

* updated pipelines and removed print statements

* Merged main into bugfix/daip2-1440-contract-amendment-num

* code moved to hybrid smart chunking

* Merged main into bugfix/daip2-1440-contract-amendment-num

* Merged main into bugfix/daip2-1440-contract-amendment-num

* removed categorical values from amendment number


Approved-by: Katon Minhas
2026-01-30 14:33:58 +00:00
Siddhant Medar a5c743f37f Merged in feature/chc-client (pull request #857)
Feature/chc client

* Add CHC client pipeline with Claims Coding/Editing Determinations extraction

- Add CHC client to pipelines with regex-based extraction
- Create extraction/claims_coding_funcs.py for section extraction
- Add chc_prompts.json with non_prompt field type
- Add CHC_PROMPTS_PATH to config
- Add tests for extraction function (11 test cases)

* Add black[jupyter] for code formatting

* Format testing notebook with black

* Remove unused prompt field from chc_prompts.json

* Merged main into feature/chc-client


Approved-by: Katon Minhas
2026-01-29 21:36:08 +00:00
Mayank Aamseek 4d6e08577c Merged in bugfix/provider-name-and-tin (pull request #855)
extract correct provider name to match all TINs

* extract correct provider name to match all TINs

* Merged main into bugfix/provider-name-and-tin


Approved-by: Katon Minhas
2026-01-29 16:25:05 +00:00
Siddhant Medar f412754167 Merged in feature/generic-parent-child (pull request #852)
Feature/generic parent child

* Refactor parent_child module with generic hierarchy lineage

- Add generic_hierarchy_lineage.py for core parent-child mapping
- Add generic_hierarchy_lineage_qc.py for QC validation engine
- Add bcbs_generic_hierarchy_lineage.py for BCBS-specific processing
- Add column_mapper.py for automatic column detection
- Rename parent_child_preprocessing.py to generic_hierarchy_lineage_preprocessing.py
- Remove old parent_child_mapping.py (replaced by generic_hierarchy_lineage.py)
- Fix import paths from src.generic_hierarchy_lineage to src.parent_child
- Replace hardcoded BCBS xwalk path with configurable parameter
- Remove global warning suppression

* Fix code review issues in parent_child module

- Fix import paths from src.generic_hierarchy_lineage to src.parent_child
- Replace hardcoded BCBS crosswalk path with configurable xwalk_path param
- Remove global warnings.filterwarnings('ignore') suppressions
- Change bare except clauses to except Exception with logging
- Add explicit client= CLI argument with backward compatible fallback
- Convert print statements to logging in bcbs_generic_hierarchy_lineage.py

* Add tests for parent_child module and fix JSON path resolution

- Add 55 unit tests covering grouping keys, parent identification,
  child ranking, and QC utilities
- Use pathlib to resolve JSON config paths relative to module location,
  allowing tests to run from any directory

* WIP: Refactor parent_child module structure

* Extract constants to centralized location for parent_child module

- Create src/constants/parent_child/ with generic.py and bcbs.py
- Update pipeline.py to use centralized constants (ASSIGNMENT_NO_PARENT, tier values, grouping key config)
- Update bcbsnc/mapping.py to use BCBS-specific constants
- Update imports in __main__.py and bcbsnc/__init__.py
- Pass client parameter from runner.py and saas/main.py

* Fix code formatting in parent_child module

* Merge remote-tracking branch 'origin/main' into feature/generic-parent-child

* Fix client detection and output path in parent_child module

* Fix typo in numeric_mappings.json filename and update logging

* Add type hints to parent_child module

* Merged main into feature/generic-parent-child


Approved-by: Katon Minhas
2026-01-28 20:18:11 +00:00
Mayank Aamseek 9e2de58ee4 Merged in bugfix/add-greater_of-in-breakout (pull request #853)
added greater_of_ind in methodology breakout

* added greater_of_ind in methodology breakout

* Merged main into bugfix/add-greater_of-in-breakout


Approved-by: Katon Minhas
2026-01-28 16:40:26 +00:00
Katon Minhas ff01deddf0 Merged in feature/remove-bcbs-main (pull request #854)
Remove main

* Remove main


Approved-by: Siddhant Medar
2026-01-28 15:36:02 +00:00
Katon Minhas 55533d5a24 Merged in feature/remove-global-lesser-process (pull request #851)
Feature/remove global lesser process

* Deprecate GLOBAL_LESSER_OF

* remove deprecated

* Remove deprecated

* Merged main into feature/remove-global-lesser-process

* Black formatter

* remove deprecated

* Remove check redundant lesser

* Remove prompt_smart_chunked

* Black format

* Update logging


Approved-by: Sha Brown
2026-01-27 18:37:38 +00:00
Mayank Aamseek 929ff4abf8 Merged in bugfix/handwritten-npi-list-updated (pull request #850)
fix ocr issue in list of NPIs

* fix ocr issue in list od NPIs

* pipeline error fixed


Approved-by: Katon Minhas
2026-01-27 17:28:46 +00:00
Katon Minhas afb6d5185d Merged in feature/lesser-table-caching-refactor-hybrid (pull request #847)
Feature/lesser table caching refactor hybrid

* chore: Remove unused duplicate main.py from shared pipeline

* fix: Correct crosswalk paths in aarete_derived.py

* chore: Remove unused documentation files from fieldExtraction

* docs: Add documentation files to documentation folder

* docs: Update README with uv setup, expanded project structure, and branching conventions

* docs: Add uv installation steps with Ubuntu/WSL emphasis

* Enable prompt caching for all remaining LLM calls

- Add _INSTRUCTION() functions for: EXHIBIT_HEADER, EXHIBIT_LINKAGE,
  EXHIBIT_TITLE_MATCH, DATE_FIX, DERIVED_TERM_DATE, CHECK_PROVIDER_NAME_MATCH,
  SPECIAL_CASE_ASSIGNMENT
- Update all invoke_claude() calls in saas and clover pipelines to use
  cache=True with corresponding _INSTRUCTION() functions
- Add new instructions to get_cacheable_instructions() for cache warming
- Update tests for new instruction functions

Functions now using caching:
- prompt_exhibit_level
- prompt_exhibit_lesser (EXHIBIT_LEVEL_LESSER_OF)
- prompt_fee_schedule_breakout
- prompt_grouper_breakout
- prompt_special_case_assignment
- prompt_exhibit_linkage
- prompt_exhibit_header
- prompt_smart_chunked (ONE_TO_ONE templates)
- prompt_date_fix
- prompt_derived_term_date
- prompt_exhibit_title_match
- provider_name_match_check

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* Reorder

* feat: Add bcbs_promise client pipeline with OFFSET_TERM extraction

- Add new bcbs_promise client with HSC-based OFFSET_TERM field extraction
- Extract full paragraph text of offset/recoupment provisions from contracts
- Derive OFFSET_INDICATOR (Y/N) from OFFSET_TERM presence
- Fix reorder_columns to preserve extra columns not in COLUMN_ORDER
- Update QC/QA output path to outputs/qc_qa/

* fix: Update dev deps and test assertions for QC/QA output path

- Add pytest/pytest-mock to dev dependencies for mypy type checking
- Update test assertions to expect outputs/qc_qa instead of qa_qc_output

* style: Apply black formatting to prompt_templates.py

* Merge main, move scripts

* Archive some scripts

* update py version

* remove .py version file

* Remove ASCII characters

* Restore testbed code

* restore tracking

* Update testbed metrics

* Enable prompt caching for CODE_LAST_CHECK, FILL_BILL_TYPE, DUAL_LOB_CHECK, and GROUPER_BREAKOUT

- Add CODE_LAST_CHECK_INSTRUCTION() for service specificity classification
- Add FILL_BILL_TYPE_INSTRUCTION() for bill type code determination
- Add DUAL_LOB_CHECK_INSTRUCTION() for Medicare/Medicaid classification
- Update code_funcs.py to use caching for CODE_LAST_CHECK, FILL_BILL_TYPE, GROUPER_BREAKOUT
- Update postprocessing_funcs.py to use caching for DUAL_LOB_CHECK
- Add new instructions to get_cacheable_instructions() for cache warming
- Add unit tests for new instruction functions

🤖 Generated with [Claude Code](https://claude.com/claude-code)

Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>

* Fix postprocessing_funcs to remove invalid columns

* Merge branch 'main' into feature/lesser-table-caching-refactor-hybrid

* Revert prompt caching changes from aed1b73c

* update formatting

* Update imports


Approved-by: Sha Brown
Approved-by: Praneel Panchigar
2026-01-26 16:52:55 +00:00
Praneel Panchigar dcec3f4b7a Merged in bugfix/preserve_lob_relationship_llm_values (pull request #846)
Bugfix/preserve lob relationship llm values

* Relationship values changes

* Fix: Preserve LLM-determined LOB relationship values

- Prevent fill_na_mapping from overwriting LLM-determined 'Inclusive' relationships
- Only set 'Exclusive' when relationship field is empty/NA (preserves LLM values)
- Default to 'Exclusive' when PROGRAM/PRODUCT is NA/blank (LLM bypassed cases)
- Condensed redundant conditional logic for better maintainability

* Changed LOB Relationship prompt to more strictly enforce an output format

* Merge remote-tracking branch 'origin/main' into bugfix/preserve_lob_relationship_llm_values


Approved-by: Katon Minhas
2026-01-23 21:53:07 +00:00
Praneel Panchigar ab31139168 Merged in bugfix/preserve_lob_relationship_llm_values (pull request #842)
Bugfix/preserve lob relationship llm values

* Relationship values changes

* Fix: Preserve LLM-determined LOB relationship values

- Prevent fill_na_mapping from overwriting LLM-determined 'Inclusive' relationships
- Only set 'Exclusive' when relationship field is empty/NA (preserves LLM values)
- Default to 'Exclusive' when PROGRAM/PRODUCT is NA/blank (LLM bypassed cases)
- Condensed redundant conditional logic for better maintainability

* Merged main into bugfix/preserve_lob_relationship_llm_values


Approved-by: Katon Minhas
2026-01-22 18:12:35 +00:00
Sha Brown 05a615e8b2 Merged in bugfix/claim_type_code (pull request #845)
Bugfix/claim type code

* Added retrieval question for hybrid smart chunking CLAIM_TYPE_CD

* Update on hybrid smart chunking retrival question

* Merged main into bugfix/claim_type_code


Approved-by: Katon Minhas
2026-01-22 16:39:02 +00:00
Katon Minhas 59e5fe4532 Merged in feature/postprocess-script (pull request #843)
Add postprocess script

* Add postprocess script


Approved-by: Siddhant Medar
2026-01-22 16:05:10 +00:00
VenkataKrishna Reddy Avula 96790f04ba Merged in restructure/facility_adjustments_group (pull request #836)
Restructure/facility adjustments group to main

* removed _PCT_RATE and _FEE_RATE in facility adjustment_breakout

* moved facility adjustments term to exhibit level

* added facility adjustmnet breakout

* Merged main into restructure/facility_adjustments_group

* Update breakout


Approved-by: Katon Minhas
2026-01-21 16:42:39 +00:00
Mayank Aamseek c62dafb7c1 Merged in bugfix/conversion-factor (pull request #841)
Bugfix/conversion factor

* fixed default value

* fixed conversion_factor prompt

* post processing error fixed

* fixed missing reimb primaries

* Merged main into bugfix/conversion-factor


Approved-by: Katon Minhas
2026-01-21 16:07:28 +00:00
Mayank Aamseek 7b05d6bfe2 Merged in bugfix/rate-escalator-name (pull request #838)
field name updated

* field name updated


Approved-by: Katon Minhas
2026-01-21 16:05:26 +00:00
Karan Desai 6679bf7f47 Merged in feature/pc-functionality (pull request #835)
bug fix for pc module

* bug fix for pc module


Approved-by: Katon Minhas
2026-01-12 21:29:53 +00:00
Mayank Aamseek 87f3d2f4f0 Merged in bugfix/mb-prompts (pull request #833)
Bugfix/mb prompts

* reimb primary prompt update

* Merge branch 'main' into bugfix/highmark-reimb-primary

* prompt reorganised

* Merged main into bugfix/mb-prompts

* mb prompt refinement

* Merge branch 'bugfix/mb-prompts' of https://bitbucket.org/aarete/doczy.ai into bugfix/mb-prompts

* Merged main into bugfix/mb-prompts

* fee schedule prompt simplified

* Merged in bugfix/methodology_breakout_reimb_pct_rate (pull request #831)

bugfix/methodology_breakout_reimb_pct_rate to bugfix/mb-prompts

* remove case rate from fill empty reimb pct rate

* Merged bugfix/mb-prompts into bugfix/methodology_breakout_reimb_pct_rate


Approved-by: Katon Minhas

* Merged main into bugfix/mb-prompts

* grouper breakout prompt changes

* Merged main into bugfix/mb-prompts

* Merged main into bugfix/mb-prompts

* Update reimb primary prompt


Approved-by: Katon Minhas
2026-01-12 18:18:42 +00:00
Karan Desai 72f633cad2 Merged in bugfix/reimb-primary (pull request #834)
Bugfix/reimb primary

* prompt update for reimb primary

* reimb primary prompt update to extract more info

* prompt update at to priotrize $ amounts

* prompt update for split tbales

* updated valid entries for reimbs

* addition to reimb primary prompt

* reimb primary pormpt update

* Merge remote-tracking branch 'origin/main' into bugfix/reimb-primary
merhing latest main in reimb-primary

* Merged main into bugfix/reimb-primary

* fix- PR comment


Approved-by: Katon Minhas
2026-01-09 22:40:45 +00:00
Faizan Mohiuddin 8c9060e425 Merged in feature/multithreading (pull request #828)
Feature/multithreading

* fixes

* Merge main into feature/multithreading

Resolved conflicts:
- Kept timing instrumentation in file_processing.py
- Kept new 3-step Exhibit-based approach for one-to-n processing
- Maintained parallelization improvements (20 workers)

Changes include:
- Timing utils integration for performance monitoring
- Increased max_workers from 5 to 20 across all components
- Parallelized code_breakout and grouper_breakout
- Fixed tin_npi_funcs function call parameters

* Fix max_workers error for empty documents

- Add check to skip parallel processing when no pages exist
- Use min(len(all_page_tasks), 20) to prevent max_workers=0
- Handles edge case of documents with no exhibits or pages

* Fix max_workers=0 errors in one_to_n_funcs

- Add checks before all ThreadPoolExecutor creations
- Prevents errors when processing empty lists:
  - carveout_and_special_case
  - breakout
  - special_case_breakout
  - filter_services_without_reimbursements
  - run_lob_relationship
- Ensures executor only created when there are items to process

* Reduce code processing parallelism to prevent API throttling

Lower max_workers from 20 to 10 for code_breakout and grouper_breakout
to prevent overwhelming Bedrock API with concurrent requests

* fixed

* Merged main into feature/multithreading

* Move documentation into folder

* Fix logging statements

* Merge branch 'main' into feature/multithreading

* Refactor for clarity

* Update previous exhibit passing logic

* properly simplify exhibits

* Merged main into feature/multithreading

* update conditional for None

* exhibit multithreading changed

* exhibit multithreading changed

* Merge remote-tracking branch 'origin/main' into feature/multithreading

* synced with main

* Parallelize dynamic assignment, refactor HSC field worker, add timing/exhibit unit tests, tidy imports/ignore helpers

* analyze_regression.py edited online with Bitbucket
* count_pages.py edited online with Bitbucket
* compare_regressed_with_baseline.py edited online with Bitbucket
* simple_testbed_compare.py edited online with Bitbucket
* run_testbed_metrics_regressed.py edited online with Bitbucket

Approved-by: Katon Minhas
2026-01-09 20:18:01 +00:00
Praneel Panchigar 69f23653a1 Merged in bugfix/dynamic_issues (pull request #832)
Bugfix/dynamic issues

* Add debug print statements for dynamic primary field discovery and 1:1 escalation tracking

* Add reimb_term to bill type CD determination context

- Updated fill_bill_type() to accept reimb_term parameter
- Modified FILL_BILL_TYPE prompt to include reimbursement term in context
- First attempt now uses: service term + reimbursement term
- Second attempt uses: service term + reimbursement term + exhibit text
- Updated extract_codes_from_service() to pass REIMB_TERM to fill_bill_type()

* Remove debug print statements from dynamic_funcs.py and file_processing.py

- Removed all DEBUG print statements for dynamic primary discovery
- Removed debug prints for field discovery status and 1:1 escalation evaluation
- Removed debug prints for exhibit level answers and final summaries
- Removed debug comment about one_to_one_fields creation
- Cleaned up all temporary debugging code

* Merge branch 'main' into bugfix/dynamic_issues

* Remove duplicate one_to_one_fields FieldSet initialization from process_file


Approved-by: Katon Minhas
2026-01-08 22:41:52 +00:00
VenkataKrishna Reddy Avula d97a31aa5b Merged in optimization/prov_info (pull request #829)
optimization_prov_info to main

* removed on signature page field which is unnecessary


Approved-by: Katon Minhas
2026-01-07 15:19:35 +00:00
VenkataKrishna Reddy Avula 874354dc44 Merged in bugfix/prov_info (pull request #826)
bugfix/prov_info to main

* added instruction to add missing info

* Merged main into bugfix/prov_info

* Prompt update


Approved-by: Katon Minhas
2026-01-06 17:40:02 +00:00
VenkataKrishna Reddy Avula a9b08388e3 Merged in bugfix/payer_name (pull request #827)
Bugfix/payer name to main

* added full context instruction for payer name

* removed print statement


Approved-by: Katon Minhas
2026-01-06 16:40:59 +00:00
Karan Desai 04a190d5b7 Merged in bugfix/hmk-issues (pull request #824)
Bugfix/hmk issues

* fix for missing AARETE_DERIVED_LOB

* fix for AARETE_DERIVED_SID

* bugfix in universal_json_load

* bugfix for normalize_state_field

*  reimb eff date assignmnet prompt update

* prompt update for missing reimb terms

* Merged main into bugfix/hmk-issues

* test case for universal_json_loads

* reveresed the prompt update for missing reimb

* prompt fix for reimb dates


Approved-by: Katon Minhas
2026-01-05 18:34:18 +00:00
VenkataKrishna Reddy Avula b0eed711fc Merged in bugfix/carveout (pull request #825)
bugfix/carveout to main

* removed specific service in base covered

* Merged main into bugfix/carveout

* Merged main into bugfix/carveout


Approved-by: Katon Minhas
2026-01-05 15:43:28 +00:00
Mayank Aamseek fd390e3907 Merged in bugfix/highmark-reimb-primary (pull request #823)
reimb primary prompt update

* reimb primary prompt update

* Merge branch 'main' into bugfix/highmark-reimb-primary

* Update reimb primary prompt


Approved-by: Katon Minhas
2026-01-05 15:41:09 +00:00
VenkataKrishna Reddy Avula 49e8964986 Merged in bugfix/provider_tin (pull request #822)
bugfix/provider_tin to main

* false positives case added to prompt


Approved-by: Katon Minhas
2026-01-05 14:15:08 +00:00
Katon Minhas 541a0f97e0 Merged in hotfix/highmark-issues (pull request #821)
Remove EXHIBIT_TEXT column (and all other invalid columns)

* Remove EXHIBIT_TEXT column (and all other invalid columns)
2025-12-29 20:36:22 +00:00
Karan Desai a8146cad11 Merged in bugfix/hmk-1-to-1 (pull request #820)
fix for redundantly populated  provider twice

* fix for redundantly populated  provider twice

* Merged main into bugfix/hmk-1-to-1


Approved-by: Katon Minhas
2025-12-26 20:44:04 +00:00
Praneel Panchigar 85f2c8cecc Merged in bugfix/bill_type_code_fixes (pull request #819)
Bugfix/bill type code fixes

* Fix LOB_PROGRAM_RELATIONSHIP extraction and preserve debugging code

- Add format requirements to LOB_RELATIONSHIP_INSTRUCTION for proper caching
- Strengthen LOB_RELATIONSHIP prompt with explicit pipe delimiter requirements
- Preserve all DEBUG blocks and print statements in dynamic_funcs.py and file_processing.py
- Update crosswalk mappings and gitignore
- Ensure format instructions are cached at API level to prevent missing pipe delimiters

* Merge main into bugfix/dynamic_issuefixes

Resolved merge conflicts in:
- fieldExtraction/src/investment/dynamic_funcs.py (merged debug print statements)
- fieldExtraction/src/investment/file_processing.py (merged debug statements and exhibit inheritance code)

All conflicts resolved successfully.

* Update FILL_BILL_TYPE prompt with place-of-service priority and optional exhibit_text

- Add optional exhibit_text parameter to FILL_BILL_TYPE function
- Update instructions to prioritize place-of-service over service-only terminology
- Add new examples: 'home dialysis' and 'dialysis' while keeping original examples
- Add conditional exhibit context section when exhibit_text is provided
- Maintains backward compatibility (exhibit_text defaults to None)

Ref: PLAN_BILL_TYPE_CD_IMPROVEMENTS.md

* Implement two-step bill type CD determination with exhibit text fallback

- Add EXHIBIT_TEXT to answer_dict in combine_one_to_n() for contextual analysis
- Update fill_bill_type() to use two-step approach:
  1. First attempt: service term only (no exhibit text)
  2. Second attempt: service term + exhibit text (only if first fails)
- Update extract_codes_from_service() to pass exhibit_text to fill_bill_type()
- Add debug print when full exhibit text context is used
- More efficient: only makes second LLM call when service term alone fails

Ref: PLAN_BILL_TYPE_CD_IMPROVEMENTS.md

* Remove debug print statement and merge main into bill_type_code_fixes

- Remove debug print statement from fill_bill_type() in code_funcs.py
- Merge latest changes from main branch
- Includes updates to prompt_templates.py (payer_name parameter)
- Includes updates to file_processing.py (provider info ordering)
- Includes updates to aarete_derived.py, crosswalk mappings, and tin_npi_funcs

* Remove remaining debug print statements from dynamic_funcs.py and file_processing.py


Approved-by: Katon Minhas
2025-12-26 18:24:57 +00:00
Praneel Panchigar b64e1fe4cc Merged in bugfix/derived_lob_changes (pull request #817)
Bugfix/derived lob changes

* Fix LOB_PROGRAM_RELATIONSHIP extraction and preserve debugging code

- Add format requirements to LOB_RELATIONSHIP_INSTRUCTION for proper caching
- Strengthen LOB_RELATIONSHIP prompt with explicit pipe delimiter requirements
- Preserve all DEBUG blocks and print statements in dynamic_funcs.py and file_processing.py
- Update crosswalk mappings and gitignore
- Ensure format instructions are cached at API level to prevent missing pipe delimiters

* Merge main into bugfix/dynamic_issuefixes

Resolved merge conflicts in:
- fieldExtraction/src/investment/dynamic_funcs.py (merged debug print statements)
- fieldExtraction/src/investment/file_processing.py (merged debug statements and exhibit inheritance code)

All conflicts resolved successfully.

* Fix AARETE_DERIVED_LOB to merge values from all crosswalks (PROGRAM and PRODUCT)

- Updated fill_na_mapping to always check both AARETE_DERIVED_PROGRAM and PRODUCT crosswalks
- Merges unique values from all sources (existing, PROGRAM, PRODUCT) instead of only filling when empty
- Ensures Duals is included when PRODUCT contains Molina Medicare Options Plus
- Deduplicates and sorts values before pipe-delimiting
- Each row processed individually for correct LOB assignment per reimbursement term

* Merged main into bugfix/derived_lob_changes


Approved-by: Katon Minhas
2025-12-22 19:40:31 +00:00
Sha Brown 77c845a82e Merged in bugfix/prov-info-issues (pull request #816)
Bugfix/prov info issues

* Initial commit

* Passed payer name to tin npi llm call to avoid assigning tin npi to payer

* Removed record that contains NO_IDENTIFIERS_FOUND from PROV_INFO_JSON

* Updated unit test for test_tin_npi_funcs

* Changed name check code back to original to be consistent

* Updated test_tin_npi_funcs.py to pass in payer_name in functions

* Add fix to unit test

* Updated output in unit tests

* Merged main into bugfix/prov-info-issues


Approved-by: Katon Minhas
2025-12-22 19:03:09 +00:00
Mayank Aamseek 7ffd62fd8e Merged in bugfix/highmark-methodology-breakout (pull request #815)
Bugfix/highmark methodology breakout

* fix reimb_term rows for dates

* reimb_primary prompt update

* prompt changes

* Merge remote-tracking branch 'origin/main' into bugfix/highmark-methodology-breakout

* prompt changes

* prompt updates

* Merge branch 'main' into bugfix/highmark-methodology-breakout

* prompt updates

* Prompt updates


Approved-by: Katon Minhas
2025-12-22 18:58:34 +00:00
VenkataKrishna Reddy Avula e8cb48643c Merged in bugfix/carveout_base (pull request #814)
bugfix/carveout_base to main

* updated base covered services


Approved-by: Katon Minhas
2025-12-18 16:51:56 +00:00
Sha Brown 46494274ff Merged in bugfix/one_to_one_and_prov_info (pull request #808)
Bugfix/one to one and prov info

* Add post-processing logic to update PROV_INFO_JSON if needed; Updated prompt to verify the number of digitis in TIN and NPI

* Added update_prov_info_with_group_name to postprocessing_funcs.py

* Updated FIRST_PAGE_PROVIDER_NAME_TEMPLATE to ask LLM to explain resoning before answering

* Updated logic to use LLM to check if provider name matches

* Use only Hybrid Smart Chunking to get provider group name

* Cleaned up code

* Removed commented out code

* Updated unit test

* Merged main into bugfix/one_to_one_and_prov_info

* Edited prompt to distinguish provider names and to enclose answer in pipes

* Updated test_tin_npi_funcs


Approved-by: Katon Minhas
2025-12-17 22:37:06 +00:00
Praneel Panchigar d3bfec0b6b Merged in bugfix/dynamic_issuefixes-clean (pull request #813)
Fix LOB_PROGRAM_RELATIONSHIP extraction and preserve debugging code

* Fix LOB_PROGRAM_RELATIONSHIP extraction and preserve debugging code

- Add format requirements to LOB_RELATIONSHIP_INSTRUCTION for proper caching
- Strengthen LOB_RELATIONSHIP prompt with explicit pipe delimiter requirements
- Preserve all DEBUG blocks and print statements in dynamic_funcs.py and file_processing.py
- Update crosswalk mappings and gitignore
- Ensure format instructions are cached at API level to prevent missing pipe delimiters

* Merge main into bugfix/dynamic_issuefixes-clean and remove all debug print statements

* Remove redundant one_to_one_fields initialization to match main


Approved-by: Katon Minhas
2025-12-17 20:42:13 +00:00
Praneel Panchigar f26b83bd0b Merged in feature/cross-exhibit-dynamic (pull request #789)
Feature/cross exhibit dynamic

* Merge branch 'feature/deprecate-haiku-3' into feature/1toN-Optimization

* fix over-filtering of lesser of

* Merged in bugfix/UT-methodology-breakout (pull request #794)

Bugfix/UT methodology breakout

* updated valid values for AARETE_DERIVED_REIMB_METHOD

* removed example reimbursements

* prompt update

* prompt update

* Merged feature/1toN-Optimization into bugfix/nv_issue_fixes

* add service term in mb prompts

* Merged feature/1toN-Optimization into bugfix/UT-methodology-breakout

* print statement removed

* Merge branch 'bugfix/UT-methodology-breakout' of https://bitbucket.org/aarete/doczy.ai into bugfix/UT-methodology-breakout

* primary prompt update

* remove duplicate prompt


Approved-by: Katon Minhas

* Merge branch 'main' into feature/1toN-Optimization

* Merge branch 'main' into feature/1toN-Optimization

* Update preprocessing to make the exhibit_chunk_mapping start at first page

* Merge remote-tracking branch 'origin/feature/1toN-Optimization' into cross-exhibit-dynamic

* Address merge request comments: refactor prompt templates and logging

- Make DYNAMIC_PRIMARY_TEXT LOB-specific instructions conditional (only show when field_name is LOB)
- Remove MEDICAID FEE SCHEDULE point from DYNAMIC_ASSIGNMENT (point #2)
- Generalize SERVICE_TERM context guidance to apply to all fields (LOB, PROGRAM, NETWORK, PRODUCT)
- Refactor proximity guidance to emphasize contextual connection over strict section boundaries
- Update point #1 to explicitly prevent inferring LOB from Programs alone
- Change exhibit inheritance logging from info to debug level

* Move function to one_to_n_funcs

* Merge row_funcs.py changes from feature/cross-exhibit-dynamic

* Merged feature/1toN-Optimization into feature/cross-exhibit-dynamic

* Merged in bugfix/UT-grouper-issues (pull request #796)

Bugfix/UT grouper issues

* updated valid values for AARETE_DERIVED_REIMB_METHOD

* removed example reimbursements

* prompt update

* prompt update

* Merged feature/1toN-Optimization into bugfix/nv_issue_fixes

* add service term in mb prompts

* Merged feature/1toN-Optimization into bugfix/UT-methodology-breakout

* print statement removed

* Merge branch 'bugfix/UT-methodology-breakout' of https://bitbucket.org/aarete/doczy.ai into bugfix/UT-methodology-breakout

* primary prompt update

* prompt update

* Merge remote-tracking branch 'origin/feature/1toN-Optimization' into bugfix/UT-grouper-issues

* removed temp changes

* removed temp changes

* Update reimb primary


Approved-by: Katon Minhas

* Merged in bugfix/validation_fixes (pull request #795)

bugfix/validation_fixes to feature/1toN-Optimization

* updated validation of clean claims reimbursement

* Merged feature/1toN-Optimization into bugfix/validation_fixes


Approved-by: Katon Minhas

* Merge branch 'feature/1toN-Optimization' into feature/cross-exhibit-dynamic

* Re-add dynamic codes and reimb-info

* Re-structure empty reimbursement prompt

* Fix lesser of check overfiltering

* Merged feature/1toN-Optimization into feature/cross-exhibit-dynamic

* Update LOB inference logic and prompt guidance

- Revert DYNAMIC_PRIMARY_TEXT to original template (remove LOB-specific conditional section)
- Add explicit guidance in DYNAMIC_ASSIGNMENT: presence of Medicaid programs does NOT imply LOB is Medicaid
- Update get_dynamic_one_to_one_fields to only check AARETE_DERIVED_LOB (not raw LOB) when skipping PROGRAM/PRODUCT/NETWORK

* Fix LOB_PROGRAM_RELATIONSHIP extraction: add pipe format instructions to LOB_RELATIONSHIP_INSTRUCTION

* Update Fidelis Essential Plan mappings in crosswalk_product_lob.json

- Add mappings for Essential Plan variants (Aliessa, EP-QHP, EP, Essential Plan)
- Essential Plan Aliessa and EP Aliessa map to Medicaid
- EP-QHP maps to Commercial
- EP and Essential Plan map to Medicaid|Commercial

* Remove debug logging statements from prompt_lob_relationship

* Merge main into feature/cross-exhibit-dynamic: resolved conflicts, removed debug statements, synced prompt_templates.py with main

* Sync non-exhibit-merge files with main before merge

* Remove test.py to match main


Approved-by: Katon Minhas
2025-12-17 19:15:49 +00:00
Karan Desai ca53ff8395 Merged in bugfix/cpt-codes (pull request #812)
Bugfix/cpt codes

* formatting fix for revenue_cd

* format fix for procedure_codes

* bugfix for revenue_codes

* working version from code main

* code formatting from main and code main

* removed dubug prints

* removed local file paths

* standardize state fields

* updated test scenerios

* minor fix

* Merge branch 'main' into bugfix/cpt-codes
merging main into cpt code fix

* checks for all fields

* reused is_empty functionality

* moved normalize_to_json_list to utils

* Merge branch 'main' into bugfix/cpt-codes
merging upto date main

* minor touches

* pytest fix


Approved-by: Katon Minhas
2025-12-17 19:05:09 +00:00
VenkataKrishna Reddy Avula f31c02517a Merged in bugfix/carveout_cd (pull request #811)
bugfix/carveout_cd to main

* added age constraint for payment carveout

* Merged main into bugfix/carveout_cd


Approved-by: Katon Minhas
2025-12-17 16:10:32 +00:00
Siddhant Medar c302c1023a Merged in bugfix/qc_qa_filetype (pull request #810)
Bugfix/qc qa filetype

* WIP: support multi-filetype

* Add graceful error handling to QC/QA standalone CLI

Wrap validation pipeline in try/except to ensure no partial results
are created/uploaded if the pipeline fails mid-execution.

* Add comprehensive QC/QA unit tests, improve coverage to 95%

- Add CLI integration tests for __main__.py (6 tests)
- Add pipeline validation tests for all validation steps (20+ tests)
- Test multi-filetype I/O (CSV, XLSX input/output)
- Test graceful error handling
- Coverage: 43% → 95%

* Merge remote-tracking branch 'origin/main' into bugfix/qc_qa_filetype


Approved-by: Katon Minhas
2025-12-16 20:56:55 +00:00