Commit Graph

89 Commits

Author SHA1 Message Date
Sha Brown 77c845a82e Merged in bugfix/prov-info-issues (pull request #816)
Bugfix/prov info issues

* Initial commit

* Passed payer name to tin npi llm call to avoid assigning tin npi to payer

* Removed record that contains NO_IDENTIFIERS_FOUND from PROV_INFO_JSON

* Updated unit test for test_tin_npi_funcs

* Changed name check code back to original to be consistent

* Updated test_tin_npi_funcs.py to pass in payer_name in functions

* Add fix to unit test

* Updated output in unit tests

* Merged main into bugfix/prov-info-issues


Approved-by: Katon Minhas
2025-12-22 19:03:09 +00:00
Sha Brown 46494274ff Merged in bugfix/one_to_one_and_prov_info (pull request #808)
Bugfix/one to one and prov info

* Add post-processing logic to update PROV_INFO_JSON if needed; Updated prompt to verify the number of digitis in TIN and NPI

* Added update_prov_info_with_group_name to postprocessing_funcs.py

* Updated FIRST_PAGE_PROVIDER_NAME_TEMPLATE to ask LLM to explain resoning before answering

* Updated logic to use LLM to check if provider name matches

* Use only Hybrid Smart Chunking to get provider group name

* Cleaned up code

* Removed commented out code

* Updated unit test

* Merged main into bugfix/one_to_one_and_prov_info

* Edited prompt to distinguish provider names and to enclose answer in pipes

* Updated test_tin_npi_funcs


Approved-by: Katon Minhas
2025-12-17 22:37:06 +00:00
Praneel Panchigar d3bfec0b6b Merged in bugfix/dynamic_issuefixes-clean (pull request #813)
Fix LOB_PROGRAM_RELATIONSHIP extraction and preserve debugging code

* Fix LOB_PROGRAM_RELATIONSHIP extraction and preserve debugging code

- Add format requirements to LOB_RELATIONSHIP_INSTRUCTION for proper caching
- Strengthen LOB_RELATIONSHIP prompt with explicit pipe delimiter requirements
- Preserve all DEBUG blocks and print statements in dynamic_funcs.py and file_processing.py
- Update crosswalk mappings and gitignore
- Ensure format instructions are cached at API level to prevent missing pipe delimiters

* Merge main into bugfix/dynamic_issuefixes-clean and remove all debug print statements

* Remove redundant one_to_one_fields initialization to match main


Approved-by: Katon Minhas
2025-12-17 20:42:13 +00:00
Praneel Panchigar f26b83bd0b Merged in feature/cross-exhibit-dynamic (pull request #789)
Feature/cross exhibit dynamic

* Merge branch 'feature/deprecate-haiku-3' into feature/1toN-Optimization

* fix over-filtering of lesser of

* Merged in bugfix/UT-methodology-breakout (pull request #794)

Bugfix/UT methodology breakout

* updated valid values for AARETE_DERIVED_REIMB_METHOD

* removed example reimbursements

* prompt update

* prompt update

* Merged feature/1toN-Optimization into bugfix/nv_issue_fixes

* add service term in mb prompts

* Merged feature/1toN-Optimization into bugfix/UT-methodology-breakout

* print statement removed

* Merge branch 'bugfix/UT-methodology-breakout' of https://bitbucket.org/aarete/doczy.ai into bugfix/UT-methodology-breakout

* primary prompt update

* remove duplicate prompt


Approved-by: Katon Minhas

* Merge branch 'main' into feature/1toN-Optimization

* Merge branch 'main' into feature/1toN-Optimization

* Update preprocessing to make the exhibit_chunk_mapping start at first page

* Merge remote-tracking branch 'origin/feature/1toN-Optimization' into cross-exhibit-dynamic

* Address merge request comments: refactor prompt templates and logging

- Make DYNAMIC_PRIMARY_TEXT LOB-specific instructions conditional (only show when field_name is LOB)
- Remove MEDICAID FEE SCHEDULE point from DYNAMIC_ASSIGNMENT (point #2)
- Generalize SERVICE_TERM context guidance to apply to all fields (LOB, PROGRAM, NETWORK, PRODUCT)
- Refactor proximity guidance to emphasize contextual connection over strict section boundaries
- Update point #1 to explicitly prevent inferring LOB from Programs alone
- Change exhibit inheritance logging from info to debug level

* Move function to one_to_n_funcs

* Merge row_funcs.py changes from feature/cross-exhibit-dynamic

* Merged feature/1toN-Optimization into feature/cross-exhibit-dynamic

* Merged in bugfix/UT-grouper-issues (pull request #796)

Bugfix/UT grouper issues

* updated valid values for AARETE_DERIVED_REIMB_METHOD

* removed example reimbursements

* prompt update

* prompt update

* Merged feature/1toN-Optimization into bugfix/nv_issue_fixes

* add service term in mb prompts

* Merged feature/1toN-Optimization into bugfix/UT-methodology-breakout

* print statement removed

* Merge branch 'bugfix/UT-methodology-breakout' of https://bitbucket.org/aarete/doczy.ai into bugfix/UT-methodology-breakout

* primary prompt update

* prompt update

* Merge remote-tracking branch 'origin/feature/1toN-Optimization' into bugfix/UT-grouper-issues

* removed temp changes

* removed temp changes

* Update reimb primary


Approved-by: Katon Minhas

* Merged in bugfix/validation_fixes (pull request #795)

bugfix/validation_fixes to feature/1toN-Optimization

* updated validation of clean claims reimbursement

* Merged feature/1toN-Optimization into bugfix/validation_fixes


Approved-by: Katon Minhas

* Merge branch 'feature/1toN-Optimization' into feature/cross-exhibit-dynamic

* Re-add dynamic codes and reimb-info

* Re-structure empty reimbursement prompt

* Fix lesser of check overfiltering

* Merged feature/1toN-Optimization into feature/cross-exhibit-dynamic

* Update LOB inference logic and prompt guidance

- Revert DYNAMIC_PRIMARY_TEXT to original template (remove LOB-specific conditional section)
- Add explicit guidance in DYNAMIC_ASSIGNMENT: presence of Medicaid programs does NOT imply LOB is Medicaid
- Update get_dynamic_one_to_one_fields to only check AARETE_DERIVED_LOB (not raw LOB) when skipping PROGRAM/PRODUCT/NETWORK

* Fix LOB_PROGRAM_RELATIONSHIP extraction: add pipe format instructions to LOB_RELATIONSHIP_INSTRUCTION

* Update Fidelis Essential Plan mappings in crosswalk_product_lob.json

- Add mappings for Essential Plan variants (Aliessa, EP-QHP, EP, Essential Plan)
- Essential Plan Aliessa and EP Aliessa map to Medicaid
- EP-QHP maps to Commercial
- EP and Essential Plan map to Medicaid|Commercial

* Remove debug logging statements from prompt_lob_relationship

* Merge main into feature/cross-exhibit-dynamic: resolved conflicts, removed debug statements, synced prompt_templates.py with main

* Sync non-exhibit-merge files with main before merge

* Remove test.py to match main


Approved-by: Katon Minhas
2025-12-17 19:15:49 +00:00
Praneel Panchigar 87a2b8b4ce Merged in issuefix/hmk-1-to-n (pull request #806)
Issuefix/hmk 1 to n

* Merge branch 'main' into feature/1toN-Optimization

* Merge branch 'main' into feature/1toN-Optimization

* Update preprocessing to make the exhibit_chunk_mapping start at first page

* Merge row_funcs.py changes from feature/cross-exhibit-dynamic

* Merged in bugfix/UT-grouper-issues (pull request #796)

Bugfix/UT grouper issues

* updated valid values for AARETE_DERIVED_REIMB_METHOD

* removed example reimbursements

* prompt update

* prompt update

* Merged feature/1toN-Optimization into bugfix/nv_issue_fixes

* add service term in mb prompts

* Merged feature/1toN-Optimization into bugfix/UT-methodology-breakout

* print statement removed

* Merge branch 'bugfix/UT-methodology-breakout' of https://bitbucket.org/aarete/doczy.ai into bugfix/UT-methodology-breakout

* primary prompt update

* prompt update

* Merge remote-tracking branch 'origin/feature/1toN-Optimization' into bugfix/UT-grouper-issues

* removed temp changes

* removed temp changes

* Update reimb primary


Approved-by: Katon Minhas

* Merged in bugfix/validation_fixes (pull request #795)

bugfix/validation_fixes to feature/1toN-Optimization

* updated validation of clean claims reimbursement

* Merged feature/1toN-Optimization into bugfix/validation_fixes


Approved-by: Katon Minhas

* Re-add dynamic codes and reimb-info

* Re-structure empty reimbursement prompt

* Fix lesser of check overfiltering

* Merged in bugfix/generic-methodology-breakout (pull request #797)

Bugfix/generic methodology breakout

* fee schedule fix

* carveout+special case prompt changed

* fee schedule changes

* prompt changes and column order


Approved-by: Katon Minhas

* pytest fix

* remove prints

* Update valid_carveouts.json to include level designations in PAYMENT_CARVEOUT definition and add adultBasic mapping for Highmark

* Fix REIMB_DATES extraction to capture only relevant date range per term

- Updated REIMB_DATES prompt to extract only the date range that applies to the specific reimbursement term
- Added proximity-based extraction: looks for dates in same section/table as the term
- Prevents cross-contamination from other contract sections
- Handles 'Same as Effective Date of Agreement' case
- Returns single date range instead of multiple comma-separated ranges

* Update prompt templates

* Update tracked files and add untracked_tests to gitignore

* Merge branch 'main' into issuefix/hmk-1-to-n

* complete merge

* complete merge

* Merge branch 'main' into issuefix/hmk-1-to-n

* Remove prints

* Update gitignore

* Revert BASE_COVERED_SERVICES and REIMB_DATES prompts to original definitions

- Revert BASE_COVERED_SERVICES definition to original concise version
- Revert REIMB_DATES prompt to original general description (specific assignment logic handled by dedicated template)
- Add DEBUG comment blocks around print statements in dynamic_funcs.py and file_processing.py for easy identification and toggling

* remove prints

* Merge branch 'main' into issuefix/hmk-1-to-n


Approved-by: Katon Minhas
2025-12-12 18:36:57 +00:00
Mayank Aamseek 1de74c9322 Merged in Optimization/reimb-primary (pull request #800)
Optimization/reimb primary

* Merged in bugfix/UT-methodology-breakout (pull request #794)

Bugfix/UT methodology breakout

* updated valid values for AARETE_DERIVED_REIMB_METHOD

* removed example reimbursements

* prompt update

* prompt update

* Merged feature/1toN-Optimization into bugfix/nv_issue_fixes

* add service term in mb prompts

* Merged feature/1toN-Optimization into bugfix/UT-methodology-breakout

* print statement removed

* Merge branch 'bugfix/UT-methodology-breakout' of https://bitbucket.org/aarete/doczy.ai into bugfix/UT-methodology-breakout

* primary prompt update

* remove duplicate prompt


Approved-by: Katon Minhas

* Merge branch 'main' into feature/1toN-Optimization

* Merge branch 'main' into feature/1toN-Optimization

* Update preprocessing to make the exhibit_chunk_mapping start at first page

* Merge row_funcs.py changes from feature/cross-exhibit-dynamic

* Merged in bugfix/UT-grouper-issues (pull request #796)

Bugfix/UT grouper issues

* updated valid values for AARETE_DERIVED_REIMB_METHOD

* removed example reimbursements

* prompt update

* prompt update

* Merged feature/1toN-Optimization into bugfix/nv_issue_fixes

* add service term in mb prompts

* Merged feature/1toN-Optimization into bugfix/UT-methodology-breakout

* print statement removed

* Merge branch 'bugfix/UT-methodology-breakout' of https://bitbucket.org/aarete/doczy.ai into bugfix/UT-methodology-breakout

* primary prompt update

* prompt update

* Merge remote-tracking branch 'origin/feature/1toN-Optimization' into bugfix/UT-grouper-issues

* removed temp changes

* removed temp changes

* Update reimb primary


Approved-by: Katon Minhas

* Merged in bugfix/validation_fixes (pull request #795)

bugfix/validation_fixes to feature/1toN-Optimization

* updated validation of clean claims reimbursement

* Merged feature/1toN-Optimization into bugfix/validation_fixes


Approved-by: Katon Minhas

* Re-add dynamic codes and reimb-info

* Re-structure empty reimbursement prompt

* Fix lesser of check overfiltering

* Merged in bugfix/generic-methodology-breakout (pull request #797)

Bugfix/generic methodology breakout

* fee schedule fix

* carveout+special case prompt changed

* fee schedule changes

* prompt changes and column order


Approved-by: Katon Minhas

* fix not_to_exceed distribution

* pytest fix

* remove prints

* Merge remote-tracking branch 'origin/feature/1toN-Optimization' into Optimization/reimb-primary

* prompt update for reimb primary

* Update mapping

* Update tests

* Update test

* row order fixed

* Merge remote-tracking branch 'origin/feature/1toN-Optimization' into Optimization/reimb-primary

* chargemaster fix

* Merge branch 'main' into Optimization/reimb-primary

* Complete merge

* Final updates


Approved-by: Katon Minhas
2025-12-10 22:24:44 +00:00
Katon Minhas 0e2622f156 Merged in feature/1toN-Optimization (pull request #799)
Feature/1toN Optimization

* Update config - remove Haiku 3

* Merged in optimization/trigger-cap (pull request #792)

Optimization/trigger cap

* trigger cap breakout added

* Merge remote-tracking branch 'origin/feature/1toN-Optimization' into optimization/trigger-cap

* carveout breakout added


Approved-by: Katon Minhas

* Merged in bugfix/utah_issue_fixes (pull request #791)

Bugfix/utah issue fixes to feature/1toN-Optimization

* updated valid values for AARETE_DERIVED_REIMB_METHOD

* removed example reimbursements

* removed reciprocatory agreements section reimbs

* Merged feature/1toN-Optimization into bugfix/utah_issue_fixes

* update reimb primary


Approved-by: Katon Minhas

* Update unit tests

* Update llm_utils

* Merge branch 'feature/deprecate-haiku-3' into feature/1toN-Optimization

* fix over-filtering of lesser of

* Merged in bugfix/UT-methodology-breakout (pull request #794)

Bugfix/UT methodology breakout

* updated valid values for AARETE_DERIVED_REIMB_METHOD

* removed example reimbursements

* prompt update

* prompt update

* Merged feature/1toN-Optimization into bugfix/nv_issue_fixes

* add service term in mb prompts

* Merged feature/1toN-Optimization into bugfix/UT-methodology-breakout

* print statement removed

* Merge branch 'bugfix/UT-methodology-breakout' of https://bitbucket.org/aarete/doczy.ai into bugfix/UT-methodology-breakout

* primary prompt update

* remove duplicate prompt


Approved-by: Katon Minhas

* Merge branch 'main' into feature/1toN-Optimization

* Merge branch 'main' into feature/1toN-Optimization

* Update preprocessing to make the exhibit_chunk_mapping start at first page

* Merge row_funcs.py changes from feature/cross-exhibit-dynamic

* Merged in bugfix/UT-grouper-issues (pull request #796)

Bugfix/UT grouper issues

* updated valid values for AARETE_DERIVED_REIMB_METHOD

* removed example reimbursements

* prompt update

* prompt update

* Merged feature/1toN-Optimization into bugfix/nv_issue_fixes

* add service term in mb prompts

* Merged feature/1toN-Optimization into bugfix/UT-methodology-breakout

* print statement removed

* Merge branch 'bugfix/UT-methodology-breakout' of https://bitbucket.org/aarete/doczy.ai into bugfix/UT-methodology-breakout

* primary prompt update

* prompt update

* Merge remote-tracking branch 'origin/feature/1toN-Optimization' into bugfix/UT-grouper-issues

* removed temp changes

* removed temp changes

* Update reimb primary


Approved-by: Katon Minhas

* Merged in bugfix/validation_fixes (pull request #795)

bugfix/validation_fixes to feature/1toN-Optimization

* updated validation of clean claims reimbursement

* Merged feature/1toN-Optimization into bugfix/validation_fixes


Approved-by: Katon Minhas

* Re-add dynamic codes and reimb-info

* Re-structure empty reimbursement prompt

* Fix lesser of check overfiltering

* Merged in bugfix/generic-methodology-breakout (pull request #797)

Bugfix/generic methodology breakout

* fee schedule fix

* carveout+special case prompt changed

* fee schedule changes

* prompt changes and column order


Approved-by: Katon Minhas

* pytest fix

* remove prints

* Update mapping

* Update tests

* Update test

* Address PR review comments: add logging, fix return types, fix variable propagation, and add tests

- Add warning log in code_funcs.py try/except block for grouper code parsing failures
- Add warning log in hybrid_smart_chunking_funcs.py except block for JSON parsing failures
- Add return type annotation to lesser_of_distribution function
- Fix first_reimbursement_page variable propagation issue by returning updated value
- Add comprehensive tests for simplify_exhibit, chunk_by_exhibit, and lesser_of_distribution functions

* Merged in bugfix/prov_info_json (pull request #802)

Bugfix/prov info json to feature/1toN-Optimization

* prov_info_json added

* remove print statements


Approved-by: Katon Minhas


Approved-by: Siddhant Medar
2025-12-10 20:01:52 +00:00
Katon Minhas 89c4447c75 Merged in hotfix/clover-errors (pull request #785)
Hotfix/clover errors

* Update function name

* Update logging

* remove excess info

* remove breakout print

* Typehints

* Don't add lists

* remove prints


Approved-by: Siddhant Medar
Approved-by: Karan Desai
2025-11-21 22:38:49 +00:00
Karan Desai f8ceb39111 Merged in feature/signatures_present (pull request #779)
Feature/signatures present

* initial commit

* shifted from pp to one_to_one

* moved func in one_to_one_funcs

* Merged main into feature/signatures_present


Approved-by: Katon Minhas
2025-11-19 18:42:04 +00:00
Siddhant Medar 55643b5a04 Merged in feature/saas-hybrid-smart-chunking (pull request #760)
Feature/saas hybrid smart chunking

* WIP: Initial work on SaaS HSC integration

* clean up hsc funcs script

* fix merging dict issue

* Merge remote-tracking branch 'origin/main' into feature/saas-hybrid-smart-chunking

* added changes as per feedback comments

* add tests and remove old smart chunking funcs

* add more tests


Approved-by: Katon Minhas
2025-11-05 15:20:33 +00:00
Katon Minhas 2b9bf50c2e Merged in feature/effective-dt (pull request #752)
Feature/effective dt

* move prompt_smart_chunked to prompt_calls.py

* refactor pass_smart_chunked_to_full_context

* Mypy fix

* Update field to EFFECTIVE_DT

* move to prompt_calls

* Update vision funcs

* Restructure prompt-based date derivation

* E2E passed

* Remove test

* Merge branch 'main' into feature/effective-dt

* remove deprecated postprocessing_func

* Docstrings

* refactor - send global lesser of to prompt_calls.py

* remove deprecated function

* more docstrings

* Merged main into feature/effective-dt

* Merged main into feature/effective-dt

* added base field for aarete derived effective date


Approved-by: VenkataKrishna Reddy Avula
2025-10-30 15:28:39 +00:00
VenkataKrishna Reddy Avula 48c2d1ad51 Merged in bugfix/claim_type_cd (pull request #749)
Bugfix/claim type cd to main

* test claim type cd

* Merged main into bugfix/claim_type_cd

* updated claim type prompt

* updated prompt

* empty one to n results initiated

* remove print statements

* src/investment/prompt_calls.py

* full context prompt update


Approved-by: Katon Minhas
2025-10-27 16:22:43 +00:00
Katon Minhas fe028eda88 Merged in hotfix/one-to-one-missing (pull request #744)
Return one_to_one results if one_to_n is empty

* Return one_to_one results if one_to_n is empty


Approved-by: Sha Brown
Approved-by: Karan Desai
2025-10-23 19:57:11 +00:00
Katon Minhas 23af15306b Merged in feature/new-model-testing (pull request #741)
Feature/new model testing

* rename and set up

* Setup

* Get dynamic primary crosswalked answers

* Generate stats for one file

* Update dynamic primary evaluation

* Multithreading

* Set up reimbursement primary evaluation

* UPdate output report generation

* Update TN calculation

* update poetry lock

* reorder reimb primary

* evaluate 3 models

* Configure 4.5

* switch to debug logging

* update type checking

* Fix mypy

* Resolve mypy

* Merged main into feature/new-model-testing


Approved-by: Karan Desai
2025-10-22 16:46:28 +00:00
Mayank Aamseek 3027d8e1d5 Merged in bugfix/mi-reimb-terms (pull request #740)
Bugfix/mi reimb terms

* test changes

* logging added

* Merge branch 'main' into bugfix/generic-reimb-terms

* parsing fixes

* fix missing reimbursements

* make service terms unique by adding additional context

* Merge branch 'main' into bugfix/mi-reimb-terms

* remomoving test changes

* remomoving test changes

* remomoving test changes

* test case for universal_json_load added


Approved-by: Katon Minhas
2025-10-22 15:21:43 +00:00
Mayank ab0c570f54 changes reversed 2025-10-15 13:41:12 +00:00
Mayank f5dc3d82a1 logging added 2025-10-15 13:39:07 +00:00
Sha Brown c56270c951 Merged in feature/field_type_config (pull request #731)
Feature/field type config

* Added field type procesing conditional logic

* Updated field type naming convention in config.py; Updated date_postprocessing to avoid column key missing issue

* Cleaned up code; Move merging functionality to one to one processing block

* Merged main into feature/field_type_config


Approved-by: Katon Minhas
2025-10-13 18:54:10 +00:00
Katon Minhas a910f27cd2 Merged in bugfix/mi-dynamic (pull request #726)
Bugfix/mi dynamic

* Refactor - split out one_to_n_cleaning

* Rework postprocess

* Update postprocessing - fill dynamic values if there is only 1 non-empty value

* Update to use string_utils

* Group by AARETE_DERIVED_LOB as well

* Update crosswalk mapping to handle | lists

* Update constants - run with MI flag


Approved-by: Siddhant Medar
2025-10-07 15:06:10 +00:00
Alex Galarce c5af046cde Merged in feature/improve-logging-docs (pull request #711)
Feature/improve logging docs

* Enhance PerFileHandler documentation and functionality for improved logging context management

* Add more docuementation

* Update logging_utils.py documentation for clarity and usage examples

* Enhance logging comments for clarity on per-file logging setup and concurrent processing

* Enhance logging comments for clarity on per-file logging context and its implications


Approved-by: Katon Minhas
2025-09-24 21:31:38 +00:00
Katon Minhas 1050287397 Merged in refactor/one-to-n (pull request #702)
Refactor/one to n

* remove print

* Update logging for fsbreakout and grouper breakout

* Additional type hints

* updated special case primary

* Remove deprecated special case check

* OUTLIER_TERMS-->OUTLIER_TERM standardization

* Applied suggestion to fieldExtraction/src/prompts/investment_prompts.json
* update unit test

* Update Carveout IND and Default IND logic

* Split out exhibit-level lesser

* Bugfix

* update lesser of exhibit prompt

* Bugfix

* exhibit lesser of varible name bugfix

* Update exhibit lesser of passing

* Update Exhibit-level prompt template

* Shorten full_context prompt length if the questions are long

* Special case assignment index

* Resolve unit tests

* updated UC IND prompt

* Remove print

* update one-to-n unit test

* added dot based date example

* updated date fix prompt

* Merged in bugfix/json_error_and_unit_of_measure (pull request #708)

Bugfix/json error and unit of measure

* json parser fixes and prompt changes

* universal json load fixed

* json error fixed

* json error fixed 2

* Merged refactor/one-to-n into bugfix/json_error_and_unit_of_measure


Approved-by: Katon Minhas


Approved-by: Alex Galarce
2025-09-24 15:56:27 +00:00
Katon Minhas 0b00cc5ddd Merged in feature/initialize-phase-5 (pull request #696)
Feature/initialize phase 5

* Initialize at Exhibit-Level

* Merge branch 'main' into feature/initialize-phase-5

* Add all breakout fields

* Prep for merge


Approved-by: Alex Galarce
2025-09-04 18:22:35 +00:00
Katon Minhas 416a9944e8 Merged in hotfix/constants-passing (pull request #695)
Hotfix/constants passing

* E2E test passed

* Merged main into hotfix/constants-passing

* Remove print


Approved-by: Alex Galarce
2025-09-04 16:01:56 +00:00
Alex Galarce 425a8ec86c Merged in feature/per-file-logging (pull request #686)
Feature/per file logging

* Add custom per-file logging handler

* Add close method in PerFileHandler to properly close file handlers

* Add return type annotation to setup_per_file_logging function

* Set up per-file logging in main and process_file functions

* Add per-file logging configuration options and update setup logic

* Refactor logging setup in testing notebook to use per-file logging and improve organization

* Clean up whitespace and improve organization in testing notebook

* Merge remote-tracking branch 'origin/main' into feature/per-file-logging

* Enhance per-file logging setup by adding debug level configuration and informative print statement

* Change logging from info to debug level in table processing functions for improved verbosity control

* isort


Approved-by: Katon Minhas
2025-08-29 14:38:15 +00:00
Katon Minhas b4de4048ca Merged in bugfix/reimb-primary-issues (pull request #682)
Bugfix/reimb primary issues

* Merge + test

* Extract (and remove) headers and footers

* Simplify prompt

* Restore

* Merged main into bugfix/reimb-primary-issues

* Successful E2E test

* Cleanup

* Return proper

* Merged main into bugfix/reimb-primary-issues


Approved-by: Alex Galarce
2025-08-27 17:42:35 +00:00
Alex Galarce 39a07f63c9 Merged in bugfix/dynamic-primary-and-pyproject (pull request #683)
pyproject requirements, prep for logging fixes

* remove chroma

* update project name to "doczy-field-extraction" in pyproject.toml

* update lock file

* replace print statements with logging for better traceability in process_file function

* replace print statements with logging for improved error tracking and information logging

* replace print statements with logging for improved error handling and traceability

* replace print statements with logging for improved error handling in llm_utils and string_utils

* remove unused table_utils module; supplanted by table_funcs and unreferenced

* replace capsys with caplog in test_invalid_input_type for improved logging of invalid input


Approved-by: Katon Minhas
2025-08-27 16:37:48 +00:00
Mayank Aamseek 94251faa03 Merged in feature/grouper-fields (pull request #658)
Feature/grouper fields

* prompt changes

* print statements removed

* merged with main

* merge issue fixed

* updated grouper prompts

* updated grouper fields in fields order

* updated breakout prompt

* Merged main into feature/grouper-fields

* reverted grouper cd prompt

* updated goruper severity prompt

* Merge remote-tracking branch 'origin/main' into feature/grouper-fields

* updated grouper breakout function

* error fixes

* Merged main into feature/grouper-fields

* Merged main into feature/grouper-fields

* updated grouper_pct_rate and grouper_base_rate

* removed empty array post processing fix

* Merged main into feature/grouper-fields

* post-processing for grouper rate modified

* updated grouper type prompt

* Merged main into feature/grouper-fields

* grouper version prompt updated

* Merged main into feature/grouper-fields

* OPPS as APC

* Merged main into feature/grouper-fields


Approved-by: Katon Minhas
2025-08-26 15:43:53 +00:00
Katon Minhas f6cefc62e3 Merged in bugfix/dynamic-primary-issues (pull request #669)
Bugfix/dynamic primary issues

* add LOB_RELATIONSHIP prompt

* Add test script

* Integrate prompt

* Update one-to-one passing rules

* Merged main into bugfix/dynamic-primary-issues

* Remove promt

* replace quotes


Approved-by: Alex Galarce
2025-08-19 16:55:09 +00:00
Katon Minhas 8c99f5d073 Merged in bugfix/dynamic-primary-issues (pull request #664)
Bugfix/dynamic primary issues

* format and restructure

* checkpoint

* Merge branch 'main' into bugfix/dynamic-primary-issues

* Update constants attributes to base class

* Revert "Update constants attributes to base class"

This reverts commit 936b1e41f2e2d445e69e840b69af08eac6661120.

* Fix mapping

* E2E test

* remove prints

* Merge branch 'main' into bugfix/dynamic-primary-issues

* Black

* Update any dynamic

* Prompt update - more leniency

* Remove prints

* Merged main into bugfix/dynamic-primary-issues

* update crosswalk

* Merged main into bugfix/dynamic-primary-issues


Approved-by: Alex Galarce
2025-08-14 16:46:28 +00:00
VenkataKrishna Reddy Avula 6c6ac91386 Merged in feature/outlier_fields (pull request #655)
Feature/outlier fields to Main

* modified existing outlier prompts and definition

* Merged main into feature/outlier_fields

* updated prompts

* updated prompts

* Merged main into feature/outlier_fields

* updated outlier definition

* added outlier terms to dynamic funcs

* added allow na for outlier terms

* added outlier breakout

* Merged main into feature/outlier_fields

* added 2 stage breakout

* error fix

* Merged main into feature/outlier_fields

* removed comments


Approved-by: Katon Minhas
2025-08-13 16:20:03 +00:00
Katon Minhas 46be909642 Merged in bugfix/dynamic-primary-issues (pull request #657)
Bugfix/dynamic primary issues

* Refactor TRIGGER_CAP special case configuration to utilize only trigger cap questions for two-stage processing

* Refactor get_special_cases to improve handling of N/A responses and merge methodology results with special case results

* Fix field type in get_special_cases for fee schedule breakout configuration

* Create fieldset.py - refactor

* Fix placeholders

* string concatenate new questions

* Update trigger cap questions output format

* Update TRIGGER_CAP_THRESHOLD_AMT prompt for clarity and specificity in reimbursement methodology

* Update constants class

* Update Field class

* Resolve constants passing

* Update field class - change Null criteria

* Add default

* Enhance documentation for get_special_cases function with detailed processing flow and architecture decisions

* Enhance get_special_cases and run_trigger_cap_breakout functions with detailed docstrings for clarity on processing methodologies

* Update constants passing

* Fix questions

* Update fs and mb questions"

* pass constanst correctly for dynamic

* Merge branch 'feature/trigger_cap_breakout' into bugfix/dynamic-primary-issues

* update constants

* only add questions if present

* ensure answer is a valid special_case

* Fix unit tests

* Remove test file


Approved-by: Alex Galarce
2025-08-11 20:47:52 +00:00
VenkataKrishna Reddy Avula 0d774acf7a Merged in bugfix/reimb_dates (pull request #649)
Bugfix/reimb dates to Main

* added reimb dates split

* Merged main into bugfix/reimb_dates

* prompt changes

* added to postprocessing

* fixed key issues

* modified split function

* removed print statements

* Merge branch 'main' into bugfix/reimb_dates

* update import

* moved split date upstream

* moved split dates to one to n funcs

* Merged main into bugfix/reimb_dates


Approved-by: Katon Minhas
2025-08-06 19:38:34 +00:00
Katon Minhas c4e519894b Merged in refactor/one-time-io (pull request #648)
Refactor/one time io

* Delete client work

* Streamline imports

* Fix list class

* Update tests

* Clear code funcs test

* Fix unit tests

* Single read code funcs

* Single-read model

* Single-load embeddings

* Successful E2E test

* update unit tests

* Update importsg

* Reload poetry.lock

* remove old exhibit header function

* Remove aarete_derived generic function

* remove align and format tables

* Remove strings to dict

* references

* Clear code

* Move preprocessing_funcs

* remove keywords

* refactor postprocessing_funcs

* Pass unit test - remove qa_qc directory

* Black and isort

* remove print


Approved-by: Alex Galarce
2025-08-05 20:46:19 +00:00
Katon Minhas c2d5db19d4 Merged in feature/doczy-codes (pull request #619)
Feature/doczy codes

* Ignore service and procedure generic terms

* UPdate PT/OT/ST codes

* Standardize stop-loss

* UPdate for multiple identical values

* Modify code funcs

* Run highest match Implicit Level first

* Fix implicit code bug

* Add PT individually to special case

* change revenue code conditions

* Merge branch 'main' into feature/doczy-codes

* Merged main into feature/doczy-codes

* Update main

* Relocate create_code_embeddings back to scripts

* Remove investment code_funcs (use codes.code_funcs now)

* Add docstrings

* stop-loss

* Update unit tests

* Remove test

* Remove reimb_dates

* fix dtypes

* Remove prints

* update prompt

* Update code breakout unit test

* Merge branch 'main' into feature/doczy-codes

* Merge branch 'main' into feature/doczy-codes


Approved-by: Alex Galarce
2025-08-01 22:36:01 +00:00
VenkataKrishna Reddy Avula 84a09002fa Merged in feature/contract_level_rate_escalator (pull request #644)
Feature/contract level rate escalator to Main

* added rate escalator fields to dynamic one to one fields

* added rate escalator breakout

* update prompts and fixed errors

* fixed pipeline error

* removed exhibit level rate escalator

* removed rate escalator parameter from reimb level fxn

* removed print statement

* updated contract level rate escalator

* Merged main into feature/contract_level_rate_escalator

* Clean up

* Cleanup

* in place combine fields


Approved-by: Alex Galarce
Approved-by: Katon Minhas
2025-08-01 22:14:00 +00:00
VenkataKrishna Reddy Avula 0df334905e Merged in bugfix/rate_escalator_reimbursements (pull request #616)
Bugfix/rate escalator reimbursements to Main

* rate_escalator_terms added

* added RE statement

* fixed pattern error

* fixed enhanced reimb term

* Merge remote-tracking branch 'origin/main' into bugfix/rate_escalator_reimbursements


Approved-by: Alex Galarce
2025-07-17 22:05:33 +00:00
Alex Galarce 2680259f22 Merged in bugfix/samaritan-missing-rows (pull request #608)
Exhibit-level Lesser Of

* Enhance exception language clarity in investment prompts

* Roll back exception language

* initial try at exhibit_lesser_of

* simplify `create_exhibit_lesser_of_row`

* Refine prompt for extracting 'lesser of' and 'not to exceed' constraints in exhibit

* Remove unused `get_service_breakout` function; it was unreferenced

* run exhibit-level answers FIRST, apply exhibit lesser-of WITHIN reimbursement_level

* Refine prompt for extracting 'lesser of' and 'not to exceed' constraints to clarify expected output and handling of multiple constraints.

* Merge remote-tracking branch 'origin/main' into bugfix/samaritan-missing-rows

* add exhibit_lesser_of="N/A" to tests

* Add exhibit_lesser_of parameter to reimbursement_level calls in tests

* restore dynamic answer retrieval for primary, reimbursement info, and code fields

* Merge remote-tracking branch 'origin/main' into bugfix/samaritan-missing-rows

* "fill in the blank" prompt attempt

* Merge remote-tracking branch 'origin/main' into bugfix/samaritan-missing-rows

* Merge remote-tracking branch 'origin/main' into feature/sonnet-4

* Merge remote-tracking branch 'origin/feature/sonnet-4' into bugfix/samaritan-missing-rows

* Merge remote-tracking branch 'origin/main' into bugfix/samaritan-missing-rows

* Remove carveout patterns from has_explicit_lesser_of_carveout function

* Add deduplicate_methodology_breakout function to remove duplicate reimbursement methodologies

* Add deduplication of methodology breakout rows in postprocess function

* Enhance create_exhibit_lesser_of_row to avoid redundancy with substring checks

* Change log level from info to debug for exhibit constraint check in create_exhibit_lesser_of_row

* Remove deduplication of methodology breakout rows from postprocess function (we handle it upstream)

* Merged main into bugfix/samaritan-missing-rows


Approved-by: Katon Minhas
2025-07-14 22:04:41 +00:00
Katon Minhas 6caf8dec2f Merged in hotfix/claim-type-cd (pull request #610)
Hotfix/claim type cd

* Fix exhibit_level prompt

* e2e test passed - successfully pulling multiple Claim Types per exhibit

* Merged main into hotfix/claim-type-cd


Approved-by: Alex Galarce
2025-07-10 19:42:55 +00:00
Katon Minhas ac6dfdb721 Merged in bugfix/mcs-issue-fixes (pull request #601)
Bugfix/mcs issue fixes

* Update code primary prompt

* Merge branch 'main' into bugfix/mcs-issue-fixes

* No longer treat non-reimbursement tables as 'tables' for the purpose of preprocessing

* No longer extract Reimb Effective Date from footer of the page

* Merge branch 'main' into bugfix/mcs-issue-fixes

* Update unit tests

* Add postprocessing step to remove Reimb Date values when the 1:1 date values are identical

* Update unit tests

* prep for merge

* Remove excess prints

* Merged main into bugfix/mcs-issue-fixes

* Merge branch 'main' into bugfix/mcs-issue-fixes

* Prep for merge

* remove print


Approved-by: Alex Galarce
2025-07-07 16:57:47 +00:00
Mayank Aamseek 5d65251851 Merged in bugfix/claim_type_1_to_1 (pull request #590)
Bugfix/claim type 1 to 1

* restored old prompt

* updated claim type cd prompt

* claim_type exhibit level added

* Merge branch 'bugfix/claim_type_1_to_1' of https://bitbucket.org/aarete/doczy.ai into bugfix/claim_type_1_to_1

* field type updated

* Merge remote-tracking branch 'origin/main' into bugfix/claim_type_1_to_1

* fixed location of updating field_type to full_context

* updated ancillary examples

* removed print statements

* Merge branch 'bugfix/claim_type_1_to_1' of https://bitbucket.org/aarete/doczy.ai into bugfix/claim_type_1_to_1

* prompt updated to avoid N/A

* Merge remote-tracking branch 'origin/main' into bugfix/claim_type_1_to_1

* prompt changed to handle multiple matches

* prompt updated

* Merge remote-tracking branch 'origin/main' into bugfix/claim_type_1_to_1


Approved-by: Alex Galarce
2025-07-01 22:32:37 +00:00
Katon Minhas 1531ad6bac Merged in feature/prov-group-tin (pull request #584)
Feature/prov group tin

* don't prompt for INTRO_PAGE

* test complete

* test 2

* Update test script

* Merge branch 'main' into feature/prov-group-tin

* Merged main into feature/prov-group-tin

* refactor

* cleaning

* Update save

* Fix return type

* Update for reimbursement_tin_npi

* Comment out reimbursement-tin-npi

* Pass unit tests

* Merge branch 'main' into feature/prov-group-tin

* Merged main into feature/prov-group-tin

* Remove length validation

* Merge branch 'main' into feature/prov-group-tin

* Remove ON_INTRO from prompt

* Restore

* Remove test

* Move reimb prov info to dynamic reimb info

* e2e test

* Fix for unit test

* remove prints

* update docstrings and typehints


Approved-by: Alex Galarce
2025-06-26 19:35:32 +00:00
Alex Galarce c84decca8d Merged in feature/rework-table-handling (pull request #575)
Feature/rework table handling

* test: add cases for configuration boundary

* test: add table post-processing tests (cases to ensure preservation of post-table text and metadata during table operations)

* test: enhance table handling tests for small and large tables, ensuring proper suffixing and splitting behavior

* Fix and clarify intent in comments

* fix: enhance handling of post-table text when combining tables

* refactor: remove commented-out code in clean_tables function for clarity

* test: update post-table text preservation assertions and add debug tracing

* refactor: remove unused remove_table_from_page function and its test

* refactor: enhance combine method documentation and clarify split_by_size_with_smart_tables logic

* Merge remote-tracking branch 'origin/main' into feature/rework-table-handling

* refactor: reduce large table threshold from 1000 to 50 rows for better table splitting

* Add simple splitting

* refactor: uncomment dynamic answer retrieval in run_one_to_n_prompts for improved functionality

* Merge remote-tracking branch 'origin/main' into feature/rework-table-handling

* refactor: remove unused table handling functions and related constants for cleaner code

* refactor: remove unused test functions in preparation for new tests

* refactor: streamline continuation table handling by utilizing existing flags

* refactor: update table combination logic to include all tables from the main page

* refactor: enhance table handling to automatically resolve column mismatches, respect row limits, and improve unit/integration test coverage

* rename clean_tables_simple to clean_tables

* Update docstrings

* refactor: rename table handling functions

* Fix indentation and add tests

* isort, black

* Merge remote-tracking branch 'origin/main' into feature/rework-table-handling


Approved-by: Katon Minhas
2025-06-20 19:46:23 +00:00
Katon Minhas d460b899e3 Merged in bugfix/dynamic-primary-issues (pull request #555)
Bugfix/dynamic primary issues

* Merge branch 'main' into bugfix/dynamic-primary-issues

* Merged main into bugfix/dynamic-primary-issues

* Crosswalk updates

* Merge branch 'main' into bugfix/dynamic-primary-issues

* dont include section information in Exhibit Header

* Add Molina Health Benefit Exchange to Product-LOB mapping

* PPO-POS separate mapping

* Prevent inference of LOB from Program or Product

* Add SoonerSelect

* Update mapping rules to allow for direct values

* update prompts

* Prep for run

* Merged main into bugfix/dynamic-primary-issues

* Update unit tests

* Update Moda Product mappings

* Remove Care1st as Product for Centene

* testing

* Mapping update

* Mapping updates

* prep for merge

* Merged main into bugfix/dynamic-primary-issues

* Merged main into bugfix/dynamic-primary-issues

* docstrings and formatting

* Get group provider for > 1

* Merged main into bugfix/dynamic-primary-issues


Approved-by: Alex Galarce
2025-06-19 17:20:39 +00:00
Alex Galarce 31e1c43f0a Merged in feature/formatted-prov-info-json (pull request #574)
Feature/formatted prov info json

* Add newline and pipe-delimited formatting for provider info JSON

* Add new investment column names for nicely-formatted PROV_INFO_JSON

* remove PROV_INFO_JSON_PIPE and rename PROV_INFO_JSON_NEWLINE to PROV_INFO_JSON_FORMATTED

* Merge remote-tracking branch 'origin/main' into feature/formatted-prov-info-json


Approved-by: Katon Minhas
2025-06-12 16:39:03 +00:00
Alex Galarce 5d5f633f02 Merged in feature/formatted-prov-info-json (pull request #572)
Feature/formatted prov info json

* Add newline and pipe-delimited formatting for provider info JSON

* Add new investment column names for nicely-formatted PROV_INFO_JSON


Approved-by: Katon Minhas
2025-06-11 23:15:31 +00:00
Alex Galarce 1c700122e8 Merged in feature/service-reimb-deduplication (pull request #568)
Add (service, methodology) deduplication with accumulator

* Add deduplication with accumulator

* Merge remote-tracking branch 'origin/main' into feature/service-reimb-deduplication

* Replace print statement with logging.debug for cross-exhibit duplicate detection

* Fix and add tests for one_to_n_funcs

* Merge remote-tracking branch 'origin/main' into feature/service-reimb-deduplication

* Add seen_pairs parameter to reimbursement_level function in tests


Approved-by: Katon Minhas
2025-06-11 20:54:48 +00:00
Katon Minhas baf5002b2c Merged in feature/exhibit-linking (pull request #564)
Feature/exhibit linking

* Initialize new exhibit header prompt

* Save

* Merge branch 'main' into feature/exhibit-linking

* update prompt

* cleanup return type

* remove test

* Test fixes

* Fix test

* Add tests

* Merge branch 'main' into feature/exhibit-linking

* Merge branch 'main' into feature/exhibit-linking

* Remove test notebooks

* Delete test notebook

* Save

* Merge branch 'main' into feature/exhibit-linking

* try except for exhbit page finding

* Prep for merge

* update test

* Remove test

* Merge branch 'main' into feature/exhibit-linking

* update unit test

* Fix unit test

* replace try except

* Update page sort

* Add docstring


Approved-by: Alex Galarce
2025-06-10 19:50:15 +00:00
Alex Galarce f84023d742 Merged in bugfix/table-preprocessing (pull request #560)
Bugfix/table preprocessing

* restructure get_exhibit_dict

* exhibit header

* update exhibit header handling

* update docstring

* bugfix

* remove test - update docstring

* Merged main into bugfix/table-preprocessing

* refactor: update page_key_sort to return float for numeric keys

* refactor: enhance get_exhibit_dict to improve chunking logic for exhibits

* refactor: update page_key_sort to return float for numeric keys

* test: add unit tests for get_exhibit_dict function

* test: fix test case 7

* Add final, difficult test

* refactor: enhance get_exhibit_dict to handle single decimal page exhibits and improve chunking logic

* pulled `one_to_n_test.py` from dynamic primary branch

* fix: update common_file_names logic to remove specific entry and log changes

* docs: update get_exhibit_dict docstring to clarify input and output formats

* test: add test for exhibit handling with decimal pages in get_exhibit_dict

* feat: add debug logging for exhibit chunk mapping and exhibit dictionary keys in one_to_n_exhibit_chunking

* Merge remote-tracking branch 'origin/main' into bugfix/table-preprocessing

* feat: enhance spatial processing rules and consistency checks for reimbursement terms

* feat: update global scan methodology to enhance context extraction for reimbursement constraints

* Merge remote-tracking branch 'origin/main' into bugfix/table-preprocessing


Approved-by: Katon Minhas
2025-06-09 21:49:45 +00:00
Alex Galarce 32d431ec4e Merged in feature/remove-exhibit-filtering (pull request #565)
strip out reimbursement exhibit identification

* strip out reimbursement exhibit identification


Approved-by: Katon Minhas
2025-06-09 21:35:20 +00:00
Alex Galarce 3be2f44cb1 Merged in feature/global-lesser-of (pull request #553)
Feature/global lesser of

* Merge remote-tracking branch 'origin/feature/reimb-primary-updates'

* First try at global "lesser of"

* Generate reimb_ids further upstream

* Implement global lesser of injection and extraction logic in file processing

* Update logic to change LESSER_OF_IND in-place for global affected rows; add docstrings

* Refactor global lesser of injection logic to create new rows for REIMB_IDs lacking lesser of logic; update original rows in place and improve tracking flags.

* Merge remote-tracking branch 'origin/main' into feature/global-lesser-of

* Add logging for global lesser of logic injection and update handling of original rows

* Refactor global lesser of logic: move functions to one_to_one_funcs.py and update inject_global_lesser_of_rows to use new implementations

* Move imports for METHODOLOGY_BREAKOUT_QUESTIONS and REIMB_TERM_BREAKOUT to the top of one_to_one_funcs.py for better organization and to remove prototype comments.

* isort

* Refactor import statement for generate_reimb_ids to use the full module path for clarity

* load field from JSON instead of hardcoded values

* Update logging levels in investment_field_testing notebook

* Merged main into feature/global-lesser-of

* Remove unnecessary comment from investment_postprocessing_funcs import statement for clarity

* Merge branch 'feature/global-lesser-of' of https://bitbucket.org/aarete/doczy.ai into feature/global-lesser-of

* Merge remote-tracking branch 'origin/main' into feature/global-lesser-of

* Remove debug print statement from process_file function

* fix call to Field.load_from_file

* Merge remote-tracking branch 'origin/main' into feature/global-lesser-of

* Add tracking flag for global lesser of logic when none is found

* explicitly sort by FILE_NAME, EXHIBIT_PAGE, and REIMB_LESSER_OF_ID columns

* Add new field type, "full_context_separate", to make sure that we don't run it twice.  Also formatting fix

* Fix iteration over separate fields in run_full_context_fields function


Approved-by: Katon Minhas
2025-06-04 19:49:10 +00:00