Commit Graph

58 Commits

Author SHA1 Message Date
VenkataKrishna Reddy Avula 0d774acf7a Merged in bugfix/reimb_dates (pull request #649)
Bugfix/reimb dates to Main

* added reimb dates split

* Merged main into bugfix/reimb_dates

* prompt changes

* added to postprocessing

* fixed key issues

* modified split function

* removed print statements

* Merge branch 'main' into bugfix/reimb_dates

* update import

* moved split date upstream

* moved split dates to one to n funcs

* Merged main into bugfix/reimb_dates


Approved-by: Katon Minhas
2025-08-06 19:38:34 +00:00
Katon Minhas c4e519894b Merged in refactor/one-time-io (pull request #648)
Refactor/one time io

* Delete client work

* Streamline imports

* Fix list class

* Update tests

* Clear code funcs test

* Fix unit tests

* Single read code funcs

* Single-read model

* Single-load embeddings

* Successful E2E test

* update unit tests

* Update importsg

* Reload poetry.lock

* remove old exhibit header function

* Remove aarete_derived generic function

* remove align and format tables

* Remove strings to dict

* references

* Clear code

* Move preprocessing_funcs

* remove keywords

* refactor postprocessing_funcs

* Pass unit test - remove qa_qc directory

* Black and isort

* remove print


Approved-by: Alex Galarce
2025-08-05 20:46:19 +00:00
Katon Minhas c2d5db19d4 Merged in feature/doczy-codes (pull request #619)
Feature/doczy codes

* Ignore service and procedure generic terms

* UPdate PT/OT/ST codes

* Standardize stop-loss

* UPdate for multiple identical values

* Modify code funcs

* Run highest match Implicit Level first

* Fix implicit code bug

* Add PT individually to special case

* change revenue code conditions

* Merge branch 'main' into feature/doczy-codes

* Merged main into feature/doczy-codes

* Update main

* Relocate create_code_embeddings back to scripts

* Remove investment code_funcs (use codes.code_funcs now)

* Add docstrings

* stop-loss

* Update unit tests

* Remove test

* Remove reimb_dates

* fix dtypes

* Remove prints

* update prompt

* Update code breakout unit test

* Merge branch 'main' into feature/doczy-codes

* Merge branch 'main' into feature/doczy-codes


Approved-by: Alex Galarce
2025-08-01 22:36:01 +00:00
VenkataKrishna Reddy Avula 84a09002fa Merged in feature/contract_level_rate_escalator (pull request #644)
Feature/contract level rate escalator to Main

* added rate escalator fields to dynamic one to one fields

* added rate escalator breakout

* update prompts and fixed errors

* fixed pipeline error

* removed exhibit level rate escalator

* removed rate escalator parameter from reimb level fxn

* removed print statement

* updated contract level rate escalator

* Merged main into feature/contract_level_rate_escalator

* Clean up

* Cleanup

* in place combine fields


Approved-by: Alex Galarce
Approved-by: Katon Minhas
2025-08-01 22:14:00 +00:00
VenkataKrishna Reddy Avula 0df334905e Merged in bugfix/rate_escalator_reimbursements (pull request #616)
Bugfix/rate escalator reimbursements to Main

* rate_escalator_terms added

* added RE statement

* fixed pattern error

* fixed enhanced reimb term

* Merge remote-tracking branch 'origin/main' into bugfix/rate_escalator_reimbursements


Approved-by: Alex Galarce
2025-07-17 22:05:33 +00:00
Alex Galarce 2680259f22 Merged in bugfix/samaritan-missing-rows (pull request #608)
Exhibit-level Lesser Of

* Enhance exception language clarity in investment prompts

* Roll back exception language

* initial try at exhibit_lesser_of

* simplify `create_exhibit_lesser_of_row`

* Refine prompt for extracting 'lesser of' and 'not to exceed' constraints in exhibit

* Remove unused `get_service_breakout` function; it was unreferenced

* run exhibit-level answers FIRST, apply exhibit lesser-of WITHIN reimbursement_level

* Refine prompt for extracting 'lesser of' and 'not to exceed' constraints to clarify expected output and handling of multiple constraints.

* Merge remote-tracking branch 'origin/main' into bugfix/samaritan-missing-rows

* add exhibit_lesser_of="N/A" to tests

* Add exhibit_lesser_of parameter to reimbursement_level calls in tests

* restore dynamic answer retrieval for primary, reimbursement info, and code fields

* Merge remote-tracking branch 'origin/main' into bugfix/samaritan-missing-rows

* "fill in the blank" prompt attempt

* Merge remote-tracking branch 'origin/main' into bugfix/samaritan-missing-rows

* Merge remote-tracking branch 'origin/main' into feature/sonnet-4

* Merge remote-tracking branch 'origin/feature/sonnet-4' into bugfix/samaritan-missing-rows

* Merge remote-tracking branch 'origin/main' into bugfix/samaritan-missing-rows

* Remove carveout patterns from has_explicit_lesser_of_carveout function

* Add deduplicate_methodology_breakout function to remove duplicate reimbursement methodologies

* Add deduplication of methodology breakout rows in postprocess function

* Enhance create_exhibit_lesser_of_row to avoid redundancy with substring checks

* Change log level from info to debug for exhibit constraint check in create_exhibit_lesser_of_row

* Remove deduplication of methodology breakout rows from postprocess function (we handle it upstream)

* Merged main into bugfix/samaritan-missing-rows


Approved-by: Katon Minhas
2025-07-14 22:04:41 +00:00
Katon Minhas 6caf8dec2f Merged in hotfix/claim-type-cd (pull request #610)
Hotfix/claim type cd

* Fix exhibit_level prompt

* e2e test passed - successfully pulling multiple Claim Types per exhibit

* Merged main into hotfix/claim-type-cd


Approved-by: Alex Galarce
2025-07-10 19:42:55 +00:00
Katon Minhas ac6dfdb721 Merged in bugfix/mcs-issue-fixes (pull request #601)
Bugfix/mcs issue fixes

* Update code primary prompt

* Merge branch 'main' into bugfix/mcs-issue-fixes

* No longer treat non-reimbursement tables as 'tables' for the purpose of preprocessing

* No longer extract Reimb Effective Date from footer of the page

* Merge branch 'main' into bugfix/mcs-issue-fixes

* Update unit tests

* Add postprocessing step to remove Reimb Date values when the 1:1 date values are identical

* Update unit tests

* prep for merge

* Remove excess prints

* Merged main into bugfix/mcs-issue-fixes

* Merge branch 'main' into bugfix/mcs-issue-fixes

* Prep for merge

* remove print


Approved-by: Alex Galarce
2025-07-07 16:57:47 +00:00
Mayank Aamseek 5d65251851 Merged in bugfix/claim_type_1_to_1 (pull request #590)
Bugfix/claim type 1 to 1

* restored old prompt

* updated claim type cd prompt

* claim_type exhibit level added

* Merge branch 'bugfix/claim_type_1_to_1' of https://bitbucket.org/aarete/doczy.ai into bugfix/claim_type_1_to_1

* field type updated

* Merge remote-tracking branch 'origin/main' into bugfix/claim_type_1_to_1

* fixed location of updating field_type to full_context

* updated ancillary examples

* removed print statements

* Merge branch 'bugfix/claim_type_1_to_1' of https://bitbucket.org/aarete/doczy.ai into bugfix/claim_type_1_to_1

* prompt updated to avoid N/A

* Merge remote-tracking branch 'origin/main' into bugfix/claim_type_1_to_1

* prompt changed to handle multiple matches

* prompt updated

* Merge remote-tracking branch 'origin/main' into bugfix/claim_type_1_to_1


Approved-by: Alex Galarce
2025-07-01 22:32:37 +00:00
Katon Minhas 1531ad6bac Merged in feature/prov-group-tin (pull request #584)
Feature/prov group tin

* don't prompt for INTRO_PAGE

* test complete

* test 2

* Update test script

* Merge branch 'main' into feature/prov-group-tin

* Merged main into feature/prov-group-tin

* refactor

* cleaning

* Update save

* Fix return type

* Update for reimbursement_tin_npi

* Comment out reimbursement-tin-npi

* Pass unit tests

* Merge branch 'main' into feature/prov-group-tin

* Merged main into feature/prov-group-tin

* Remove length validation

* Merge branch 'main' into feature/prov-group-tin

* Remove ON_INTRO from prompt

* Restore

* Remove test

* Move reimb prov info to dynamic reimb info

* e2e test

* Fix for unit test

* remove prints

* update docstrings and typehints


Approved-by: Alex Galarce
2025-06-26 19:35:32 +00:00
Alex Galarce c84decca8d Merged in feature/rework-table-handling (pull request #575)
Feature/rework table handling

* test: add cases for configuration boundary

* test: add table post-processing tests (cases to ensure preservation of post-table text and metadata during table operations)

* test: enhance table handling tests for small and large tables, ensuring proper suffixing and splitting behavior

* Fix and clarify intent in comments

* fix: enhance handling of post-table text when combining tables

* refactor: remove commented-out code in clean_tables function for clarity

* test: update post-table text preservation assertions and add debug tracing

* refactor: remove unused remove_table_from_page function and its test

* refactor: enhance combine method documentation and clarify split_by_size_with_smart_tables logic

* Merge remote-tracking branch 'origin/main' into feature/rework-table-handling

* refactor: reduce large table threshold from 1000 to 50 rows for better table splitting

* Add simple splitting

* refactor: uncomment dynamic answer retrieval in run_one_to_n_prompts for improved functionality

* Merge remote-tracking branch 'origin/main' into feature/rework-table-handling

* refactor: remove unused table handling functions and related constants for cleaner code

* refactor: remove unused test functions in preparation for new tests

* refactor: streamline continuation table handling by utilizing existing flags

* refactor: update table combination logic to include all tables from the main page

* refactor: enhance table handling to automatically resolve column mismatches, respect row limits, and improve unit/integration test coverage

* rename clean_tables_simple to clean_tables

* Update docstrings

* refactor: rename table handling functions

* Fix indentation and add tests

* isort, black

* Merge remote-tracking branch 'origin/main' into feature/rework-table-handling


Approved-by: Katon Minhas
2025-06-20 19:46:23 +00:00
Katon Minhas d460b899e3 Merged in bugfix/dynamic-primary-issues (pull request #555)
Bugfix/dynamic primary issues

* Merge branch 'main' into bugfix/dynamic-primary-issues

* Merged main into bugfix/dynamic-primary-issues

* Crosswalk updates

* Merge branch 'main' into bugfix/dynamic-primary-issues

* dont include section information in Exhibit Header

* Add Molina Health Benefit Exchange to Product-LOB mapping

* PPO-POS separate mapping

* Prevent inference of LOB from Program or Product

* Add SoonerSelect

* Update mapping rules to allow for direct values

* update prompts

* Prep for run

* Merged main into bugfix/dynamic-primary-issues

* Update unit tests

* Update Moda Product mappings

* Remove Care1st as Product for Centene

* testing

* Mapping update

* Mapping updates

* prep for merge

* Merged main into bugfix/dynamic-primary-issues

* Merged main into bugfix/dynamic-primary-issues

* docstrings and formatting

* Get group provider for > 1

* Merged main into bugfix/dynamic-primary-issues


Approved-by: Alex Galarce
2025-06-19 17:20:39 +00:00
Alex Galarce 31e1c43f0a Merged in feature/formatted-prov-info-json (pull request #574)
Feature/formatted prov info json

* Add newline and pipe-delimited formatting for provider info JSON

* Add new investment column names for nicely-formatted PROV_INFO_JSON

* remove PROV_INFO_JSON_PIPE and rename PROV_INFO_JSON_NEWLINE to PROV_INFO_JSON_FORMATTED

* Merge remote-tracking branch 'origin/main' into feature/formatted-prov-info-json


Approved-by: Katon Minhas
2025-06-12 16:39:03 +00:00
Alex Galarce 5d5f633f02 Merged in feature/formatted-prov-info-json (pull request #572)
Feature/formatted prov info json

* Add newline and pipe-delimited formatting for provider info JSON

* Add new investment column names for nicely-formatted PROV_INFO_JSON


Approved-by: Katon Minhas
2025-06-11 23:15:31 +00:00
Alex Galarce 1c700122e8 Merged in feature/service-reimb-deduplication (pull request #568)
Add (service, methodology) deduplication with accumulator

* Add deduplication with accumulator

* Merge remote-tracking branch 'origin/main' into feature/service-reimb-deduplication

* Replace print statement with logging.debug for cross-exhibit duplicate detection

* Fix and add tests for one_to_n_funcs

* Merge remote-tracking branch 'origin/main' into feature/service-reimb-deduplication

* Add seen_pairs parameter to reimbursement_level function in tests


Approved-by: Katon Minhas
2025-06-11 20:54:48 +00:00
Katon Minhas baf5002b2c Merged in feature/exhibit-linking (pull request #564)
Feature/exhibit linking

* Initialize new exhibit header prompt

* Save

* Merge branch 'main' into feature/exhibit-linking

* update prompt

* cleanup return type

* remove test

* Test fixes

* Fix test

* Add tests

* Merge branch 'main' into feature/exhibit-linking

* Merge branch 'main' into feature/exhibit-linking

* Remove test notebooks

* Delete test notebook

* Save

* Merge branch 'main' into feature/exhibit-linking

* try except for exhbit page finding

* Prep for merge

* update test

* Remove test

* Merge branch 'main' into feature/exhibit-linking

* update unit test

* Fix unit test

* replace try except

* Update page sort

* Add docstring


Approved-by: Alex Galarce
2025-06-10 19:50:15 +00:00
Alex Galarce f84023d742 Merged in bugfix/table-preprocessing (pull request #560)
Bugfix/table preprocessing

* restructure get_exhibit_dict

* exhibit header

* update exhibit header handling

* update docstring

* bugfix

* remove test - update docstring

* Merged main into bugfix/table-preprocessing

* refactor: update page_key_sort to return float for numeric keys

* refactor: enhance get_exhibit_dict to improve chunking logic for exhibits

* refactor: update page_key_sort to return float for numeric keys

* test: add unit tests for get_exhibit_dict function

* test: fix test case 7

* Add final, difficult test

* refactor: enhance get_exhibit_dict to handle single decimal page exhibits and improve chunking logic

* pulled `one_to_n_test.py` from dynamic primary branch

* fix: update common_file_names logic to remove specific entry and log changes

* docs: update get_exhibit_dict docstring to clarify input and output formats

* test: add test for exhibit handling with decimal pages in get_exhibit_dict

* feat: add debug logging for exhibit chunk mapping and exhibit dictionary keys in one_to_n_exhibit_chunking

* Merge remote-tracking branch 'origin/main' into bugfix/table-preprocessing

* feat: enhance spatial processing rules and consistency checks for reimbursement terms

* feat: update global scan methodology to enhance context extraction for reimbursement constraints

* Merge remote-tracking branch 'origin/main' into bugfix/table-preprocessing


Approved-by: Katon Minhas
2025-06-09 21:49:45 +00:00
Alex Galarce 32d431ec4e Merged in feature/remove-exhibit-filtering (pull request #565)
strip out reimbursement exhibit identification

* strip out reimbursement exhibit identification


Approved-by: Katon Minhas
2025-06-09 21:35:20 +00:00
Alex Galarce 3be2f44cb1 Merged in feature/global-lesser-of (pull request #553)
Feature/global lesser of

* Merge remote-tracking branch 'origin/feature/reimb-primary-updates'

* First try at global "lesser of"

* Generate reimb_ids further upstream

* Implement global lesser of injection and extraction logic in file processing

* Update logic to change LESSER_OF_IND in-place for global affected rows; add docstrings

* Refactor global lesser of injection logic to create new rows for REIMB_IDs lacking lesser of logic; update original rows in place and improve tracking flags.

* Merge remote-tracking branch 'origin/main' into feature/global-lesser-of

* Add logging for global lesser of logic injection and update handling of original rows

* Refactor global lesser of logic: move functions to one_to_one_funcs.py and update inject_global_lesser_of_rows to use new implementations

* Move imports for METHODOLOGY_BREAKOUT_QUESTIONS and REIMB_TERM_BREAKOUT to the top of one_to_one_funcs.py for better organization and to remove prototype comments.

* isort

* Refactor import statement for generate_reimb_ids to use the full module path for clarity

* load field from JSON instead of hardcoded values

* Update logging levels in investment_field_testing notebook

* Merged main into feature/global-lesser-of

* Remove unnecessary comment from investment_postprocessing_funcs import statement for clarity

* Merge branch 'feature/global-lesser-of' of https://bitbucket.org/aarete/doczy.ai into feature/global-lesser-of

* Merge remote-tracking branch 'origin/main' into feature/global-lesser-of

* Remove debug print statement from process_file function

* fix call to Field.load_from_file

* Merge remote-tracking branch 'origin/main' into feature/global-lesser-of

* Add tracking flag for global lesser of logic when none is found

* explicitly sort by FILE_NAME, EXHIBIT_PAGE, and REIMB_LESSER_OF_ID columns

* Add new field type, "full_context_separate", to make sure that we don't run it twice.  Also formatting fix

* Fix iteration over separate fields in run_full_context_fields function


Approved-by: Katon Minhas
2025-06-04 19:49:10 +00:00
Katon Minhas ca01d49452 Merged in bugfix/dropped-pages (pull request #554)
Bugfix/dropped pages

* Rearrange

* Update table preprocessing

* Remove print

* Remove page start

* test for table issue

* Merged main into bugfix/dropped-pages

* prints and documentation

* Pipeline fix

* remove get_pages_to_process


Approved-by: Alex Galarce
2025-06-03 20:47:42 +00:00
Alex Galarce 806c272dc5 Merged in feature/further-reimb-primary-improvements (pull request #546)
Feature/further reimb primary improvements

* Refactor identify_reimbursement_exhibits to return both EXPLICIT and IMPLICIT values

* Moving from # Debugging lines to `logger`

* Enhance "lesser of" and exception language instructions in reimbursement terms

* clarify unclear print/logging statements

* Replace debugging print statements with logging in get_reimbursement_primary function

* Add logging for page processing in run_one_to_n_prompts function


Approved-by: Katon Minhas
2025-05-29 22:33:12 +00:00
Alex Galarce 9db0fb328e Merged in bugfix/identify-tables (pull request #544)
Don't consider table-subpages each individually for reimbursement exhibit identification

* Refine EXHIBIT_CHECK prompt

* test: rework `get_exhibit_pages` for table subpages

* Fix indentation

* Experimental table de-chunking

* Increase max_tokens for Claude model in exhibit page processing

* fix get_pages_to_process docstring

* refactor: improved docstring for get_exhibit_pages

* move `get_pages_to_process` to `preprocess.py`

* test: add unit tests for get_pages_to_process function

* Comment out debugging lines

* Changed prompt that was breading universal_json_load parsing

* fix: handle "no results" case in get_reimbursement_primary function

* fixed docstring


Approved-by: Katon Minhas
2025-05-28 16:57:23 +00:00
Katon Minhas 098c10b9ca Merged in bugfix/dynamic-primary-issues (pull request #541)
Bugfix/dynamic primary issues

* Update reimbursement exhibit detection

* Remove prints

* Update prompt

* Decrease header size

* Prompt update - only return dynamic 1:1 values if applied to entire contract

* Testing

* pipe-delimited

* Merged main into bugfix/dynamic-primary-issues

* Fix Product->LOB mapping

* Remove test

* remove commented crosswalk code

* Remove print


Approved-by: Alex Galarce
2025-05-22 15:04:19 +00:00
Alex Galarce 55fa4f2e79 Merged in feature/reimb-primary-improvements (pull request #524)
Reimbursement Exhibit Prefiltering

* Add IDENTIFY_REIMBURSEMENT_EXHIBITS function to analyze exhibit headers for reimbursement information

* Add Zone.Identifier to .gitignore to prevent Windows zone information files from being tracked

* Add functionality to identify reimbursement exhibits

* Add functionality to identify reimbursement exhibits in run_one_to_n_prompts

* Output YAML parsing

* Refactor identify_reimbursement_exhibits to improve YAML parsing and error handling

* New reimbursement exhibit logic

* Update IDENTIFY_REIMBURSEMENT_EXHIBITS_PROMPT to return PAGE NUMBERS instead of exhibit headers

* Fix identify_reimbursement_exhibits to return page numbers as strings after YAML parsing

* Remove debugging print statement for reimbursement exhibits in run_one_to_n_prompts

* Remove commented debugging print statements in identify_reimbursement_exhibits

* Increase read_timeout in EC2 configuration from 2000 to 4000

* Merge remote-tracking branch 'origin/main' into feature/reimb-primary-improvements

* Remove debugging print statements in get_reimbursement_primary function

* mypy error

* Fix return type in identify_reimbursement_exhibits to ensure keys are returned as a list

* Merge remote-tracking branch 'origin/main' into feature/reimb-primary-improvements


Approved-by: Katon Minhas
2025-05-13 20:22:43 +00:00
Katon Minhas e51b599925 Merged in feature/medicare-medicaid-postprocess (pull request #521)
Feature/medicare medicaid postprocess

* Initial test

* add to postprocessing funcs

* Remove from postprocessing

* Merged main into feature/medicare-medicaid-postprocess


Approved-by: Alex Galarce
2025-05-07 23:03:16 +00:00
Alex Galarce cbd7c9c7d0 Merged in bugfix/add_name_to_full_context_if_no_identifiers (pull request #520)
Bugfix/add name to full context if no identifiers

* Add handling for missing TIN/NPI identifiers in one-to-one results

* Update provider name handling in one-to-one results for group providers


Approved-by: Katon Minhas
2025-05-07 22:53:27 +00:00
VenkataKrishna Reddy Avula 0d6bdf9539 Merged in bugfix/effective_date_fix (pull request #504)
DRAFT: Bugfix/effective date fix

* Merged in bugfix/extra-fields (pull request #500)

Do not run Exhibit level prompt if there are no exhibit level fields

* Do not run Exhibit level prompt if there are no exhibit level fields

* Update dynamic test

* Remove test from one-to-n test module

* Update test

* Fix unit test


Approved-by: Alex Galarce

* Merged in bugfix/tin-npi-prompt-fixes (pull request #499)

Bugfix/tin npi prompt fixes

* Change delimiter for other provider information from comma to pipe

* filter out invalid providers

* Update extraction and formatting instructions for provider entities in TIN_NPI_TEMPLATE

* Merged main into bugfix/tin-npi-small-bugfixes

* Refactor deduplication logic to use string_utils for checking empty provider fields

* Merge branch 'bugfix/tin-npi-small-bugfixes' of https://bitbucket.org/aarete/doczy.ai into bugfix/tin-npi-small-bugfixes

* Update deduplication logic to exclude providers with 'UNKNOWN' TIN, NPI, or NAME

* Merge remote-tracking branch 'origin/main' into bugfix/tin-npi-small-bugfixes

* Fix deduplication logic to not drop subsequent names

* Fix formatting of provider names in deduplication logic

* add payer name filtering

* reorder execution order to allow to find payer_name first

* Add handling for previously unfound group providers in identify_group_provider function

* Merge remote-trackin…
* Merged in feature/update-testbed-script (pull request #501)

Feature/update testbed script

* Update preprocessing

* Update preprocessing

* remove quit

* Update date for filename

* Merged main into feature/update-testbed-script


Approved-by: Alex Galarce

* Merged in bugfix/smart-chunks (pull request #503)

missing of text in chunks fixed

* missing of text in chunks fixed


Approved-by: Katon Minhas

* Merged in feature/rework-reimb-tin-npi (pull request #502)

Feature/rework reimb tin npi

* use LLM for reimb_[tin/npi/name]

* Merge remote-tracking branch 'origin/main' into feature/rework-reimb-tin-npi

* Merge remote-tracking branch 'origin/main' into feature/rework-reimb-tin-npi

* deduplicate tin/name/npi before adding to prompt

* Pull out prompt and llm_response in methodology_breakout_single_row for easier debugging

* Refactor reimbursement_tin_npi to handle multiple providers and update valid TIN, NPI, and NAME fields accordingly

* Reorder REIMB_PROV_NAME field in investment_prompts.json

* Update REIMB_PROV_NAME prompt to specify valid values for provider names

* Remove valid_values constraint from prov_name to raise accuracy

* Remove debugging print statements from reimbursement_tin_npi and related functions

* Remove debugging print statements from run_provider_info_fields function

* Merge remote-tracking branch 'origin/main' into feature/rework-reimb-tin-npi

* Remove debugging print statem…
* Merged in feature/preformat-single-quotes-input (pull request #506)

Replace double quotes with single quotes in context strings for parsability

* Replace double quotes with single quotes in context strings for parsability

* Merged main into feature/preformat-single-quotes-input


Approved-by: Katon Minhas

* updated sig pages fnxn

* Merge branch 'main' into bugfix/effective_date_fix

* poetry add pymupdf

* created vision funcs

* Merged main into bugfix/effective_date_fix

* fixed pipeline issue

* updated string_utils_test

* updated test values

* updated extract signature page fxn

* updated test values

* updated test values

* Merged main into bugfix/effective_date_fix

* Prompt template in all-caps

* remove debugging prints

* Merged main into bugfix/effective_date_fix

* Change vision to False by default

* Change vision to False by default

* fix tests

* fix tests


Approved-by: Katon Minhas
2025-05-07 18:21:09 +00:00
Alex Galarce 4191a5616e Merged in feature/rework-reimb-tin-npi (pull request #502)
Feature/rework reimb tin npi

* use LLM for reimb_[tin/npi/name]

* Merge remote-tracking branch 'origin/main' into feature/rework-reimb-tin-npi

* Merge remote-tracking branch 'origin/main' into feature/rework-reimb-tin-npi

* deduplicate tin/name/npi before adding to prompt

* Pull out prompt and llm_response in methodology_breakout_single_row for easier debugging

* Refactor reimbursement_tin_npi to handle multiple providers and update valid TIN, NPI, and NAME fields accordingly

* Reorder REIMB_PROV_NAME field in investment_prompts.json

* Update REIMB_PROV_NAME prompt to specify valid values for provider names

* Remove valid_values constraint from prov_name to raise accuracy

* Remove debugging print statements from reimbursement_tin_npi and related functions

* Remove debugging print statements from run_provider_info_fields function

* Merge remote-tracking branch 'origin/main' into feature/rework-reimb-tin-npi

* Remove debugging print statements from get_reimbursement_primary function

* Merge remote-tracking branch 'origin/main' into feature/rework-reimb-tin-npi

* Refactor reimbursement_tin_npi to always add providers (when we have any) to reimbursement_level_fields


Approved-by: Katon Minhas
2025-05-02 20:13:02 +00:00
Alex Galarce 4844dcfd50 Merged in feature/tin-npi-prompt-formatting (pull request #490)
Collect TIN/NPI/Name page-by-page and run a second pass for IS_GROUP

* configure first pass: no providers marked IS_GROUP.  Allows simplification of prompt

* Add group provider identification logic in second pass

* group identification: don't take 20% of pages.  Take first 3 and last 3

* Refine group provider identification logic to use pipes for JSON formatting and improve error handling

* fix mypy error for `identify_group_provider`

* Merge remote-tracking branch 'origin/main' into feature/tin-npi-prompt-formatting

* Add "PROV_INFO_JSON" to investment column order

* Refactor group provider identification to use signature page extraction and fix JSON format

* move signature page extraction to `string_utils.py` and add unit tests for it

* Update identify_group_provider to return provider name along with TIN and NPI in JSON format

* Update identify_group_provider to always use group-provided name when available

* move out GROUP_TIN_NPI_TEMPLATE to `investment_prompts.py`

* Remove deprecated PROV_GROUP_TIN_CHECK and PROV_GROUP_NPI_CHECK prompts

* Move out provider info fields from `one_to_one_funcs` to `tin_npi_funcs`

* isort

* remove debug print statements

* Remove debug print statements from provider info functions

* add note to regex_utils.py about where functions are used

* Add page_key_sort function and corresponding tests for sorting page keys

* Refactor identify_group_provider to sort pages and avoid duplicates in extracted sections

* Remove debug print statements from identify_group_provider function

* Merge remote-tracking branch 'origin/main' into feature/tin-npi-prompt-formatting

* extract_signature_page function: prioritize Textract marker for signatures and fallback to keyword search

* fix mypy error

* Update test cases to reflect changes in signature extraction logic


Approved-by: Katon Minhas
2025-04-25 20:28:52 +00:00
Katon Minhas af1a16cc0c Merged in field/dynamic-codes (pull request #475)
Field/dynamic codes

* remove test

* Merge branch 'main' into field/dynamic-codes

* Merged main into field/dynamic-codes

* Update test

* Fix unit test

* Merged main into field/dynamic-codes

* Merged main into field/dynamic-codes

* Merged main into field/dynamic-codes

* Update fields with crosswalks

* Update mapping to return empty string if no mapping

* Update dynamic_funcs

* Restructure

* Update file_processing

* Update dynamic primary

* Fix dynamic primary

* Update base fields

* Update one-to-n process

* Merged main into field/dynamic-codes

* genericized get_dynamic_answers

* Update tests

* Remove test file

* dynamic_funcs cleanup

* remove prints

* add exhibit_header

* Fix imports


Approved-by: Alex Galarce
2025-04-23 22:00:02 +00:00
Alex Galarce c8e64bcd4b Merged in feature/align-tin-npi (pull request #485)
Feature/align tin npi

* Add merge_provider_info function to consolidate provider data into one-to-one results

* Merge remote-tracking branch 'origin/main' into feature/align-tin-npi

* Remove debug print statement from run_one_to_one_prompts function

* turn run_regex_fields into an orchestrator function by breaking parts of it into separate functions

* Remove unused import of get_matches function and add docstring to chunk_on_matches for clarity

* Add docstring to run_regex_fields for improved clarity and documentation

* fix provider name keys in merge_provider_info

* Remove unused functions get_matches, tin_npi_prompt, and clean_tin_npi from tin_npi_funcs.py

* Set provider name to "UNKNOWN" if cleaned name is empty in clean_provider_info

* Add unit tests for clean_provider_info function to validate data processing

* Enhance docstring for get_all_matches function

* Add tests for get_all_matches and chunk_on_matches functions

* Merge remote-tracking branch 'origin/main' into feature/align-tin-npi

* Implement merge_provider_info function to consolidate provider data into one-to-one results

* Refactor merge_provider_info to simplify list conversion for group and other provider information

* Remove unnecessary TIN and NPI postprocessing step from postprocess function

* Remove clean_tin_npi_other function and its invocation from postprocess function

* Use json prompts for TIN/NPI/Name and inject them into TIN_NPI_TEMPLATE

* Rename run_regex_fields to run_provider_info_fields

* Enhance TIN_NPI_TEMPLATE to include detailed instructions for extracting provider entities and their identifying information, including IS_GROUP logic for main contracting parties.

* moved TIN/NPI regexes to regex_patterns.py

* Remove unnecessary blank line in investment_values.py

* Remove debug print statements from run_provider_info_fields and get_provider_info functions

* Enhance clean_provider_info to handle various representations of IS_GROUP as boolean

* Enhance get_provider_info to include error handling for JSON parsing and ensure consistent return format


Approved-by: Katon Minhas
2025-04-18 20:50:25 +00:00
Katon Minhas a9da72871e Merged in feature/dynamic-one-to-one (pull request #461)
Feature/dynamic one to one

* Pass N/A to one-to-one

* E2E test passed

* Merged main into feature/dynamic-one-to-one

* Pass ANY non N/A answer to reimbursement-level

* Update test

* Remove test

* Merge branch 'main' into feature/dynamic-one-to-one

* Wrap clean_dates

* Revert postprocess

* Merged main into feature/dynamic-one-to-one

* Merged main into feature/dynamic-one-to-one

* Remove prints


Approved-by: Alex Galarce
2025-04-08 18:29:39 +00:00
Katon Minhas 73b106b041 Merged in field_group/optimize_dynamic (pull request #459)
field group/optimize dynamic

* refactor tin/npi

* Refactor dynamic

* Refactor dynamic

* testing

* Save

* Merge branch 'main' into field_group/optimize_dynamic

* Merge branch 'main' into field_group/optimize_dynamic

* Prompt improvements

* save

* Move fill na mapping to AArete-Derived section

* MR prep

* Fix pipeline

* Merged main into field_group/optimize_dynamic

* Save

* Split crosswalk and aarete-derived

* Merge prep

* Pipeline fixes

* update test


Approved-by: Alex Galarce
2025-03-31 20:07:19 +00:00
Katon Minhas f8b8a580ff Merged in fields/additional-initializations (pull request #447)
fields/additional initializations

* Set up test file

* New fields + new field names

* Add initial prompts to json

* Column renaming - remove CONTRACT_ prefix

* update tests

* testing

* Fix regex pattern in json.loads

* Merge branch 'hotfix/file-errors' into refactor/new-field-names

* add UNIT_OF_MEASURE

* initialize new values

* Add rate change interval

* Merge branch 'main' into fields/additional-initializations

* remove test

* replace field names

* update REIMB_ID

* update date columns

* update reimbursement effective dt

* Revert adhoc scripts

* Revert client files

* Update postprocessing_funcs

* revert regex_utils

* Fix valid issue

* UNIT_OF_MEASURE

* REIMB_TERM

* Revert postprocessing funcs tests


Approved-by: Alex Galarce
2025-03-21 18:11:55 +00:00
Mayank Aamseek 138ac82fa2 Merged in hotfix/file-errors (pull request #448)
Hotfix/file errors

* Set up test file

* Fix regex pattern in json.loads

*  bug fix No valid JSON object found in the input string

* bug fixes

* moved try statement to parse fn

* invalid literal for int() with base 10 error fixed

* remove test

* remove excess print statements

* Raise error

* Remove todo

* Remove print statement


Approved-by: Katon Minhas
2025-03-19 22:11:27 +00:00
Alex Galarce 1c71592ca9 Merged in feature/unique-identifier-for-reimbursements (pull request #443)
Generate unique IDs for reimbursements

* Add reimbursement ID generation and update function signatures for unique identification

* Merge remote-tracking branch 'origin/main' into feature/unique-identifier-for-reimbursements


Approved-by: Katon Minhas
2025-03-17 20:38:50 +00:00
Katon Minhas 60c3104119 Merged in field/tin-npi-breakout (pull request #435)
field/tin npi breakout

* additional comments added

* pipeline error fixed

* final bug fixes during testing

* Merge branch 'main' into field/tin-npi-breakout

* formatting change - no error otherwise

* Merge branch 'field/tin-npi-breakout' of https://bitbucket.org/aarete/doczy.ai into field/tin-npi-breakout
pulled latest changes

* Streamline tin_npi_funcs

* Prompt changes

* remove punctuation

* Remove duplicates after cleaning

* Prompt mods

* Update values

* Prompt mods - allow for N/A

* Prompt for ALL NPIs

* Change clean function

* Allow multiple group TIN/NPI

* Prompt update

* Merged main into field/tin-npi-breakout

* remove test file

* Fix return type

* Restore testbed

* Merged main into field/tin-npi-breakout

* tin_npi_funcs.py edited online with Bitbucket
* Removed commented function

* Remove regex fields from one-to-one


Approved-by: Alex Galarce
2025-03-06 20:50:36 +00:00
Alex Galarce 2f96963162 Merged in feature/reimbursement-primary-optimizization (pull request #428)
Fix load_embeddings()

* move datetime_str function to string_utils module

* enhance load_embeddings to create directory structure and handle download errors

* Merged main into feature/reimbursement-primary-optimizization


Approved-by: Katon Minhas
2025-03-04 22:08:16 +00:00
Katon Minhas 1220f450ac Merged in field/initialize-prov-specialty (pull request #417)
field/initialize prov specialty

* Run One-To-N before One-To-One

* Switch PROV_TYPE to Exhibit-Level

* Modify PROV_TYPE prompt

* Add REIMBURSEMENT_PROV_NAME prompt

* prompt modification

* Prompt change

* Merge branch 'main' into field/initialize-reimb-prov-name

* Merge branch 'main' into field/initialize-prov-specialty

* Initialized Prov_Specialty

* Remove print statements

* Removed testing variables

* Remove test statements

* add CONTRACT_PROV_SPECIALTY

* prompt changes

* Update crosswalks

* Add mapping for lists


Approved-by: Alex Galarce
2025-02-26 14:14:30 +00:00
Katon Minhas ba6420102b Merged in field/reconfigure-prov-type (pull request #414)
Field/reconfigure prov type

* Run One-To-N before One-To-One

* Switch PROV_TYPE to Exhibit-Level

* Modify PROV_TYPE prompt

* Merge branch 'main' into field/reconfigure-prov-type


Approved-by: Alex Galarce
2025-02-25 17:11:02 +00:00
Katon Minhas a43f574239 Merged in hotfix/remove-print-statements (pull request #411)
Hotfix/remove print statements

* Remove print statements

* added llm_answer_raw

* prompt edit


Approved-by: Alex Galarce
2025-02-21 20:24:34 +00:00
Katon Minhas 81dfe7896b Merged in field/stop-loss (pull request #407)
StopLoss, Outliers, and Sequestration

* changed any method to all method in list handling

* code mapping moved outside to run once

* Merge branch 'feature/daip2-97' into bugfix/proc-code-lists

* merge conflict fixed

* merge conflicts fixed

* incorporated latest changes from main

* Merged main into bugfix/proc-code-lists

* Move load all dataset to io_utils

* Fix list is_empty

* Updated poetry.lock

* Bugfix

* Docstring

* Docstrings

* Merge branch 'main' into bugfix/proc-code-lists

* Merge branch 'bugfix/proc-code-lists' into bugfix/dynamic-assignment-issue

* Add outliers, stop-loss, and sequestration to reimbursement-level primary prompt template

* Add stop loss fields

* Add outlier fields

* Add sequestration fields

* Remove test

* Docstrings

* Update test for no default exhibit_level_answer_dict

* Merge branch 'bugfix/dynamic-assignment-issue' into field/stop-loss

* Merge branch 'main' into field/stop-loss

* Merged main into field/stop-loss


Approved-by: Alex Galarce
2025-02-21 19:55:12 +00:00
Katon Minhas 92029cdabe Merged in bugfix/proc-code-lists (pull request #403)
Bugfix/proc code lists

* Add admit type code

* add proc mod description

* Add bill type pos mapping

* Merged main into feature/proc-desc

* Uncomment test code

* Bugfix

* script optimisation for codes

* modified is empty function in string utils

* Remove test.py

* bug fix to handle list of proc_codes

* Merged main into bugfix/proc-code-lists

* changed any method to all method in list handling

* code mapping moved outside to run once

* Merge branch 'feature/daip2-97' into bugfix/proc-code-lists

* merge conflict fixed

* merge conflicts fixed

* incorporated latest changes from main

* Merged main into bugfix/proc-code-lists

* Move load all dataset to io_utils

* Fix list is_empty

* Updated poetry.lock

* Docstring

* Docstrings

* Merge branch 'main' into bugfix/proc-code-lists

* bug fix


Approved-by: Mayank Aamseek
Approved-by: Alex Galarce
2025-02-21 14:44:36 +00:00
Alex Galarce 073995a945 Merged in hotfix/fix_chroma_add_timestamps (pull request #405)
Hotfix/fix chroma add timestamps

* re-add chroma persistence

* add timestamps

* Merged main into hotfix/fix_chroma_add_timestamps

* add timestamp to file processing log message

* forgot parentheses to actually write datetime


Approved-by: Katon Minhas
2025-02-20 20:19:53 +00:00
Alex Galarce a18d562d34 Merged in hotfix/fix-merge-when-no-reimb (pull request #401)
change join to outer so 1:1 merge works when no reimb

* change join to outer so 1:1 merge works when no reimb


Approved-by: Katon Minhas
2025-02-19 19:59:27 +00:00
Alex Galarce b0c1730a0a Merged in hotfix/remove-persistent-chroma (pull request #398)
remove chroma persistence and add print statements

* remove chroma persistence and add print statements

* add print statement for file writing completion and remove commented-out code


Approved-by: Katon Minhas
2025-02-14 22:43:01 +00:00
Mayank Aamseek 8533d3cf78 Merged in feature/daip2-101 (pull request #384)
DRAFT: missing group tin fixed

* missing group tin fixed

* use tin prompt in full context when N/A

* Merged main into feature/daip2-101


Approved-by: Katon Minhas
2025-02-07 20:13:41 +00:00
Katon Minhas 71e0eb5c45 Merged in feature/reimbursement-level-tins (pull request #379)
Initialize Reimbursement-Level TINs

* Initialized Reimbursement-Level TIN and NPI

* Update unit tests

* Docstrings and cleanup

* Update test


Approved-by: Alex Galarce
2025-02-03 21:50:50 +00:00
Alex Galarce 55f96e77fc Merged in feature/contract_to_aarete_derived_fields (pull request #376)
Feature/contract to aarete derived fields

* aarete derived prompt template added

* get_aarete_derived function added

* fixed valid values issue

* fixed valid values

* modified aarete derived for dataframe

* Merge remote-tracking branch 'origin/main' into feature/contract_to_aarete_derived_fields

* add commented-out test shell

* Fixed one_to_n file running

* updated PIPE_FORMAT_INSTRUCTIONS

* Fix conflict

* Update FieldSet class filters

* Updated crosswalk mapping integration

* Merged main into feature/contract_to_aarete_derived_fields

* Merge branch 'main' into feature/contract_to_aarete_derived_fields

* Merged main into feature/contract_to_aarete_derived_fields

* Removed test functionality

* aareted derived llm values for unique column values

* investment_prompts.py edited online with Bitbucket
* Removed print statements

* Changed load_from_file - field_set = FieldSet(file_path=fields.json, relationship=one-to-one, field_type=True, base_field=True)

* Throw type error

* Remove ellipsis


Approved-by: Katon Minhas
2025-02-03 17:29:59 +00:00
Katon Minhas 66f429d0db Merged in field/tin-npi-initialization (pull request #374)
Initialize 1:1 TIN and NPI Fields (Using CNC methodology)

* 1:1 TIN and NPI fields added (using CNC methodology)

* Merge branch 'main' into field/tin-npi-initialization

* Merged main into field/tin-npi-initialization

* Merged main into field/tin-npi-initialization

* Merged main into field/tin-npi-initialization


Approved-by: Alex Galarce
2025-01-31 22:40:17 +00:00