Table Continuation Fix
* Add EXHIBIT_HEADER_MARKERS for improved document header recognition in table handling
* Refactor EXHIBIT_HEADER_MARKERS definition for improved clarity and organization
* remove unused page span tracking code; add unit tests for new functionality
* remove debugging logs from combine_continuous_tables function
* Refactor logging setup and update test file configuration in investment field testing notebook
* Merge remote-tracking branch 'origin/main' into feature/further-phase-1-and-2-fixes
Approved-by: Katon Minhas
Feature/global lesser of
* Merge remote-tracking branch 'origin/feature/reimb-primary-updates'
* First try at global "lesser of"
* Generate reimb_ids further upstream
* Implement global lesser of injection and extraction logic in file processing
* Update logic to change LESSER_OF_IND in-place for global affected rows; add docstrings
* Refactor global lesser of injection logic to create new rows for REIMB_IDs lacking lesser of logic; update original rows in place and improve tracking flags.
* Merge remote-tracking branch 'origin/main' into feature/global-lesser-of
* Add logging for global lesser of logic injection and update handling of original rows
* Refactor global lesser of logic: move functions to one_to_one_funcs.py and update inject_global_lesser_of_rows to use new implementations
* Move imports for METHODOLOGY_BREAKOUT_QUESTIONS and REIMB_TERM_BREAKOUT to the top of one_to_one_funcs.py for better organization and to remove prototype comments.
* isort
* Refactor import statement for generate_reimb_ids to use the full module path for clarity
* load field from JSON instead of hardcoded values
* Update logging levels in investment_field_testing notebook
* Merged main into feature/global-lesser-of
* Remove unnecessary comment from investment_postprocessing_funcs import statement for clarity
* Merge branch 'feature/global-lesser-of' of https://bitbucket.org/aarete/doczy.ai into feature/global-lesser-of
* Merge remote-tracking branch 'origin/main' into feature/global-lesser-of
* Remove debug print statement from process_file function
* fix call to Field.load_from_file
* Merge remote-tracking branch 'origin/main' into feature/global-lesser-of
* Add tracking flag for global lesser of logic when none is found
* explicitly sort by FILE_NAME, EXHIBIT_PAGE, and REIMB_LESSER_OF_ID columns
* Add new field type, "full_context_separate", to make sure that we don't run it twice. Also formatting fix
* Fix iteration over separate fields in run_full_context_fields function
Approved-by: Katon Minhas
Rework reimbursement ID to associate parts of "lesser of" (or other such coupled statement) with one another
* refactor reimbursement ID generation with hashed service terms
* reduce hash length for unique reimbursement ID generation
* update reimbursement ID format to include 'exh_pg' for clarity
* clean up testing.ipynb
* one more formatting fix to testing.ipynb
* Move reimb_id generation to postprocessing
* move reimb_id function to postprocessing_funcs.py
* refactor generate_reimb_ids to group by FILE_NAME and EXHIBIT_PAGE for ID generation
* clean testing.ipynb further
Approved-by: Katon Minhas
Better visibility and testing functionality for optimizing prompts
* first commit
* Enhance prompts and processing for contract termination dates
- Updated prompt to include 'year to year' as a valid example output.
- Implemented case-insensitive processing for contract text and field keywords for smart chunking.
- Clarified prompt for derived termination date to specify it is exclusive of renewals
* Merge remote-tracking branch 'origin/main' into test/check-doc-dates-against-test-bed
* Update termination date prompt tests
* Refine keywords for contract termination date prompt in investment prompts
* Docs on using the testing notebook
* add 'input_files' option to test_params
* Merge remote-tracking branch 'origin/main' into test/check-doc-dates-against-test-bed
* comment out debug line
* refactor: adjust input_files filtering logic in main function
Approved-by: Katon Minhas
Test/check doc dates against test bed
* first commit
* Enhance prompts and processing for contract termination dates
- Updated prompt to include 'year to year' as a valid example output.
- Implemented case-insensitive processing for contract text and field keywords for smart chunking.
- Clarified prompt for derived termination date to specify it is exclusive of renewals
* Merge remote-tracking branch 'origin/main' into test/check-doc-dates-against-test-bed
* Update termination date prompt tests
* Refine keywords for contract termination date prompt in investment prompts
Approved-by: Katon Minhas
Refactor/daip2-9 code refactor
* add missing import
* refactor ac_smart_chunking.py
* update tests
* removed old and unused imports
* removed outdated import
* forgot to import re
* refactor bottom up funcs
* remove unused imports from file_processing.py
* refactor dict_operations.py
* remove unused import from conditional_funcs.py
* replace top_down_funcs with one_to_n_funcs and remove top_down_funcs.py
* remove duplicate import from one_to_n_funcs.py
* refactor all utils.py imports
* refactor error handling by removing last code in utils and integrating InvalidDateException into postprocessing_funcs
* refactor import in claude_funcs.py to use string_funcs directly and (hopefully) resolve circular import
* refactor regex_funcs.py to use postprocessing_funcs for add_hyphen_if_needed calls
* refactor hotfix_helper_funcs.py to use postprocessing_funcs for add_hyphen_if_needed calls
* refactor postprocess.py to remove unused imports
* add numpy import to string_funcs
* fix tests after utils refactor
* move adhoc from tests to scripts
* remove unused imports
* remove scripts from mypy checking
* isort
* Merged main into refactor/daip-2-9-code-refactor
Approved-by: Katon Minhas
Feature/self repair main
* fixes
* fixes
* fixes
* fixes
* fixes
* fixes
* fixes
* fixes
* new testing
* new testing
* added unit tests under fieldExtraction/tests/main-unit-tests
* unit test fixes
* unit test fixes
* merge into main
* modified consolidate_output and slight changes to qa_qc_helpers to accomodate qaqc functionality w faizan and michael's changes
* merging self repair into main
* synced with main now
* fixes
* provider billing removed
* provider billing removed
* provider billing fixed
* commented out load dotenv in config
* Merged main into feature/self-repair-main
* modified check_s3_bucket_exists function by returning False if credentials not found for pipeline
* Merge branch 'feature/self-repair-main' of https://bitbucket.org/aarete/doczy.ai into feature/self-repair-main
Approved-by: Michael McGuinness
QCQA Additions
* tested on batch 10, small fixes + change to s3 output
* Merged main into QCQA_Additions
* unit tests for qc qa helpers
* static error fix
* another static fix
* static fix again
* fixed None check
* static fix none
* Merged main into QCQA_Additions
* Small changes based on Katon's review
Approved-by: Katon Minhas
QCQA Integration
* Merged in bugfix/npi_hotfix (pull request #305)
double filter for duplicates
* double filter for duplicates
Approved-by: Katon Minhas
* Merged in chore/npi (pull request #309)
npi hotfix into main
* npi hotfix into main
* npi hotfix into main
Approved-by: Alex Galarce
* Merged in chore/irs (pull request #310)
irs hotfix into main
* irs hotfix into main
* pass top sheet to irs hotfix func
* docstring updates
Approved-by: Alex Galarce
* slight fix to methodology short fix
* Merged in bugfix/all_cnc_hotfixes (pull request #302)
DRAFT: Bugfix/all cnc hotfixes
* Pipeline fix - health plan state
* Update for prompt vs non-prompt fixes
* Revert utils.find_regex_matches()
* Remove prints
* Non-prompt run
* Health Plan State, IRS, NPI, LOB, Lesser, Prov 2, Default, Rate Standard, Contract Effective Date, Term Group
* Merged in bugfix/agreement_name (pull request #294)
contract title fixed
* contract title fixed
* Merged bugfix/all_cnc_hotfixes into bugfix/agreement_name
Approved-by: Katon Minhas
* All hotfixes added
* Fixed utils
* fix for clean_contract_effective_date
* Merged in default-fix (pull request #296)
moved default funcs to the top, dropped intermediate columns
* moved default funcs to the top, dropped intermediate columns
Approved-by: Katon Minhas
* quick fix to dropping extra columns
* Merged in bugfix/effective_date_meridian (pull request #297)
fixed effective date for meridian contracts
* fixed effective date for meridian contracts
Approved-by: Kat…
* Merged in hotfix/irs_tweak (pull request #303)
returning None instead on str(N/A)
* returning None instead on str(N/A)
Approved-by: Katon Minhas
* Merged in bugfix/npi_hotfix (pull request #305)
double filter for duplicates
* double filter for duplicates
Approved-by: Katon Minhas
* Merged in bugfix/all_cnc_hotfixes (pull request #302)
DRAFT: Bugfix/all cnc hotfixes
* Pipeline fix - health plan state
* Update for prompt vs non-prompt fixes
* Revert utils.find_regex_matches()
* Remove prints
* Non-prompt run
* Health Plan State, IRS, NPI, LOB, Lesser, Prov 2, Default, Rate Standard, Contract Effective Date, Term Group
* Merged in bugfix/agreement_name (pull request #294)
contract title fixed
* contract title fixed
* Merged bugfix/all_cnc_hotfixes into bugfix/agreement_name
Approved-by: Katon Minhas
* All hotfixes added
* Fixed utils
* fix for clean_contract_effective_date
* Merged in default-fix (pull request #296)
moved default funcs to the top, dropped intermediate columns
* moved default funcs to the top, dropped intermediate columns
Approved-by: Katon Minhas
* quick fix to dropping extra columns
* Merged in bugfix/effective_date_meridian (pull request #297)
fixed effective date for meridian contracts
* fixed effective date for meridian contracts
Approved-by: Kat…
* Merged in hotfix/irs_tweak (pull request #303)
returning None instead on str(N/A)
* returning None instead on str(N/A)
Approved-by: Katon Minhas
* Merged in bugfix/npi_hotfix (pull request #305)
double filter for duplicates
* double filter for duplicates
Approved-by: Katon Minhas
* Merged in chore/npi (pull request #309)
npi hotfix into main
* npi hotfix into main
* npi hotfix into main
Approved-by: Alex Galarce
* Merged in chore/irs (pull request #310)
irs hotfix into main
* irs hotfix into main
* pass top sheet to irs hotfix func
* docstring updates
Approved-by: Alex Galarce
* Merged in bugfix/clean_short_methodology (pull request #312)
changed regex pattern in get_short_methodology
* changed regex pattern in
Approved-by: Katon Minhas
* Merged in hotfix/filename_fix (pull request #314)
filename corrected
* filename corrected
* Merged main into hotfix/filename_fix
Approved-by: Katon Minhas
* Merge remote-tracking branch 'remotes/origin/main' into HEAD
* rearranged qcqa scripts/funcs and prep for merge w main, moved lots of stuff to postprocess
* slight change to icm_number_fix
* transitioned some qcqa funcs to postprocessing, added fields to qcqa report
* Merged main into QCQA_Integration
* reorganization of qa qc helpers, consolidate_output, qa_qc from Katon's feedback
* adhoc and at scale qcqa ran successfully
* fix to consolidate_output
* Merge remote-tracking branch 'remotes/origin/main' into QCQA_Integration
* Merged main into QCQA_Integration
* small changes to npi_check, changed name of config.CONTRACT_FILE_NAME to config.FILE_NAME_COLUMN
Approved-by: Katon Minhas
Feature/hotfix stitch
* Add hotfix scripts
* Merged main into feature/hotfix_stitch
* Moved stitching scripts to scripts folder
* Update for 2
* Fixed flat fee issue
* Fix payer name issue
* All hotfix updates
* Merged main into feature/hotfix_stitch
* Add rapidfuzz
Approved-by: Alex Galarce
DRAFT: Bugfix/all cnc hotfixes
* Pipeline fix - health plan state
* Update for prompt vs non-prompt fixes
* Revert utils.find_regex_matches()
* Remove prints
* Non-prompt run
* Health Plan State, IRS, NPI, LOB, Lesser, Prov 2, Default, Rate Standard, Contract Effective Date, Term Group
* Merged in bugfix/agreement_name (pull request #294)
contract title fixed
* contract title fixed
* Merged bugfix/all_cnc_hotfixes into bugfix/agreement_name
Approved-by: Katon Minhas
* All hotfixes added
* Fixed utils
* fix for clean_contract_effective_date
* Merged in default-fix (pull request #296)
moved default funcs to the top, dropped intermediate columns
* moved default funcs to the top, dropped intermediate columns
Approved-by: Katon Minhas
* quick fix to dropping extra columns
* Merged in bugfix/effective_date_meridian (pull request #297)
fixed effective date for meridian contracts
* fixed effective date for meridian contracts
Approved-by: Katon Minhas
* Default rate and Lesser fix
* Merged in hotfix/irs_npi_updated (pull request #298)
updated irs and npi hotfix funcs
* updated irs and npi hotfix funcs
* Merged bugfix/all_cnc_hotfixes into hotfix/irs_npi_updated
Approved-by: Katon Minhas
* non-functional commit - default only
* debug commit - premerge
* Batch 1 Dup File Reconciliation
* Default rate fix
* Full CNC 1A Run Config
* Standard column refactor
* Working stitching for 1A
* Merged in Alex-Galarce/cnc_hotfix_effective_date_utilspy-edited-1732659983687 (pull request #301)
Remove "license" from effective date smart chunking
* Remove "license" from effective date smart chunking
* Merged bugfix/all_cnc_hotfixes into Alex-Galarce/cnc_hotfix_effective_date_utilspy-edited-1732659983687
Approved-by: Katon Minhas
* Merged in hotfix/irs_batch2_feedback (pull request #300)
IRS fix for feedback on batch2
* IRS fix for feedback on batch2
* Merged bugfix/all_cnc_hotfixes into hotfix/irs_batch2_feedback
Approved-by: Katon Minhas
* Merged main into bugfix/all_cnc_hotfixes
Approved-by: Katon Minhas
grouped keywords and IRS regex chunking
* abandon old branch and rebuild
* removed print for field names, prompts
* pushing as part of leftovers
* IRS regex chunking
* Merged in feature/chunk_term_clean (pull request #262)
chunk_log_added
* chunk_log_added
* Merge remote-tracking branch 'remotes/origin/feature_ac_chunking_clean' into feature/chunk_term_clean
Approved-by: Alex Galarce
* added function for regex based IRS chunking
* shifted regex_match_chunk function to utils
* added helper functions for regex based chunking
* replaced tin_regex function to ac_funcs
* removed prompt for irs_others
* added regex chunking for irs
* removed irs group from smart chunking
* testing functionality for regex based irs fields
* updated return N/A condition
* Merged main into feature_ac_chunking_clean
* file_processing.py edited online with Bitbucket
* added back conditional prompts
* minor fixed for PR
* remove install types
* mering clean branch
* with passed test casses
* added flag for regex based tin execution
Approved-by: Michael McGuinness
Katon Minhas