Feature/daip2-7-crosswalk
* add crosswalk framework
* remove erroneous print statements
* fix mypy error, format to black and isort
* add README.md
* Merged main into feature/daip2-7-crosswalk
Approved-by: Katon Minhas
slight change to prompt, tested on 83 files
* slight change to prompt, tested on 83 files
* Merged main into bugfix/default-rate-prompt-adjustment
Approved-by: Katon Minhas
Feature/self repair main
* fixes
* fixes
* fixes
* fixes
* fixes
* fixes
* fixes
* fixes
* new testing
* new testing
* added unit tests under fieldExtraction/tests/main-unit-tests
* unit test fixes
* unit test fixes
* merge into main
* modified consolidate_output and slight changes to qa_qc_helpers to accomodate qaqc functionality w faizan and michael's changes
* merging self repair into main
* synced with main now
* fixes
* provider billing removed
* provider billing removed
* provider billing fixed
* commented out load dotenv in config
* Merged main into feature/self-repair-main
* modified check_s3_bucket_exists function by returning False if credentials not found for pipeline
* Merge branch 'feature/self-repair-main' of https://bitbucket.org/aarete/doczy.ai into feature/self-repair-main
Approved-by: Michael McGuinness
fix logic error when effective date reformatting is applied twice when the date is already filled
* fix logic error when effective date reformatting is applied twice when the date is already filled
Approved-by: Katon Minhas
move openpyxl from dev dependencies to general dependencies bc we use it for qaqc now
* move openpyxl from dev dependencies to general dependencies bc we use it for qaqc now
Approved-by: Katon Minhas
Hotfix/Exhibit
* Initial draft - Exhibit test
* Update logic
* Add Exhibit page_num column, add pipe format to prompt
* Working test
* Update LOB to check new Exhibit
* Greenlight file commit - needs minor efficiency improvement
* Reordering, intermediate file saving, Exhibit+LOB Fix
* Merged main into Exhibit branch
* removed excess print statements
* Merged main into hotfix/Exhibit
* Updated valid lists for mapping, reconfigured for rerun filter
* Extra print
* added fixes for batch 7
* added fixes for batch 7
* Merged main into hotfix/Exhibit
* apply_cnc_hotfix.py edited online with Bitbucket
Approved-by: Alex Galarce
QCQA Additions
* tested on batch 10, small fixes + change to s3 output
* Merged main into QCQA_Additions
* unit tests for qc qa helpers
* static error fix
* another static fix
* static fix again
* fixed None check
* static fix none
* Merged main into QCQA_Additions
* Small changes based on Katon's review
Approved-by: Katon Minhas
Bugfix/termGroupHotfix
* Minor term group fix
* Added xlsb to gitignore, updated apply_cnc_hotfix for Filename prefix, added xlsb read functionality to utils
* Date bugfix
* Fixed effective date bug
* Effective date bug fixes
* Fix agreement name field name bug
* Further bug fixes
* update poetry for pyxlsb
* Write to excel after each file is completed
* Fix for restarts
Approved-by: Alex Galarce
log information logic added to hotfix for contract effective date
* log information logic added to hotfix for contract effective date
Approved-by: Katon Minhas
QCQA Integration
* Merged in bugfix/npi_hotfix (pull request #305)
double filter for duplicates
* double filter for duplicates
Approved-by: Katon Minhas
* Merged in chore/npi (pull request #309)
npi hotfix into main
* npi hotfix into main
* npi hotfix into main
Approved-by: Alex Galarce
* Merged in chore/irs (pull request #310)
irs hotfix into main
* irs hotfix into main
* pass top sheet to irs hotfix func
* docstring updates
Approved-by: Alex Galarce
* slight fix to methodology short fix
* Merged in bugfix/all_cnc_hotfixes (pull request #302)
DRAFT: Bugfix/all cnc hotfixes
* Pipeline fix - health plan state
* Update for prompt vs non-prompt fixes
* Revert utils.find_regex_matches()
* Remove prints
* Non-prompt run
* Health Plan State, IRS, NPI, LOB, Lesser, Prov 2, Default, Rate Standard, Contract Effective Date, Term Group
* Merged in bugfix/agreement_name (pull request #294)
contract title fixed
* contract title fixed
* Merged bugfix/all_cnc_hotfixes into bugfix/agreement_name
Approved-by: Katon Minhas
* All hotfixes added
* Fixed utils
* fix for clean_contract_effective_date
* Merged in default-fix (pull request #296)
moved default funcs to the top, dropped intermediate columns
* moved default funcs to the top, dropped intermediate columns
Approved-by: Katon Minhas
* quick fix to dropping extra columns
* Merged in bugfix/effective_date_meridian (pull request #297)
fixed effective date for meridian contracts
* fixed effective date for meridian contracts
Approved-by: Kat…
* Merged in hotfix/irs_tweak (pull request #303)
returning None instead on str(N/A)
* returning None instead on str(N/A)
Approved-by: Katon Minhas
* Merged in bugfix/npi_hotfix (pull request #305)
double filter for duplicates
* double filter for duplicates
Approved-by: Katon Minhas
* Merged in bugfix/all_cnc_hotfixes (pull request #302)
DRAFT: Bugfix/all cnc hotfixes
* Pipeline fix - health plan state
* Update for prompt vs non-prompt fixes
* Revert utils.find_regex_matches()
* Remove prints
* Non-prompt run
* Health Plan State, IRS, NPI, LOB, Lesser, Prov 2, Default, Rate Standard, Contract Effective Date, Term Group
* Merged in bugfix/agreement_name (pull request #294)
contract title fixed
* contract title fixed
* Merged bugfix/all_cnc_hotfixes into bugfix/agreement_name
Approved-by: Katon Minhas
* All hotfixes added
* Fixed utils
* fix for clean_contract_effective_date
* Merged in default-fix (pull request #296)
moved default funcs to the top, dropped intermediate columns
* moved default funcs to the top, dropped intermediate columns
Approved-by: Katon Minhas
* quick fix to dropping extra columns
* Merged in bugfix/effective_date_meridian (pull request #297)
fixed effective date for meridian contracts
* fixed effective date for meridian contracts
Approved-by: Kat…
* Merged in hotfix/irs_tweak (pull request #303)
returning None instead on str(N/A)
* returning None instead on str(N/A)
Approved-by: Katon Minhas
* Merged in bugfix/npi_hotfix (pull request #305)
double filter for duplicates
* double filter for duplicates
Approved-by: Katon Minhas
* Merged in chore/npi (pull request #309)
npi hotfix into main
* npi hotfix into main
* npi hotfix into main
Approved-by: Alex Galarce
* Merged in chore/irs (pull request #310)
irs hotfix into main
* irs hotfix into main
* pass top sheet to irs hotfix func
* docstring updates
Approved-by: Alex Galarce
* Merged in bugfix/clean_short_methodology (pull request #312)
changed regex pattern in get_short_methodology
* changed regex pattern in
Approved-by: Katon Minhas
* Merged in hotfix/filename_fix (pull request #314)
filename corrected
* filename corrected
* Merged main into hotfix/filename_fix
Approved-by: Katon Minhas
* Merge remote-tracking branch 'remotes/origin/main' into HEAD
* rearranged qcqa scripts/funcs and prep for merge w main, moved lots of stuff to postprocess
* slight change to icm_number_fix
* transitioned some qcqa funcs to postprocessing, added fields to qcqa report
* Merged main into QCQA_Integration
* reorganization of qa qc helpers, consolidate_output, qa_qc from Katon's feedback
* adhoc and at scale qcqa ran successfully
* fix to consolidate_output
* Merge remote-tracking branch 'remotes/origin/main' into QCQA_Integration
* Merged main into QCQA_Integration
* small changes to npi_check, changed name of config.CONTRACT_FILE_NAME to config.FILE_NAME_COLUMN
Approved-by: Katon Minhas
New Prompt Method (starting with contract effective date)
* add todos for Siddhant
* moved effective date prompts to prompts.py and added docstring and type hints
* moved effective date prompts to prompts.py and added docstring and type hints
* fixed conflict
Co-authored-by: Alex Galarce <agalarce@aarete.com>
* modified contract effective date logic in existing smart chunking
* Merged main into chore/contract_effective_date
* updated extract_text_from_delimiters function and unit tests for it
* Merge remote-tracking branch 'origin/main' into chore/contract_effective_date
* added comments to preprocessing_funcs
* preprocessing_funcs.py edited online with Bitbucket
* utils.py edited online with Bitbucket
* add debug line
* add NA handling into fix_date_formatting()
* add debug print statements
* worked on review comments
* fixed date checking
* fixed date checking and added unit tests
* Merged main into chore/contract_effective_date
* added logic to handle case when smart chunking returns empty context for effective date
* added comments to file_processing.py
* optimized imports and removed a TODO
* Merged main into chore/contract_effective_date
* made prompt changes for effective date
* keywords.py edited online with Bitbucket
* Removed excess print statements
Approved-by: Katon Minhas
refactored code to add recent health plan state hotfixes into the existing postprocessing pipeline
* refactored code to add recent health plan state hotfixes into the existing postprocessing pipeline
* Merged main into chore/health_plan_state
* Merged main into chore/health_plan_state
* add debug print statement
* debug print statements
* more debug print statements
* print df columns for debug
* move health plan state hotfix functionality to postprocessing_funcs.py
* add better error printing
* add error message for parent agreement code
* remove dots in state abbreviations
* remove unused import
* also look for state abbreviations
* Merged main into chore/health_plan_state
* remove print statements
* Merged main into chore/health_plan_state
Approved-by: Katon Minhas
agreement_name hotfix to main
* agreement_name hotfix to main
* group name changed to AGREEMENT_NAME
* Merged main into feature/agreement_name_chunking
Approved-by: Alex Galarce
Feature/hotfix stitch
* Add hotfix scripts
* Merged main into feature/hotfix_stitch
* Moved stitching scripts to scripts folder
* Update for 2
* Fixed flat fee issue
* Fix payer name issue
* All hotfix updates
* Merged main into feature/hotfix_stitch
* Add rapidfuzz
Approved-by: Alex Galarce
DRAFT: Bugfix/all cnc hotfixes
* Pipeline fix - health plan state
* Update for prompt vs non-prompt fixes
* Revert utils.find_regex_matches()
* Remove prints
* Non-prompt run
* Health Plan State, IRS, NPI, LOB, Lesser, Prov 2, Default, Rate Standard, Contract Effective Date, Term Group
* Merged in bugfix/agreement_name (pull request #294)
contract title fixed
* contract title fixed
* Merged bugfix/all_cnc_hotfixes into bugfix/agreement_name
Approved-by: Katon Minhas
* All hotfixes added
* Fixed utils
* fix for clean_contract_effective_date
* Merged in default-fix (pull request #296)
moved default funcs to the top, dropped intermediate columns
* moved default funcs to the top, dropped intermediate columns
Approved-by: Katon Minhas
* quick fix to dropping extra columns
* Merged in bugfix/effective_date_meridian (pull request #297)
fixed effective date for meridian contracts
* fixed effective date for meridian contracts
Approved-by: Katon Minhas
* Default rate and Lesser fix
* Merged in hotfix/irs_npi_updated (pull request #298)
updated irs and npi hotfix funcs
* updated irs and npi hotfix funcs
* Merged bugfix/all_cnc_hotfixes into hotfix/irs_npi_updated
Approved-by: Katon Minhas
* non-functional commit - default only
* debug commit - premerge
* Batch 1 Dup File Reconciliation
* Default rate fix
* Full CNC 1A Run Config
* Standard column refactor
* Working stitching for 1A
* Merged in Alex-Galarce/cnc_hotfix_effective_date_utilspy-edited-1732659983687 (pull request #301)
Remove "license" from effective date smart chunking
* Remove "license" from effective date smart chunking
* Merged bugfix/all_cnc_hotfixes into Alex-Galarce/cnc_hotfix_effective_date_utilspy-edited-1732659983687
Approved-by: Katon Minhas
* Merged in hotfix/irs_batch2_feedback (pull request #300)
IRS fix for feedback on batch2
* IRS fix for feedback on batch2
* Merged bugfix/all_cnc_hotfixes into hotfix/irs_batch2_feedback
Approved-by: Katon Minhas
* Merged main into bugfix/all_cnc_hotfixes
Approved-by: Katon Minhas
Bugfix/tracking bug fixes
* runtime tracking bug fix
* fixes to dashboard
* for ec2 host
* updated to match config without ac fields param
* Merge remote-tracking branch 'remotes/origin/main' into HEAD
* removed redundant get active runs function
Approved-by: Katon Minhas
term group postprocessing fix
* term group postprocessing fix
* docstring added
* Merged main into bugfix/term_group
* Merged main into bugfix/term_group
Approved-by: Alex Galarce
Approved-by: Katon Minhas
removed ac fields as it was removed from main from batch tracking
* removed ac fields as it was removed from main from batch tracking
Approved-by: Alex Galarce
Initial top sheet conditional commit
* Initial top sheet conditional commit
* Merged main into feature/topSheetConditional
* preprocessing_funcs.py edited online with Bitbucket
* Updated quick review docstring
Approved-by: Alex Galarce
Refactor/tidy claude funcs
* pull out create_cost_log_entry() into its own function
* add docstring
* isort
* Merged main into refactor/tidy-claude-funcs
* Merged main into refactor/tidy-claude-funcs
Approved-by: Michael McGuinness
Approved-by: Katon Minhas
Feature/add runtime tracking
* added return true
* added fixes and other features
* added fixes and other features
* added fixes and other features
* added fixes and other features
* added fixes and other features
* added fixes and other features
* test
* test
* test
* test
* test
* test
* added
* added
* added potential fix
* added potential fix
* functions
* added potential fix
* fixes
* Merged main into feature/add-runtime-tracking
* moved streamlit dashabord to streamlit folder
* Merge branch 'feature/add-runtime-tracking' of https://bitbucket.org/aarete/doczy.ai into feature/add-runtime-tracking
* remove src/valid.py to keep src empty
* add psutil to requirements
Approved-by: Michael McGuinness
update inputs for smart chunking
* update inputs for smart chunking
* updated test case inputs
* updated test case inputs
* Add keywords back to argument for ac smart chunking
* Test case fix
Approved-by: Katon Minhas
Bugfix/health plan state
* minor bugfix to cover an edge-case
* updated state_check function and unit tests
* Merged main into bugfix/health_plan_state
* Merged main into bugfix/health_plan_state
Approved-by: Katon Minhas
Add cost logging
* add inspect logging
* change comment
* add simple token estimation
* rudimentary costlog
* compute cost
* float formatting
* put costlog to s3
* float formatting attempt 2
* fix formatting, remove a couple prints
* add missing comma
* add filename
* make costlog a .csv instead of a .log
* use csv library instead of from-scratch csv writing
* switch to import math, add test
* pyproject.toml Added src to pythonpath
* Merged main into feature/costlog
* add jupyter to requirements, add test path to pyproject.toml
* Merge remote-tracking branch 'refs/remotes/origin/feature/costlog' into feature/costlog
* Merge remote-tracking branch 'origin/main' into feature/costlog
* merge in main
* add TODOs
* Merged main into feature/costlog
* Merged main into feature/costlog
Approved-by: Michael McGuinness
Approved-by: Katon Minhas
term group fixed
* term group fixed
* Merged main into bugfix/group_chunk_json_parsing
* file_processing.py edited online with Bitbucket
* Merged main into bugfix/group_chunk_json_parsing
Approved-by: Alex Galarce
Update for full-context AC chunks to return exact contract_text instead of concatenated text_dict pages
* Update for full-context AC chunks to return exact contract_text instead of concatenated text_dict pages
* Updated test cases
* Updated case 1
* Merged main into bugfix/full_context_check
Approved-by: Alex Galarce
Feature/add dotenv
* transferred sensitive information to .env file from src/config.py
* replace model IDs, AWS account info
* Merge remote-tracking branch 'origin/main' into feature/add-dotenv
* add explanatory comment
* Merge remote-tracking branch 'origin/main' into feature/add-dotenv
* remove tests that were just for local
Approved-by: Katon Minhas
Bugfix/clean full context in smart chunking
* move full context bugfix to clean branch
* took out commented code
* black formating
* Add explanatory comments to file_processing.py
Approved-by: Alex Galarce
plan state hotfix
* updated state_check function
* added a wrapper function apply_state_check_to_df function to enable state_check function to work with dataframe and also wrote unit tests for both the functions.
* Merged main into hotfix/plan_state
* append extra column with new health plan state corrected
* Merge remote-tracking branch 'refs/remotes/origin/hotfix/plan_state' into hotfix/plan_state
* fix tests
* move apply_state_check_to_df() to cnc_hotfix.py
* move all functionality out of utils and into cnc_hotfix.py
* Merge remote-tracking branch 'origin/main' into hotfix/plan_state
* remove duplicated pythonpath in pyproject.toml
* format cnc_hotfix.py
* changed to use valid.STATE_MAP to not duplicate functionality
Approved-by: Katon Minhas
grouped keywords and IRS regex chunking
* abandon old branch and rebuild
* removed print for field names, prompts
* pushing as part of leftovers
* IRS regex chunking
* Merged in feature/chunk_term_clean (pull request #262)
chunk_log_added
* chunk_log_added
* Merge remote-tracking branch 'remotes/origin/feature_ac_chunking_clean' into feature/chunk_term_clean
Approved-by: Alex Galarce
* added function for regex based IRS chunking
* shifted regex_match_chunk function to utils
* added helper functions for regex based chunking
* replaced tin_regex function to ac_funcs
* removed prompt for irs_others
* added regex chunking for irs
* removed irs group from smart chunking
* testing functionality for regex based irs fields
* updated return N/A condition
* Merged main into feature_ac_chunking_clean
* file_processing.py edited online with Bitbucket
* added back conditional prompts
* minor fixed for PR
* remove install types
* mering clean branch
* with passed test casses
* added flag for regex based tin execution
Approved-by: Michael McGuinness
Add split_and_write_csv function to utils
* Add split_and_write_csv function to utils
* Merge branch 'main' into feature/outputSplit
Approved-by: Alex Galarce
Alex Galarce