DRAFT: Feature/split out error log
* Refactor result handling to separate successful and error results in processing
* Add forced exception for testing in safe_process_file function
* Merged main into feature/split-out-error-log
* main.py edited online with Bitbucket
* Add support for error output type in write_s3 function
Approved-by: Katon Minhas
Hotfix/file errors
* Set up test file
* Fix regex pattern in json.loads
* bug fix No valid JSON object found in the input string
* bug fixes
* moved try statement to parse fn
* invalid literal for int() with base 10 error fixed
* remove test
* remove excess print statements
* Raise error
* Remove todo
* Remove print statement
Approved-by: Katon Minhas
Bugfix/table failed files
* improved align and format table prompt
* Successful E2E test on San Joaquin - without running align and format table prompt
* San Joaquin e2e test passed
* Merged main into bugfix/table-error-fix
* Merge branch 'main' into bugfix/table-error-fix
* Add same number of columns condition
* Merged main into bugfix/table-error-fix
* Clean investment_values
* Remove import
* docstrings
* Remove test doc
* Add back original ALIGN_AND_FORMAT_TABLES prompt for client work
* Remove import
* Add config reference
* expected string or bytes-like object, got 'int'
* Merged main into bugfix/table-error-fix
* Merge branch 'bugfix/table-error-fix' into bugfix/table-failed-files
* Update universal json load
* update test
* Merge branch 'main' into bugfix/table-failed-files
* Position 0 error fixed
* update table_utils
* remove test.py
* revert table_utils test
* remove table_utils_test
Approved-by: Alex Galarce
Bugfix/table error fix
* handle multipage table data correctly
* same metadata for all tables on the page
* Multiple tables on page, non-continuous, split
* Add back align_and_format_tables
* update unit tests
* Merged main into bugfix/table-error-fix
* improved align and format table prompt
* Successful E2E test on San Joaquin - without running align and format table prompt
* San Joaquin e2e test passed
* Merged main into bugfix/table-error-fix
* Merge branch 'main' into bugfix/table-error-fix
* Add same number of columns condition
* Merged main into bugfix/table-error-fix
* Clean investment_values
* Remove import
* docstrings
* Remove test doc
* Add back original ALIGN_AND_FORMAT_TABLES prompt for client work
* Remove import
* Add config reference
* Merged main into bugfix/table-error-fix
* Move table_utils to investment.table_funcs
* restore table_utils.py
* Update tests
* update preprocess
Approved-by: Alex Galarce
Generate unique IDs for reimbursements
* Add reimbursement ID generation and update function signatures for unique identification
* Merge remote-tracking branch 'origin/main' into feature/unique-identifier-for-reimbursements
Approved-by: Katon Minhas
New Reimbursement Primary Prompt
* Enhance reimbursement primary extraction prompts with detailed formatting and instruction guidelines
* Clarify 'lesser of' statement application in reimbursement methods with detailed examples and guidelines
* Add examples to 'lesser of' statement application in investment prompts
* up row_limit to 20
* hybrid prompt for global table info for reimb_primary
* Merged main into feature/new-reimb-primary-prompt
* most explicit prompt
* update default row_limit in clean_tables function to 6
* Merge remote-tracking branch 'origin/main' into feature/new-reimb-primary-prompt
* Add detailed guidelines for extracting complex payment structures in anesthesia services
* Clarify handling of specific plan names in reimbursement prompts
* removed a couple examples to save tokens
Approved-by: Katon Minhas
Refactor fuzzy matching logic to use partial token sort ratio and update matched pairs structure
* Refactor fuzzy matching logic to use partial token sort ratio and update matched pairs structure
* Merged main into feature/change-1-to-n-comparison-metric
Approved-by: Katon Minhas
Small fix to comparisons (lowercase comparison string fields)
* Add accuracy calculation to fuzzy matching evaluation
* reimbursement primary accuracy test
* add match_threshold control
* clarify metrics
* Merged main into reimb-primary-accuracy-analysis
* Merge branch 'reimb-primary-accuracy-analysis' of https://bitbucket.org/aarete/doczy.ai into reimb-primary-accuracy-analysis
* add 'billed charges' to reimbursement keywords and update TODO for page checking
* add comparisons dictionary
* normalize field comparison by converting strings to lowercase
* deduplicate data sources in notebook
* Merge remote-tracking branch 'origin/main' into feature/add-billed-charges-to-reimb-keywords
Approved-by: Katon Minhas
Feature/daip2 125
* grouper prompt changed
* grouper code, type and desc updated
* removed test file
* Merged main into feature/daip2-125
* Merge remote-tracking branch 'origin/main' into feature/daip2-125
Approved-by: Alex Galarce
very small error logging fix
* Handle ValueError in code range parsing and provide fallback
* Implement safe file processing with error handling and concurrent execution
* Merge remote-tracking branch 'origin/main' into bugfix/improve-stability
* more explicit error printing
* Merge remote-tracking branch 'origin/main' into bugfix/improve-stability
Approved-by: Katon Minhas
updated prompt to get the group provider
* updated prompt to get the group provider
* Merged main into bugfix/individual-prov-group-name
Approved-by: Alex Galarce
Bugfix/improve stability
* Handle ValueError in code range parsing and provide fallback
* Implement safe file processing with error handling and concurrent execution
* Merge remote-tracking branch 'origin/main' into bugfix/improve-stability
Approved-by: Katon Minhas
Feature/field renaming
* Update column names
* Update field names
* Update field names
* Merge branch 'main' into feature/field-renaming
* _DATE to _DT in field names
* _CD_DESC to _DESC
* NOT_TO_EXCEED_IND added as valid field
* Remove print statements
Approved-by: Alex Galarce
Fix load_embeddings()
* move datetime_str function to string_utils module
* enhance load_embeddings to create directory structure and handle download errors
* Merged main into feature/reimbursement-primary-optimizization
Approved-by: Katon Minhas
Feature/test 1 to n fields
* Add function to evaluate one-to-n field comparisons and calculate metrics
* Add fuzzy matching for one-to-n field comparisons and enhance evaluation metrics
* Refactor similarity calculation to use token sort ratio for improved field comparison
* Add matched labels to evaluation metrics in one-to-n fuzzy matching
Approved-by: Katon Minhas
Feature/override aa derived eff date
* Implement special handling for AARETE_DERIVED_TERMINATION_DATE based on CONTRACT_AUTO_RENEWAL_IND
* Refine prompts for CONTRACT_AUTO_RENEWAL_IND and CONTRACT_AUTO_RENEWAL_TERM for clarity
* move special handling for derived effective date to postprocess step
* Remove unnecessary blank line in get_crosswalk_and_derived function
Approved-by: Katon Minhas
Feature/standardize column order
* add column reordering
* Standardize column order
* Merged main into feature/standardize-column-order
Approved-by: Alex Galarce
Change from AARETE_DERIVED_ to CONTRACT_- FEE_SCHEDULE_VERSION
* Change from AARETE_DERIVED_ to CONTRACT_- FEE_SCHEDULE_VERSION
Approved-by: Alex Galarce
Hotfix/code description handling
* Update date formatting in prompts to YYYY/MM/DD
* fix tests for new date formatting
* Enhance code parsing to support comma-separated values in get_code_description function
* Merged main into hotfix/code-description-handling
Approved-by: Katon Minhas
Field/reconfigure prov type
* Run One-To-N before One-To-One
* Switch PROV_TYPE to Exhibit-Level
* Modify PROV_TYPE prompt
* Merge branch 'main' into field/reconfigure-prov-type
Approved-by: Alex Galarce
Update date formatting in prompts to YYYY/MM/DD
* Update date formatting in prompts to YYYY/MM/DD
* fix tests for new date formatting
Approved-by: Katon Minhas
StopLoss, Outliers, and Sequestration
* changed any method to all method in list handling
* code mapping moved outside to run once
* Merge branch 'feature/daip2-97' into bugfix/proc-code-lists
* merge conflict fixed
* merge conflicts fixed
* incorporated latest changes from main
* Merged main into bugfix/proc-code-lists
* Move load all dataset to io_utils
* Fix list is_empty
* Updated poetry.lock
* Bugfix
* Docstring
* Docstrings
* Merge branch 'main' into bugfix/proc-code-lists
* Merge branch 'bugfix/proc-code-lists' into bugfix/dynamic-assignment-issue
* Add outliers, stop-loss, and sequestration to reimbursement-level primary prompt template
* Add stop loss fields
* Add outlier fields
* Add sequestration fields
* Remove test
* Docstrings
* Update test for no default exhibit_level_answer_dict
* Merge branch 'bugfix/dynamic-assignment-issue' into field/stop-loss
* Merge branch 'main' into field/stop-loss
* Merged main into field/stop-loss
Approved-by: Alex Galarce
Bugfix/dynamic assignment issue
* Merged main into feature/proc-desc
* Uncomment test code
* Bugfix
* script optimisation for codes
* modified is empty function in string utils
* Remove test.py
* bug fix to handle list of proc_codes
* Merged main into bugfix/proc-code-lists
* changed any method to all method in list handling
* code mapping moved outside to run once
* Merge branch 'feature/daip2-97' into bugfix/proc-code-lists
* merge conflict fixed
* merge conflicts fixed
* incorporated latest changes from main
* Merged main into bugfix/proc-code-lists
* Move load all dataset to io_utils
* Fix list is_empty
* Updated poetry.lock
* Bugfix
* Docstring
* Docstrings
* Merge branch 'main' into bugfix/proc-code-lists
* Merge branch 'bugfix/proc-code-lists' into bugfix/dynamic-assignment-issue
* Update test for no default exhibit_level_answer_dict
* Merged main into bugfix/dynamic-assignment-issue
Approved-by: Alex Galarce
Bugfix/proc code lists
* Add admit type code
* add proc mod description
* Add bill type pos mapping
* Merged main into feature/proc-desc
* Uncomment test code
* Bugfix
* script optimisation for codes
* modified is empty function in string utils
* Remove test.py
* bug fix to handle list of proc_codes
* Merged main into bugfix/proc-code-lists
* changed any method to all method in list handling
* code mapping moved outside to run once
* Merge branch 'feature/daip2-97' into bugfix/proc-code-lists
* merge conflict fixed
* merge conflicts fixed
* incorporated latest changes from main
* Merged main into bugfix/proc-code-lists
* Move load all dataset to io_utils
* Fix list is_empty
* Updated poetry.lock
* Docstring
* Docstrings
* Merge branch 'main' into bugfix/proc-code-lists
* bug fix
Approved-by: Mayank Aamseek
Approved-by: Alex Galarce
Field/initialize prov type
* Code map bug fixes
* docstrings
* Initialize Rev and Diag descriptions
* Add 0 to front for rev codes
* Comments
* Merged main into feature/proc-desc
* Comments
* Separate grouper and diag codes
* add grouper version
* Bugfix"
* Add admit type code
* add proc mod description
* Add bill type pos mapping
* Merged main into feature/proc-desc
* Uncomment test code
* Bugfix
* Remove test.py
* Service Breakout prompt template
* list bugfix
* Merged main into feature/proc-desc
* concurrency
* Merge branch 'feature/proc-desc' into field/initialize-prov-type
* Added service_breakout function - only contains aarete prov-type rn
* Merge branch 'main' into field/initialize-prov-type
* Docstring
Approved-by: Alex Galarce
Client runs dashboard updates
* new dashboard code
* dashboard updates
* Merged main into client-runs-dashboard-updates
* Merged main into client-runs-dashboard-updates
Approved-by: Alex Galarce
Feature/test bed facilitation for 1-1 fields
* start comparing output to label data
* Enhance load_test_files.py to merge and reorder labels and predictions, outputting to CSV
* Refactor load_test_files.py to streamline label and prediction loading, sorting, and CSV output for date and TIN/NPI fields
* rename to one_to_one_comparison
* Merged main into feature/test-bed-facilitation
* rename notebook_config
* Merge branch 'feature/test-bed-facilitation' of https://bitbucket.org/aarete/doczy.ai into feature/test-bed-facilitation
* Merged main into feature/test-bed-facilitation
Approved-by: Katon Minhas
Document dates optimization
* simplified file processing for testing
* modified derived_term_date_prompt
* fixed day values with spaces or leading zeros
* modified contract termination date prompt
* updated termination date prompt
* reranking of chunks for better context
* exclude last date while calculating term date
* clean up
* clean up 2
* Merge branch 'main' into document_dates_optimization
pulled latest changes from main
* Merged main into document_dates_optimization
* fix unit tests
* Merged main into document_dates_optimization
* refactor: update termination date prompt for clarity and accuracy
* Merged main into document_dates_optimization
* fix pipeline
* Merge remote-tracking branch 'origin/main' into document_dates_optimization
* Merge branch 'document_dates_optimization' of https://bitbucket.org/aarete/doczy.ai into document_dates_optimization
Approved-by: Katon Minhas
Feature/daip2 95
* testing carveout
* Merge branch 'main' into feature/daip2-95
pulled latest changes from main
* CONTRACT_DEFAULT_IND logic changed
* Merge branch 'main' into feature/daip2-95
pulled latest changes from main
* prompt updated to remove mention of last carveout
* Merged main into feature/daip2-95
* Merged main into feature/daip2-95
* prompt updated
* Merge branch 'feature/daip2-95' of https://bitbucket.org/aarete/doczy.ai into feature/daip2-95
latest changes pulled
Approved-by: Katon Minhas
Feature/daip2 101
* missing group tin fixed
* use tin prompt in full context when N/A
* fixed N/A TIN and applied same to NPI
* null tin fixed
* Merged main into feature/daip2-101
* if PROV_GROUP_TIN is still N/A, pick first value from tin_other
* Merge branch 'feature/daip2-101' of https://bitbucket.org/aarete/doczy.ai into feature/daip2-101
pulled latest changes from main
Approved-by: Katon Minhas
Feature/new prompts for methodology
* Added reimb. effective date and termination date
* Added prompt for identifying 'not to exceed' limitations in investment prompts
* Refactor field names for reimbursement effective and termination dates in investment prompts
* Clarified documentation for handling multiple reimbursement methods in investment prompts
Approved-by: Katon Minhas
Alex Galarce