Feature/update testbed metrics
* Clean up filename filter function
* Strip out unused tests
* Strip out page-level comparison
* Add 1:1 fields back to analysis:
* Run through
* Merged main into feature/update-testbed-metrics
* Merged main into feature/update-testbed-metrics
* Merged main into feature/update-testbed-metrics
* Merged main into feature/update-testbed-metrics
* Merged main into feature/update-testbed-metrics
Approved-by: Faizan Mohiuddin
Feature/black and isort
* black and isort constants
* black and isort codes
* black and isort crosswalks
* black and isort investment
* black and isort prompts
* isort testbed
* isort tracking
* black and isort utils
* poetry and black tests
Approved-by: Katon Minhas
Bugfix/dynamic primary issues
* Refactor TRIGGER_CAP special case configuration to utilize only trigger cap questions for two-stage processing
* Refactor get_special_cases to improve handling of N/A responses and merge methodology results with special case results
* Fix field type in get_special_cases for fee schedule breakout configuration
* Create fieldset.py - refactor
* Fix placeholders
* string concatenate new questions
* Update trigger cap questions output format
* Update TRIGGER_CAP_THRESHOLD_AMT prompt for clarity and specificity in reimbursement methodology
* Update constants class
* Update Field class
* Resolve constants passing
* Update field class - change Null criteria
* Add default
* Enhance documentation for get_special_cases function with detailed processing flow and architecture decisions
* Enhance get_special_cases and run_trigger_cap_breakout functions with detailed docstrings for clarity on processing methodologies
* Update constants passing
* Fix questions
* Update fs and mb questions"
* pass constanst correctly for dynamic
* Merge branch 'feature/trigger_cap_breakout' into bugfix/dynamic-primary-issues
* update constants
* only add questions if present
* ensure answer is a valid special_case
* Fix unit tests
* Remove test file
Approved-by: Alex Galarce
Refactor/one time io
* Delete client work
* Streamline imports
* Fix list class
* Update tests
* Clear code funcs test
* Fix unit tests
* Single read code funcs
* Single-read model
* Single-load embeddings
* Successful E2E test
* update unit tests
* Update importsg
* Reload poetry.lock
* remove old exhibit header function
* Remove aarete_derived generic function
* remove align and format tables
* Remove strings to dict
* references
* Clear code
* Move preprocessing_funcs
* remove keywords
* refactor postprocessing_funcs
* Pass unit test - remove qa_qc directory
* Black and isort
* remove print
Approved-by: Alex Galarce
bugfix/special_case_breakout_test_metrics to main
* updated special case breakout fields to testbed metrics
* Merged main into bugfix/special_case_breakout_test_metrics
Approved-by: Alex Galarce
Fix data quality issues in testbed preprocessing and enhance error reporting
* Fix data quality issues in testbed preprocessing and enhance error reporting
* Merge remote-tracking branch 'origin/main' into bugfix/fix_nonunique_one_to_ones_and_strip_columns
Approved-by: Siddhant Medar
Feature/prov group tin
* don't prompt for INTRO_PAGE
* test complete
* test 2
* Update test script
* Merge branch 'main' into feature/prov-group-tin
* Merged main into feature/prov-group-tin
* refactor
* cleaning
* Update save
* Fix return type
* Update for reimbursement_tin_npi
* Comment out reimbursement-tin-npi
* Pass unit tests
* Merge branch 'main' into feature/prov-group-tin
* Merged main into feature/prov-group-tin
* Remove length validation
* Merge branch 'main' into feature/prov-group-tin
* Remove ON_INTRO from prompt
* Restore
* Remove test
* Move reimb prov info to dynamic reimb info
* e2e test
* Fix for unit test
* remove prints
* update docstrings and typehints
Approved-by: Alex Galarce
Bugfix/fee rate conversion rate
* printing incorrect recall values
* values responsible for low recall added in spreadsheet
* changing match list to unmatch list for breakouts
* recall values added for 1-to-N
* print statements removed
* removed print statements
* Merged main into feature/test_files
* output formatting fix
* fixed cases where conversion factor was shown as fee rate
* Merged main into bugfix/fee_rate_conversion_rate
* Merged main into bugfix/fee_rate_conversion_rate
* unit of measure fixed
* Merge branch 'bugfix/fee_rate_conversion_rate' of https://bitbucket.org/aarete/doczy.ai into bugfix/fee_rate_conversion_rate
* Merged main into bugfix/fee_rate_conversion_rate
* functionality to create and compare multiple rows for AARETE_DERIVED_LOB added
* Merge branch 'bugfix/fee_rate_conversion_rate' of https://bitbucket.org/aarete/doczy.ai into bugfix/fee_rate_conversion_rate
* Merged main into bugfix/fee_rate_conversion_rate
Approved-by: Katon Minhas
Feature/test files
* printing incorrect recall values
* values responsible for low recall added in spreadsheet
* changing match list to unmatch list for breakouts
* recall values added for 1-to-N
* print statements removed
* removed print statements
* Merged main into feature/test_files
* output formatting fix
* Merged main into feature/test_files
Approved-by: Katon Minhas
Feature/fuzzy match no numbers
* Add fuzzy matching function for string comparison and enhance term extraction result evaluation
* Enhance term extraction results evaluation by adding counts and readable strings for matched, false negatives, and false positives
* Adjust text similarity threshold to 65 and improve readability of comparison results in term extraction evaluation
* Merged main into feature/fuzzy-match-no-numbers
Approved-by: Katon Minhas
Refactor methodology breakout prompt, small testbed cleanup
* Refactor REIMB_TERM_BREAKOUT prompt. Include service AND methodology. Change to sections formatting.
* rename some columns in `Reimbursement Primary` tab of test output
Approved-by: Katon Minhas
Methodology Primary - New Prompt and Test Bed Utilities
* Add IDENTIFY_REIMBURSEMENT_EXHIBITS function to analyze exhibit headers for reimbursement information
* Add Zone.Identifier to .gitignore to prevent Windows zone information files from being tracked
* Add functionality to identify reimbursement exhibits
* Add functionality to identify reimbursement exhibits in run_one_to_n_prompts
* Output YAML parsing
* Refactor identify_reimbursement_exhibits to improve YAML parsing and error handling
* New reimbursement exhibit logic
* Update IDENTIFY_REIMBURSEMENT_EXHIBITS_PROMPT to return PAGE NUMBERS instead of exhibit headers
* Fix identify_reimbursement_exhibits to return page numbers as strings after YAML parsing
* Remove debugging print statement for reimbursement exhibits in run_one_to_n_prompts
* Remove commented debugging print statements in identify_reimbursement_exhibits
* Increase read_timeout in EC2 configuration from 2000 to 4000
* Merge remote-tracking branch 'origin/main' into feature/reimb-primary-improvements
* Remove debugging print statements in get_reimbursement_primary function
* mypy error
* Fix return type in identify_reimbursement_exhibits to ensure keys are returned as a list
* Merge remote-tracking branch 'origin/main' into feature/reimb-primary-improvements
* Refactor REIMBURSEMENT_LEVEL_PRIMARY prompt
* Merge remote-tracking branch 'origin/main' into feature/explore-methodology-primary
* Rename and document row comparison function
* Merge remote-tracking branch 'origin/main' into feature/explore-methodology-primary
* Add reimbursement primary analysis and comparison function to testbed
* Enhance reimbursement exhibit identification prompt with explicit and implicit indicators
* Rename ground truth count and pairs in term extraction results for clarity
* shorten methdology_primary_row_comparison notebook
Approved-by: Katon Minhas
Update testbed data sources and enhance metric calculations for blank fields
* Update testbed data sources and enhance metric calculations for blank fields
* Add order-independent provider analysis
* add tin-npi normalization and verbose
* Enhance provider field evaluation by excluding "UNKNOWN" and "NO_IDENTIFIERS_FOUND" values
* List-based method
* Enhance provider field evaluation by incorporating frequency-based metrics and excluding "UNKNOWN" values
* Fix splits
* fix headers
* fix comma splitting
* Merged main into tin-npi-acc-testing
* Merged main into tin-npi-acc-testing
* Refine provider metrics logging to exclude empty predicted values
* Merge branch 'tin-npi-acc-testing' of https://bitbucket.org/aarete/doczy.ai into tin-npi-acc-testing
* Merged main into tin-npi-acc-testing
* last commit before changing back to sets
* Refactor evaluation metrics calculation to use set operations (while still displaying counters for info)
* Merge branch 'tin-npi-acc-testing' of https://bitbucket.org/aarete/doczy.ai into tin-npi-acc-testing
Approved-by: Katon Minhas
Add function to group provider information into JSON array format post-hoc from the test bed
* Add function to group provider information into JSON array format
* Add example notebook
Approved-by: Katon Minhas
Feature/update testbed script
* Update preprocessing
* Update preprocessing
* remove quit
* Update date for filename
* Merged main into feature/update-testbed-script
Approved-by: Alex Galarce
Mayank Aamseek