Feature/black and isort
* black and isort constants
* black and isort codes
* black and isort crosswalks
* black and isort investment
* black and isort prompts
* isort testbed
* isort tracking
* black and isort utils
* poetry and black tests
Approved-by: Katon Minhas
io_utils documentation
* add docstring to read_local function and handle file read errors gracefully by returning None on failure
* update read_local function to support .xlsx files and improve error handling
* add docstring to read_s3 function to clarify its purpose and return value
* refactor read_s3 and read_s3_csv functions to improve error handling and add type hints
* improve error handling in read_s3_csv function for CSV file parsing
* enhance read_input function with error handling and detailed docstring
* Add TODOs for read_input_csv and filter_already_processed functions; add type hints and docstring for remove_txt_extension
* enhance write_local function documentation; make docstrings more consistent when there is a "notes" section
* enhance write_local and write_s3 functions; improve documentation, add error output type handling, and ensure consistent CSV writing options
* refactor test_io_utils.py; reorganize imports and enhance read_input_local test with additional mock for file existence
Approved-by: Katon Minhas
Refactor/one to n
* remove print
* Update logging for fsbreakout and grouper breakout
* Additional type hints
* updated special case primary
* Remove deprecated special case check
* OUTLIER_TERMS-->OUTLIER_TERM standardization
* Applied suggestion to fieldExtraction/src/prompts/investment_prompts.json
* update unit test
* Update Carveout IND and Default IND logic
* Split out exhibit-level lesser
* Bugfix
* update lesser of exhibit prompt
* Bugfix
* exhibit lesser of varible name bugfix
* Update exhibit lesser of passing
* Update Exhibit-level prompt template
* Shorten full_context prompt length if the questions are long
* Special case assignment index
* Resolve unit tests
* updated UC IND prompt
* Remove print
* update one-to-n unit test
* added dot based date example
* updated date fix prompt
* Merged in bugfix/json_error_and_unit_of_measure (pull request #708)
Bugfix/json error and unit of measure
* json parser fixes and prompt changes
* universal json load fixed
* json error fixed
* json error fixed 2
* Merged refactor/one-to-n into bugfix/json_error_and_unit_of_measure
Approved-by: Katon Minhas
Approved-by: Alex Galarce
Bugfix/trigger cap fixes
* stoploss prompt added again
* stoploss prompt added again
* stoploss prompt updated
* reverted back to one stage for trigger cap cases
* Merge branch 'main' into bugfix/trigger_cap_fixes
* pipeline error fixed
* incorrect discount special case fixed
Approved-by: Alex Galarce
Feature/tin enhancements
* Add OCR-correctable TIN patterns and substitution mapping
* Merge remote-tracking branch 'origin/main' into feature/tin-enhancements
* Refactor OCR-correctable TIN pattern and enhance documentation for clarity
* Enhance get_all_matches function with type hints and improve documentation; add get_all_matches_with_ocr for OCR-correctable matches
* Add attempt_ocr_correction function for TIN OCR error handling and validation
* Merge remote-tracking branch 'origin/main' into feature/tin-enhancements
* Integrate OCR correction for TIN extraction in run_provider_info_fields; log corrections for monitoring
* Refactor OCR-correctable patterns in regex_patterns.py; add NPI_OCR_PATTERN and enhance documentation for clarity
* Enhance OCR correction in attempt_ocr_correction to support both TIN and NPI; update validation logic for identifier lengths
* Integrate NPI OCR correction in run_provider_info_fields; log corrections for monitoring
* Enhance logging in run_provider_info_fields to summarize TIN and NPI extraction quality; include counts of exact and OCR-corrected identifiers
* Add unit and integration tests for OCR correction
* isort
* Merge remote-tracking branch 'origin/main' into feature/tin-enhancements
* Merged main into feature/tin-enhancements
Approved-by: Katon Minhas
pyproject requirements, prep for logging fixes
* remove chroma
* update project name to "doczy-field-extraction" in pyproject.toml
* update lock file
* replace print statements with logging for better traceability in process_file function
* replace print statements with logging for improved error tracking and information logging
* replace print statements with logging for improved error handling and traceability
* replace print statements with logging for improved error handling in llm_utils and string_utils
* remove unused table_utils module; supplanted by table_funcs and unreferenced
* replace capsys with caplog in test_invalid_input_type for improved logging of invalid input
Approved-by: Katon Minhas
Bugfix/too many files
* Implement caching for field data loading to optimize performance and prevent "too many open files" errors
* Documentation
* isort
* fix breaking test to clear cache and ensure mock data is actually being used
* Merge remote-tracking branch 'origin/main' into bugfix/too-many-files
* Implement thread-safe vectorstore caching and cleanup in field context extraction
* Refactor imports in smart_chunking_funcs.py for clarity and organization
* Update module docstring in smart_chunking_funcs.py
* Fix mypy errors
* Fix variable naming and error handling in run_smart_chunked_fields function
* Reorganize imports and enhance module documentation for clarity
* Add logging for error handling in field_context function
Approved-by: Katon Minhas
Remove cost logging functionality and related tests
* Remove cost logging functionality and related tests
* Remove cost log entry test and related CSV writer mock
* Merged main into feature/remove-costlog
Approved-by: Katon Minhas
Bugfix/uom
* Add "Per Procedure" to valid unit of measure list
* Enhance unit of measure prompt for clarity and specificity in reimbursement methodologies
* Refine unit of measure prompt to ensure defaulting to 'Per Unit' regardless of procedure codes for flat rate methodologies
* Merge remote-tracking branch 'origin/main' into bugfix/uom
* Merged main into bugfix/uom
* Enhance logic for setting UNIT_OF_MEASURE to 'Per Unit' for Flat Rate reimbursement, handling all edge cases including NaN and empty strings.
* Enhance is_empty function to handle additional empty value cases for pd.Series and strings
* Enhance test cases for is_empty function to cover additional whitespace and null-like values
* Merge branch 'bugfix/uom' of https://bitbucket.org/aarete/doczy.ai into bugfix/uom
* Enhance type checking for is_empty function with overloads for str, list, and pd.Series
* Overload fix
* Merged main into bugfix/uom
Approved-by: Katon Minhas
Bugfix/dynamic primary issues
* Update get prompt for direct valid value setting
* prep for merge
* Merge branch 'main' into bugfix/dynamic-primary-issues
* Remove test
* Update unit tests with real constants object
* Ready for review
* direct replace {valid_values}
Approved-by: Alex Galarce
Bugfix/carveout cd
* Update field class - change Null criteria
* Add default
* Enhance documentation for get_special_cases function with detailed processing flow and architecture decisions
* Enhance get_special_cases and run_trigger_cap_breakout functions with detailed docstrings for clarity on processing methodologies
* Update constants passing
* Fix questions
* Update fs and mb questions"
* pass constanst correctly for dynamic
* Merge branch 'feature/trigger_cap_breakout' into bugfix/dynamic-primary-issues
* update constants
* only add questions if present
* ensure answer is a valid special_case
* Set carveout indicator for two-stage process; prepped set_carveout_indicators and run_methodology_breakout_with_carveout_indicators to replace process_carveout_or_special_case
* Fix unit tests
* Remove test file
* Refactor carveout handling: streamline indicator setting and enhance methodology breakout integration
* Refactor get_carveout_and_special_fields: ensure methodology breakout runs regardless of carveout/special case configs; streamline handling of indicators and questions
* Remove deprecated process_carveout_or_special_case function
* Merge remote-tracking branch 'origin/bugfix/dynamic-primary-issues' into bugfix/carveout-cd
* Add tests for carveout indicator setting and special cases processing
* Add tests for unrecognized carveout handling and legacy stop-loss processing
* Refactor special case configurations: streamline STOP_LOSS handling and update test cases for methodology prompt templates
* Refactor special case handling: improve argument formatting in get_special_cases and run_methodology_breakout_with_carveout_indicators functions
* Add test for run_methodology_breakout_with_carveout_indicators function
* Merge remote-tracking branch 'origin/main' into bugfix/carveout-cd
Approved-by: Katon Minhas
Bugfix/dynamic primary issues
* Refactor TRIGGER_CAP special case configuration to utilize only trigger cap questions for two-stage processing
* Refactor get_special_cases to improve handling of N/A responses and merge methodology results with special case results
* Fix field type in get_special_cases for fee schedule breakout configuration
* Create fieldset.py - refactor
* Fix placeholders
* string concatenate new questions
* Update trigger cap questions output format
* Update TRIGGER_CAP_THRESHOLD_AMT prompt for clarity and specificity in reimbursement methodology
* Update constants class
* Update Field class
* Resolve constants passing
* Update field class - change Null criteria
* Add default
* Enhance documentation for get_special_cases function with detailed processing flow and architecture decisions
* Enhance get_special_cases and run_trigger_cap_breakout functions with detailed docstrings for clarity on processing methodologies
* Update constants passing
* Fix questions
* Update fs and mb questions"
* pass constanst correctly for dynamic
* Merge branch 'feature/trigger_cap_breakout' into bugfix/dynamic-primary-issues
* update constants
* only add questions if present
* ensure answer is a valid special_case
* Fix unit tests
* Remove test file
Approved-by: Alex Galarce
Refactor/one time io
* Delete client work
* Streamline imports
* Fix list class
* Update tests
* Clear code funcs test
* Fix unit tests
* Single read code funcs
* Single-read model
* Single-load embeddings
* Successful E2E test
* update unit tests
* Update importsg
* Reload poetry.lock
* remove old exhibit header function
* Remove aarete_derived generic function
* remove align and format tables
* Remove strings to dict
* references
* Clear code
* Move preprocessing_funcs
* remove keywords
* refactor postprocessing_funcs
* Pass unit test - remove qa_qc directory
* Black and isort
* remove print
Approved-by: Alex Galarce
Implement deduplication of provider columns in DataFrame and add corresponding unit tests
* Implement deduplication of provider columns in DataFrame and add corresponding unit tests
Approved-by: Katon Minhas
Table Continuation Fix
* Add EXHIBIT_HEADER_MARKERS for improved document header recognition in table handling
* Refactor EXHIBIT_HEADER_MARKERS definition for improved clarity and organization
* remove unused page span tracking code; add unit tests for new functionality
* remove debugging logs from combine_continuous_tables function
* Refactor logging setup and update test file configuration in investment field testing notebook
* Merge remote-tracking branch 'origin/main' into feature/further-phase-1-and-2-fixes
Approved-by: Katon Minhas
Exhibit-level Lesser Of
* Enhance exception language clarity in investment prompts
* Roll back exception language
* initial try at exhibit_lesser_of
* simplify `create_exhibit_lesser_of_row`
* Refine prompt for extracting 'lesser of' and 'not to exceed' constraints in exhibit
* Remove unused `get_service_breakout` function; it was unreferenced
* run exhibit-level answers FIRST, apply exhibit lesser-of WITHIN reimbursement_level
* Refine prompt for extracting 'lesser of' and 'not to exceed' constraints to clarify expected output and handling of multiple constraints.
* Merge remote-tracking branch 'origin/main' into bugfix/samaritan-missing-rows
* add exhibit_lesser_of="N/A" to tests
* Add exhibit_lesser_of parameter to reimbursement_level calls in tests
* restore dynamic answer retrieval for primary, reimbursement info, and code fields
* Merge remote-tracking branch 'origin/main' into bugfix/samaritan-missing-rows
* "fill in the blank" prompt attempt
* Merge remote-tracking branch 'origin/main' into bugfix/samaritan-missing-rows
* Merge remote-tracking branch 'origin/main' into feature/sonnet-4
* Merge remote-tracking branch 'origin/feature/sonnet-4' into bugfix/samaritan-missing-rows
* Merge remote-tracking branch 'origin/main' into bugfix/samaritan-missing-rows
* Remove carveout patterns from has_explicit_lesser_of_carveout function
* Add deduplicate_methodology_breakout function to remove duplicate reimbursement methodologies
* Add deduplication of methodology breakout rows in postprocess function
* Enhance create_exhibit_lesser_of_row to avoid redundancy with substring checks
* Change log level from info to debug for exhibit constraint check in create_exhibit_lesser_of_row
* Remove deduplication of methodology breakout rows from postprocess function (we handle it upstream)
* Merged main into bugfix/samaritan-missing-rows
Approved-by: Katon Minhas
Bugfix/mcs issue fixes
* debugging and initial prompt change
* Remove redundant reimb info
* Merged main into bugfix/mcs-issue-fixes
* Prep for merge
* update unit tests
Approved-by: Alex Galarce
Feature/provider state
* Add prompt for extracting provider state information in investment prompts
* Add state normalization function and integrate into context field extraction
* black format string_utils.py; add tests for normalize_state_to_abbreviation function
* black format string_utils_test.py
* Add PROVIDER_STATE to investment column order
* Merged main into feature/provider-state
Approved-by: Katon Minhas
Bugfix/mcs issue fixes
* Update code primary prompt
* Merge branch 'main' into bugfix/mcs-issue-fixes
* No longer treat non-reimbursement tables as 'tables' for the purpose of preprocessing
* No longer extract Reimb Effective Date from footer of the page
* Merge branch 'main' into bugfix/mcs-issue-fixes
* Update unit tests
* Add postprocessing step to remove Reimb Date values when the 1:1 date values are identical
* Update unit tests
* prep for merge
* Remove excess prints
* Merged main into bugfix/mcs-issue-fixes
* Merge branch 'main' into bugfix/mcs-issue-fixes
* Prep for merge
* remove print
Approved-by: Alex Galarce
Bugfix/fix overzealous duplication
* Refactor duplicate processing logic to scope seen pairs by base page, enhancing clarity and accuracy in deduplication.
* Enhance deduplication tests to include base page identifiers in seen pairs
* Merged main into bugfix/fix-overzealous-duplication
Approved-by: Katon Minhas
Feature/split reimbursements
* Update reimbursement test cases to reflect accurate reimbursement terms and improve deduplication logic
* move VALIDATE_REIMBURSEMENTS_PROMPT to investment_prompts.py
* black, isort formatting
* Remove IDENTIFY_REIMBURSEMENT_EXHIBITS_PROMPT function (unused)
* Add MODEL_ID_CLAUDE4_SONNET for new model integration (doesn't work right now)
* pipe delimited-output and parsing
* Merge remote-tracking branch 'origin/main' into feature/split-and-filter-reimbursements
* Merge remote-tracking branch 'origin/main' into feature/split-reimbursements
* Add functionality to split compound reimbursement terms into distinct methodologies
* splitting logic for reimbursement terms
* put in split compound reimbursement step
* Fix JSON output formatting in reimbursement terms example
* reimbursement splitting: move prompt to investment_prompts.py, move keywords to investment_values.py
* Merge remote-tracking branch 'origin/main' into feature/split-reimbursements
* Merge remote-tracking branch 'origin/main' into feature/split-reimbursements
* Change order (split -> filter over filter -> split), improve logging
* Refactor reimbursement filtering logic to use a two-stage approach with pattern-based negative filtering and LLM validation
* Replace clear reimbursement indicators with obvious non-reimbursement indicators for improved clarity in reimbursement filtering
* Simplify reimbursement filtering by removing pattern-based checks and relying solely on LLM validation
* Remove obvious non-reimbursement indicators from reimbursement filtering keywords
* Enhance reimbursement level test by adding LLM validation and updating mock return values for service terms
* Merge remote-tracking branch 'origin/main' into feature/split-reimbursements
* Fix failing test
* Merge remote-tracking branch 'origin/main' into feature/split-reimbursements
* update mock return values to fix failed tests
Approved-by: Siddhant Medar
Feature/filter reimbursements
* First-pass implementation for reimbursement filtering
* Enhance reimbursement level processing by filtering out services without reimbursement terms and logging when no valid pairs are found.
* Update mock return values in reimbursement level tests to reflect new reimbursement terms
* Merge remote-tracking branch 'origin/main' into feature/split-and-filter-reimbursements
* Refine reimbursement filtering by adding 'billed' to indicators and ensuring REIMB_TERM is a string before processing.
* Refactor is_empty function signature to support multiple input types and clarify return values
* adding LLM review for ambiguous reimbursement methodology cases
* more stringent keywords for stage 1 of reimbursement filtering
* refining rate and cost patterns for reimbursement filtering
* Enhance LLM response handling in reimbursement methodology check to improve accuracy and logging for ambiguous cases.
* Merge remote-tracking branch 'origin/main' into feature/split-and-filter-reimbursements
* Update reimbursement test cases to reflect accurate reimbursement terms and improve deduplication logic
* move VALIDATE_REIMBURSEMENTS_PROMPT to investment_prompts.py
* black, isort formatting
* Remove IDENTIFY_REIMBURSEMENT_EXHIBITS_PROMPT function (unused)
* Add MODEL_ID_CLAUDE4_SONNET for new model integration (doesn't work right now)
* pipe delimited-output and parsing
* Merge remote-tracking branch 'origin/main' into feature/split-and-filter-reimbursements
* Refactor reimbursement filtering by moving clear indicators to constants
Approved-by: Katon Minhas
Feature/prov group tin
* don't prompt for INTRO_PAGE
* test complete
* test 2
* Update test script
* Merge branch 'main' into feature/prov-group-tin
* Merged main into feature/prov-group-tin
* refactor
* cleaning
* Update save
* Fix return type
* Update for reimbursement_tin_npi
* Comment out reimbursement-tin-npi
* Pass unit tests
* Merge branch 'main' into feature/prov-group-tin
* Merged main into feature/prov-group-tin
* Remove length validation
* Merge branch 'main' into feature/prov-group-tin
* Remove ON_INTRO from prompt
* Restore
* Remove test
* Move reimb prov info to dynamic reimb info
* e2e test
* Fix for unit test
* remove prints
* update docstrings and typehints
Approved-by: Alex Galarce
Feature/rework table handling
* test: add cases for configuration boundary
* test: add table post-processing tests (cases to ensure preservation of post-table text and metadata during table operations)
* test: enhance table handling tests for small and large tables, ensuring proper suffixing and splitting behavior
* Fix and clarify intent in comments
* fix: enhance handling of post-table text when combining tables
* refactor: remove commented-out code in clean_tables function for clarity
* test: update post-table text preservation assertions and add debug tracing
* refactor: remove unused remove_table_from_page function and its test
* refactor: enhance combine method documentation and clarify split_by_size_with_smart_tables logic
* Merge remote-tracking branch 'origin/main' into feature/rework-table-handling
* refactor: reduce large table threshold from 1000 to 50 rows for better table splitting
* Add simple splitting
* refactor: uncomment dynamic answer retrieval in run_one_to_n_prompts for improved functionality
* Merge remote-tracking branch 'origin/main' into feature/rework-table-handling
* refactor: remove unused table handling functions and related constants for cleaner code
* refactor: remove unused test functions in preparation for new tests
* refactor: streamline continuation table handling by utilizing existing flags
* refactor: update table combination logic to include all tables from the main page
* refactor: enhance table handling to automatically resolve column mismatches, respect row limits, and improve unit/integration test coverage
* rename clean_tables_simple to clean_tables
* Update docstrings
* refactor: rename table handling functions
* Fix indentation and add tests
* isort, black
* Merge remote-tracking branch 'origin/main' into feature/rework-table-handling
Approved-by: Katon Minhas
Bugfix/dynamic primary issues
* Merge branch 'main' into bugfix/dynamic-primary-issues
* Merged main into bugfix/dynamic-primary-issues
* Crosswalk updates
* Merge branch 'main' into bugfix/dynamic-primary-issues
* dont include section information in Exhibit Header
* Add Molina Health Benefit Exchange to Product-LOB mapping
* PPO-POS separate mapping
* Prevent inference of LOB from Program or Product
* Add SoonerSelect
* Update mapping rules to allow for direct values
* update prompts
* Prep for run
* Merged main into bugfix/dynamic-primary-issues
* Update unit tests
* Update Moda Product mappings
* Remove Care1st as Product for Centene
* testing
* Mapping update
* Mapping updates
* prep for merge
* Merged main into bugfix/dynamic-primary-issues
* Merged main into bugfix/dynamic-primary-issues
* docstrings and formatting
* Get group provider for > 1
* Merged main into bugfix/dynamic-primary-issues
Approved-by: Alex Galarce
Add (service, methodology) deduplication with accumulator
* Add deduplication with accumulator
* Merge remote-tracking branch 'origin/main' into feature/service-reimb-deduplication
* Replace print statement with logging.debug for cross-exhibit duplicate detection
* Fix and add tests for one_to_n_funcs
* Merge remote-tracking branch 'origin/main' into feature/service-reimb-deduplication
* Add seen_pairs parameter to reimbursement_level function in tests
Approved-by: Katon Minhas
Feature/exhibit linking
* Initialize new exhibit header prompt
* Save
* Merge branch 'main' into feature/exhibit-linking
* update prompt
* cleanup return type
* remove test
* Test fixes
* Fix test
* Add tests
* Merge branch 'main' into feature/exhibit-linking
* Merge branch 'main' into feature/exhibit-linking
* Remove test notebooks
* Delete test notebook
* Save
* Merge branch 'main' into feature/exhibit-linking
* try except for exhbit page finding
* Prep for merge
* update test
* Remove test
* Merge branch 'main' into feature/exhibit-linking
* update unit test
* Fix unit test
* replace try except
* Update page sort
* Add docstring
Approved-by: Alex Galarce
Don't consider table-subpages each individually for reimbursement exhibit identification
* Refine EXHIBIT_CHECK prompt
* test: rework `get_exhibit_pages` for table subpages
* Fix indentation
* Experimental table de-chunking
* Increase max_tokens for Claude model in exhibit page processing
* fix get_pages_to_process docstring
* refactor: improved docstring for get_exhibit_pages
* move `get_pages_to_process` to `preprocess.py`
* test: add unit tests for get_pages_to_process function
* Comment out debugging lines
* Changed prompt that was breading universal_json_load parsing
* fix: handle "no results" case in get_reimbursement_primary function
* fixed docstring
Approved-by: Katon Minhas
find_overlap function added to handle inconsistent chunk size
* find_overlap function added to handle inconsistent chunk size
* set min chunk overlap needed to remove overlap to 3
* Merge branch 'main' into bugfix/smart-chunking
* pipeline error fixed
* Merge branch 'main' into bugfix/smart-chunking
* spelling fix
Approved-by: Katon Minhas
DRAFT: Bugfix/effective date fix
* Merged in bugfix/extra-fields (pull request #500)
Do not run Exhibit level prompt if there are no exhibit level fields
* Do not run Exhibit level prompt if there are no exhibit level fields
* Update dynamic test
* Remove test from one-to-n test module
* Update test
* Fix unit test
Approved-by: Alex Galarce
* Merged in bugfix/tin-npi-prompt-fixes (pull request #499)
Bugfix/tin npi prompt fixes
* Change delimiter for other provider information from comma to pipe
* filter out invalid providers
* Update extraction and formatting instructions for provider entities in TIN_NPI_TEMPLATE
* Merged main into bugfix/tin-npi-small-bugfixes
* Refactor deduplication logic to use string_utils for checking empty provider fields
* Merge branch 'bugfix/tin-npi-small-bugfixes' of https://bitbucket.org/aarete/doczy.ai into bugfix/tin-npi-small-bugfixes
* Update deduplication logic to exclude providers with 'UNKNOWN' TIN, NPI, or NAME
* Merge remote-tracking branch 'origin/main' into bugfix/tin-npi-small-bugfixes
* Fix deduplication logic to not drop subsequent names
* Fix formatting of provider names in deduplication logic
* add payer name filtering
* reorder execution order to allow to find payer_name first
* Add handling for previously unfound group providers in identify_group_provider function
* Merge remote-trackin…
* Merged in feature/update-testbed-script (pull request #501)
Feature/update testbed script
* Update preprocessing
* Update preprocessing
* remove quit
* Update date for filename
* Merged main into feature/update-testbed-script
Approved-by: Alex Galarce
* Merged in bugfix/smart-chunks (pull request #503)
missing of text in chunks fixed
* missing of text in chunks fixed
Approved-by: Katon Minhas
* Merged in feature/rework-reimb-tin-npi (pull request #502)
Feature/rework reimb tin npi
* use LLM for reimb_[tin/npi/name]
* Merge remote-tracking branch 'origin/main' into feature/rework-reimb-tin-npi
* Merge remote-tracking branch 'origin/main' into feature/rework-reimb-tin-npi
* deduplicate tin/name/npi before adding to prompt
* Pull out prompt and llm_response in methodology_breakout_single_row for easier debugging
* Refactor reimbursement_tin_npi to handle multiple providers and update valid TIN, NPI, and NAME fields accordingly
* Reorder REIMB_PROV_NAME field in investment_prompts.json
* Update REIMB_PROV_NAME prompt to specify valid values for provider names
* Remove valid_values constraint from prov_name to raise accuracy
* Remove debugging print statements from reimbursement_tin_npi and related functions
* Remove debugging print statements from run_provider_info_fields function
* Merge remote-tracking branch 'origin/main' into feature/rework-reimb-tin-npi
* Remove debugging print statem…
* Merged in feature/preformat-single-quotes-input (pull request #506)
Replace double quotes with single quotes in context strings for parsability
* Replace double quotes with single quotes in context strings for parsability
* Merged main into feature/preformat-single-quotes-input
Approved-by: Katon Minhas
* updated sig pages fnxn
* Merge branch 'main' into bugfix/effective_date_fix
* poetry add pymupdf
* created vision funcs
* Merged main into bugfix/effective_date_fix
* fixed pipeline issue
* updated string_utils_test
* updated test values
* updated extract signature page fxn
* updated test values
* updated test values
* Merged main into bugfix/effective_date_fix
* Prompt template in all-caps
* remove debugging prints
* Merged main into bugfix/effective_date_fix
* Change vision to False by default
* Change vision to False by default
* fix tests
* fix tests
Approved-by: Katon Minhas
Do not run Exhibit level prompt if there are no exhibit level fields
* Do not run Exhibit level prompt if there are no exhibit level fields
* Update dynamic test
* Remove test from one-to-n test module
* Update test
* Fix unit test
Approved-by: Alex Galarce
Collect TIN/NPI/Name page-by-page and run a second pass for IS_GROUP
* configure first pass: no providers marked IS_GROUP. Allows simplification of prompt
* Add group provider identification logic in second pass
* group identification: don't take 20% of pages. Take first 3 and last 3
* Refine group provider identification logic to use pipes for JSON formatting and improve error handling
* fix mypy error for `identify_group_provider`
* Merge remote-tracking branch 'origin/main' into feature/tin-npi-prompt-formatting
* Add "PROV_INFO_JSON" to investment column order
* Refactor group provider identification to use signature page extraction and fix JSON format
* move signature page extraction to `string_utils.py` and add unit tests for it
* Update identify_group_provider to return provider name along with TIN and NPI in JSON format
* Update identify_group_provider to always use group-provided name when available
* move out GROUP_TIN_NPI_TEMPLATE to `investment_prompts.py`
* Remove deprecated PROV_GROUP_TIN_CHECK and PROV_GROUP_NPI_CHECK prompts
* Move out provider info fields from `one_to_one_funcs` to `tin_npi_funcs`
* isort
* remove debug print statements
* Remove debug print statements from provider info functions
* add note to regex_utils.py about where functions are used
* Add page_key_sort function and corresponding tests for sorting page keys
* Refactor identify_group_provider to sort pages and avoid duplicates in extracted sections
* Remove debug print statements from identify_group_provider function
* Merge remote-tracking branch 'origin/main' into feature/tin-npi-prompt-formatting
* extract_signature_page function: prioritize Textract marker for signatures and fallback to keyword search
* fix mypy error
* Update test cases to reflect changes in signature extraction logic
Approved-by: Katon Minhas
Field/initialize patient age
* Update split_text so that it works on single-page contracts (discovered when testing)
* Add PATIENT_AGE_RANGE intermediate column
* Remove PATIENT_AGE_RANGE in postprocessing
* Dont convert columns to int type
* Add unit tests
* Remove test
* Update unit test
* Update unit test
* Update unit test
* Update unit test
Approved-by: Alex Galarce
Field/dynamic codes
* remove test
* Merge branch 'main' into field/dynamic-codes
* Merged main into field/dynamic-codes
* Update test
* Fix unit test
* Merged main into field/dynamic-codes
* Merged main into field/dynamic-codes
* Merged main into field/dynamic-codes
* Update fields with crosswalks
* Update mapping to return empty string if no mapping
* Update dynamic_funcs
* Restructure
* Update file_processing
* Update dynamic primary
* Fix dynamic primary
* Update base fields
* Update one-to-n process
* Merged main into field/dynamic-codes
* genericized get_dynamic_answers
* Update tests
* Remove test file
* dynamic_funcs cleanup
* remove prints
* add exhibit_header
* Fix imports
Approved-by: Alex Galarce
Bugfix/flatten singleton list ints problem
* Fix flatten_singleton_string_list to handle non-list int strings and improve test coverage
* Fix test_flatten_singleton_string_list to handle integer input correctly
Approved-by: Katon Minhas
Feature/align tin npi
* Add merge_provider_info function to consolidate provider data into one-to-one results
* Merge remote-tracking branch 'origin/main' into feature/align-tin-npi
* Remove debug print statement from run_one_to_one_prompts function
* turn run_regex_fields into an orchestrator function by breaking parts of it into separate functions
* Remove unused import of get_matches function and add docstring to chunk_on_matches for clarity
* Add docstring to run_regex_fields for improved clarity and documentation
* fix provider name keys in merge_provider_info
* Remove unused functions get_matches, tin_npi_prompt, and clean_tin_npi from tin_npi_funcs.py
* Set provider name to "UNKNOWN" if cleaned name is empty in clean_provider_info
* Add unit tests for clean_provider_info function to validate data processing
* Enhance docstring for get_all_matches function
* Add tests for get_all_matches and chunk_on_matches functions
* Merge remote-tracking branch 'origin/main' into feature/align-tin-npi
* Implement merge_provider_info function to consolidate provider data into one-to-one results
* Refactor merge_provider_info to simplify list conversion for group and other provider information
* Remove unnecessary TIN and NPI postprocessing step from postprocess function
* Remove clean_tin_npi_other function and its invocation from postprocess function
* Use json prompts for TIN/NPI/Name and inject them into TIN_NPI_TEMPLATE
* Rename run_regex_fields to run_provider_info_fields
* Enhance TIN_NPI_TEMPLATE to include detailed instructions for extracting provider entities and their identifying information, including IS_GROUP logic for main contracting parties.
* moved TIN/NPI regexes to regex_patterns.py
* Remove unnecessary blank line in investment_values.py
* Remove debug print statements from run_provider_info_fields and get_provider_info functions
* Enhance clean_provider_info to handle various representations of IS_GROUP as boolean
* Enhance get_provider_info to include error handling for JSON parsing and ensure consistent return format
Approved-by: Katon Minhas
Alex Galarce