Bugfix/payer name provider state to main
* updated provider state
* updated prompt
* Merged main into bugfix/payer_name_provider_state
* updated smart chunked field types
* added post processing for provider state
* field check added
* filter by field_type instead of retrieval question
* Merged main into bugfix/payer_name_provider_state
Approved-by: Katon Minhas
Bugfix/ut errors
* Update universal_json_load and unit tests
* Update tests
* Fix unit tests
* remove test
* remove imports
* Update function
* update pd Series is_empty
* Update reimbursement check
* added some test cases
* Merged main into bugfix/ut-errors
* Merged main into bugfix/ut-errors
* remove import
* Merged main into bugfix/ut-errors
Approved-by: Karan Desai
Bugfix/mi reimb terms
* test changes
* logging added
* Merge branch 'main' into bugfix/generic-reimb-terms
* parsing fixes
* fix missing reimbursements
* make service terms unique by adding additional context
* Merge branch 'main' into bugfix/mi-reimb-terms
* remomoving test changes
* remomoving test changes
* remomoving test changes
* test case for universal_json_load added
Approved-by: Katon Minhas
Feature/black and isort
* black and isort constants
* black and isort codes
* black and isort crosswalks
* black and isort investment
* black and isort prompts
* isort testbed
* isort tracking
* black and isort utils
* poetry and black tests
Approved-by: Katon Minhas
Refactor/one to n
* remove print
* Update logging for fsbreakout and grouper breakout
* Additional type hints
* updated special case primary
* Remove deprecated special case check
* OUTLIER_TERMS-->OUTLIER_TERM standardization
* Applied suggestion to fieldExtraction/src/prompts/investment_prompts.json
* update unit test
* Update Carveout IND and Default IND logic
* Split out exhibit-level lesser
* Bugfix
* update lesser of exhibit prompt
* Bugfix
* exhibit lesser of varible name bugfix
* Update exhibit lesser of passing
* Update Exhibit-level prompt template
* Shorten full_context prompt length if the questions are long
* Special case assignment index
* Resolve unit tests
* updated UC IND prompt
* Remove print
* update one-to-n unit test
* added dot based date example
* updated date fix prompt
* Merged in bugfix/json_error_and_unit_of_measure (pull request #708)
Bugfix/json error and unit of measure
* json parser fixes and prompt changes
* universal json load fixed
* json error fixed
* json error fixed 2
* Merged refactor/one-to-n into bugfix/json_error_and_unit_of_measure
Approved-by: Katon Minhas
Approved-by: Alex Galarce
pyproject requirements, prep for logging fixes
* remove chroma
* update project name to "doczy-field-extraction" in pyproject.toml
* update lock file
* replace print statements with logging for better traceability in process_file function
* replace print statements with logging for improved error tracking and information logging
* replace print statements with logging for improved error handling and traceability
* replace print statements with logging for improved error handling in llm_utils and string_utils
* remove unused table_utils module; supplanted by table_funcs and unreferenced
* replace capsys with caplog in test_invalid_input_type for improved logging of invalid input
Approved-by: Katon Minhas
Bugfix/uom
* Add "Per Procedure" to valid unit of measure list
* Enhance unit of measure prompt for clarity and specificity in reimbursement methodologies
* Refine unit of measure prompt to ensure defaulting to 'Per Unit' regardless of procedure codes for flat rate methodologies
* Merge remote-tracking branch 'origin/main' into bugfix/uom
* Merged main into bugfix/uom
* Enhance logic for setting UNIT_OF_MEASURE to 'Per Unit' for Flat Rate reimbursement, handling all edge cases including NaN and empty strings.
* Enhance is_empty function to handle additional empty value cases for pd.Series and strings
* Enhance test cases for is_empty function to cover additional whitespace and null-like values
* Merge branch 'bugfix/uom' of https://bitbucket.org/aarete/doczy.ai into bugfix/uom
* Enhance type checking for is_empty function with overloads for str, list, and pd.Series
* Overload fix
* Merged main into bugfix/uom
Approved-by: Katon Minhas
Refactor/one time io
* Delete client work
* Streamline imports
* Fix list class
* Update tests
* Clear code funcs test
* Fix unit tests
* Single read code funcs
* Single-read model
* Single-load embeddings
* Successful E2E test
* update unit tests
* Update importsg
* Reload poetry.lock
* remove old exhibit header function
* Remove aarete_derived generic function
* remove align and format tables
* Remove strings to dict
* references
* Clear code
* Move preprocessing_funcs
* remove keywords
* refactor postprocessing_funcs
* Pass unit test - remove qa_qc directory
* Black and isort
* remove print
Approved-by: Alex Galarce
Hotfix/defensive crosswalk handling
* Enhance crosswalk functionality with flatten_to_strings helper and improve value handling in apply_crosswalk
* Improve apply_crosswalk function to return original value as fallback when no mapping is found
* Enhance apply_crosswalk function to handle empty values and improve defensive programming tests
* move flatten_to_strings helper function to string_utils
Approved-by: Katon Minhas
Sonnet 4 and model aliasing
* Add EXHIBIT_HEADER_MARKERS for improved document header recognition in table handling
* Refactor EXHIBIT_HEADER_MARKERS definition for improved clarity and organization
* remove unused page span tracking code; add unit tests for new functionality
* remove debugging logs from combine_continuous_tables function
* Refactor logging setup and update test file configuration in investment field testing notebook
* Merge remote-tracking branch 'origin/main' into feature/further-phase-1-and-2-fixes
* Merge remote-tracking branch 'origin/main' into feature/further-phase-1-and-2-fixes
* Refactor model ID handling and add support for model aliases in llm_utils
* Update model IDs to use alias in LLM invocation functions in one_to_n_funcs
* change pre-and-post-processing to new model alias method
* Update model ID references to use 'sonnet_latest' in LLM invocation functions
* Update LLM invocation functions to use 'sonnet_latest' model ID
* Update LLM invocation functions to use 'sonnet_latest' and 'haiku_latest' model IDs
* update invoke_claude function docstring
* Fix caller frame to address mypy error
Approved-by: Katon Minhas
Feature/provider state
* Add prompt for extracting provider state information in investment prompts
* Add state normalization function and integrate into context field extraction
* black format string_utils.py; add tests for normalize_state_to_abbreviation function
* black format string_utils_test.py
* Add PROVIDER_STATE to investment column order
* Merged main into feature/provider-state
Approved-by: Katon Minhas
Feature/filter reimbursements
* First-pass implementation for reimbursement filtering
* Enhance reimbursement level processing by filtering out services without reimbursement terms and logging when no valid pairs are found.
* Update mock return values in reimbursement level tests to reflect new reimbursement terms
* Merge remote-tracking branch 'origin/main' into feature/split-and-filter-reimbursements
* Refine reimbursement filtering by adding 'billed' to indicators and ensuring REIMB_TERM is a string before processing.
* Refactor is_empty function signature to support multiple input types and clarify return values
* adding LLM review for ambiguous reimbursement methodology cases
* more stringent keywords for stage 1 of reimbursement filtering
* refining rate and cost patterns for reimbursement filtering
* Enhance LLM response handling in reimbursement methodology check to improve accuracy and logging for ambiguous cases.
* Merge remote-tracking branch 'origin/main' into feature/split-and-filter-reimbursements
* Update reimbursement test cases to reflect accurate reimbursement terms and improve deduplication logic
* move VALIDATE_REIMBURSEMENTS_PROMPT to investment_prompts.py
* black, isort formatting
* Remove IDENTIFY_REIMBURSEMENT_EXHIBITS_PROMPT function (unused)
* Add MODEL_ID_CLAUDE4_SONNET for new model integration (doesn't work right now)
* pipe delimited-output and parsing
* Merge remote-tracking branch 'origin/main' into feature/split-and-filter-reimbursements
* Refactor reimbursement filtering by moving clear indicators to constants
Approved-by: Katon Minhas
DRAFT: Bugfix/effective date fix
* Merged in bugfix/extra-fields (pull request #500)
Do not run Exhibit level prompt if there are no exhibit level fields
* Do not run Exhibit level prompt if there are no exhibit level fields
* Update dynamic test
* Remove test from one-to-n test module
* Update test
* Fix unit test
Approved-by: Alex Galarce
* Merged in bugfix/tin-npi-prompt-fixes (pull request #499)
Bugfix/tin npi prompt fixes
* Change delimiter for other provider information from comma to pipe
* filter out invalid providers
* Update extraction and formatting instructions for provider entities in TIN_NPI_TEMPLATE
* Merged main into bugfix/tin-npi-small-bugfixes
* Refactor deduplication logic to use string_utils for checking empty provider fields
* Merge branch 'bugfix/tin-npi-small-bugfixes' of https://bitbucket.org/aarete/doczy.ai into bugfix/tin-npi-small-bugfixes
* Update deduplication logic to exclude providers with 'UNKNOWN' TIN, NPI, or NAME
* Merge remote-tracking branch 'origin/main' into bugfix/tin-npi-small-bugfixes
* Fix deduplication logic to not drop subsequent names
* Fix formatting of provider names in deduplication logic
* add payer name filtering
* reorder execution order to allow to find payer_name first
* Add handling for previously unfound group providers in identify_group_provider function
* Merge remote-trackin…
* Merged in feature/update-testbed-script (pull request #501)
Feature/update testbed script
* Update preprocessing
* Update preprocessing
* remove quit
* Update date for filename
* Merged main into feature/update-testbed-script
Approved-by: Alex Galarce
* Merged in bugfix/smart-chunks (pull request #503)
missing of text in chunks fixed
* missing of text in chunks fixed
Approved-by: Katon Minhas
* Merged in feature/rework-reimb-tin-npi (pull request #502)
Feature/rework reimb tin npi
* use LLM for reimb_[tin/npi/name]
* Merge remote-tracking branch 'origin/main' into feature/rework-reimb-tin-npi
* Merge remote-tracking branch 'origin/main' into feature/rework-reimb-tin-npi
* deduplicate tin/name/npi before adding to prompt
* Pull out prompt and llm_response in methodology_breakout_single_row for easier debugging
* Refactor reimbursement_tin_npi to handle multiple providers and update valid TIN, NPI, and NAME fields accordingly
* Reorder REIMB_PROV_NAME field in investment_prompts.json
* Update REIMB_PROV_NAME prompt to specify valid values for provider names
* Remove valid_values constraint from prov_name to raise accuracy
* Remove debugging print statements from reimbursement_tin_npi and related functions
* Remove debugging print statements from run_provider_info_fields function
* Merge remote-tracking branch 'origin/main' into feature/rework-reimb-tin-npi
* Remove debugging print statem…
* Merged in feature/preformat-single-quotes-input (pull request #506)
Replace double quotes with single quotes in context strings for parsability
* Replace double quotes with single quotes in context strings for parsability
* Merged main into feature/preformat-single-quotes-input
Approved-by: Katon Minhas
* updated sig pages fnxn
* Merge branch 'main' into bugfix/effective_date_fix
* poetry add pymupdf
* created vision funcs
* Merged main into bugfix/effective_date_fix
* fixed pipeline issue
* updated string_utils_test
* updated test values
* updated extract signature page fxn
* updated test values
* updated test values
* Merged main into bugfix/effective_date_fix
* Prompt template in all-caps
* remove debugging prints
* Merged main into bugfix/effective_date_fix
* Change vision to False by default
* Change vision to False by default
* fix tests
* fix tests
Approved-by: Katon Minhas
Collect TIN/NPI/Name page-by-page and run a second pass for IS_GROUP
* configure first pass: no providers marked IS_GROUP. Allows simplification of prompt
* Add group provider identification logic in second pass
* group identification: don't take 20% of pages. Take first 3 and last 3
* Refine group provider identification logic to use pipes for JSON formatting and improve error handling
* fix mypy error for `identify_group_provider`
* Merge remote-tracking branch 'origin/main' into feature/tin-npi-prompt-formatting
* Add "PROV_INFO_JSON" to investment column order
* Refactor group provider identification to use signature page extraction and fix JSON format
* move signature page extraction to `string_utils.py` and add unit tests for it
* Update identify_group_provider to return provider name along with TIN and NPI in JSON format
* Update identify_group_provider to always use group-provided name when available
* move out GROUP_TIN_NPI_TEMPLATE to `investment_prompts.py`
* Remove deprecated PROV_GROUP_TIN_CHECK and PROV_GROUP_NPI_CHECK prompts
* Move out provider info fields from `one_to_one_funcs` to `tin_npi_funcs`
* isort
* remove debug print statements
* Remove debug print statements from provider info functions
* add note to regex_utils.py about where functions are used
* Add page_key_sort function and corresponding tests for sorting page keys
* Refactor identify_group_provider to sort pages and avoid duplicates in extracted sections
* Remove debug print statements from identify_group_provider function
* Merge remote-tracking branch 'origin/main' into feature/tin-npi-prompt-formatting
* extract_signature_page function: prioritize Textract marker for signatures and fallback to keyword search
* fix mypy error
* Update test cases to reflect changes in signature extraction logic
Approved-by: Katon Minhas
Enhance universal_json_load function with detailed error handling and traceback logging
* Enhance universal_json_load function with detailed error handling and traceback logging
* Add type hint for string_dict parameter in universal_json_load function
* Update return type annotation for universal_json_load function to reflect possible return values
Approved-by: Katon Minhas
Enhance is_empty function to handle pandas Series and improve empty value checks
* Enhance is_empty function to handle pandas Series and improve empty value checks
* Enhance is_empty function to support pandas Series and add pd_mask parameter for flexible empty checks
Approved-by: Katon Minhas
Hotfix/file errors
* Set up test file
* Fix regex pattern in json.loads
* bug fix No valid JSON object found in the input string
* bug fixes
* moved try statement to parse fn
* invalid literal for int() with base 10 error fixed
* remove test
* remove excess print statements
* Raise error
* Remove todo
* Remove print statement
Approved-by: Katon Minhas
Bugfix/table failed files
* improved align and format table prompt
* Successful E2E test on San Joaquin - without running align and format table prompt
* San Joaquin e2e test passed
* Merged main into bugfix/table-error-fix
* Merge branch 'main' into bugfix/table-error-fix
* Add same number of columns condition
* Merged main into bugfix/table-error-fix
* Clean investment_values
* Remove import
* docstrings
* Remove test doc
* Add back original ALIGN_AND_FORMAT_TABLES prompt for client work
* Remove import
* Add config reference
* expected string or bytes-like object, got 'int'
* Merged main into bugfix/table-error-fix
* Merge branch 'bugfix/table-error-fix' into bugfix/table-failed-files
* Update universal json load
* update test
* Merge branch 'main' into bugfix/table-failed-files
* Position 0 error fixed
* update table_utils
* remove test.py
* revert table_utils test
* remove table_utils_test
Approved-by: Alex Galarce
Bugfix/table error fix
* handle multipage table data correctly
* same metadata for all tables on the page
* Multiple tables on page, non-continuous, split
* Add back align_and_format_tables
* update unit tests
* Merged main into bugfix/table-error-fix
* improved align and format table prompt
* Successful E2E test on San Joaquin - without running align and format table prompt
* San Joaquin e2e test passed
* Merged main into bugfix/table-error-fix
* Merge branch 'main' into bugfix/table-error-fix
* Add same number of columns condition
* Merged main into bugfix/table-error-fix
* Clean investment_values
* Remove import
* docstrings
* Remove test doc
* Add back original ALIGN_AND_FORMAT_TABLES prompt for client work
* Remove import
* Add config reference
* Merged main into bugfix/table-error-fix
* Move table_utils to investment.table_funcs
* restore table_utils.py
* Update tests
* update preprocess
Approved-by: Alex Galarce
Small fix to comparisons (lowercase comparison string fields)
* Add accuracy calculation to fuzzy matching evaluation
* reimbursement primary accuracy test
* add match_threshold control
* clarify metrics
* Merged main into reimb-primary-accuracy-analysis
* Merge branch 'reimb-primary-accuracy-analysis' of https://bitbucket.org/aarete/doczy.ai into reimb-primary-accuracy-analysis
* add 'billed charges' to reimbursement keywords and update TODO for page checking
* add comparisons dictionary
* normalize field comparison by converting strings to lowercase
* deduplicate data sources in notebook
* Merge remote-tracking branch 'origin/main' into feature/add-billed-charges-to-reimb-keywords
Approved-by: Katon Minhas
Fix load_embeddings()
* move datetime_str function to string_utils module
* enhance load_embeddings to create directory structure and handle download errors
* Merged main into feature/reimbursement-primary-optimizization
Approved-by: Katon Minhas
Bugfix/proc code lists
* Add admit type code
* add proc mod description
* Add bill type pos mapping
* Merged main into feature/proc-desc
* Uncomment test code
* Bugfix
* script optimisation for codes
* modified is empty function in string utils
* Remove test.py
* bug fix to handle list of proc_codes
* Merged main into bugfix/proc-code-lists
* changed any method to all method in list handling
* code mapping moved outside to run once
* Merge branch 'feature/daip2-97' into bugfix/proc-code-lists
* merge conflict fixed
* merge conflicts fixed
* incorporated latest changes from main
* Merged main into bugfix/proc-code-lists
* Move load all dataset to io_utils
* Fix list is_empty
* Updated poetry.lock
* Docstring
* Docstrings
* Merge branch 'main' into bugfix/proc-code-lists
* bug fix
Approved-by: Mayank Aamseek
Approved-by: Alex Galarce
feature/dynamic and generalized branch
* included txt files for nltk_data
* move nltk_data to src
* Fix last upload count
* Last upload count bugfix
* Fixed B processing
* Remove client-specific postprocessing
* split consolidate_output
* Run by file
* Fix output
* Reconfigure smart_chunk fields
* Add Full Context
* Fix merge conflicts
* Regex, Smart-Chunked, and Full working - not adding smart-chunked-->full when necessary
* Modernized run_full_context_fields()
* Switched set to list in field_context
* Move fields from smart_chunked to full_context as part of 'field_context' function
* Working version with placeholders
* Update poetry and pyproject
* Update s3 output
* Remove deprecated unit test
* Updated error messages
* Updated smart chunk ac name to one to one
* Update dependencies - end-to-end test for write s3 functional
* Add basic multithreading
* Send individual output to s3/local
Approved-by: Alex Galarce
Feature/integrate standard and complex
* last edits to merge_tables_draft_v2 before moving to table_funcs
* migrate to table_utils.py
* fix typehint that caused mypy error
* fix mypy errors
* black and isort
* fix error with multi-table pages
* update poetry.lock
* add tests, fix bug when table start marker or table end marker is missing
* added more tests
* tests for get_str_dictionaries_from_text()
* fix edge case in get_str_dictionaries_from_text
* fix mypy error
* more tests
* add tests, add handling for invalid dictionaries
* add tests
* add tests for insert_column_headers()
* update requirements
* add new simple/complex logic to preprocess.py
* remove files used solely for testing
* split pages into sub-pages by tables
* add table split on end marker. Also add tests
* add docstrings, change control flow
* remove unused tests, add TODO for new tests
* get in prompt changes and intermediate decisions
* TODO for future enhancement for get_exhibit_pages
Approved-by: Katon Minhas
feature/simplify b field chunking
* Add chunk_by_exhibit function and unit tests
* Refactor preprocess
* Updated to run BU primary on entire Exhibit
* Makes BU secondary compatible with exhibit chunk mapping
* Added unit tests for get_exhibit_chunk
* Trimmed TD funcs
* Downstream changes to run_top_down
* Removed conditional funcs
* Merged main into feature/simplify-b-field-chunking
* removed conditional funcs import
* Merged main into feature/simplify-b-field-chunking
* Fixed run_bottom_up_primary arguments
* Fixed TD primary input
* Fixed grouped keyword mappings
* Remove filename tracking until postprocessing
* Removed postprocessing for client columns
* Removed col-based postprocessing
* Merged main into feature/simplify-b-field-chunking
* Fixed NoneType
* Add docstrings
* Updated unit tests for removal of 'Filename'
* Updated comments
Approved-by: Alex Galarce
Task/unit tests investment and client
* move io_utils.py
* add __init__.py
* refactor: move string_funcs and update import paths to use utils namespace
* rename string_funcs to string_utils
* move and rename table_utils.py
* refactor: rename claude_funcs to llm_utils in multiple files
* update unit tests
* added unit tests for src/string_utils.py
* added unit tests for src/table_utils.py
* added unit tests for src/llm_utils.py
* added unit tests for src/io_utils.py
* Merge branch 'main' into task/unit-tests-investment-and-client; TODO: fix unit tests
* Unit tests to fix
* fix unit tests
* fix invalid references
Approved-by: Alex Galarce
Refactor/split investment and client
* refactor: clean up tests
* Renamed tests/qcqa to tests/qa_qc for consistency
* comment out faulty test - see docstring note at top
* split to client/investment
* add __init__.py to investment and client to avoid mypy confusion
* fix imports
* update imports for consistency across investment module
* move back to one config
* try changing import to relative
* add init.py to src
* try relative import
* try one more sys.path.append
* fix path for client main
* fix imports in file_processing
* make prompts common
* move keywords out to `src`
* move smart_chunking_funcs to `src`
* fix mypy error
* start moving to explicit imports
* remove old keywords and fix imports for keywords and prompts
* change all imports to full paths
* fix unit tests
* add readme for new way of running things
* isort after all that import work
* remove unused sys.path.append from some tests
Approved-by: Katon Minhas
VenkataKrishna Reddy Avula