Feature/signatures present
* initial commit
* shifted from pp to one_to_one
* moved func in one_to_one_funcs
* Merged main into feature/signatures_present
Approved-by: Katon Minhas
Feature/effective dt
* move prompt_smart_chunked to prompt_calls.py
* refactor pass_smart_chunked_to_full_context
* Mypy fix
* Update field to EFFECTIVE_DT
* move to prompt_calls
* Update vision funcs
* Restructure prompt-based date derivation
* E2E passed
* Remove test
* Merge branch 'main' into feature/effective-dt
* remove deprecated postprocessing_func
* Docstrings
* refactor - send global lesser of to prompt_calls.py
* remove deprecated function
* more docstrings
* Merged main into feature/effective-dt
* Merged main into feature/effective-dt
* added base field for aarete derived effective date
Approved-by: VenkataKrishna Reddy Avula
Bugfix/claim type cd to main
* test claim type cd
* Merged main into bugfix/claim_type_cd
* updated claim type prompt
* updated prompt
* empty one to n results initiated
* remove print statements
* src/investment/prompt_calls.py
* full context prompt update
Approved-by: Katon Minhas
Bugfix/mi reimb terms
* test changes
* logging added
* Merge branch 'main' into bugfix/generic-reimb-terms
* parsing fixes
* fix missing reimbursements
* make service terms unique by adding additional context
* Merge branch 'main' into bugfix/mi-reimb-terms
* remomoving test changes
* remomoving test changes
* remomoving test changes
* test case for universal_json_load added
Approved-by: Katon Minhas
Feature/field type config
* Added field type procesing conditional logic
* Updated field type naming convention in config.py; Updated date_postprocessing to avoid column key missing issue
* Cleaned up code; Move merging functionality to one to one processing block
* Merged main into feature/field_type_config
Approved-by: Katon Minhas
Bugfix/mi dynamic
* Refactor - split out one_to_n_cleaning
* Rework postprocess
* Update postprocessing - fill dynamic values if there is only 1 non-empty value
* Update to use string_utils
* Group by AARETE_DERIVED_LOB as well
* Update crosswalk mapping to handle | lists
* Update constants - run with MI flag
Approved-by: Siddhant Medar
Feature/improve logging docs
* Enhance PerFileHandler documentation and functionality for improved logging context management
* Add more docuementation
* Update logging_utils.py documentation for clarity and usage examples
* Enhance logging comments for clarity on per-file logging setup and concurrent processing
* Enhance logging comments for clarity on per-file logging context and its implications
Approved-by: Katon Minhas
Refactor/one to n
* remove print
* Update logging for fsbreakout and grouper breakout
* Additional type hints
* updated special case primary
* Remove deprecated special case check
* OUTLIER_TERMS-->OUTLIER_TERM standardization
* Applied suggestion to fieldExtraction/src/prompts/investment_prompts.json
* update unit test
* Update Carveout IND and Default IND logic
* Split out exhibit-level lesser
* Bugfix
* update lesser of exhibit prompt
* Bugfix
* exhibit lesser of varible name bugfix
* Update exhibit lesser of passing
* Update Exhibit-level prompt template
* Shorten full_context prompt length if the questions are long
* Special case assignment index
* Resolve unit tests
* updated UC IND prompt
* Remove print
* update one-to-n unit test
* added dot based date example
* updated date fix prompt
* Merged in bugfix/json_error_and_unit_of_measure (pull request #708)
Bugfix/json error and unit of measure
* json parser fixes and prompt changes
* universal json load fixed
* json error fixed
* json error fixed 2
* Merged refactor/one-to-n into bugfix/json_error_and_unit_of_measure
Approved-by: Katon Minhas
Approved-by: Alex Galarce
Feature/initialize phase 5
* Initialize at Exhibit-Level
* Merge branch 'main' into feature/initialize-phase-5
* Add all breakout fields
* Prep for merge
Approved-by: Alex Galarce
Feature/per file logging
* Add custom per-file logging handler
* Add close method in PerFileHandler to properly close file handlers
* Add return type annotation to setup_per_file_logging function
* Set up per-file logging in main and process_file functions
* Add per-file logging configuration options and update setup logic
* Refactor logging setup in testing notebook to use per-file logging and improve organization
* Clean up whitespace and improve organization in testing notebook
* Merge remote-tracking branch 'origin/main' into feature/per-file-logging
* Enhance per-file logging setup by adding debug level configuration and informative print statement
* Change logging from info to debug level in table processing functions for improved verbosity control
* isort
Approved-by: Katon Minhas
pyproject requirements, prep for logging fixes
* remove chroma
* update project name to "doczy-field-extraction" in pyproject.toml
* update lock file
* replace print statements with logging for better traceability in process_file function
* replace print statements with logging for improved error tracking and information logging
* replace print statements with logging for improved error handling and traceability
* replace print statements with logging for improved error handling in llm_utils and string_utils
* remove unused table_utils module; supplanted by table_funcs and unreferenced
* replace capsys with caplog in test_invalid_input_type for improved logging of invalid input
Approved-by: Katon Minhas
Feature/grouper fields
* prompt changes
* print statements removed
* merged with main
* merge issue fixed
* updated grouper prompts
* updated grouper fields in fields order
* updated breakout prompt
* Merged main into feature/grouper-fields
* reverted grouper cd prompt
* updated goruper severity prompt
* Merge remote-tracking branch 'origin/main' into feature/grouper-fields
* updated grouper breakout function
* error fixes
* Merged main into feature/grouper-fields
* Merged main into feature/grouper-fields
* updated grouper_pct_rate and grouper_base_rate
* removed empty array post processing fix
* Merged main into feature/grouper-fields
* post-processing for grouper rate modified
* updated grouper type prompt
* Merged main into feature/grouper-fields
* grouper version prompt updated
* Merged main into feature/grouper-fields
* OPPS as APC
* Merged main into feature/grouper-fields
Approved-by: Katon Minhas
Bugfix/dynamic primary issues
* format and restructure
* checkpoint
* Merge branch 'main' into bugfix/dynamic-primary-issues
* Update constants attributes to base class
* Revert "Update constants attributes to base class"
This reverts commit 936b1e41f2e2d445e69e840b69af08eac6661120.
* Fix mapping
* E2E test
* remove prints
* Merge branch 'main' into bugfix/dynamic-primary-issues
* Black
* Update any dynamic
* Prompt update - more leniency
* Remove prints
* Merged main into bugfix/dynamic-primary-issues
* update crosswalk
* Merged main into bugfix/dynamic-primary-issues
Approved-by: Alex Galarce
Bugfix/dynamic primary issues
* Refactor TRIGGER_CAP special case configuration to utilize only trigger cap questions for two-stage processing
* Refactor get_special_cases to improve handling of N/A responses and merge methodology results with special case results
* Fix field type in get_special_cases for fee schedule breakout configuration
* Create fieldset.py - refactor
* Fix placeholders
* string concatenate new questions
* Update trigger cap questions output format
* Update TRIGGER_CAP_THRESHOLD_AMT prompt for clarity and specificity in reimbursement methodology
* Update constants class
* Update Field class
* Resolve constants passing
* Update field class - change Null criteria
* Add default
* Enhance documentation for get_special_cases function with detailed processing flow and architecture decisions
* Enhance get_special_cases and run_trigger_cap_breakout functions with detailed docstrings for clarity on processing methodologies
* Update constants passing
* Fix questions
* Update fs and mb questions"
* pass constanst correctly for dynamic
* Merge branch 'feature/trigger_cap_breakout' into bugfix/dynamic-primary-issues
* update constants
* only add questions if present
* ensure answer is a valid special_case
* Fix unit tests
* Remove test file
Approved-by: Alex Galarce
Bugfix/reimb dates to Main
* added reimb dates split
* Merged main into bugfix/reimb_dates
* prompt changes
* added to postprocessing
* fixed key issues
* modified split function
* removed print statements
* Merge branch 'main' into bugfix/reimb_dates
* update import
* moved split date upstream
* moved split dates to one to n funcs
* Merged main into bugfix/reimb_dates
Approved-by: Katon Minhas
Refactor/one time io
* Delete client work
* Streamline imports
* Fix list class
* Update tests
* Clear code funcs test
* Fix unit tests
* Single read code funcs
* Single-read model
* Single-load embeddings
* Successful E2E test
* update unit tests
* Update importsg
* Reload poetry.lock
* remove old exhibit header function
* Remove aarete_derived generic function
* remove align and format tables
* Remove strings to dict
* references
* Clear code
* Move preprocessing_funcs
* remove keywords
* refactor postprocessing_funcs
* Pass unit test - remove qa_qc directory
* Black and isort
* remove print
Approved-by: Alex Galarce
Exhibit-level Lesser Of
* Enhance exception language clarity in investment prompts
* Roll back exception language
* initial try at exhibit_lesser_of
* simplify `create_exhibit_lesser_of_row`
* Refine prompt for extracting 'lesser of' and 'not to exceed' constraints in exhibit
* Remove unused `get_service_breakout` function; it was unreferenced
* run exhibit-level answers FIRST, apply exhibit lesser-of WITHIN reimbursement_level
* Refine prompt for extracting 'lesser of' and 'not to exceed' constraints to clarify expected output and handling of multiple constraints.
* Merge remote-tracking branch 'origin/main' into bugfix/samaritan-missing-rows
* add exhibit_lesser_of="N/A" to tests
* Add exhibit_lesser_of parameter to reimbursement_level calls in tests
* restore dynamic answer retrieval for primary, reimbursement info, and code fields
* Merge remote-tracking branch 'origin/main' into bugfix/samaritan-missing-rows
* "fill in the blank" prompt attempt
* Merge remote-tracking branch 'origin/main' into bugfix/samaritan-missing-rows
* Merge remote-tracking branch 'origin/main' into feature/sonnet-4
* Merge remote-tracking branch 'origin/feature/sonnet-4' into bugfix/samaritan-missing-rows
* Merge remote-tracking branch 'origin/main' into bugfix/samaritan-missing-rows
* Remove carveout patterns from has_explicit_lesser_of_carveout function
* Add deduplicate_methodology_breakout function to remove duplicate reimbursement methodologies
* Add deduplication of methodology breakout rows in postprocess function
* Enhance create_exhibit_lesser_of_row to avoid redundancy with substring checks
* Change log level from info to debug for exhibit constraint check in create_exhibit_lesser_of_row
* Remove deduplication of methodology breakout rows from postprocess function (we handle it upstream)
* Merged main into bugfix/samaritan-missing-rows
Approved-by: Katon Minhas
Hotfix/claim type cd
* Fix exhibit_level prompt
* e2e test passed - successfully pulling multiple Claim Types per exhibit
* Merged main into hotfix/claim-type-cd
Approved-by: Alex Galarce
Bugfix/mcs issue fixes
* Update code primary prompt
* Merge branch 'main' into bugfix/mcs-issue-fixes
* No longer treat non-reimbursement tables as 'tables' for the purpose of preprocessing
* No longer extract Reimb Effective Date from footer of the page
* Merge branch 'main' into bugfix/mcs-issue-fixes
* Update unit tests
* Add postprocessing step to remove Reimb Date values when the 1:1 date values are identical
* Update unit tests
* prep for merge
* Remove excess prints
* Merged main into bugfix/mcs-issue-fixes
* Merge branch 'main' into bugfix/mcs-issue-fixes
* Prep for merge
* remove print
Approved-by: Alex Galarce
Feature/prov group tin
* don't prompt for INTRO_PAGE
* test complete
* test 2
* Update test script
* Merge branch 'main' into feature/prov-group-tin
* Merged main into feature/prov-group-tin
* refactor
* cleaning
* Update save
* Fix return type
* Update for reimbursement_tin_npi
* Comment out reimbursement-tin-npi
* Pass unit tests
* Merge branch 'main' into feature/prov-group-tin
* Merged main into feature/prov-group-tin
* Remove length validation
* Merge branch 'main' into feature/prov-group-tin
* Remove ON_INTRO from prompt
* Restore
* Remove test
* Move reimb prov info to dynamic reimb info
* e2e test
* Fix for unit test
* remove prints
* update docstrings and typehints
Approved-by: Alex Galarce
Feature/rework table handling
* test: add cases for configuration boundary
* test: add table post-processing tests (cases to ensure preservation of post-table text and metadata during table operations)
* test: enhance table handling tests for small and large tables, ensuring proper suffixing and splitting behavior
* Fix and clarify intent in comments
* fix: enhance handling of post-table text when combining tables
* refactor: remove commented-out code in clean_tables function for clarity
* test: update post-table text preservation assertions and add debug tracing
* refactor: remove unused remove_table_from_page function and its test
* refactor: enhance combine method documentation and clarify split_by_size_with_smart_tables logic
* Merge remote-tracking branch 'origin/main' into feature/rework-table-handling
* refactor: reduce large table threshold from 1000 to 50 rows for better table splitting
* Add simple splitting
* refactor: uncomment dynamic answer retrieval in run_one_to_n_prompts for improved functionality
* Merge remote-tracking branch 'origin/main' into feature/rework-table-handling
* refactor: remove unused table handling functions and related constants for cleaner code
* refactor: remove unused test functions in preparation for new tests
* refactor: streamline continuation table handling by utilizing existing flags
* refactor: update table combination logic to include all tables from the main page
* refactor: enhance table handling to automatically resolve column mismatches, respect row limits, and improve unit/integration test coverage
* rename clean_tables_simple to clean_tables
* Update docstrings
* refactor: rename table handling functions
* Fix indentation and add tests
* isort, black
* Merge remote-tracking branch 'origin/main' into feature/rework-table-handling
Approved-by: Katon Minhas
Bugfix/dynamic primary issues
* Merge branch 'main' into bugfix/dynamic-primary-issues
* Merged main into bugfix/dynamic-primary-issues
* Crosswalk updates
* Merge branch 'main' into bugfix/dynamic-primary-issues
* dont include section information in Exhibit Header
* Add Molina Health Benefit Exchange to Product-LOB mapping
* PPO-POS separate mapping
* Prevent inference of LOB from Program or Product
* Add SoonerSelect
* Update mapping rules to allow for direct values
* update prompts
* Prep for run
* Merged main into bugfix/dynamic-primary-issues
* Update unit tests
* Update Moda Product mappings
* Remove Care1st as Product for Centene
* testing
* Mapping update
* Mapping updates
* prep for merge
* Merged main into bugfix/dynamic-primary-issues
* Merged main into bugfix/dynamic-primary-issues
* docstrings and formatting
* Get group provider for > 1
* Merged main into bugfix/dynamic-primary-issues
Approved-by: Alex Galarce
Feature/formatted prov info json
* Add newline and pipe-delimited formatting for provider info JSON
* Add new investment column names for nicely-formatted PROV_INFO_JSON
* remove PROV_INFO_JSON_PIPE and rename PROV_INFO_JSON_NEWLINE to PROV_INFO_JSON_FORMATTED
* Merge remote-tracking branch 'origin/main' into feature/formatted-prov-info-json
Approved-by: Katon Minhas
Feature/formatted prov info json
* Add newline and pipe-delimited formatting for provider info JSON
* Add new investment column names for nicely-formatted PROV_INFO_JSON
Approved-by: Katon Minhas
Add (service, methodology) deduplication with accumulator
* Add deduplication with accumulator
* Merge remote-tracking branch 'origin/main' into feature/service-reimb-deduplication
* Replace print statement with logging.debug for cross-exhibit duplicate detection
* Fix and add tests for one_to_n_funcs
* Merge remote-tracking branch 'origin/main' into feature/service-reimb-deduplication
* Add seen_pairs parameter to reimbursement_level function in tests
Approved-by: Katon Minhas
Feature/exhibit linking
* Initialize new exhibit header prompt
* Save
* Merge branch 'main' into feature/exhibit-linking
* update prompt
* cleanup return type
* remove test
* Test fixes
* Fix test
* Add tests
* Merge branch 'main' into feature/exhibit-linking
* Merge branch 'main' into feature/exhibit-linking
* Remove test notebooks
* Delete test notebook
* Save
* Merge branch 'main' into feature/exhibit-linking
* try except for exhbit page finding
* Prep for merge
* update test
* Remove test
* Merge branch 'main' into feature/exhibit-linking
* update unit test
* Fix unit test
* replace try except
* Update page sort
* Add docstring
Approved-by: Alex Galarce
Feature/global lesser of
* Merge remote-tracking branch 'origin/feature/reimb-primary-updates'
* First try at global "lesser of"
* Generate reimb_ids further upstream
* Implement global lesser of injection and extraction logic in file processing
* Update logic to change LESSER_OF_IND in-place for global affected rows; add docstrings
* Refactor global lesser of injection logic to create new rows for REIMB_IDs lacking lesser of logic; update original rows in place and improve tracking flags.
* Merge remote-tracking branch 'origin/main' into feature/global-lesser-of
* Add logging for global lesser of logic injection and update handling of original rows
* Refactor global lesser of logic: move functions to one_to_one_funcs.py and update inject_global_lesser_of_rows to use new implementations
* Move imports for METHODOLOGY_BREAKOUT_QUESTIONS and REIMB_TERM_BREAKOUT to the top of one_to_one_funcs.py for better organization and to remove prototype comments.
* isort
* Refactor import statement for generate_reimb_ids to use the full module path for clarity
* load field from JSON instead of hardcoded values
* Update logging levels in investment_field_testing notebook
* Merged main into feature/global-lesser-of
* Remove unnecessary comment from investment_postprocessing_funcs import statement for clarity
* Merge branch 'feature/global-lesser-of' of https://bitbucket.org/aarete/doczy.ai into feature/global-lesser-of
* Merge remote-tracking branch 'origin/main' into feature/global-lesser-of
* Remove debug print statement from process_file function
* fix call to Field.load_from_file
* Merge remote-tracking branch 'origin/main' into feature/global-lesser-of
* Add tracking flag for global lesser of logic when none is found
* explicitly sort by FILE_NAME, EXHIBIT_PAGE, and REIMB_LESSER_OF_ID columns
* Add new field type, "full_context_separate", to make sure that we don't run it twice. Also formatting fix
* Fix iteration over separate fields in run_full_context_fields function
Approved-by: Katon Minhas
Feature/further reimb primary improvements
* Refactor identify_reimbursement_exhibits to return both EXPLICIT and IMPLICIT values
* Moving from # Debugging lines to `logger`
* Enhance "lesser of" and exception language instructions in reimbursement terms
* clarify unclear print/logging statements
* Replace debugging print statements with logging in get_reimbursement_primary function
* Add logging for page processing in run_one_to_n_prompts function
Approved-by: Katon Minhas
Don't consider table-subpages each individually for reimbursement exhibit identification
* Refine EXHIBIT_CHECK prompt
* test: rework `get_exhibit_pages` for table subpages
* Fix indentation
* Experimental table de-chunking
* Increase max_tokens for Claude model in exhibit page processing
* fix get_pages_to_process docstring
* refactor: improved docstring for get_exhibit_pages
* move `get_pages_to_process` to `preprocess.py`
* test: add unit tests for get_pages_to_process function
* Comment out debugging lines
* Changed prompt that was breading universal_json_load parsing
* fix: handle "no results" case in get_reimbursement_primary function
* fixed docstring
Approved-by: Katon Minhas
Reimbursement Exhibit Prefiltering
* Add IDENTIFY_REIMBURSEMENT_EXHIBITS function to analyze exhibit headers for reimbursement information
* Add Zone.Identifier to .gitignore to prevent Windows zone information files from being tracked
* Add functionality to identify reimbursement exhibits
* Add functionality to identify reimbursement exhibits in run_one_to_n_prompts
* Output YAML parsing
* Refactor identify_reimbursement_exhibits to improve YAML parsing and error handling
* New reimbursement exhibit logic
* Update IDENTIFY_REIMBURSEMENT_EXHIBITS_PROMPT to return PAGE NUMBERS instead of exhibit headers
* Fix identify_reimbursement_exhibits to return page numbers as strings after YAML parsing
* Remove debugging print statement for reimbursement exhibits in run_one_to_n_prompts
* Remove commented debugging print statements in identify_reimbursement_exhibits
* Increase read_timeout in EC2 configuration from 2000 to 4000
* Merge remote-tracking branch 'origin/main' into feature/reimb-primary-improvements
* Remove debugging print statements in get_reimbursement_primary function
* mypy error
* Fix return type in identify_reimbursement_exhibits to ensure keys are returned as a list
* Merge remote-tracking branch 'origin/main' into feature/reimb-primary-improvements
Approved-by: Katon Minhas
Feature/medicare medicaid postprocess
* Initial test
* add to postprocessing funcs
* Remove from postprocessing
* Merged main into feature/medicare-medicaid-postprocess
Approved-by: Alex Galarce
Bugfix/add name to full context if no identifiers
* Add handling for missing TIN/NPI identifiers in one-to-one results
* Update provider name handling in one-to-one results for group providers
Approved-by: Katon Minhas
Katon Minhas