jacob-mathison/doczyai-pipelines
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Activity

Merged in refactor/split-investment-and-client (pull request #347)

Browse Source 
Refactor/split investment and client

* refactor: clean up tests

* Renamed tests/qcqa to tests/qa_qc for consistency

* comment out faulty test - see docstring note at top

* split to client/investment

* add __init__.py to investment and client to avoid mypy confusion

* fix imports

* update imports for consistency across investment module

* move back to one config

* try changing import to relative

* add init.py to src

* try relative import

* try one more sys.path.append

* fix path for client main

* fix imports in file_processing

* make prompts common

* move keywords out to `src`

* move smart_chunking_funcs to `src`

* fix mypy error

* start moving to explicit imports

* remove old keywords and fix imports for keywords and prompts

* change all imports to full paths

* fix unit tests

* add readme for new way of running things

* isort after all that import work

* remove unused sys.path.append from some tests


Approved-by: Katon Minhas
This commit is contained in:
Alex Galarce
2025-01-08 21:59:47 +00:00
parent b1e8b8db34
commit 53b38ea15c
60 changed files with 1444 additions and 268 deletions
Download Patch File Download Diff File Expand all files Collapse all files
fieldExtraction/src/preprocessing_funcs.py
+7 -12
View File
@@ -2,15 +2,10 @@ import re
from collections import defaultdict
from itertools import chain, count, groupby
import claude_funcs
import config
import keywords
import prompts
import smart_chunking_funcs
import string_funcs
import valid
from valid import DERIVED_INDICATOR_FIELDS
from regex_patterns import PIPE_PATTERN
import src.utils.llm_utils as llm_utils
import src.utils.string_utils as string_utils
from src import config, keywords, prompts, smart_chunking_funcs
from src.regex.regex_patterns import PIPE_PATTERN
def clean_newlines(contract_text: str) -> str:
@@ -66,7 +61,7 @@ def clean_law_symbols(contract_text):
def chunk_consecutive_og(text_dict, exhibit_pages):
# If needed - extend page 1 to page 1 and 2, then cut chunking off after (edge case: compensation terms not found on first page of exhibit)
reimbursement_pages = [page_num for page_num in text_dict.keys() if string_funcs.contains_reimbursement(text_dict, page_num)]
reimbursement_pages = [page_num for page_num in text_dict.keys() if string_utils.contains_reimbursement(text_dict, page_num)]
page_dict = {}
current_exhibit = None
@@ -106,7 +101,7 @@ def chunk_consecutive(text_dict, exhibit_pages):
reimbursement_pages = {
page_num
for page_num in text_dict.keys()
if string_funcs.contains_reimbursement(text_dict, page_num)
if string_utils.contains_reimbursement(text_dict, page_num)
}
page_dict = {}
@@ -304,7 +299,7 @@ def get_exhibit_pages(text_dict, filename):
exhibit_pages = []
for page_num, page in text_dict.items():
prompt = prompts.EXHIBIT_CHECK(page[0:100])
claude_answer_raw = claude_funcs.invoke_claude(
claude_answer_raw = llm_utils.invoke_claude(
prompt, config.MODEL_ID_CLAUDE3_HAIKU, filename, max_tokens=10
)
claude_answer_extracted = re.findall(pattern=PIPE_PATTERN, string=claude_answer_raw)[-1]