jacob-mathison/doczyai-pipelines
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Activity

Merged in bugfix/table-preprocessing (pull request #560)

Browse Source 
Bugfix/table preprocessing

* restructure get_exhibit_dict

* exhibit header

* update exhibit header handling

* update docstring

* bugfix

* remove test - update docstring

* Merged main into bugfix/table-preprocessing

* refactor: update page_key_sort to return float for numeric keys

* refactor: enhance get_exhibit_dict to improve chunking logic for exhibits

* refactor: update page_key_sort to return float for numeric keys

* test: add unit tests for get_exhibit_dict function

* test: fix test case 7

* Add final, difficult test

* refactor: enhance get_exhibit_dict to handle single decimal page exhibits and improve chunking logic

* pulled `one_to_n_test.py` from dynamic primary branch

* fix: update common_file_names logic to remove specific entry and log changes

* docs: update get_exhibit_dict docstring to clarify input and output formats

* test: add test for exhibit handling with decimal pages in get_exhibit_dict

* feat: add debug logging for exhibit chunk mapping and exhibit dictionary keys in one_to_n_exhibit_chunking

* Merge remote-tracking branch 'origin/main' into bugfix/table-preprocessing

* feat: enhance spatial processing rules and consistency checks for reimbursement terms

* feat: update global scan methodology to enhance context extraction for reimbursement constraints

* Merge remote-tracking branch 'origin/main' into bugfix/table-preprocessing


Approved-by: Katon Minhas
This commit is contained in:
Alex Galarce
2025-06-09 21:49:45 +00:00
parent 32d431ec4e
commit f84023d742
8 changed files with 663 additions and 111 deletions
Download Patch File Download Diff File Expand all files Collapse all files
fieldExtraction/src/preprocessing_funcs.py
+150 -3
View File
@@ -298,14 +298,161 @@ def chunk_by_exhibit(text_dict: dict,
if len(exhibit_pages) == 0:
return {key : key for key in text_dict.keys()}
exhibit_dict = {}
exhibit_chunk_mapping = {}
current_exhibit = "0"
for page_num in text_dict.keys():
if page_num in exhibit_pages:
current_exhibit = page_num
exhibit_dict[page_num] = current_exhibit
exhibit_chunk_mapping[page_num] = current_exhibit
else:
exhibit_dict[page_num] = current_exhibit
exhibit_chunk_mapping[page_num] = current_exhibit
return exhibit_chunk_mapping
def get_exhibit_dict(text_dict: dict,
exhibit_chunk_mapping: dict
) -> dict:
"""
Creates a dictionary of exhibit texts, concatenating pages with the same exhibit number,
but separating decimal pages into their own chunks.
Args:
text_dict (dict): Dictionary mapping page numbers to page text.
exhibit_chunk_mapping (dict): Dictionary mapping page numbers to exhibit numbers.
Returns:
dict: Dictionary mapping exhibit chunk keys to combined text. Format for keys:
"{exhibit}.{index}"
NOTE: If exhibit identifiers contain decimals (e.g., "23.0") output keys will
follow the same pattern (resulting in "23.0.0", "23.0.1", etc.)
Guidelines:
- Every exhibit chunk contains at most 1 table page in it (i.e. one page with a decimal)
- Tables are inserted in chronological order within the document flow.
- Multiple decimal pages are separated into individual chunks
- Each chunk gets a new key: "{exhibit}.{index}" (e.g. "2.0", "2.1", "2.2", etc.)
Algorithm:
For each exhibit:
1. Identify base pages (no decimals) and table pages (with decimals).
2. For each decimal page, create a chunk containing:
- All base pages that came before the decimal page (in document order)
- The specific decimal page
- All base pages that came after the decimal page (in document order)
Example 1:
Input mapping {'1.0': '1.0', '2': '2', '3.0': '2', '3.1': '2', '3.2': '2', '4.0': '2'}
- '1.0': text from '1.0' (single page exhibit)
- '2.0': text from '2' + text from '3.0' (base page 2, then table 3.0)
- '2.1': text from '2' + text from '3.1' (base page 2, then table 3.1)
- '2.2': text from '2' + text from '3.2' (base page 2, then table 3.2)
- '2.3': text from '2' + text from '4.0' (base page 2, then table 4.0)
Example 2:
Input mapping {'1.0': '1.0', '2': '2', '3.0': '2', '3.1': '2', '3.2': '2', '4': '2', '5' : '2'}
- '1.0': text from '1.0'
- '2.0': text from '2' + text from '3.0' + text from '4' + text from '5'
- '2.1': text from '2' + text from '3.1' + text from '4' + text from '5'
- '2.2': text from '2' + text from '3.2' + text from '4' + text from '5'
- '2.3': text from '2' + text from '4' + text from '5'
Note: Tables 3.0, 3.1, and 3.2 appear between pages 2 and 4 in document flow.
Other cases:
- No decimal pages: creates single chunk for the exhibit.
- Single decimal page exhibit: uses original page as the chunk key.
"""
exhibit_dict = {}
# Group pages by exhibit
exhibit_pages = {}
for page, exhibit in exhibit_chunk_mapping.items():
if exhibit not in exhibit_pages:
exhibit_pages[exhibit] = []
exhibit_pages[exhibit].append(page)
# Process each exhibit
for exhibit in exhibit_pages:
pages = sorted(exhibit_pages[exhibit], key=string_utils.page_key_sort)
base_pages = [p for p in pages if '.' not in p]
decimal_pages = [p for p in pages if '.' in p]
# Special case: single decimal page exhibit
if len(decimal_pages) == 1 and not base_pages:
exhibit_dict[decimal_pages[0]] = text_dict[decimal_pages[0]]
continue
# Case: No decimal pages - create single chunk with .0 suffix
if not decimal_pages:
exhibit_dict[f"{exhibit}.0"] = "\n".join(text_dict[p] for p in pages)
continue
# Case: With decimal pages, create chunks for each decimal page
chunk_index = 0
for decimal_page in decimal_pages:
chunk_key = f"{exhibit}.{chunk_index}"
# Get decimal page base number for chronological positioning
decimal_base = int(decimal_page.split('.')[0])
# Build chunk with pages in chronological order
chunk_parts = []
# Add base pages before decimal page
for base_page in base_pages:
if int(base_page) <= decimal_base:
chunk_parts.append(text_dict[base_page])
# Add the decimal page itself
chunk_parts.append(text_dict[decimal_page])
# Add base pages after decimal page
for base_page in base_pages:
if int(base_page) > decimal_base:
chunk_parts.append(text_dict[base_page])
exhibit_dict[chunk_key] = "\n".join(chunk_parts)
chunk_index += 1
# Add final chunk with just base pages (if there are base pages after all decimal pages)
if base_pages and decimal_pages:
# Check if there are base pages that come after all decimal pages
max_decimal_base = max(int(dp.split('.')[0]) for dp in decimal_pages)
remaining_base_pages = [bp for bp in base_pages if int(bp) > max_decimal_base]
if remaining_base_pages:
chunk_key = f"{exhibit}.{chunk_index}"
# Include ALL base pages in the final chunk
exhibit_dict[chunk_key] = "\n".join(text_dict[bp] for bp in base_pages)
return exhibit_dict
def get_exhibit_headers(exhibit_dict, filename):
"""
Extracts the exhibit header using an LLM prompt.
Args:
exhibit_chunk (str): A chunk of exhibit text to analyze.
filename (str): The name of the file being processed.
Returns:
str: The extracted exhibit header.
"""
exhibit_header_dict = {}
base_pages = {}
for page_num, exhibit_chunk in exhibit_dict.items():
base_page_num = page_num.split('.')[0]
if base_page_num in base_pages.keys():
exhibit_header_dict[page_num] = base_pages[base_page_num]
else:
prompt = investment_prompts.EXHIBIT_HEADER(exhibit_chunk[0:400])
llm_answer_raw = llm_utils.invoke_claude(
prompt, config.MODEL_ID_CLAUDE35_SONNET, filename
)
llm_answer_final = string_utils.extract_text_from_delimiters(
llm_answer_raw, Delimiter.PIPE
)
exhibit_header_dict[page_num] = llm_answer_final
base_pages[base_page_num] = llm_answer_final
return exhibit_header_dict