jacob-mathison/doczyai-pipelines
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Activity

Merged in feature/simplify-b-field-chunking (pull request #348)

Browse Source 
feature/simplify b field chunking

* Add chunk_by_exhibit function and unit tests

* Refactor preprocess

* Updated to run BU primary on entire Exhibit

* Makes BU secondary compatible with exhibit chunk mapping

* Added unit tests for get_exhibit_chunk

* Trimmed TD funcs

* Downstream changes to run_top_down

* Removed conditional funcs

* Merged main into feature/simplify-b-field-chunking

* removed conditional funcs import

* Merged main into feature/simplify-b-field-chunking

* Fixed run_bottom_up_primary arguments

* Fixed TD primary input

* Fixed grouped keyword mappings

* Remove filename tracking until postprocessing

* Removed postprocessing for client columns

* Removed col-based postprocessing

* Merged main into feature/simplify-b-field-chunking

* Fixed NoneType

* Add docstrings

* Updated unit tests for removal of 'Filename'

* Updated comments


Approved-by: Alex Galarce
This commit is contained in:
Katon Minhas
2025-01-10 17:24:44 +00:00
parent b2c4bef039
commit 3bb6abbcef
10 changed files with 278 additions and 391 deletions
Download Patch File Download Diff File Expand all files Collapse all files
fieldExtraction/tests/string_utils_test.py
+9 -9
View File
@@ -96,9 +96,9 @@ class TestPrimaryStringToDict:
}
filename = "test_file.txt"
expected_output = [
{"key1": "value1", "page_num": "1", "Filename": "test_file.txt"},
{"key2": "value2", "page_num": "1", "Filename": "test_file.txt"},
{"key3": "value3", "page_num": "2", "Filename": "test_file.txt"},
{"key1": "value1", "page_num": "1"},
{"key2": "value2", "page_num": "1"},
{"key3": "value3", "page_num": "2"},
]
result = primary_string_to_dict(string_dict, filename)
assert result == expected_output
@@ -135,8 +135,8 @@ class TestPrimaryStringToDict:
}
filename = "test_file.txt"
expected_output = [
{"key1": "value1", "page_num": "1", "Filename": "test_file.txt"},
{"key2": "value2", "page_num": "2", "Filename": "test_file.txt"},
{"key1": "value1", "page_num": "1"},
{"key2": "value2", "page_num": "2"},
]
result = primary_string_to_dict(string_dict, filename)
assert result == expected_output
@@ -155,10 +155,10 @@ class TestPrimaryStringToDict:
}
filename = "test_file.txt"
expected_output = [
{"key1": "value1", "page_num": "1", "Filename": "test_file.txt"},
{"key2": "value2", "page_num": "1", "Filename": "test_file.txt"},
{"key3": "value3", "page_num": "2", "Filename": "test_file.txt"},
{"key4": "value4", "page_num": "2", "Filename": "test_file.txt"},
{"key1": "value1", "page_num": "1"},
{"key2": "value2", "page_num": "1"},
{"key3": "value3", "page_num": "2"},
{"key4": "value4", "page_num": "2"},
]
result = primary_string_to_dict(string_dict, filename)
assert result == expected_output