Katon Minhas
063c95d6ac
feature/dynamic and generalized branch * included txt files for nltk_data * move nltk_data to src * Fix last upload count * Last upload count bugfix * Fixed B processing * Remove client-specific postprocessing * split consolidate_output * Run by file * Fix output * Reconfigure smart_chunk fields * Add Full Context * Fix merge conflicts * Regex, Smart-Chunked, and Full working - not adding smart-chunked-->full when necessary * Modernized run_full_context_fields() * Switched set to list in field_context * Move fields from smart_chunked to full_context as part of 'field_context' function * Working version with placeholders * Update poetry and pyproject * Update s3 output * Remove deprecated unit test * Updated error messages * Updated smart chunk ac name to one to one * Update dependencies - end-to-end test for write s3 functional * Add basic multithreading * Send individual output to s3/local Approved-by: Alex Galarce
Field Extraction
Running the Code
This project uses Python's module system for imports. To run the code properly:
-
DO NOT run files directly like this:
python src/client/main.py # ❌ This won't work -
INSTEAD, use Python's module flag (
-m) from the project root:python -m src.client.main # ✅ This is correct
Why?
The code uses absolute imports (e.g., from src.utils import llm_utils) to maintain a clear and consistent package structure. Running with python -m ensures Python can properly resolve these imports.
Development
- All imports should use the
src.prefix (e.g.,from src.utils import llm_utils) - Always run code from the project root directory using the
-mflag - Tests are configured to handle these imports automatically via pytest settings in pyproject.toml
Configuration
The project uses poetry for dependency management and pytest for testing. Key configurations in pyproject.toml:
[tool.pytest.ini_options]
pythonpath=["."]
testpaths = [
"tests"
]
Dependencies
To install dependencies:
poetry install
To run tests:
poetry run pytest
To run type checking:
poetry run mypy .