Alex Galarce
357bdadd1b
Hotfix/fix nltk locaiton * remove incorrectly placed NLTK tokenizers * fix NLTK location, only download english punkt * update .gitignore to disallow all .tab files except english Approved-by: Katon Minhas
NLTK Data
This directory contains NLTK model data used by our project, specifically the punkt_tab tokenizer. The data is version controlled to ensure consistent behavior across environments and avoid download steps during deployment.
Updating Models
If you need to update the NLTK models, run the following script from fieldExtraction:
# update_nltk.py
import nltk
import os
# Ensure we're downloading to the right place
nltk.download("punkt_tab", download_dir="./nltk_data")
After running this script, commit any updated files to version control.
Why Version Control?
We version control NLTK data to:
- Ensure consistent model versions across all environments
- Remove network dependencies during deployment
- Enable offline development
- Avoid download steps after cloning