Alex Galarce
357bdadd1b
Hotfix/fix nltk locaiton * remove incorrectly placed NLTK tokenizers * fix NLTK location, only download english punkt * update .gitignore to disallow all .tab files except english Approved-by: Katon Minhas
26 lines
808 B
Markdown
26 lines
808 B
Markdown
# NLTK Data
|
|
|
|
This directory contains NLTK model data used by our project, specifically the `punkt_tab` tokenizer. The data is version controlled to ensure consistent behavior across environments and avoid download steps during deployment.
|
|
|
|
## Updating Models
|
|
|
|
If you need to update the NLTK models, run the following script from `fieldExtraction`:
|
|
|
|
```python
|
|
# update_nltk.py
|
|
import nltk
|
|
import os
|
|
|
|
# Ensure we're downloading to the right place
|
|
nltk.download("punkt_tab", download_dir="./nltk_data")
|
|
```
|
|
|
|
After running this script, commit any updated files to version control.
|
|
|
|
## Why Version Control?
|
|
|
|
We version control NLTK data to:
|
|
- Ensure consistent model versions across all environments
|
|
- Remove network dependencies during deployment
|
|
- Enable offline development
|
|
- Avoid download steps after cloning |