feat: deploy training data to zoe root
CI / validate (push) Successful in 38s

This commit is contained in:
2026-07-22 09:13:54 -05:00
parent 1d683ab380
commit 8a697cf64f
11 changed files with 124 additions and 22 deletions
+15 -11
View File
@@ -8,8 +8,9 @@ scripts/generate_training_data.py
This script reads the generated dictionary artifacts from this project, optionally runs a
NeMo Data Designer expansion pass on Mimir via SSH, and writes a timestamped raw training
dataset into the `workspace_Data` pipeline, then registers it in `manifest.llm.json` so the
Training Monitor picks it up automatically.
dataset into `/Volumes/Zoe/custom-local-llm/training-data`, then registers it in
`/Volumes/Zoe/custom-local-llm/manifest.llm.json` so local training tools can point at one
stable artifact root.
---
@@ -46,10 +47,10 @@ Step 2 (remote — Mimir, optional): │
--dataset-name module_controller_intents_<timestamp>
SCP expanded dataset back
Step 3 (local — workspace_Data):
Write to data/module_controller_intents_<timestamp>/ ◄──────────────
Step 3 (local — Zoe custom-local-llm):
Write to training-data/module_controller_intents_<timestamp>/ ◄─────┘
Patch manifest.llm.json with timestamped entry
Training Monitor picks it up as a new Data Source
Training tools pick it up from the Zoe artifact root
```
---
@@ -59,7 +60,7 @@ Step 3 (local — workspace_Data): │
Every dataset generated by this script is tagged with a **Unix timestamp at generation time**.
The timestamp is embedded in:
1. **Output directory name:** `data/module_controller_intents_<unix_ts>/`
1. **Output directory name:** `training-data/module_controller_intents_<unix_ts>/`
2. **Manifest entry key:** `"module_controller_intents_<unix_ts>"`
3. **Manifest entry field:** `"generated_at": <unix_ts>`
4. **Every JSONL record's metadata:** `"generated_at": <unix_ts>`
@@ -167,7 +168,7 @@ columns:
"redistribution_allowed": false,
"estimated_tokens": null,
"status": "generated_local",
"local_path": "data/module_controller_intents_<unix_ts>",
"local_path": "training-data/module_controller_intents_<unix_ts>",
"local_format": "jsonl",
"generator": "mp-ai-module-controller/scripts/generate_training_data.py",
"sdg_engine": "data-designer",
@@ -191,14 +192,14 @@ columns:
# Seed only (no NeMo expansion — fast, local, 260 records):
python3 scripts/generate_training_data.py \
--dictionary-dir dictionary/ \
--output-dir ../workspace_Data/data/ \
--manifest ../workspace_Data/manifest.llm.json
--output-dir /Volumes/Zoe/custom-local-llm/training-data/ \
--manifest /Volumes/Zoe/custom-local-llm/manifest.llm.json
# Full SDG run via Mimir (requires NVIDIA_API_KEY on Mimir):
python3 scripts/generate_training_data.py \
--dictionary-dir dictionary/ \
--output-dir ../workspace_Data/data/ \
--manifest ../workspace_Data/manifest.llm.json \
--output-dir /Volumes/Zoe/custom-local-llm/training-data/ \
--manifest /Volumes/Zoe/custom-local-llm/manifest.llm.json \
--sdg-host 100.80.52.47 \
--sdg-user aaron-pressey \
--sdg-key ~/.ssh/silma_orson_ed25519 \
@@ -206,6 +207,9 @@ python3 scripts/generate_training_data.py \
# Dry run (prints what would be generated, writes nothing):
python3 scripts/generate_training_data.py --dry-run
# Woodpecker deploy target:
scripts/deploy_training_data.sh
```
---