This commit is contained in:
+15
-11
@@ -8,8 +8,9 @@ scripts/generate_training_data.py
|
||||
|
||||
This script reads the generated dictionary artifacts from this project, optionally runs a
|
||||
NeMo Data Designer expansion pass on Mimir via SSH, and writes a timestamped raw training
|
||||
dataset into the `workspace_Data` pipeline, then registers it in `manifest.llm.json` so the
|
||||
Training Monitor picks it up automatically.
|
||||
dataset into `/Volumes/Zoe/custom-local-llm/training-data`, then registers it in
|
||||
`/Volumes/Zoe/custom-local-llm/manifest.llm.json` so local training tools can point at one
|
||||
stable artifact root.
|
||||
|
||||
---
|
||||
|
||||
@@ -46,10 +47,10 @@ Step 2 (remote — Mimir, optional): │
|
||||
--dataset-name module_controller_intents_<timestamp>
|
||||
SCP expanded dataset back
|
||||
│
|
||||
Step 3 (local — workspace_Data): │
|
||||
Write to data/module_controller_intents_<timestamp>/ ◄──────────────┘
|
||||
Step 3 (local — Zoe custom-local-llm): │
|
||||
Write to training-data/module_controller_intents_<timestamp>/ ◄─────┘
|
||||
Patch manifest.llm.json with timestamped entry
|
||||
Training Monitor picks it up as a new Data Source
|
||||
Training tools pick it up from the Zoe artifact root
|
||||
```
|
||||
|
||||
---
|
||||
@@ -59,7 +60,7 @@ Step 3 (local — workspace_Data): │
|
||||
Every dataset generated by this script is tagged with a **Unix timestamp at generation time**.
|
||||
The timestamp is embedded in:
|
||||
|
||||
1. **Output directory name:** `data/module_controller_intents_<unix_ts>/`
|
||||
1. **Output directory name:** `training-data/module_controller_intents_<unix_ts>/`
|
||||
2. **Manifest entry key:** `"module_controller_intents_<unix_ts>"`
|
||||
3. **Manifest entry field:** `"generated_at": <unix_ts>`
|
||||
4. **Every JSONL record's metadata:** `"generated_at": <unix_ts>`
|
||||
@@ -167,7 +168,7 @@ columns:
|
||||
"redistribution_allowed": false,
|
||||
"estimated_tokens": null,
|
||||
"status": "generated_local",
|
||||
"local_path": "data/module_controller_intents_<unix_ts>",
|
||||
"local_path": "training-data/module_controller_intents_<unix_ts>",
|
||||
"local_format": "jsonl",
|
||||
"generator": "mp-ai-module-controller/scripts/generate_training_data.py",
|
||||
"sdg_engine": "data-designer",
|
||||
@@ -191,14 +192,14 @@ columns:
|
||||
# Seed only (no NeMo expansion — fast, local, 260 records):
|
||||
python3 scripts/generate_training_data.py \
|
||||
--dictionary-dir dictionary/ \
|
||||
--output-dir ../workspace_Data/data/ \
|
||||
--manifest ../workspace_Data/manifest.llm.json
|
||||
--output-dir /Volumes/Zoe/custom-local-llm/training-data/ \
|
||||
--manifest /Volumes/Zoe/custom-local-llm/manifest.llm.json
|
||||
|
||||
# Full SDG run via Mimir (requires NVIDIA_API_KEY on Mimir):
|
||||
python3 scripts/generate_training_data.py \
|
||||
--dictionary-dir dictionary/ \
|
||||
--output-dir ../workspace_Data/data/ \
|
||||
--manifest ../workspace_Data/manifest.llm.json \
|
||||
--output-dir /Volumes/Zoe/custom-local-llm/training-data/ \
|
||||
--manifest /Volumes/Zoe/custom-local-llm/manifest.llm.json \
|
||||
--sdg-host 100.80.52.47 \
|
||||
--sdg-user aaron-pressey \
|
||||
--sdg-key ~/.ssh/silma_orson_ed25519 \
|
||||
@@ -206,6 +207,9 @@ python3 scripts/generate_training_data.py \
|
||||
|
||||
# Dry run (prints what would be generated, writes nothing):
|
||||
python3 scripts/generate_training_data.py --dry-run
|
||||
|
||||
# Woodpecker deploy target:
|
||||
scripts/deploy_training_data.sh
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
Reference in New Issue
Block a user