feat: enrich training deploy via mimir sdg
CI / validate (push) Successful in 37s

This commit is contained in:
2026-07-22 09:37:40 -05:00
parent 8a697cf64f
commit fec9e5239b
11 changed files with 370 additions and 91 deletions
+2 -1
View File
@@ -33,6 +33,7 @@ Commands:
- `scripts/precommit-check.sh`
- `scripts/generate_training_data.py --dry-run`
- `scripts/deploy_training_data.sh`
- `ENABLE_MIMIR_SDG=1 scripts/deploy_training_data.sh`
- `git config core.hooksPath .githooks`
- `.codex/environments/environment.toml`
@@ -60,4 +61,4 @@ Codex environment: `.codex/environments/environment.toml` defines setup plus Bui
CI and hooks: `.gitea/workflows/ci.yml`, `.woodpecker.yml`, and `.githooks/pre-commit` all use `scripts/precommit-check.sh` to verify formatting, tests, and dictionary generation.
SDG bridge: `scripts/generate_training_data.py` reads `dictionary/actions.index.json`, `dictionary/model.codebook.json`, and `dictionary/model.examples.jsonl`, then writes instruction-format datasets to `/Volumes/Zoe/custom-local-llm/training-data/module_controller_intents_<unix_ts>/` and patches `/Volumes/Zoe/custom-local-llm/manifest.llm.json`. Use `--dry-run` to validate without writes. Optional Mimir expansion requires `--sdg-host`, `--sdg-user`, and `--sdg-key`; failed remote expansion leaves local seeds and writes `SDG_ERROR.txt`. Woodpecker deploys the export with `scripts/deploy_training_data.sh`.
SDG bridge: `scripts/generate_training_data.py` reads `dictionary/actions.index.json`, `dictionary/model.codebook.json`, and `dictionary/model.examples.jsonl`, then writes instruction-format datasets to `/Volumes/Zoe/custom-local-llm/training-data/module_controller_intents_<unix_ts>/` and patches `/Volumes/Zoe/custom-local-llm/manifest.llm.json`. Use `--dry-run` to validate without writes. Seed-only `train.jsonl` and `validation.jsonl` are useful for routing checks but need enrichment for serious training. Optional Mimir expansion requires `--sdg-host`, `--sdg-user`, and `--sdg-key`; it runs Data Designer over SSH, sources Mimir's Data Designer env file, uses `--sdg-model-alias` default `nvidia-text`, copies back `expanded.raw.jsonl`, and normalizes it to `expanded.jsonl`. Failed remote expansion leaves local seeds and writes `SDG_ERROR.txt`. Woodpecker deploys enriched data with `ENABLE_MIMIR_SDG=1 scripts/deploy_training_data.sh`.