jacob-mathison/mp-ai-module-controller
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Activity

feat: enrich training deploy via mimir sdg
CI / validate (push) Successful in 37s
Details

Browse Source
This commit is contained in:
Jacob Mathison
2026-07-22 09:37:40 -05:00
parent 8a697cf64f
commit fec9e5239b
11 changed files with 370 additions and 91 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+9 -3
View File
@@ -27,6 +27,7 @@ cargo run -- run write_note --payload '{"filename":"hello.txt","content":"hello"
scripts/precommit-check.sh
python3 scripts/generate_training_data.py --dry-run
scripts/deploy_training_data.sh
ENABLE_MIMIR_SDG=1 scripts/deploy_training_data.sh
```
`serve` starts:
@@ -40,6 +41,9 @@ Optional environment variables:
- `LLAMA_CONTEXT_SIZE`: appended as `--ctx-size`.
- `LLAMA_EXTRA_ARGS`: shell-split and appended to `llama-server`.
- `CONTROLLER_LOG_LEVEL`: tracing log level, default `info`.
- `ENABLE_MIMIR_SDG`: set to `1` for Mimir/Data Designer expansion during training-data deploy.
- `MIMIR_SDG_HOST`, `MIMIR_SDG_USER`, `MIMIR_SDG_KEY`, `MIMIR_SDG_MODEL_ALIAS`, `MIMIR_SDG_NUM_RECORDS`: SDG deploy overrides.
- `NVIDIA_API_KEY`: optional local Data Designer provider credential. Keep it in ignored `.env` only.
## Dictionary
@@ -134,12 +138,14 @@ python3 scripts/generate_training_data.py \
--manifest /Volumes/Zoe/custom-local-llm/manifest.llm.json
```
Optional Mimir/Data Designer expansion is enabled only when `--sdg-host`, `--sdg-user`, and `--sdg-key` are supplied. The bridge does not start or restart Mimir inference services.
Optional Mimir/Data Designer expansion is enabled only when `--sdg-host`, `--sdg-user`, and `--sdg-key` are supplied. The deploy helper turns this on with `ENABLE_MIMIR_SDG=1` and defaults to Mimir at `100.80.52.47`, user `aaron-pressey`, key `~/.ssh/silma_orson_ed25519`, model alias `nvidia-text`, and `1000` expanded rows. Override those with `MIMIR_SDG_HOST`, `MIMIR_SDG_USER`, `MIMIR_SDG_KEY`, `MIMIR_SDG_MODEL_ALIAS`, and `MIMIR_SDG_NUM_RECORDS`.
Remote expansion uses SSH to run Mimir's Data Designer CLI. It sources `/home/aaron-pressey/.config/home-grown-llm-data/data-designer.env`, uses the configured model alias, copies back `expanded.raw.jsonl`, and normalizes it to `expanded.jsonl`. The bridge does not start or restart Mimir inference services.
If remote expansion fails, the timestamped local seed dataset remains in place, `SDG_ERROR.txt` is written in that dataset directory, and the manifest entry is marked `generated_local_remote_failed`.
Woodpecker deploys a new training-data export to Zoe on `main` pushes by running:
Woodpecker deploys a new enriched training-data export to Zoe on `main` pushes by running:
```sh
scripts/deploy_training_data.sh
ENABLE_MIMIR_SDG=1 scripts/deploy_training_data.sh
```