259 lines
9.4 KiB
Markdown
259 lines
9.4 KiB
Markdown
# SDG Bridge Plan
|
||
|
||
## Bridge Script Filename
|
||
|
||
```
|
||
scripts/generate_training_data.py
|
||
```
|
||
|
||
This script reads the generated dictionary artifacts from this project, optionally runs a
|
||
NeMo Data Designer expansion pass on Mimir via SSH, and writes a timestamped raw training
|
||
dataset into `/Volumes/Zoe/custom-local-llm/training-data`, then registers it in
|
||
`/Volumes/Zoe/custom-local-llm/manifest.llm.json` so local training tools can point at one
|
||
stable artifact root.
|
||
|
||
---
|
||
|
||
## Mimir — Data Designer Status
|
||
|
||
Mimir (`100.80.52.47`) has **Data Designer 0.8.0** installed and ready:
|
||
|
||
| Component | Location | Status |
|
||
|-----------|----------|--------|
|
||
| CLI | `/home/aaron-pressey/.venvs/data-designer/bin/data-designer` | ✅ installed |
|
||
| Backend adapter | `/home/aaron-pressey/.local/share/home-grown-llm-data/nemo_data_designer_backend.py` | ✅ present |
|
||
| Storage | `/mnt/storage/data-designer/` | ✅ mounted |
|
||
| Model configs | `/mnt/storage/data-designer/model_configs.yaml` | ✅ configured |
|
||
| Config env | `/home/aaron-pressey/.config/home-grown-llm-data/data-designer.env` | ✅ present |
|
||
|
||
**Current model provider:** NVIDIA API through the `nvidia-text` alias, mapped to
|
||
`nvidia/nemotron-3-nano-30b-a3b`. Mimir now has `NVIDIA_API_KEY` in its Data Designer env file.
|
||
Use `MIMIR_SDG_MODEL_ALIAS=openrouter-text` only as a fallback. Local llama.cpp on port 8081 is
|
||
configured but not currently running.
|
||
|
||
---
|
||
|
||
## What the Bridge Does
|
||
|
||
```
|
||
Step 1 (local — this project):
|
||
dictionary/actions.index.json ─┐
|
||
dictionary/model.codebook.json ├─► generate_training_data.py ─► seed JSONL
|
||
dictionary/model.examples.jsonl ─┘ (20 records/action, 260 total)
|
||
│
|
||
Step 2 (remote — Mimir, optional): │
|
||
SCP seed JSONL + DD config to Mimir ◄────────┘
|
||
SSH: source Data Designer env, then data-designer create module_controller.py
|
||
--num-records <N>
|
||
--dataset-name module_controller_intents_<timestamp>
|
||
--artifact-path /mnt/storage/data-designer/managed-assets/module_controller_artifacts_<timestamp>
|
||
--output-format jsonl
|
||
--no-tui
|
||
SCP expanded dataset back
|
||
│
|
||
Step 3 (local — Zoe custom-local-llm): │
|
||
Write to training-data/module_controller_intents_<timestamp>/ ◄─────┘
|
||
Patch manifest.llm.json with timestamped entry
|
||
Training tools pick it up from the Zoe artifact root
|
||
```
|
||
|
||
---
|
||
|
||
## Timestamp Tagging
|
||
|
||
Every dataset generated by this script is tagged with a **Unix timestamp at generation time**.
|
||
The timestamp is embedded in:
|
||
|
||
1. **Output directory name:** `training-data/module_controller_intents_<unix_ts>/`
|
||
2. **Manifest entry key:** `"module_controller_intents_<unix_ts>"`
|
||
3. **Manifest entry field:** `"generated_at": <unix_ts>`
|
||
4. **Every JSONL record's metadata:** `"generated_at": <unix_ts>`
|
||
|
||
This means you can run the bridge multiple times (after adding actions, changing examples,
|
||
or expanding via NeMo) and the Training Monitor will show each run as a distinct, traceable
|
||
Data Source. No clobbering, no ambiguity.
|
||
|
||
---
|
||
|
||
## Output Record Schema
|
||
|
||
Each output record uses the `instruction` schema profile:
|
||
|
||
```json
|
||
{
|
||
"instruction": "Map the following natural language request to the correct action and payload.",
|
||
"input": "hash hello with sha256",
|
||
"output": "{\"action_id\": \"hash_string\", \"payload\": {\"text\": \"hello\", \"algorithm\": \"sha256\"}}",
|
||
"metadata": {
|
||
"action_id": "hash_string",
|
||
"compact": "{\"v\":\"1.0.0\",\"c\":\"<codebook_checksum>\",\"a\":\"A007\",\"p\":{\"F010\":\"hello\",\"F001\":\"sha256\"}}",
|
||
"source": "seed_intent_example",
|
||
"split": "train",
|
||
"generated_at": 1784660000
|
||
}
|
||
}
|
||
```
|
||
|
||
---
|
||
|
||
## Generation Strategy
|
||
|
||
For each action in `actions.index.json`:
|
||
|
||
1. **Seed examples** — emit each `intent_example` directly (source: `seed_intent_example`)
|
||
2. **Alias variants** — one phrase per alias using the alias as the verb (source: `alias_variant`)
|
||
3. **Payload field variants** — for enum fields, one record per enum value; for numeric fields,
|
||
low/mid/high values (source: `field_variant`)
|
||
4. **Template expansion** — 5 records per action from `synthetic_template` with filled placeholders
|
||
(source: `template_expansion`)
|
||
|
||
**Minimum seed target:** ~20 records × 13 actions = 260 records.
|
||
**After NeMo expansion:** configurable via `--num-records` (default: 1000).
|
||
|
||
Split: 90% train / 10% validation, partitioned by action so all 13 actions appear in both splits.
|
||
|
||
The deterministic seed files are intentionally small. They are good for validating the action
|
||
surface and dispatch schema, but they are not diverse enough for serious payload extraction
|
||
training by themselves. Use the Mimir-expanded `expanded.jsonl` file when training a model
|
||
that needs to generalize across phrasing, payload values, and action families.
|
||
|
||
---
|
||
|
||
## Mimir Data Designer Config (auto-generated by bridge)
|
||
|
||
The bridge writes this config to Mimir before running generation:
|
||
|
||
```python
|
||
# auto-generated by generate_training_data.py - do not edit by hand
|
||
# generated_at: <unix_ts>
|
||
|
||
from data_designer.config import DataDesignerConfigBuilder, LocalFileSeedSource
|
||
|
||
|
||
ACTION_IDS = ["calculate", "..."]
|
||
|
||
|
||
def load_config_builder():
|
||
builder = DataDesignerConfigBuilder(
|
||
model_configs="/mnt/storage/data-designer/model_configs.yaml",
|
||
)
|
||
builder.with_seed_dataset(
|
||
LocalFileSeedSource(
|
||
path="/mnt/storage/data-designer/managed-assets/module_controller_seed_<unix_ts>.jsonl"
|
||
)
|
||
)
|
||
builder.add_column(
|
||
name="expanded_record",
|
||
column_type="llm-structured",
|
||
model_alias="nvidia-text",
|
||
prompt="Generate one new strict instruction dataset row from the seed input/output.",
|
||
output_format={
|
||
"type": "object",
|
||
"required": ["instruction", "input", "output", "metadata"],
|
||
"properties": {
|
||
"instruction": {"type": "string"},
|
||
"input": {"type": "string"},
|
||
"output": {"type": "string"},
|
||
"metadata": {
|
||
"type": "object",
|
||
"required": ["action_id", "source", "split", "generated_at"],
|
||
"properties": {
|
||
"action_id": {"type": "string", "enum": ACTION_IDS}
|
||
},
|
||
},
|
||
},
|
||
},
|
||
)
|
||
return builder
|
||
```
|
||
|
||
Data Designer writes rows that include the generated `expanded_record` column. The bridge
|
||
copies that raw file back as `expanded.raw.jsonl`, extracts valid instruction rows, and writes
|
||
the model-training file as `expanded.jsonl`.
|
||
|
||
---
|
||
|
||
## Manifest Entry (auto-patched by bridge)
|
||
|
||
```json
|
||
"module_controller_intents_<unix_ts>": {
|
||
"name": "Module Controller Intents",
|
||
"category": "synthetic_sft",
|
||
"schema_profile": "instruction",
|
||
"source_url": null,
|
||
"acquisition_method": "local_synthetic_generator",
|
||
"license_spdx": "UNLICENSED",
|
||
"redistribution_allowed": false,
|
||
"estimated_tokens": null,
|
||
"status": "generated_local",
|
||
"local_path": "training-data/module_controller_intents_<unix_ts>",
|
||
"local_format": "jsonl",
|
||
"generator": "mp-ai-module-controller/scripts/generate_training_data.py",
|
||
"sdg_engine": "data-designer",
|
||
"sdg_host": "mimir (100.80.52.47)",
|
||
"sdg_model": "nvidia-text",
|
||
"sdg_model_alias": "nvidia-text",
|
||
"expanded_path": "/Volumes/Zoe/custom-local-llm/training-data/module_controller_intents_<unix_ts>/expanded.jsonl",
|
||
"codebook_version": "1.0.0",
|
||
"codebook_checksum": "<from model.codebook.json>",
|
||
"registry_checksum": "<from actions.index.json>",
|
||
"generated_at": <unix_ts>,
|
||
"doc": "data/module_controller_intents_<unix_ts>/README.md"
|
||
}
|
||
```
|
||
|
||
---
|
||
|
||
## CLI Usage
|
||
|
||
```bash
|
||
# From mp-ai-module-controller project root:
|
||
|
||
# Seed only (no NeMo expansion — fast, local, 260 records):
|
||
python3 scripts/generate_training_data.py \
|
||
--dictionary-dir dictionary/ \
|
||
--output-dir /Volumes/Zoe/custom-local-llm/training-data/ \
|
||
--manifest /Volumes/Zoe/custom-local-llm/manifest.llm.json
|
||
|
||
# Full SDG run via Mimir (requires NVIDIA_API_KEY on Mimir):
|
||
python3 scripts/generate_training_data.py \
|
||
--dictionary-dir dictionary/ \
|
||
--output-dir /Volumes/Zoe/custom-local-llm/training-data/ \
|
||
--manifest /Volumes/Zoe/custom-local-llm/manifest.llm.json \
|
||
--sdg-host 100.80.52.47 \
|
||
--sdg-user aaron-pressey \
|
||
--sdg-key ~/.ssh/silma_orson_ed25519 \
|
||
--sdg-model-alias nvidia-text \
|
||
--num-records 1000
|
||
|
||
# Dry run (prints what would be generated, writes nothing):
|
||
python3 scripts/generate_training_data.py --dry-run
|
||
|
||
# Woodpecker deploy target:
|
||
ENABLE_MIMIR_SDG=1 scripts/deploy_training_data.sh
|
||
```
|
||
|
||
---
|
||
|
||
## Prerequisites
|
||
|
||
- Python 3.11+ (stdlib only: `json`, `pathlib`, `hashlib`, `time`, `argparse`, `subprocess`)
|
||
- SSH access to Mimir via `~/.ssh/silma_orson_ed25519` (for NeMo expansion pass)
|
||
- `/home/aaron-pressey/.config/home-grown-llm-data/data-designer.env` present on Mimir with provider credentials
|
||
- Data Designer provider available for the selected alias. Current default: `nvidia-text`.
|
||
- `dictionary/` files current — run `cargo run -- dictionary generate` first if registry changed
|
||
|
||
---
|
||
|
||
## Staleness Check
|
||
|
||
The manifest entry embeds both `codebook_checksum` and `generated_at`. If you add actions:
|
||
|
||
```bash
|
||
cargo run -- dictionary generate # updates codebook_checksum
|
||
python3 scripts/generate_training_data.py # new timestamp → new manifest entry
|
||
```
|
||
|
||
The Training Monitor will show both the old and new dataset. You can deprecate the old one
|
||
by setting its `status` to `"superseded"` in the manifest.
|