mp-ai-module-controller/SDG_BRIDGE_PLAN.md

SDG Bridge Plan

Bridge Script Filename

scripts/generate_training_data.py

This script reads the generated dictionary artifacts from this project, optionally runs a NeMo Data Designer expansion pass on Mimir via SSH, and writes a timestamped raw training dataset into /Volumes/Zoe/custom-local-llm/training-data, then registers it in /Volumes/Zoe/custom-local-llm/manifest.llm.json so local training tools can point at one stable artifact root.

---

Mimir — Data Designer Status

Mimir (100.80.52.47) has Data Designer 0.8.0 installed and ready:

Component	Location	Status
CLI	/home/aaron-pressey/.venvs/data-designer/bin/data-designer	✅ installed
Backend adapter	/home/aaron-pressey/.local/share/home-grown-llm-data/nemo_data_designer_backend.py	✅ present
Storage	/mnt/storage/data-designer/	✅ mounted
Model configs	/mnt/storage/data-designer/model_configs.yaml	✅ configured
Config env	/home/aaron-pressey/.config/home-grown-llm-data/data-designer.env	✅ present

Current model provider: NVIDIA API through the nvidia-text alias, mapped to nvidia/nemotron-3-nano-30b-a3b. Mimir now has NVIDIA_API_KEY in its Data Designer env file. Use MIMIR_SDG_MODEL_ALIAS=openrouter-text only as a fallback. Local llama.cpp on port 8081 is configured but not currently running.

---

What the Bridge Does

Step 1 (local — this project):
  dictionary/actions.index.json     ─┐
  dictionary/model.codebook.json     ├─► generate_training_data.py ─► seed JSONL
  dictionary/model.examples.jsonl   ─┘     (20 records/action, 260 total)
                                               │
Step 2 (remote — Mimir, optional):             │
  SCP seed JSONL + DD config to Mimir ◄────────┘
  SSH: source Data Designer env, then data-designer create module_controller.py
         --num-records <N>
         --dataset-name module_controller_intents_<timestamp>
         --artifact-path /mnt/storage/data-designer/managed-assets/module_controller_artifacts_<timestamp>
         --output-format jsonl
         --no-tui
  SCP expanded dataset back
                                               │
Step 3 (local — Zoe custom-local-llm):         │
  Write to training-data/module_controller_intents_<timestamp>/ ◄─────┘
  Patch manifest.llm.json with timestamped entry
  Training tools pick it up from the Zoe artifact root

---

Timestamp Tagging

Every dataset generated by this script is tagged with a Unix timestamp at generation time. The timestamp is embedded in:

1. Output directory name: training-data/module_controller_intents_<unix_ts>/
2. Manifest entry key: "module_controller_intents_<unix_ts>"
3. Manifest entry field: "generated_at": <unix_ts>
4. Every JSONL record's metadata: "generated_at": <unix_ts>

This means you can run the bridge multiple times (after adding actions, changing examples, or expanding via NeMo) and the Training Monitor will show each run as a distinct, traceable Data Source. No clobbering, no ambiguity.

---

Output Record Schema

Each output record uses the instruction schema profile:

{
  "instruction": "Map the following natural language request to the correct action and payload.",
  "input": "hash hello with sha256",
  "output": "{\"action_id\": \"hash_string\", \"payload\": {\"text\": \"hello\", \"algorithm\": \"sha256\"}}",
  "metadata": {
    "action_id": "hash_string",
    "compact": "{\"v\":\"1.0.0\",\"c\":\"<codebook_checksum>\",\"a\":\"A007\",\"p\":{\"F010\":\"hello\",\"F001\":\"sha256\"}}",
    "source": "seed_intent_example",
    "split": "train",
    "generated_at": 1784660000
  }
}

---

Generation Strategy

For each action in actions.index.json:

1. Seed examples — emit each intent_example directly (source: seed_intent_example)
2. Alias variants — one phrase per alias using the alias as the verb (source: alias_variant)
3. Payload field variants — for enum fields, one record per enum value; for numeric fields, low/mid/high values (source: field_variant)
4. Template expansion — 5 records per action from synthetic_template with filled placeholders (source: template_expansion)

Minimum seed target: ~20 records × 13 actions = 260 records. After NeMo expansion: configurable via --num-records (default: 1000).

Split: 90% train / 10% validation, partitioned by action so all 13 actions appear in both splits.

The deterministic seed files are intentionally small. They are good for validating the action surface and dispatch schema, but they are not diverse enough for serious payload extraction training by themselves. Use the Mimir-expanded expanded.jsonl file when training a model that needs to generalize across phrasing, payload values, and action families.

---

Mimir Data Designer Config (auto-generated by bridge)

The bridge writes this config to Mimir before running generation:

# auto-generated by generate_training_data.py - do not edit by hand
# generated_at: <unix_ts>

from data_designer.config import DataDesignerConfigBuilder, LocalFileSeedSource


ACTION_IDS = ["calculate", "..."]


def load_config_builder():
    builder = DataDesignerConfigBuilder(
        model_configs="/mnt/storage/data-designer/model_configs.yaml",
    )
    builder.with_seed_dataset(
        LocalFileSeedSource(
            path="/mnt/storage/data-designer/managed-assets/module_controller_seed_<unix_ts>.jsonl"
        )
    )
    builder.add_column(
        name="expanded_record",
        column_type="llm-structured",
        model_alias="nvidia-text",
        prompt="Generate one new strict instruction dataset row from the seed input/output.",
        output_format={
            "type": "object",
            "required": ["instruction", "input", "output", "metadata"],
            "properties": {
                "instruction": {"type": "string"},
                "input": {"type": "string"},
                "output": {"type": "string"},
                "metadata": {
                    "type": "object",
                    "required": ["action_id", "source", "split", "generated_at"],
                    "properties": {
                        "action_id": {"type": "string", "enum": ACTION_IDS}
                    },
                },
            },
        },
    )
    return builder

Data Designer writes rows that include the generated expanded_record column. The bridge copies that raw file back as expanded.raw.jsonl, extracts valid instruction rows, and writes the model-training file as expanded.jsonl.

---

Manifest Entry (auto-patched by bridge)

"module_controller_intents_<unix_ts>": {
  "name": "Module Controller Intents",
  "category": "synthetic_sft",
  "schema_profile": "instruction",
  "source_url": null,
  "acquisition_method": "local_synthetic_generator",
  "license_spdx": "UNLICENSED",
  "redistribution_allowed": false,
  "estimated_tokens": null,
  "status": "generated_local",
  "local_path": "training-data/module_controller_intents_<unix_ts>",
  "local_format": "jsonl",
  "generator": "mp-ai-module-controller/scripts/generate_training_data.py",
  "sdg_engine": "data-designer",
  "sdg_host": "mimir (100.80.52.47)",
  "sdg_model": "nvidia-text",
  "sdg_model_alias": "nvidia-text",
  "expanded_path": "/Volumes/Zoe/custom-local-llm/training-data/module_controller_intents_<unix_ts>/expanded.jsonl",
  "codebook_version": "1.0.0",
  "codebook_checksum": "<from model.codebook.json>",
  "registry_checksum": "<from actions.index.json>",
  "generated_at": <unix_ts>,
  "doc": "data/module_controller_intents_<unix_ts>/README.md"
}

---

CLI Usage

# From mp-ai-module-controller project root:

# Seed only (no NeMo expansion — fast, local, 260 records):
python3 scripts/generate_training_data.py \
  --dictionary-dir dictionary/ \
  --output-dir /Volumes/Zoe/custom-local-llm/training-data/ \
  --manifest /Volumes/Zoe/custom-local-llm/manifest.llm.json

# Full SDG run via Mimir (requires NVIDIA_API_KEY on Mimir):
python3 scripts/generate_training_data.py \
  --dictionary-dir dictionary/ \
  --output-dir /Volumes/Zoe/custom-local-llm/training-data/ \
  --manifest /Volumes/Zoe/custom-local-llm/manifest.llm.json \
  --sdg-host 100.80.52.47 \
  --sdg-user aaron-pressey \
  --sdg-key ~/.ssh/silma_orson_ed25519 \
  --sdg-model-alias nvidia-text \
  --num-records 1000

# Dry run (prints what would be generated, writes nothing):
python3 scripts/generate_training_data.py --dry-run

# Woodpecker deploy target:
ENABLE_MIMIR_SDG=1 scripts/deploy_training_data.sh

---

Prerequisites

• Python 3.11+ (stdlib only: json, pathlib, hashlib, time, argparse, subprocess)
• SSH access to Mimir via ~/.ssh/silma_orson_ed25519 (for NeMo expansion pass)
• /home/aaron-pressey/.config/home-grown-llm-data/data-designer.env present on Mimir with provider credentials
• Data Designer provider available for the selected alias. Current default: nvidia-text.
• dictionary/ files current — run cargo run -- dictionary generate first if registry changed

---

Staleness Check

The manifest entry embeds both codebook_checksum and generated_at. If you add actions:

cargo run -- dictionary generate          # updates codebook_checksum
python3 scripts/generate_training_data.py # new timestamp → new manifest entry

The Training Monitor will show both the old and new dataset. You can deprecate the old one by setting its status to "superseded" in the manifest.