mp-ai-module-controller/SDG_BRIDGE_PLAN.md

# SDG Bridge Plan

## Bridge Script Filename

```
scripts/generate_training_data.py
```

This script reads the generated dictionary artifacts from this project, optionally runs a
NeMo Data Designer expansion pass on Mimir via SSH, and writes a timestamped raw training
dataset into `/Volumes/Zoe/custom-local-llm/training-data`, then registers it in
`/Volumes/Zoe/custom-local-llm/manifest.llm.json` so local training tools can point at one
stable artifact root.

---

## Mimir — Data Designer Status

Mimir (`100.80.52.47`) has **Data Designer 0.8.0** installed and ready:

| Component | Location | Status |
|-----------|----------|--------|
| CLI | `/home/aaron-pressey/.venvs/data-designer/bin/data-designer` | ✅ installed |
| Backend adapter | `/home/aaron-pressey/.local/share/home-grown-llm-data/nemo_data_designer_backend.py` | ✅ present |
| Storage | `/mnt/storage/data-designer/` | ✅ mounted |
| Model configs | `/mnt/storage/data-designer/model_configs.yaml` | ✅ configured |
| Config env | `/home/aaron-pressey/.config/home-grown-llm-data/data-designer.env` | ✅ present |

**Current model provider:** NVIDIA API through the `nvidia-text` alias, mapped to
`nvidia/nemotron-3-nano-30b-a3b`. Mimir now has `NVIDIA_API_KEY` in its Data Designer env file.
Use `MIMIR_SDG_MODEL_ALIAS=openrouter-text` only as a fallback. Local llama.cpp on port 8081 is
configured but not currently running.

---

## What the Bridge Does

```
Step 1 (local — this project):
  dictionary/actions.index.json     ─┐
  dictionary/model.codebook.json     ├─► generate_training_data.py ─► seed JSONL
  dictionary/model.examples.jsonl   ─┘     (20 records/action, 260 total)
                                               │
Step 2 (remote — Mimir, optional):             │
  SCP seed JSONL + DD config to Mimir ◄────────┘
  SSH: source Data Designer env, then data-designer create module_controller.py
         --num-records <N>
         --dataset-name module_controller_intents_<timestamp>
         --artifact-path /mnt/storage/data-designer/managed-assets/module_controller_artifacts_<timestamp>
         --output-format jsonl
         --no-tui
  SCP expanded dataset back
                                               │
Step 3 (local — Zoe custom-local-llm):         │
  Write to training-data/module_controller_intents_<timestamp>/ ◄─────┘
  Patch manifest.llm.json with timestamped entry
  Training tools pick it up from the Zoe artifact root
```

---

## Timestamp Tagging

Every dataset generated by this script is tagged with a **Unix timestamp at generation time**.
The timestamp is embedded in:

1. **Output directory name:** `training-data/module_controller_intents_<unix_ts>/`
2. **Manifest entry key:** `"module_controller_intents_<unix_ts>"`
3. **Manifest entry field:** `"generated_at": <unix_ts>`
4. **Every JSONL record's metadata:** `"generated_at": <unix_ts>`

This means you can run the bridge multiple times (after adding actions, changing examples,
or expanding via NeMo) and the Training Monitor will show each run as a distinct, traceable
Data Source. No clobbering, no ambiguity.

---

## Output Record Schema

Each output record uses the `instruction` schema profile:

```json
{
  "instruction": "Map the following natural language request to the correct action and payload.",
  "input": "hash hello with sha256",
  "output": "{\"action_id\": \"hash_string\", \"payload\": {\"text\": \"hello\", \"algorithm\": \"sha256\"}}",
  "metadata": {
    "action_id": "hash_string",
    "compact": "{\"v\":\"1.0.0\",\"c\":\"<codebook_checksum>\",\"a\":\"A007\",\"p\":{\"F010\":\"hello\",\"F001\":\"sha256\"}}",
    "source": "seed_intent_example",
    "split": "train",
    "generated_at": 1784660000
  }
}
```

---

## Generation Strategy

For each action in `actions.index.json`:

1. **Seed examples** — emit each `intent_example` directly (source: `seed_intent_example`)
2. **Alias variants** — one phrase per alias using the alias as the verb (source: `alias_variant`)
3. **Payload field variants** — for enum fields, one record per enum value; for numeric fields,
   low/mid/high values (source: `field_variant`)
4. **Template expansion** — 5 records per action from `synthetic_template` with filled placeholders
   (source: `template_expansion`)

**Minimum seed target:** ~20 records × 13 actions = 260 records.
**After NeMo expansion:** configurable via `--num-records` (default: 1000).

Split: 90% train / 10% validation, partitioned by action so all 13 actions appear in both splits.

The deterministic seed files are intentionally small. They are good for validating the action
surface and dispatch schema, but they are not diverse enough for serious payload extraction
training by themselves. Use the Mimir-expanded `expanded.jsonl` file when training a model
that needs to generalize across phrasing, payload values, and action families.

---

## Mimir Data Designer Config (auto-generated by bridge)

The bridge writes this config to Mimir before running generation:

```python
# auto-generated by generate_training_data.py - do not edit by hand
# generated_at: <unix_ts>

from data_designer.config import DataDesignerConfigBuilder, LocalFileSeedSource


ACTION_IDS = ["calculate", "..."]


def load_config_builder():
    builder = DataDesignerConfigBuilder(
        model_configs="/mnt/storage/data-designer/model_configs.yaml",
    )
    builder.with_seed_dataset(
        LocalFileSeedSource(
            path="/mnt/storage/data-designer/managed-assets/module_controller_seed_<unix_ts>.jsonl"
        )
    )
    builder.add_column(
        name="expanded_record",
        column_type="llm-structured",
        model_alias="nvidia-text",
        prompt="Generate one new strict instruction dataset row from the seed input/output.",
        output_format={
            "type": "object",
            "required": ["instruction", "input", "output", "metadata"],
            "properties": {
                "instruction": {"type": "string"},
                "input": {"type": "string"},
                "output": {"type": "string"},
                "metadata": {
                    "type": "object",
                    "required": ["action_id", "source", "split", "generated_at"],
                    "properties": {
                        "action_id": {"type": "string", "enum": ACTION_IDS}
                    },
                },
            },
        },
    )
    return builder
```

Data Designer writes rows that include the generated `expanded_record` column. The bridge
copies that raw file back as `expanded.raw.jsonl`, extracts valid instruction rows, and writes
the model-training file as `expanded.jsonl`.

---

## Manifest Entry (auto-patched by bridge)

```json
"module_controller_intents_<unix_ts>": {
  "name": "Module Controller Intents",
  "category": "synthetic_sft",
  "schema_profile": "instruction",
  "source_url": null,
  "acquisition_method": "local_synthetic_generator",
  "license_spdx": "UNLICENSED",
  "redistribution_allowed": false,
  "estimated_tokens": null,
  "status": "generated_local",
  "local_path": "training-data/module_controller_intents_<unix_ts>",
  "local_format": "jsonl",
  "generator": "mp-ai-module-controller/scripts/generate_training_data.py",
  "sdg_engine": "data-designer",
  "sdg_host": "mimir (100.80.52.47)",
  "sdg_model": "nvidia-text",
  "sdg_model_alias": "nvidia-text",
  "expanded_path": "/Volumes/Zoe/custom-local-llm/training-data/module_controller_intents_<unix_ts>/expanded.jsonl",
  "codebook_version": "1.0.0",
  "codebook_checksum": "<from model.codebook.json>",
  "registry_checksum": "<from actions.index.json>",
  "generated_at": <unix_ts>,
  "doc": "data/module_controller_intents_<unix_ts>/README.md"
}
```

---

## CLI Usage

```bash
# From mp-ai-module-controller project root:

# Seed only (no NeMo expansion — fast, local, 260 records):
python3 scripts/generate_training_data.py \
  --dictionary-dir dictionary/ \
  --output-dir /Volumes/Zoe/custom-local-llm/training-data/ \
  --manifest /Volumes/Zoe/custom-local-llm/manifest.llm.json

# Full SDG run via Mimir (requires NVIDIA_API_KEY on Mimir):
python3 scripts/generate_training_data.py \
  --dictionary-dir dictionary/ \
  --output-dir /Volumes/Zoe/custom-local-llm/training-data/ \
  --manifest /Volumes/Zoe/custom-local-llm/manifest.llm.json \
  --sdg-host 100.80.52.47 \
  --sdg-user aaron-pressey \
  --sdg-key ~/.ssh/silma_orson_ed25519 \
  --sdg-model-alias nvidia-text \
  --num-records 1000

# Dry run (prints what would be generated, writes nothing):
python3 scripts/generate_training_data.py --dry-run

# Woodpecker deploy target:
ENABLE_MIMIR_SDG=1 scripts/deploy_training_data.sh
```

---

## Prerequisites

- Python 3.11+ (stdlib only: `json`, `pathlib`, `hashlib`, `time`, `argparse`, `subprocess`)
- SSH access to Mimir via `~/.ssh/silma_orson_ed25519` (for NeMo expansion pass)
- `/home/aaron-pressey/.config/home-grown-llm-data/data-designer.env` present on Mimir with provider credentials
- Data Designer provider available for the selected alias. Current default: `nvidia-text`.
- `dictionary/` files current — run `cargo run -- dictionary generate` first if registry changed

---

## Staleness Check

The manifest entry embeds both `codebook_checksum` and `generated_at`. If you add actions:

```bash
cargo run -- dictionary generate          # updates codebook_checksum
python3 scripts/generate_training_data.py # new timestamp → new manifest entry
```

The Training Monitor will show both the old and new dataset. You can deprecate the old one
by setting its `status` to `"superseded"` in the manifest.