171 lines
4.5 KiB
Markdown
171 lines
4.5 KiB
Markdown
# Local LLM Guide
|
|
|
|
Two LLM models run locally on Tabitha via llama.cpp with Metal (Apple Silicon GPU) acceleration. Both expose OpenAI-compatible REST APIs — drop-in replacements for `api.openai.com` endpoints.
|
|
|
|
**llama.cpp GitHub:** https://github.com/ggml-org/llama.cpp
|
|
**llama.cpp API docs:** https://github.com/ggml-org/llama.cpp/blob/master/tools/server/README.md
|
|
|
|
---
|
|
|
|
## Running the Servers
|
|
|
|
```bash
|
|
cd /Users/Tabitha/.openclaw
|
|
npm run api:llm
|
|
```
|
|
|
|
This starts both models in the background. Logs go to stdout/stderr.
|
|
|
|
---
|
|
|
|
## Models
|
|
|
|
### Gemma-4-E4B — Fast / Light
|
|
| Field | Value |
|
|
|-------|-------|
|
|
| **Model** | Google Gemma 4 E4B (4-billion parameter, efficient variant) |
|
|
| **Endpoint** | `http://localhost:8080` |
|
|
| **API** | OpenAI-compatible (`/v1/chat/completions`, `/v1/completions`, `/v1/models`) |
|
|
| **Use case** | Quick tasks, code generation, summarization — fast response |
|
|
| **Acceleration** | Apple Metal (GPU) |
|
|
|
|
### Gemma-4-26B — Capable / Reasoning
|
|
| Field | Value |
|
|
|-------|-------|
|
|
| **Model** | Google Gemma 4 27B |
|
|
| **Endpoint** | `http://localhost:8081` |
|
|
| **API** | OpenAI-compatible |
|
|
| **Use case** | Complex reasoning, architecture questions, longer context tasks |
|
|
| **Acceleration** | Apple Metal (GPU) |
|
|
|
|
---
|
|
|
|
## API Usage
|
|
|
|
Both models speak the OpenAI API format:
|
|
|
|
```bash
|
|
# Chat completion — Gemma-4-E4B (fast)
|
|
curl -s http://localhost:8080/v1/chat/completions \
|
|
-H "Content-Type: application/json" \
|
|
-d '{
|
|
"model": "gemma-4-e4b",
|
|
"messages": [{"role": "user", "content": "Explain this code: ..."}]
|
|
}' | python3 -m json.tool
|
|
|
|
# Chat completion — Gemma-4-26B (capable)
|
|
curl -s http://localhost:8081/v1/chat/completions \
|
|
-H "Content-Type: application/json" \
|
|
-d '{
|
|
"model": "gemma-4-26b",
|
|
"messages": [{"role": "user", "content": "Design a REST API for..."}]
|
|
}' | python3 -m json.tool
|
|
|
|
# List available models
|
|
curl -s http://localhost:8080/v1/models
|
|
curl -s http://localhost:8081/v1/models
|
|
```
|
|
|
|
---
|
|
|
|
## Python (openai SDK)
|
|
|
|
```python
|
|
from openai import OpenAI
|
|
|
|
# Fast model
|
|
fast = OpenAI(base_url="http://localhost:8080/v1", api_key="local")
|
|
response = fast.chat.completions.create(
|
|
model="gemma-4-e4b",
|
|
messages=[{"role": "user", "content": "Write a Python function to..."}]
|
|
)
|
|
print(response.choices[0].message.content)
|
|
|
|
# Capable model
|
|
smart = OpenAI(base_url="http://localhost:8081/v1", api_key="local")
|
|
response = smart.chat.completions.create(
|
|
model="gemma-4-26b",
|
|
messages=[{"role": "user", "content": "Architect a microservice system for..."}],
|
|
temperature=0.7,
|
|
max_tokens=2048
|
|
)
|
|
print(response.choices[0].message.content)
|
|
```
|
|
|
|
---
|
|
|
|
## Using with AI Coding Tools
|
|
|
|
### Claude Code CLI
|
|
```bash
|
|
export ANTHROPIC_BASE_URL=http://localhost:8080/v1
|
|
export ANTHROPIC_API_KEY=local
|
|
claude --model gemma-4-e4b
|
|
```
|
|
|
|
### Codex CLI
|
|
```bash
|
|
OPENAI_BASE_URL=http://localhost:8080/v1 \
|
|
OPENAI_API_KEY=local \
|
|
codex --model gemma-4-e4b "explain this codebase"
|
|
```
|
|
|
|
### Cursor (IDE)
|
|
Settings → Models → Add Model:
|
|
- Base URL: `http://localhost:8080/v1`
|
|
- API Key: `local`
|
|
- Model name: `gemma-4-e4b`
|
|
|
|
Repeat for port 8081 / `gemma-4-26b`.
|
|
|
|
### LangChain / any OpenAI-compatible client
|
|
```python
|
|
from langchain_openai import ChatOpenAI
|
|
|
|
llm = ChatOpenAI(
|
|
base_url="http://localhost:8080/v1",
|
|
api_key="local",
|
|
model="gemma-4-e4b"
|
|
)
|
|
```
|
|
|
|
---
|
|
|
|
## Health Check
|
|
|
|
```bash
|
|
# Check if servers are responding
|
|
curl -s http://localhost:8080/health && echo " — E4B OK"
|
|
curl -s http://localhost:8081/health && echo " — 26B OK"
|
|
|
|
# Or check model list
|
|
curl -s http://localhost:8080/v1/models | python3 -c "import sys,json; [print(m['id']) for m in json.load(sys.stdin)['data']]"
|
|
```
|
|
|
|
---
|
|
|
|
## Model Storage
|
|
|
|
Models are stored on Theodora (hot SSD cache):
|
|
```
|
|
~/.cache/huggingface → /Volumes/Theodora/models/huggingface (symlink)
|
|
~/.cache/llama.cpp → /Volumes/Theodora/models/llama-cpp (symlink)
|
|
```
|
|
|
|
Archive copies on Hagia:
|
|
```
|
|
/Volumes/Hagia/models/huggingface/
|
|
/Volumes/Hagia/models/llama-cpp/
|
|
```
|
|
|
|
---
|
|
|
|
## Notes for AI Agents
|
|
|
|
- The servers run **on the host** (not in Docker/k8s) — they are accessible from any process on Tabitha
|
|
- Theodora (SSD) must be mounted for fast model loading — `ls /Volumes/Theodora` to verify
|
|
- If a server is not responding, run `cd /Users/Tabitha/.openclaw && npm run api:llm` to start it
|
|
- **Gemma-4-E4B** is the default fast option — use it for code tasks, summarization, quick Q&A
|
|
- **Gemma-4-26B** costs more compute — use it for complex reasoning, architecture, long context
|
|
- Both are private and local — no data leaves the machine
|