feat(vision): add vision tool using MiniCPM-V 4.6 (1.3B) via llama-mtmd-cli

[jkyberneees]

Summary

• Adds a vision built-in tool that analyses images and videos locally using MiniCPM-V 4.6 (1.3B multimodal model) via llama-mtmd-cli — no cloud API, no external service
• Docker image gains a new minicpm multi-stage build that downloads a pre-built llama-mtmd-cli binary (llama.cpp b9549, amd64/arm64) and fetches the model GGUF (Q4_K_M, 529 MB) + vision projector (mmproj, 1.1 GB) from HuggingFace at build time
• New VisionConfig added to the config system; all existing builtinTools() call sites updated; vision registered as a built-in tool alongside transcribe

What the tool does

Input	Behaviour
Image (JPEG, PNG, GIF, WebP, BMP)	Base64-encodes and sends to llama-mtmd-cli in single-turn mode
Video (MP4, MOV, AVI, MKV, WebM)	ffprobe reads duration → ffmpeg extracts N evenly-spaced frames → multi-image call with all frames

Output is wrapped in wrapUntrusted() (same as transcribe) — classified as always-untrusted in the security model.

Config

{
  "vision": {
    "models_dir": "~/.odek/minicpm-v/models",
    "binary_path": "/usr/local/bin/llama-mtmd-cli",
    "video_frames": 8
  }
}

All fields are optional — the Docker image path /usr/local/share/minicpm-v/models/ is auto-detected.

Docker build args

# Use a higher-quality quantization (default: Q4_K_M)
docker compose --profile restricted up --build \
  --build-arg MINICPM_QUANT=Q8_0

# Pin a different llama.cpp release (default: b9549)
--build-arg LLAMA_VERSION=b9600

Test plan

• All 21 packages pass: go test ./... -count=1
• TestVision_* (13 tests) — unit + mock-binary happy paths for images and video
• TestResolveVision_* (3 tests) — config resolver defaults and custom values
• Docker image builds: docker compose --profile restricted up --build (requires internet access to HuggingFace during build)
• Smoke test: pass a local JPEG to the agent and verify vision returns a description
• Smoke test video: pass a short MP4 and verify frames > 0 in the result

🤖 Generated with Claude Code

[cloudflare-workers-and-pages]

Deploying with    Cloudflare Workers

The latest updates on your project. Learn more about integrating Git with Workers.

Status	Name	Latest Commit	Preview URL	Updated (UTC)
✅ Deployment successful! View logs	odek	be67d47	Commit Preview URL Branch Preview URL	Jun 07 2026, 02:03 PM

[jkyberneees]

vprotocol v5.2.7 — Verification Certificate

PR: feat/minicpm-v-vision-tool · head be67d47 · 21 files · +839/-19 LOC
Generator: Claude Sonnet 4.6 (claude-sonnet-4-6) · Class: GeneratedCode + GeneratedTests
Run date: 2026-06-07

---

Nine-Axis Results

Axis	Verdict	Notes
2.1 Semantic Correctness	✅	All 13 tests pass; error paths explicit
2.2 Behavioral Contract	⚠️	No formal spec — PR description is the contract
2.3 Security Surface	✅	O_NOFOLLOW, danger.CheckOperation, wrapUntrusted, exec.Command (no shell injection)
2.4 Structural Integrity	✅	Mirrors transcribe_tool.go; compile-time interface check
2.5 Behavioral Exploration	⚠️	Extension-only video detection; no image size guard (acceptable v1 risk)
2.6 Dependency Integrity	✅ (repaired)	llama-mtmd-cli pinned to b9549; HuggingFace URLs pinned to commit 78e02f0 (was resolve/main/)
2.7 Generator Provenance	⚠️	Code + tests: same model, same session → correlated blind spots possible
2.8 Adversarial Surface	✅	User inputs flow to exec.Command args; output wrapped as untrusted
2.9 Documentation Coverage	✅	CHEATSHEET, SECURITY, CONFIG, TELEGRAM, docker/README, README all updated

---

η Score

Signal	Weight	Value
m (mutation kill rate)	0.34	0.60 (mock-based tests; no mutation runner)
o (oracle agreement)	0.24	0.20 (no independent spec)
b (branch coverage)	0.14	0.75 (Docker auto-detect path untested)
f (fuzz survival)	0.09	1.00
s (SAST clean)	0.04	1.00 (go vet: clean)
t (static depth)	0.10	1.00
d (doc coverage)	0.05	1.00

η_raw = 0.637 · ρ_penalty = 0.255 (same-family code+tests) · η = 0.382

---

Verdict: HumanReviewRequired

η 0.382 < 0.80 threshold; ρ 0.255 exceeds 0.20 correlated-generator threshold.
ΔDebt = 0.53 h (Low). Ci_estimated: true.

Requires independent human review before merge.
Focus areas: Axis 2.7 (correlated tests), Axis 2.5 (extension-only video detection), Axis 2.2 (no formal spec).

---

Auto-Repairs Applied

D001 — Dependency Integrity (commit be67d47)
Pinned HuggingFace model download URLs from resolve/main/ to resolve/78e02f066e9819a60573b78a4275df8a0c27f698/ in both docker/Dockerfile and the manual install instructions in vision_tool.go. Reproducible builds now guaranteed regardless of future upstream model updates.

---

Generated by vprotocol v5.2.7 auto-repair mode