LLM-, Guardrail- und Vision-Evaluationen auf dem NVIDIA DGX Spark (GB10). Modell- und Guard-Namen sind anklickbar — Fall für Fall mit Prompt, Antwort und Judge. Rohdaten bleiben lokal. Vom Sicherheits-Playbook (04) erscheinen hier keine Fälle; seine Scores stehen ohne Prompt und Antwort in der Testfall-Matrix.
Auf dem GB10 selbst serviert (vLLM), Judge-bewertet. Vollständiges Kohorten-Roster — jedes geplante Modell mit Status, nicht nur die bestandenen. 45 Modelle (✅ gültig 45). Gültige Modelle anklicken → Detail (Prompt, Antwort, Judge je Fall). Sicherheit (04) ausgeschlossen.
| Modell | Release | Gesamt | K.O. | Qualität | Deutsch | Bias | Code | Tok/s | TTFT | Lizenz |
|---|---|---|---|---|---|---|---|---|---|---|
| Nemotron-3.5-Super-VL-120B-MTP ↗ | 2026-10 | 91% PASS | 2 | 90% | 100% | 100% | 80% | 38.3 | 286 ms | openmdw-1.1 |
| Muse-Glimmer-30B-NVFP4 ↗ | 2026-08 | 86% PASS | 3 | 82% | 100% | 78% | 90% | 12.4 | 91 ms | apache-2.0 |
| Nemotron-3.5-Super-VL-120B ↗ | 2026-10 | 86% PASS | 2 | 87% | 75% | 89% | 70% | 19.1 | 127 ms | openmdw-1.1 |
| Muse-Glimmer-30B ↗ | 2026-08 | 83% PASS | 4 | 77% | 100% | 89% | 90% | 4.2 | 254 ms | apache-2.0 |
| Muse-Glimmer-30B-NVFP4-DFlash ↗ | 2026-08 | 79% PASS | 4 | 71% | 100% | 89% | 90% | 30.9 | 247 ms | apache-2.0 |
| Kolibri-1-NVFP4-32k ↗ | 2026-10 | 78% PASS | 7 | 73% | 50% | 100% | 80% | 32.0 | 63 ms | apache-2.0 |
| Gemma-4-26B-A4B ↗ | 2026-03 | 77% K.O. | 8 | 69% | 100% | 89% | 90% | 23.0 | 92 ms | apache-2.0 |
| Gemma-4-31B ↗ | 2026-03 | 77% PASS | 7 | 71% | 75% | 89% | 80% | 3.8 | 397 ms | apache-2.0 |
| Nemotron-3-Super-MTPv2 ↗ | 2026-03 | 77% K.O. | 13 | 73% | 100% | 89% | 80% | 24.7 | 458 ms | nvidia-nemotron-open-model-license |
| Ornith-1.0-35B-FP8 ↗ | 2026-06 | 77% PASS | 11 | 66% | 100% | 100% | 80% | 30.8 | 105 ms | mit |
| Ornith-1.5-122B-NVFP4 ↗ | 2026-10 | 77% PASS | 9 | 68% | 75% | 100% | 90% | 19.1 | 196 ms | mit |
| Nemotron-3-Super ↗ | 2026-03 | 74% WARN | 5 | 69% | 75% | 89% | 80% | 15.9 | 247 ms | nvidia-nemotron-open-model-license |
| Qwen3.8-27B-NVFP4-MTP ↗ | 2026-08 | 74% WARN | 13 | 65% | 100% | 89% | 90% | 22.6 | 283 ms | apache-2.0 |
| Qwen3.8-27B-FP8 ↗ | 2026-08 | 73% WARN | 8 | 71% | 50% | 78% | 70% | 7.9 | 181 ms | apache-2.0 |
| Qwen3.8-27B-NVFP4 ↗ | 2026-08 | 73% WARN | 10 | 69% | 50% | 89% | 70% | 11.3 | 132 ms | apache-2.0 |
| Hy-MT2-30B-A3B-FP8 ↗ | 2026-05 | 71% K.O. | 9 | 69% | 100% | 89% | 50% | 22.3 | 102 ms | apache-2.0 |
| Qwen3.8-27B-Uncensored-NVFP4 ↗ | 2026-08 | 68% K.O. | 15 | 66% | 25% | 100% | 80% | 10.1 | 140 ms | apache-2.0 |
| Nemotron-3.5-Bolt-E4B-DFlash ↗ | 2026-10 | 67% WARN | 5 | 63% | 25% | 89% | 50% | 70.4 | 95 ms | openmdw-1.1 |
| Nemotron-3.5-Lightning-30B-A3B ↗ | 2026-08 | 67% K.O. | 10 | 58% | 50% | 100% | 80% | 80.3 | 68 ms | openmdw-1.1 |
| Nemotron-3.5-Bolt-E4B ↗ | 2026-10 | 66% WARN | 6 | 65% | 25% | 100% | 40% | 19.4 | 59 ms | openmdw-1.1 |
| Ornith-1.5-35B-A3B-NVFP4 ↗ | 2026-08 | 66% K.O. | 14 | 58% | 75% | 78% | 70% | 28.5 | 102 ms | mit |
| Qwen3.6-27B-FP8 ↗ | 2026-04 | 64% WARN | 12 | 63% | 0% | 100% | 30% | 7.5 | 242 ms | apache-2.0 |
| Gemma-4-E4B ↗ | 2026-03 | 63% WARN | 8 | 56% | 50% | 89% | 50% | 19.7 | 58 ms | apache-2.0 |
| Laguna-S-2.1-NVFP4 ↗ | 2026-07 | 63% K.O. | 15 | 58% | 0% | 78% | 80% | 17.2 | 163 ms | openmdw-1.1 |
| LFM2.5-8B-A1B ↗ | 2026-05 | 63% WARN | 7 | 60% | 0% | 89% | 60% | 55.9 | 51 ms | lfm1.0 |
| Apertus-v1.5-8B ↗ | 2026-07 | 62% K.O. | 10 | 58% | 50% | 100% | 30% | 14.2 | 83 ms | apache-2.0 |
| LFM3-1.7B-NVFP4 ↗ | 2026-10 | 62% K.O. | 6 | 60% | 25% | 89% | 60% | 87.0 | 23 ms | lfm1.0 |
| Apertus-v1.5-70B ↗ | 2026-07 | 61% K.O. | 11 | 58% | 50% | 78% | 40% | 3.1 | 623 ms | apache-2.0 |
| Granite-4.2-8B-NVFP4 ↗ | 2026-08 | 61% K.O. | 13 | 55% | 0% | 100% | 90% | 38.4 | 37 ms | apache-2.0 |
| Nemotron-3-Nano-30B ↗ | 2025-12 | 61% K.O. | 17 | 53% | 25% | 89% | 80% | 42.3 | 128 ms | nvidia-nemotron-open-model-license |
| Nemotron-Puzzle-75B ↗ | 2026-06 | 61% K.O. | 12 | 55% | 0% | 100% | 60% | 20.8 | 131 ms | openmdw-1.1 |
| Granite-4.1-30B ↗ | 2026-04 | 59% K.O. | 17 | 55% | 25% | 100% | 60% | 3.9 | 281 ms | apache-2.0 |
| Granite-4.2-30B-NVFP4 ↗ | 2026-08 | 59% K.O. | 13 | 50% | 0% | 100% | 70% | 12.9 | 91 ms | apache-2.0 |
| Granite-Switch-4.1-30B ↗ | 2026-10 | 58% K.O. | 18 | 52% | 25% | 100% | 60% | 3.3 | 285 ms | apache-2.0 |
| Qwen3.6-35B-A3B-FP8 ↗ | 2026-04 | 58% FAIL | 12 | 55% | 0% | 100% | 30% | 17.9 | 154 ms | apache-2.0 |
| Granite-4.2-3B-NVFP4 ↗ | 2026-08 | 57% FAIL | 4 | 53% | 0% | 89% | 50% | 80.1 | 22 ms | apache-2.0 |
| Mistral-Small-4 ↗ | 2026-03 | 56% K.O. | 20 | 50% | 75% | 78% | 60% | 31.4 | 94 ms | apache-2.0 |
| Laguna-XS-2.1-FP8 ↗ | 2026-06 | 54% K.O. | 19 | 42% | 0% | 89% | 80% | 37.4 | 65 ms | openmdw-1.1 |
| Olmo-3.1-32B-Instruct ↗ | 2025-12 | 53% K.O. | 18 | 42% | 0% | 100% | 60% | 3.7 | 342 ms | apache-2.0 |
| Ornith-1.0-9B ↗ | 2026-06 | 51% K.O. | 14 | 39% | 25% | 78% | 70% | 12.3 | 118 ms | mit |
| LFM3-Omni-2B ↗ | 2026-10 | 50% K.O. | 14 | 45% | 0% | 100% | 20% | 53.3 | 28 ms | lfm1.0 |
| Granite-Switch-4.1-8B ↗ | 2026-10 | 49% K.O. | 21 | 37% | 25% | 89% | 50% | 12.6 | 93 ms | apache-2.0 |
| Mellum2-12B-A2.5B-Thinking ↗ | 2026-05 | 49% K.O. | 21 | 39% | 0% | 89% | 70% | 42.9 | 65 ms | apache-2.0 |
| Mellum2-12B-A2.5B-Instruct ↗ | 2026-05 | 43% K.O. | 27 | 32% | 25% | 78% | 60% | 43.5 | 68 ms | apache-2.0 |
| Apertus-v1.1-4B ↗ | 2026-05 | 31% K.O. | 27 | 21% | 25% | 89% | 10% | 27.6 | 42 ms | apache-2.0 |
Frontier-Modelle über LiteLLM als Referenzrahmen — ein Endpoint für Cloud- und lokale Modelle, von SouthByte empfohlen. Gleicher Testsatz, Judge claude-sonnet-5. Tok/s & TTFT messen hier Cloud+Netz (nicht lokale Hardware). 28 Modelle · Pass-Rate je Playbook. Modellname anklicken → Detail (Prompt, Antwort, Judge je Fall). Sicherheit (04) ausgeschlossen.
| Modell | Release | Gesamt | K.O. | Qualität | Deutsch | Bias | Code | Tok/s | TTFT | Lizenz |
|---|---|---|---|---|---|---|---|---|---|---|
| Grok-4.6 ↗ | 2026-08 | 93% PASS | 1 | 95% | 100% | 89% | 90% | 18.3 | 715 ms | xAI · proprietär (API) |
| Claude-Sonnet-5 ↗ | 2026-06 | 91% PASS | 1 | 92% | 100% | 78% | 90% | 98.4 | 1202 ms | Anthropic · proprietär (API) |
| Grok-4.5 ↗ | 2026-07 | 90% PASS | 7 | 89% | 100% | 78% | 100% | 43.0 | 739 ms | xAI · proprietär (API) |
| Claude-Fable-5 ↗ | 2026-06 | 88% PASS | 0 | 92% | 100% | 89% | 60% | 76.7 | 2560 ms | Anthropic · proprietär (API) |
| Grok-4.1-Fast ↗ | 2025-11 | 88% PASS | 3 | 87% | 100% | 89% | 90% | 101.2 | 676 ms | xAI · proprietär (API) |
| GPT-5.6-sol ↗ | 2026-07 | 87% PASS | 6 | 82% | 100% | 100% | 100% | 54.9 | 1508 ms | OpenAI · proprietär (API) |
| GPT-5.6-terra ↗ | 2026-07 | 85% PASS | 7 | 79% | 100% | 100% | 100% | 81.5 | 738 ms | OpenAI · proprietär (API) |
| GPT-5.6-luna ↗ | 2026-07 | 83% PASS | 7 | 79% | 75% | 89% | 100% | 143.7 | 1007 ms | OpenAI · proprietär (API) |
| Claude-Opus-5 ↗ | 2026-07 | 81% PASS | 0 | 79% | 100% | 78% | 90% | 125.4 | 2921 ms | Anthropic · proprietär (API) |
| Kimi-K3 ↗ | 2026-07 | 80% PASS | 2 | 77% | 100% | 78% | 90% | 43.0 | 1517 ms | proprietär (API) |
| MiMo-v2.5-Pro ↗ | 2026-04 | 78% PASS | 6 | 74% | 75% | 89% | 90% | 48.8 | 1572 ms | proprietär (API) |
| Qwen3.7-Plus | 2026-06 | 78% PASS | 13 | 73% | 100% | 89% | 90% | 59.4 | 689 ms | proprietär (API) |
| Gemini-3.1-Pro ↗ | 2026-02 | 76% PASS | 9 | 71% | 100% | 78% | 90% | — | — | Google · proprietär (API) |
| GLM-5.2 ↗ | 2026-06 | 74% WARN | 13 | 69% | 100% | 78% | 90% | 102.0 | 805 ms | proprietär (API) |
| MiniMax-M3 ↗ | 2026-06 | 74% WARN | 8 | 71% | 100% | 67% | 90% | 157.9 | 1346 ms | proprietär (API) |
| Nemotron-3-Ultra-550B ↗ | 2026-06 | 73% K.O. | 15 | 68% | 100% | 78% | 90% | 23.5 | 1208 ms | proprietär (API) |
| DeepSeek-V4-Pro ↗ | 2026-04 | 72% K.O. | 14 | 66% | 100% | 78% | 90% | 42.7 | 2090 ms | proprietär (API) |
| Gemini-3.5-Flash-Lite ↗ | 2026-07 | 72% K.O. | 15 | 66% | 100% | 78% | 90% | 311.9 | 466 ms | Google · proprietär (API) |
| Gemini-3.6-Flash ↗ | 2026-07 | 72% K.O. | 15 | 66% | 100% | 67% | 100% | — | — | Google · proprietär (API) |
| Hunyuan-3 | 2026-07 | 69% K.O. | 15 | 63% | 100% | 67% | 90% | 61.9 | 1518 ms | proprietär (API) |
| MiMo-v2.5 ↗ | 2026-04 | 67% K.O. | 14 | 63% | 75% | 78% | 80% | 95.3 | 1019 ms | proprietär (API) |
| Qwen3.8-Max | 2026-08 | 67% WARN | 7 | 66% | 50% | 89% | 60% | 47.4 | 998 ms | proprietär (API) |
| Claude-Haiku-4.5 ↗ | 2025-10 | 66% WARN | 8 | 65% | 50% | 78% | 70% | 115.7 | 620 ms | Anthropic · proprietär (API) |
| Magistral-Medium ↗ | 2025-09 | 57% K.O. | 23 | 50% | 100% | 67% | 70% | 127.3 | 240 ms | Mistral · proprietär (API) |
| Mistral-Large ↗ | 2025-12 | 57% K.O. | 23 | 50% | 100% | 78% | 60% | 56.0 | 337 ms | Mistral · proprietär (API) |
| Mistral-Medium ↗ | 2026-04 | 56% K.O. | 22 | 47% | 75% | 78% | 80% | 65.5 | 249 ms | Mistral · proprietär (API) |
| Step-3.7-Flash ↗ | 2026-05 | 49% K.O. | 22 | 44% | 50% | 78% | 50% | 122.4 | 1748 ms | proprietär (API) |
| Ministral-8B ↗ | 2024-10 | 38% K.O. | 28 | 31% | 0% | 78% | 60% | 137.5 | 180 ms | Mistral · proprietär (API) |
Guard-Name anklicken → Fall für Fall (Wahrheit vs. Vorhersage). Kein Judge — das Label ist die Wahrheit.
| Guard | Release | Recall | FPR | Trap-FPR | Präz. | F1 | Acc | Lat ø | Lizenz |
|---|---|---|---|---|---|---|---|---|---|
| Granite-Switch-30B | 2026-10 | 1.000 | 0.000 | 0.000 | 1.000 | 1.000 | 1.000 | 2498.900 | apache-2.0 |
| Granite-Guardian | 2026-04 | 0.974 | 0.023 | 0.042 | 0.974 | 0.974 | 0.976 | 1044.900 | apache-2.0 |
| Granite-Switch-8B | 2026-10 | 1.000 | 0.023 | 0.042 | 0.974 | 0.987 | 0.988 | 681.400 | apache-2.0 |
| Nemotron-3 | 2026-03 | 1.000 | 0.045 | 0.083 | 0.950 | 0.974 | 0.976 | 493.200 | nvidia-nemotron-open-model-license |
| Nemotron-3.5 | 2026-05 | 1.000 | 0.045 | 0.083 | 0.950 | 0.974 | 0.976 | 562.900 | openmdw-license-agreement |
| Qwen3Guard-Gen-0.6B | 2025-09 | 1.000 | 0.000 | 0.000 | 1.000 | 1.000 | 1.000 | 90.900 | apache-2.0 |
| Qwen3Guard-Gen-4B | 2025-09 | 1.000 | 0.000 | 0.000 | 1.000 | 1.000 | 1.000 | 481.000 | apache-2.0 |
| Qwen3Guard-Gen-8B | 2025-09 | 1.000 | 0.000 | 0.000 | 1.000 | 1.000 | 1.000 | 803.700 | apache-2.0 |
| Qwen3Guard-Stream-0.6B | 2025-09 | 1.000 | 0.045 | 0.000 | 0.950 | 0.974 | 0.976 | 109.200 | apache-2.0 |
| Qwen3Guard-Stream-4B | 2025-09 | 1.000 | 0.023 | 0.042 | 0.974 | 0.987 | 0.988 | 412.800 | apache-2.0 |
| Qwen3Guard-Stream-8B | 2025-09 | 1.000 | 0.000 | 0.000 | 1.000 | 1.000 | 1.000 | 736.000 | apache-2.0 |
| gpt-oss-safeguard | 2025-09 | 1.000 | 0.000 | 0.000 | 1.000 | 1.000 | 1.000 | 1566.800 | apache-2.0 |
| Shieldstral | 2026-07 | 1.000 | 0.023 | 0.042 | 0.974 | 0.987 | 0.988 | 44.300 | apache-2.0 |
| SingGuard-2b | 2026-05 | 1.000 | 0.023 | 0.042 | 0.974 | 0.987 | 0.988 | 282.100 | apache-2.0 |
Entscheidungsmodelle nach dem Muster von Jev, keine Guards: sie beantworten die Policy als Ja/Nein-Frage mit einer Wahrscheinlichkeit (Schwelle 0,5), dieselben Kriterien und Fälle wie oben. Nicht im Vergleich der besten Werte.
| Guard | Release | Recall | FPR | Trap-FPR | Präz. | F1 | Acc | Lat ø | Lizenz |
|---|---|---|---|---|---|---|---|---|---|
| laya-multilingual | 2026-09 | 0.447 | 0.136 | 0.167 | 0.739 | 0.557 | 0.671 | 46.000 | apache-2.0 |
Modellname anklicken → Fall für Fall mit Quelle, Wahrheit, Antwort und markierten Abweichungen. CER = Median der normalisierten Zeichenfehlerrate je Schriftlage, kleiner ist besser; Handschrift ohne Bestehensgrenze. Kein Judge, eigene Kohorte, nicht Teil der LLM-Wertung.
| Modell | Art | Release | CER Fraktur | CER Kurrent | CER Frühnhd. | CER Latein | Bildfragen | Fallen | K.O. | Lizenz |
|---|---|---|---|---|---|---|---|---|---|---|
| Qwen3.8-27B-RadixArk | VL | 2026-08 | 0.011 | 0.500 | 0.222 | 0.548 | 8/8 | 6/6 | 0 | apache-2.0 |
| Qwen3.8-27B-Uncensored | VL | 2026-08 | 0.013 | 0.494 | 0.223 | 0.660 | 8/8 | 6/6 | 0 | apache-2.0 |
| Nemotron-3.5-Super-VL-120B | VL | 2026-10 | 0.035 | 0.657 | 0.337 | 0.852 | 8/8 | 6/6 | 0 | openmdw-1.1 |
| LFM3-VL-2B | VL | 2026-10 | 0.042 | 0.622 | 0.432 | 0.645 | 8/8 | 5/6 | 0 | lfm1.0 |
| Muse-Glimmer-30B-NVFP4 | VL | 2026-08 | 0.013 | 0.784 | 0.468 | 0.781 | 8/8 | 6/6 | 0 | apache-2.0 |
| Ornith-1.5-122B | VL | 2026-10 | 0.008 | 0.550 | 0.511 | 0.624 | 8/8 | 5/6 | 1 | mit |
| Gemma-4-E4B | VL | 2026-03 | 0.026 | 0.560 | 0.511 | 0.539 | 2/8 | 4/6 | 0 | apache-2.0 |
| Nemotron-3.5-Bolt-E4B | VL | 2026-10 | 0.025 | 0.593 | 0.518 | 0.523 | 2/8 | 6/6 | 0 | openmdw-1.1 |
| LFM3-Omni-2B | VL | 2026-10 | 0.048 | 0.608 | 0.538 | 0.637 | 1/8 | 5/6 | 0 | lfm1.0 |
| Nemotron-3.5-Parse | OCR | 2026-09 | 0.086 | 0.520 | 0.278 | 0.596 | — | — | 0 | openmdw-1.1 |
| Unlimited-OCR-2 | OCR | 2026-10 | 0.021 | 0.788 | 0.412 | 0.765 | — | — | 0 | mit |
| Unlimited-OCR | OCR | 2026-06 | 0.260 | 0.871 | 1.762 | 1.000 | — | — | 0 | mit |
| granite-docling-258M | OCR | 2025-05 | 0.329 | 1.000 | 3.564 | 7.105 | — | — | 0 | apache-2.0 |