A 35B model on a 4 GB GPU: from 6 to 37 tokens per second

The problem

A 35B-parameter MoE model on a home server with a GTX 1650 (4 GB) generated only 6 tokens per second, and most of the serving configs were broken.

What I built

• Measure before changing anything: the container's 4 GB swap was 100% full, so the 21.7 GB model was being paged to disk.
• Removing swap, raising the memory cap and lowering host swappiness took it from 6 to 34 tokens/s on its own.
• Rebuilt llama.cpp in a separate worktree so production never broke, then tuned how layers split between GPU and CPU.
• Five tuned, switchable endpoints, and a code-quality evaluation of five models with hidden graders.

Measured result

• Decode speed: 6.05 → 36.8 tokens/s. Prefill: 2.57 → about 140 tokens/s.
• Cold time-to-first-token on a 12k prompt: 109.9 s → 85.7 s (−22%).
• It now runs nightly automated jobs at zero API cost. Honestly: it's correct on small jobs but much slower, and it struggled with multi-file tasks and some Arabic text.