vLLM Blog
25 stories
vLLM BlogToolsAnnouncing Humming: Fast, Flexible Kernels for Quantized Inference Introducing Humming, a quantized inference kernel library from Ant Group and the vLLM community, with broad GPU and quantization support, dense and MoE performance results, and vLLM integration.
vLLM BlogToolsIntroducing vLLM TorchTPU vllm-torchtpu is a new vLLM backend for Google Cloud TPUs built on TorchTPU, with Pallas kernels, multi-host parallelism, KV cache offloading, and disaggregated serving.
vLLM BlogToolsAchieving near-zero-downtime scaling in Elastic Expert Parallelism How asynchronous preparation and NIXL EP enabled near-zero-downtime Elastic EP scaling in vLLM.
vLLM BlogToolsHyper Optimizing Kimi K3 in vLLM: Moving Fast and Responsibly How agent-assisted CuTe DSL optimizations accelerate Kimi K3 decode on NVIDIA GB300, and how scoped changes, correctness tests, and reproducible benchmarks help ship them responsibly.
vLLM BlogToolsScaling Kimi K3 on AMD Instinct MI355X: Matching NVIDIA HGX B300 Throughput per GPU on AgentX Inside the Kimi K3 optimization effort: understanding the bottlenecks and the decisions that made inference faster
vLLM BlogToolsServing Reflection Beam on vLLM from Day 0 vLLM supports Reflection's Beam-1.0 (501B total, 23B active) from day 0, with agent-driven optimization that cut decode latency 2.71× and DFlash2 speculative decoding with adaptive verification.
vLLM BlogToolsDay 0 support for Muse Spark Open on vLLM Day-0 vLLM support for Muse Spark Open: 816B MoE, 1M context, multimodal inputs, DSpark speculative decoding, NVIDIA and AMD serving, initial TPU support, and LoRA RL.
vLLM BlogToolsAnnouncing Day 0 Support for NVIDIA Nemotron 3.5 Super VL on vLLM Run NVIDIA Nemotron 3.5 Super VL on vLLM for multimodal agent workflows with image, document, and video understanding, OpenAI-compatible serving, and multi-token prediction.
vLLM BlogToolsAdaptive Verification Without a Confidence Head An online-fitted acceptance estimator that lets every draft-model speculator use Adaptive Verification, by predicting the target's acceptance probability from the drafter's own logits and refitting…
vLLM BlogToolsPush-Based KV Transfer to Improve TTFT and TPOT for Disaggregated Inference in vLLM A push-based NIXL connector for vLLM disaggregated serving: the prefill instance writes KV directly into decode memory over RDMA, taking the router relay, decode-side allocation, and transfer setup…
vLLM BlogToolsFinish RL Training Faster: Bitwise-Consistent, High-Performance, Agent-Driven We make vLLM rollout and the training forward give bitwise identical logprobs on DeepSeek-V4-Flash and Nemotron-3.5-Lightning-30B-A3B (NVFP4), keep 91% of rollout throughput on DeepSeek-V4-Flash, and…
vLLM BlogToolsImprove TTFT for Multi-Turn Agentic Workloads with Bi-Directional KV Transfer in vLLM Bi-directional KV transfer lets vLLM prefill nodes reuse previously computed KV from decode nodes on later conversation turns, cutting redundant prefill recompute and reducing TTFT by up to 3.3x on…
vLLM BlogToolsvLLM Support for NVIDIA Vera Rubin NVL72: 7.8x Throughput over GB200 NVL72 vLLM now runs on NVIDIA Vera Rubin NVL72 with day-0 model support, Rubin-tuned FlashInfer kernels, and locality-aware MoE, delivering 7.8x the per-GPU throughput of GB200 NVL72 on AgentX.
vLLM BlogToolsDeepSeek-V4.1-Flash on vLLM: 5x Agentic Throughput Since Day 0 Within three weeks of release, vLLM made DeepSeek-V4.1-Flash 1.9x faster at low concurrency and lifted its throughput 5x on SemiAnalysis AgentX, with SWA bounded replay, CUDA graphs, DeepSeek's new…
vLLM BlogToolsTaking vLLM Apart: A Practical Guide to Disaggregated Serving What disaggregated serving actually buys you, how to run it end to end in vLLM today with prefill/decode plus the new GPU-less frontend and the things we're still working on.
vLLM BlogToolsWatermarking in vLLM How vLLM implements distribution-preserving Gumbel-max text watermarking with efficient GPU kernels, statistical detection, speculative decoding, and repeated-context safeguards.
vLLM BlogToolsReducing vLLM startup time with one uv flag Shipping Python bytecode saved about 18 seconds per startup in H100 server benchmarks, with images and models already local.
vLLM BlogToolsAnnouncing vllm-metal: Concurrent Serving on Apple Silicon vllm-metal brings vLLM's paged, continuously batched serving stack to Apple Silicon, with flatter TTFT under concurrent agent load, batched MTP, and automatic M5 prefill acceleration.
vLLM BlogToolsPD Serving of Qwen3.8-2.4T How vLLM reaches 5K throughput and 180 interactivity on Qwen3.8-2.4T with GB300 NVL72 PD serving and how to reproduce results yourself.
vLLM BlogToolsScaling Multi-GPU Video Captioning with PyNvVideoCodec and vLLM How to leverage NVIDIA Hardware Video Decoders to Achieve Multi-GPU Scaling in Video Captioning and Description tasks.
vLLM BlogToolsHow we trained the fastest DSpark for Kimi-K3 using GB300 NVL72 How Speculators and Mooncake enabled multi-node DSpark training for Kimi K3.
vLLM BlogToolsvLLM x Novita AI: Chord, Faster INT4 MoE for Kimi K2.x. Up to 1.3x on H200, 2.15x on Untuned B300 Novita AI has open-sourced Chord, a high-performance W4A16 MoE CUDA kernel for Kimi K2.x serving shapes, with a Humming-compatible indexed path and grouped SM90 operators.
vLLM BlogToolsvime × RL-Kernel × AMD: Bitwise Train–Rollout Consistency on ROCm vime and RL-Kernel align selected-token logprobs bit for bit across Megatron training and vLLM rollout on AMD Instinct MI300X, with zero mismatches across 200 GRPO steps.
vLLM BlogToolsKimi K3 Performance Optimizations in vLLM: The Road to 2.8× Throughput Kimi K3 serving optimizations across scheduling, KDA prefix caching, ReplaySSM state recovery, PD disaggregation and state offload, parallelism, MoE, and GPU kernels.
Nothing matches this filter yet.