Virtualized Elastic KV Cache for Dynamic GPU Sharing and Beyond

Make huge neural nets fit in memory

USP: Unified (a.k.a. Hybrid, 2D) Sequence Parallel Attention for Long Context Transformers Model Training and Inference

DeepSpeed is a deep learning optimization library that makes distributed training and inference easy, efficient, and effective.

Self-Adapting Language Models

📚LeetCUDA: Modern CUDA Learn Notes with PyTorch for Beginners🐑, 200+ CUDA Kernels, Tensor Cores, HGEMM, FA-2 MMA.🎉

Triton implementation of Flash Attention2.0

Memory-efficient multi layer perceptron implementation in OpenAI Triton.

GPU programming related news and material links

Material for gpu-mode lectures

Skywork-R1V is an advanced multimodal AI model series developed by Skywork AI (Kunlun Inc.), specializing in vision-language reasoning.

Video-R1: Reinforcing Video Reasoning in MLLMs [🔥the first paper to explore R1 for video]

SkyReels-V2: Infinite-length Film Generative model

Official implementation of "Fast-dLLM: Training-free Acceleration of Diffusion LLM by Enabling KV Cache and Parallel Decoding"

Official PyTorch implementation for "Large Language Diffusion Models"

Ring attention implementation with flash attention

ring-attention experiments

Pie: Programmable LLM Serving

verl: Volcano Engine Reinforcement Learning for LLMs

PyTorch library for cost-effective, fast and easy serving of MoE models.

Fast CUDA matrix multiplication from scratch

Domain-specific language designed to streamline the development of high-performance GPU/CPU/Accelerators kernels

EasyR1: An Efficient, Scalable, Multi-Modality RL Training Framework based on veRL

A curated list of Multi-Modal Reinforcement Learning resources (continually updated)

Qwen2.5-Omni is an end-to-end multimodal model by Qwen team at Alibaba Cloud, capable of understanding text, audio, vision, video, and performing real-time speech generation.

[TMLR 2025] Latte: Latent Diffusion Transformer for Video Generation.

Official PyTorch Implementation of "Scalable Diffusion Models with Transformers"

[ICLR 2025] FasterCache: Training-Free Video Diffusion Model Acceleration with High Quality

Wan: Open and Advanced Large-Scale Video Generative Models