Whether you're hand-tuning CUDA replacements, building compiler backends, or exploring what's next for AI kernel agents — this is your day.
The fourth annual Triton Developer Conference brings together GPU kernel engineers, AI compiler researchers, and hardware ecosystem partners for technical talks and conversation that shape Triton's roadmap. And for the first time, we're co-located with PyTorch Conference — attend both for the complete experience from kernel to model.
© 2024 Meta. Meta Platforms, Inc., 1601 Willow Rd. Menlo Park, CA 94025 or where the Meta Products are offered to you by Meta Platforms Ireland Limited, Serpentine Avenue, Ballsbridge, Dublin 4, Ireland.
Meta Data Policy - Terms - Cookies -
How to use this module
1. Clone this module. You will need one clone per speaker.
2. Rename your clones. Update each module with one speaker's information.
3. Link the bios. Hover over each"view bio" and click the "edit" icon. Using the drop downs, link the speaker to the corresponding bio module.
4. Test each module. After you've linked each bio, use the "test" icon to ensure each link matches the correct speaker.
Lixun Zhang is a Senior Manager at AMD, leading a team focused on Triton compiler development and GPU kernel performance optimization. His work spans compiler architecture, instruction scheduling, register allocation, and high-performance Triton/Gluon kernels for AMD GPUs. He is the author of the AMD gfx950 Gluon tutorials and has worked extensively on techniques for achieving near-peak performance on AMD GPUs. He is passionate about making high-performance GPU programming more accessible through better compiler abstractions, programming models, and developer education.
Thierry Moreau is the Triton Compiler Engineering Manager at NVIDIA. Before joining NVIDIA, he co-founded OctoAI, a startup built around Apache TVM. He is a founding engineer and PMC member of the Apache TVM open-source deep learning compiler project, and holds a Ph.D. in computer architecture from the University of Washington.
Manman works on Triton compiler performance at Meta, focusing on closing the gap between Python-authored GPU kernels and hand-tuned throughput. Before Triton, she worked in LLVM on CPU performance and code size for server and mobile workloads. Manman holds a PhD from Stanford, where the research focus was programming models for explicitly managed memory hierarchies and high-performance architecture.
Masahiro Masuda is a Senior Software Engineer on the Triton compiler team at NVIDIA. Prior to joining NVIDIA, he worked on deep learning compilation and generative AI acceleration at OctoAI. He received his M.S. degree in computer science from University of Tsukuba, where he focused on the intersection between programming languages and cryptography.
Alessandro is a Senior Software Engineer in Red Hat's Emerging Technologies (Office of the CTO), where he works on ML systems and GPU-kernel tooling across Triton, PyTorch, and Helion. His work focuses on kernel autotuning, caching, deployment-oriented performance improvements. He is also a Helion committer and active open-source contributor, and has presented his work at technical conferences including DevConf.US and the 2025 Triton Conference.
Erwei Wang is a Senior Member of Technical Staff in AMD Research and Advanced Development (RAD), where he leads and contributes to compiler and software infrastructure for AMD's next-generation NPUs and spatial accelerators. His work focuses on developing open-source compiler frameworks, including MLIR-AIR and Triton-XDNA, to improve programmability, performance portability, and deployment of AI workloads on heterogeneous hardware platforms. He is particularly interested in compiler optimizations, parallel programming models, and runtime systems for machine learning accelerators. He actively collaborates with academic and industry partners to advance open-source ecosystems and next-generation AI computing platforms.
Adam Straw is a Senior Software Engineer on the Triton Compiler team at NVIDIA. He began his career at Intel in post-silicon validation, then joined OctoAI, where he applied Apache TVM to optimize deep learning models for various hardware architectures including NVIDIA GPUs. Adam holds a B.S. in Computer Engineering from Iowa State University.
Jiaqi Xu is a Research Scientist at Meta, driving GPU efficiency for Meta’s largest recommendation system model. Her research focuses on large-scale recommendation system training and spans the full stack, from model–hardware co-design and ML compiler optimization to low-level GPU optimization, including kernel and parallelism optimization. She works across model and system layers to improve GPU efficiency for production-scale AI training.
Pengzhan Zhao is a Member of Technical Staff at AMD, where he develops high-performance GPU kernels and Triton compiler support for AI inference on AMD Instinct GPUs. He is a co-creator of TokenSpeed and a core contributor to its kernel stack, focusing on attention, MoE, and multi-silicon optimization for agentic workloads.
Dimple Prajapati is a Principal Member of Technical Staff in AMD's AI Group, where she leads the Triton-native multi-GPU communication workstream. She builds compiler and runtime technology for large-scale AI across Triton, PyTorch, and ROCm, with upstream contributions to PyTorch Symmetric Memory and rocSHMEM.
Saagar is a software engineer at Sublinear Systems, where he leads GPU performance optimization and research focusing on ultra-long context attention. His interests include program analysis and systems engineering, inasmuch as he can't shut up about them if someone brings them up in a conversation. He can usually be found somewhere around the Bay Area or, when time permits, arguing with people online about how to write software.
Ph.D. in Theoretical Condensed Matter Physics, with research focused on strongly correlated electron systems. Previously an algorithm engineer at Sophgo, an AI chip company, specializing in performance optimization of the TPU software stack. Currently developing high-performance GPU kernels at the Beijing Academy of Artificial Intelligence (BAAI).
Francesco Fusco is a Senior Research Scientist at IBM Research AI in Zurich, where he has spent more than a decade designing efficient model architectures and building the systems that run them, with work spanning recommender systems, efficient NLP, and LLMs. He is a co-creator of pNLP-Mixer, an all-MLP architecture for low-latency language understanding, contributed to the training of IBM's Granite LLMs, and lately has brought performance optimizations to hybrid-model inference in vLLM. He holds a PhD from ETH Zurich and is a serial inventor with patents in AI, networking, databases, and compression algorithms.
Nikhil Stephen is a Senior Staff Engineer at Marvell specializing in AI/ML architecture, hardware-software co-design, and compiler stack development. He has contributed to multiple generations of ML inference accelerator platforms, with deep focus on building the software stack — particularly the compiler layers that connect AI/ML workloads to high-performance silicon. More recently, his work has shifted to data-center scale simulation infrastructure, tackling challenges across scale-up, scale-out, and memory tiering domains.
I'm a Principal Member of Technical Staff at AMD Research, where I design, architect, and tech-lead system-level software, libraries, and runtimes for next-generation computing. I lead cross-organizational teams and coordinate initiatives across ML frameworks, kernel engineering, distributed systems, and research teams. I designed and architected Iris, an open-source multi-GPU programming framework, from scratch and tech-lead it with a cross-organizational team. I also created and architected IntelliKit and IntelliPerf, leading their development from inception, and serve as tooling lead for AMD's company-wide ML for performance engineering initiative, coordinating strategy across multiple engineering teams.
Before joining AMD, I earned my Ph.D. at UC Davis under Professor John Owens, designing and building concurrent GPU data structures that resulted in four first-author publications. I'm passionate about parallel and heterogeneous computing, performance analysis, and AI-assisted development, coding across the stack from Python to assembly, with native expertise in HIP/CUDA C++. I currently serve on the review board for IEEE Transactions on Parallel and Distributed Systems and contribute to the OpenSHMEM specification workgroup.
Created Flow (static type checker for JavaScript). Led development of torch.export (ahead-of-time graph capture for PyTorch). Implemented type checking for PyTorch tensor shapes (and more recently, extending it for CuTe layouts).
Used to publish research in the areas of programming languages and computer security a long time back.
The next era of AI won't be defined by a single chip — it'll run everywhere. As CTO of Kernelize, Simon Waters leads the team building the open software layer that makes any model portable to any hardware. He has been building compiler for over two decades targeting a wide variety of architectures, including GPUs, NPUs and high-level synthesis for chip design.
The next era of AI won't be defined by a single chip — it'll run everywhere. As CTO of Kernelize, Simon Waters leads the team building the open software layer that makes any model portable to any hardware. He has been building compiler for over two decades targeting a wide variety of architectures, including GPUs, NPUs and high-level synthesis for chip design.
Yash Nishant works on AI Systems with a focus on inference optimization. His work centers on memory systems and interconnects for next-generation AI infrastructure, along with their software implications, including ML framework integration and optimization. His talk introduces Photonic Fabric Memory Appliance, a terabyte-scale disaggregated and shared memory system designed to address the memory capacity bottleneck in AI inference.
Getting high performance on GPUs has traditionally required extensive kernel tuning and architecture-specific optimizations. With AMD MI450 and Triton Gluon, many workloads achieve strong performance out of the box, allowing developers to focus more on algorithms and less on low-level tuning.
In this talk, we'll examine what enables this shift. We'll discuss how the MI450 architecture and the Triton Gluon programming model work together to improve default kernel performance, reducing the amount of manual optimization required compared to previous GPU generations.
We'll present performance results across representative GEMM workloads, explain the key architectural and compiler changes behind the improvements, and discuss where manual tuning still provides value. Finally, we'll share lessons learned from bringing up Gluon on MI450 and what these changes mean for future Triton kernel development on AMD GPUs.
NVIDIA’s latest generation of Rubin GPUs introduces new opportunities for pushing performance to new heights. This talk provides a developer-oriented introduction to Rubin from the perspective of high-performance kernel authors using Triton and Gluon.
While some architectural details are not yet public at the time of abstract submission, they are expected to be made public by the time of the summit. The talk will focus on Rubin’s to-be disclosed architectural features and hardware capabilities, and on what they mean in practice for programmers building performance-critical kernels.
Through several examples, we will demonstrate how Rubin’s latest features can translate into measurable performance gains.
Attendees will leave with a practical understanding of Rubin’s optimization patterns for Triton/Gluon and demanding AI workloads.
This talk presents Meta’s version of the Triton compiler for NVIDIA GPUs, with a focus on increased support for the latest hardware and kernel authoring techniques. We will present our latest performance numbers and provide insights into the techniques empowering these results, such as data partitioning, epilogue subtiling, and automatic warp specialization with memory planning. In addition, we will discuss how we have enabled several advanced hardware features including 2-CTA execution, multi-CTA reductions, and Cluster Launch Control, without sacrificing the core Triton programming experience.
Our central design principle is to separate decision-making from program transformation. Performance-critical choices are extracted into an explicit, autotunable search space, while compiler transformations preserve semantics for any valid decision set. This architecture maximizes flexibility, enables advanced tuning models, guarantees correctness by construction, and reduces the time developers spend manually rewriting kernels.
In an increasingly agentic world, the bar for performance has never been higher, and some kernels may need further low-level optimization beyond what Triton can offer. To address this, we provide support for emitting TLX from Triton, creating one combined Triton ecosystem focused on optimizing the performance/complexity trade-off. Together, these techniques push Triton performance toward the limits of modern NVIDIA GPUs without sacrificing programmability.
One year ago at the last Triton Conference, like our forefathers in the Stone Age carving pictures on to bones, we wrote code by hand. Now a year later, Codex and other AI agents have transformed the way we develop the Triton compiler and programming language. In this talk, we will discuss our experiencing learning how to productively vibe code in a compiler code base, and how to leverage Codex to make code changes every compiler and programming language engineer dreams of, like refactoring legacy, load-bearing code. We discuss how we finally unified the intra-kernel memory model in the compiler, how we removed legacy features from the language like `can_reorder`, and how Codex has massively accelerated our pace of development on the compiler. We give a brief overview of the kinds of major Codex-assisted contributions we have made, such as our various sanitizers, our much improved barrier insertion and optimization pass, and broad improvements to our backend code generation, all of which would never have happened without accelerating AI progress.
LLM inference performance is often constrained by critical operators and compiler-generated inefficiencies, particularly in Mixture-of-Experts workloads. Achieving consistently high performance across diverse AI accelerators requires coordinated optimization at both the operator and compiler levels.
This talk presents three Triton-based optimization techniques. First, we optimize critical MoE operators through workload-aware kernel design, improved data movement, efficient expert dispatch, and fused computation. Second, we introduce compiler-level layout optimizations that eliminate redundant layout conversions and improve memory-access efficiency across more than 50 operators. Third, we present instruction-scheduling optimizations that restructure dependency chains, increase instruction-level parallelism, and better overlap memory operations with computation.
Together, these techniques enable optimized Triton operators to outperform equivalent implementations written in native accelerator programming languages. They also deliver over 10% end-to-end LLM inference performance improvement across multiple workloads and AI accelerator platforms.
Triton has made high-performance kernel development far more accessible, but building a production-grade ecosystem requires more than optimizing isolated kernels. It demands broad operator coverage, systematic performance engineering, model-level integration, and portability across rapidly evolving accelerator backends.
This talk presents FlagGems, the world’s largest open-source Triton operator library, comprising more than 500 operators and extensive support for Triton-TLE, an extended Triton language for portable optimization across diverse AI hardware. We will share how FlagGems combines reusable Triton implementations with shape-aware specialization, fusion, automated code generation, and architecture-specific optimization. Attention kernels used by leading open-source models including Qwen, Kimi, DeepSeek have reached state-of-the-art performance, delivering up to 80× speedups over prior Triton implementations. Across NVIDIA GPUs, Huawei Ascend, Moore Threads, and other accelerators, many FlagGems operators also outperform vendor-provided closed-source C implementations.
We will then show how this operator foundation enabled Day-0 migration of nearly every major open-source model released in the first half of 2026—including GLM, DeepSeek, Qwen, Hunyuan, Minimax, MiniCPM, etc. —across up to 10 different accelerator platforms. These experiences reveal what Triton needs to support continuously changing model architectures, emerging operator patterns, and heterogeneous backends at production scale.
Finally, through collaboration with the vLLM and SGLang communities, we are building unified cross-accelerator plugins that bring this large operator library directly into widely used inference frameworks. This connects hardware backends, Triton language extensions, operator optimization, and model-serving frameworks into a shared upstream-downstream ecosystem.
Attendees will learn practical techniques for scaling and validating Triton operator libraries, deciding where portable kernels end and hardware specialization begins, and using model-driven requirements to advance Triton itself. FlagGems demonstrates how a large, open operator foundation can help Triton grow from a kernel programming language into a truly cross-accelerator AI software ecosystem.
Hybrid architectures that interleave attention with state-space or linear-attention layers are quickly becoming the default recipe for frontier open-weight models. Models like Qwen and Nemotron all ship variants keeping a recurrent state alongside a KV cache. While this extra state makes hybrid models attractive at long context, it makes standard serving primitives—like prefix caching and speculative decoding—surprisingly challenging: every accepted draft step and every cache hit forces the runtime to decide, per request and per layer, whether to copy or roll back a state. This talk traces the journey we took in vLLM to shrink this overhead to a negligible fraction of the decode step, with Triton as the vehicle at every stage. Our starting point was a Python CPU loop called at every decode step to issue state copies — a design that forced device-to-host synchronizations and introduced a GPU bubble. Rewriting this loop as a single fused Triton kernel that consumes all per-request metadata directly on-device increased the decoding throughput by more than 20% on widely used models such as Qwen3.5 with speculative decoding and prefix caching enabled. We then further optimize the kernel: widening the copy body to uint64 transactions pushed HBM utilization to ~84% of peak on H100 and GB200, and tiling the copy in a 3D grid delivered a further 6× kernel throughput at small batches on GB200. The result is that prefix caching and speculative decoding, the two features that hybrid models were reputed to "not get for free" now run at effectively no overhead. We share the microbenchmark methodology, the end-to-end results and the design lessons.
Problem: Serving long-context LLMs has made the KV cache — not model weights — the memory bottleneck. At 128K context a single request's KV cache runs to tens of gigabytes; at a million tokens it reaches the hundreds of gigabytes, more than any one accelerator can hold. Providers already serve these workloads by spreading KV across many GPUs. The natural next step is a shared memory pool that many accelerators access over a fast fabric — stored once, not copied per accelerator. But such a pool is software-coherent: a write by one accelerator isn't automatically visible to others, so producer/consumer ordering must be established explicitly, not by hardware. Appliances do this with a hardware rendezvous — a signal (atomic increment) and a wait (block until a value) — but it lives below the kernel, invisible to the languages used to write inference kernels.
Solution: We bring the rendezvous into Triton. Signal and wait map directly onto Triton's existing scope=""sys"" atomics (atomic_add, atomic_poll). Using the Triton Plugin Extensions system, with no fork, we add a small, portable API so one kernel can safely read and write pooled KV memory shared across accelerators. The same family composes upward — handoff, barriers, and collective coordination across N participants — with release/acquire ordering. We show how it coordinates common shared-KV patterns: prefix sharing, disaggregated prefill/decode, and context-parallel serving. Finally, we map the design onto a real hardware rendezvous and outline what a memory-centric Triton target still needs — chiefly tag addressability and memory layouts.
Triton has redefined how developers write high-performance GPU kernels through a high-level programming model. Its abstractions are built around the dynamically scheduled execution of GPUs and do not directly align with emerging spatial accelerators such as NPUs. This work explores how to extend Triton beyond its original design scope to support spatial architectures using AMD Ryzen AI as a case study. The key motivation is to improve the programmability of spatial accelerators using Triton’s programming model while exposing these architectures to a broader developer community. This bridges developer productivity with the performance potential of spatial execution. This talk presents a software stack that closes this gap across compiler, runtime, and developer workflows. We introduce Triton-XDNA, a compilation flow built on MLIR-AIR and triton-shared that progressively lowers Triton programs from GPU-style implicit semantics into explicit spatial schedules. MLIR-AIR explicitly represents tiling, data movement, synchronization, and placement, enabling efficient mapping to static architectures without sacrificing programmability. We present a unified HSA-based runtime for heterogeneous systems, enabling coordinated dispatch across GPUs and NPUs. This runtime reconciles different architectures and provides a unified programming interface and execution model for composing workloads across devices. To further assist heterogeneous device programming, we explore agent-driven Triton development. Using MCP-based tooling and LLM-guided workflows, developers can iteratively refine kernels and optimize workloads, complementing traditional autotuning. This talk argues that Triton’s future is heterogeneous, driven by system co-design and a unified runtime. Attendees will understand where explicit compilation is required for spatial architectures, how runtime systems enable cross-device execution, and how agent-driven workflows reshape kernel development.
Floating-point addition is non-associative, so a reduction's exact result depends on the order its elements are combined. That order is controlled by the Triton compiler as a byproduct of its layout system, so bitwise results shift as the compiler evolves or as a kernel is tiled differently. Today, torch.compile gives no guarantee that its reductions match PyTorch eager, or even stay stable across compiler versions.
This is increasingly important to frontier labs, who want bitwise determinism: DeepSeek-V4's numerics section[1] and Thinking Machines' ""Defeating Nondeterminism in LLM Inference"" [2] both show how reduction ordering breaks reproducibility, batch-invariant inference, and staying on-policy in RL. It is also a practical lever for iteration speed, validating the numerics of a large model is expensive and time-consuming, whereas bitwise equivalence lets engineers verify a local change without re-checking the whole model.
To address this we introduce a user-controlled, fixed reduction ordering in Meta's Triton fork the inner-tree reduction, which combines neighbors in a balanced power-of-two tree, keeping results stable across autotuning (we aim to upstream this ahead of the conference). We then build on it to make torch.compile reproduce PyTorch eager's reduction bit-for-bit.
On the eager side we add an inner-tree reduction kernel and expose a small, runtime-free planning API describing its tiling the per-batch tile size and split count, a foundational building block toward our eventual goal of full eager/compile bitwise equivalence. On the Inductor side we add a numerics=""strict"" codegen path that consumes the same API, so eager and torch.compile derive identical tiling by construction. It emits the inner-tree ordering and restructures accumulation into eager's ""tree-within-a-batch, linear-across-batches"" form, spanning persistent, looped, and split reductions. We verify bitwise equivalence live against PyTorch eager at near-parity performance.
NVIDIA is introducing a new CUDA Tile IR backend for OpenAI Triton that lets developers compile Triton kernels directly to CUDA Tile IR instead of PTX without requiring code rewrites or deep CUDA expertise. CUDA Tile extends the CUDA programming model to enable first-class support for tile programming.
This session explains how CUDA Tile, introduced with CUDA 13.1, shifts GPU programming from traditional SIMT thread management to a tile-centric model that targets NVIDIA Tensor Cores with a portable, MLIR-based intermediate representation called CUDA Tile IR. We will walk through the Triton-to-TileIR backend, a new bridge that allows existing Triton deep learning kernels—already expressed in terms of blocks of data and computation—to be compiled and executed on the CUDA Tile IR path, preserving high-level tile semantics throughout the compilation pipeline. Attendees will see how this integration extends the Triton compiler ecosystem, how it enables them to exploit modern NVIDIA GPU features while minimizing hardware-specific code.
Key takeaways for attendees include: understanding the core concepts behind CUDA Tile and CUDA Tile IR; learning how Triton’s Python-based DSL maps naturally onto Tile IR abstractions; and discovering how the Triton-to-TileIR backend can deliver high performance with improved forward portability across future NVIDIA architectures. The session will also highlight practical steps to get started from source, discuss current limitations.
Modern AI workloads demand fine-grained overlap of computation and communication across multiple GPUs. The barrier to achieving this isn't hardware — today's systems are physically capable. The barrier is abstraction mismatch: Triton's tile-based programming model has no corresponding tile-based communication primitive. Developers are forced to choose between low-level HIP/CUDA communication libraries that demand substantial engineering effort, or high-level wrappers that sacrifice compiler co-optimization.
Iris eliminates this mismatch. It is a multi-GPU communication library implemented entirely in Python and Triton — no external communication library dependencies — providing symmetric memory operations at tile granularity. When communication primitives live in the same semantic space as computation, complex overlap patterns become straightforward extensions of existing kernels rather than heroic engineering efforts.
In this talk, I will cover the symmetric heap model at the core of Iris — how it modernizes proven OpenSHMEM concepts for tile-based GPU programming with a pointer translation mechanism that keeps the API minimal and familiar. I will then walk through a taxonomy of compute-communication overlap patterns, from bulk-synchronous to producer-consumer stream concurrency to fused workgroup specialization, showing how each is expressed in Iris with just a few additional lines in an existing Triton kernel. I will demonstrate what native implementation enables that external libraries cannot, and close with evaluation results on AMD multi-GPU systems and the open-source roadmap.
Attendees will leave with a concrete mental model of why fine-grained overlap is hard today, how Iris's abstractions make it tractable in Triton, and practical guidance for their own distributed workloads.
Helion, a Python-embedded DSL that compiles to Triton and other GPU kernel backends, depends on autotuning for high-performance kernels—but it dominates the edit-run-measure loop. Full random search takes 10–20 minutes per compile; the fast default finishes in seconds yet yields kernels 14–31% slower. The current cache worsens this by tying results to exact source code, so renaming a variable or editing a comment invalidates configs. Each kernel's best config is also saved only on the machine that tuned it, so teammates can't reuse it.
We present FROM_BEST_AVAILABLE, a warm-start strategy now default for Helion's quick autotuning. It seeds the search with prior configs matched by input characteristics and device, using a match key decoupled from source identity. Integrated with RemoteCacheBackend, it merges and deduplicates local and remote entries by flat configuration, so a winning config from one machine becomes a warm-start candidate team-wide. In one case, autotuning dropped from 1238 to 92 seconds while matching full-search quality.
You'll leave with a reference repository and the knowledge to implement shared caching, saving your team hundreds of hours of compile time.
Modern attention kernels are limited not only by matrix throughput but by workload scheduling, data movement, synchronization, and state residency. Dense Multi-head Latent Attention (MLA) decoding faces irregular KV-cache lengths and numerically coupled split-KV results, while Kimi Delta Attention (KDA) prefill combines highly parallel chunk preparation with a serial recurrent-state path.
We present two optimizations built in the Triton ecosystem with Triton-TLE. TLE complements Triton's Python workflow, tl.dot, autotuning, and TensorDescriptor support with composable abstractions for explicit shared-memory tensors, asynchronous movement, typed producer-consumer pipes, warp-role partitioning, and synchronization.
For dense FlashMLA, a schedule-compute-combine design first balances KV-block work across SM partitions and splits long requests. A warp-specialized producer moves Q and KV tiles into shared memory, while two consumers process adjacent KV blocks and complementary output halves, exchanging online-softmax statistics and weights. Split outputs and log-sum-exp values are then merged by a numerically stable combine kernel. On H800, this implementation averages 97.5% of DeepSeek CUDA performance and 1.66x the Triton baseline across standard-context tests.
For KDA, we retain FlashKDA's two-kernel chunked algorithm but optimize each critical path differently. K1 uses explicit shared-memory residency, asynchronous loads, tensor-level computation, and CTA autotuning to reduce barrier-induced idle time and improve effective issue rate. K2 keeps the FP32 master state in registers across chunks, while load/store pipes and specialized load, MMA, and store roles overlap peripheral movement around the unavoidable serial recurrence. Across 12 H800 configurations, TLE outperforms FlashKDA in every case, with a 1.38x geometric-mean speedup and about 1.4x on key workloads.
We also optimize other attention kernels, including GDN/GDN2, GLA, and NSA. Attendees will learn how to express CUDA-style cooperative execution while remaining within the PyTorch and Triton ecosystems.
AOT-T(ensor) is a lightweight, general-purpose compilation framework that bundles optimized GPU binaries directly with the model, eliminating JIT runtime overhead and making model inference ready for C++-only serving runtimes. Neither Triton nor NVIDIA's CuteDSL natively supports the ahead-of-time compilation required for latency-sensitive, Python-free deployment. AOT-T fills this gap with a unified, language-agnostic workflow supporting both Triton and CuteDSL, while resolving longstanding challenges around static builds, version management, and debugging. It is currently deployed on multiple production models at Meta and is on track for open-source release.
The Triton compiler does an enormous amount of work to turn a @triton.jit kernel into fast PTX. Triton’s autotuner does more on top, sweeping the kernel's meta-parameters like block size, number of warps, number of stages and so on. But the final lowering step of turning PTX into SASS has so far been left untouched and relies on default PTXAS heuristics which often leaves a lot of performance on the table.
This poster introduces CompileIQ, NVIDIA’s freely available hyper-parameter optimization engine for compiler heuristics which now allows the developer to access PTXAS’s advanced controls interface - giving the developer finer grained control of register allocation strategy, instruction scheduling, loop unrolling and other code-generation heuristics which suits the specific workload at hand.
Triton developers already think about kernel autotuning as a search, and treating the compiler itself as a tunable entity is a natural extension of their mental model. Instead of relying on PTXAS defaults, they can now benefit from bespoke optimizations for their own workload.
In addition, CompileIQ supports a hybrid or mixed-search space optimization: the Triton kernel configurations and PTXAS controls are explored jointly, so algorithmic and backend compiler choices are co-optimized leading to the best performance uplifts.
CompileIQ already has native support in Triton using the existing `ptx_options` API, and has shown meaningful performance uplifts in critical attention workloads across various different architectures.
Triton has become the de facto standard for operator development in the industry. With the rapid development of diverse computing capabilities, in addition to GPUs, NPUs as new backends also require Triton's support. Based on features such as pipeline scheduling, efficient memory access, and extreme reuse of on-chip cache on Ascend, we have achieved performance nearly comparable to hand-written Kernels, and it has already been deployed in many top-tier OTT customers. We would like to share our experiences and lessons with the community as a vision for the next phase.
An increasing number of hardware backends now support Triton. In addition to NVIDIA and AMD, new backends including Ascend have been successively onboarded. Previously, TritonParse only supported in-tree backends. Emerging backends such as Ascend could not leverage TritonParse to parse compilation pipelines, resolve source-code mappings, or reproduce runtime issues. Since the original analysis logic was tightly coupled to specific backend implementations, integrating new backends remained challenging.
To resolve this issue, we have fully decoupled the entire processing workflow from backend-specific logic. We implement a unified, backend-agnostic pipeline that covers trace parsing, IR extraction, IR analysis, case reproduction generation, and frontend visualization. All backend-specific discrepancies are encapsulated in independent backend adapters. Each adapter explicitly defines the corresponding compilation stages, source mapping resolution rules, required IR analysis passes, and the method to generate readable assembly from binary artifacts. Adapters support unified registration, on-demand instantiation and mutual isolation.
With this extensible mechanism, TritonParse now stably supports the compilation chains of NVIDIA, AMD, and Ascend, with all backends working independently without conflict. Onboarding a new backend only requires implementing and registering a corresponding adapter, with no changes to the core framework. Hardware vendors with Triton-compatible backends can integrate their backend with minimal effort, acquiring complete and consistent operator analysis and debugging capabilities equivalent to mainstream official backends.
FlagTree DevTools provides debugging and performance-analysis capabilities for Triton kernels compiled by FlagTree. It includes FlagTree Debugger, which correlates Triton source and compiler IR with runtime values and memory access information, and FlagTree Profiler, which records execution context, timing, and hardware performance metrics. Together, these tools provide a consistent workflow for locating correctness issues and performance bottlenecks across supported accelerator backends.
Hybrid architectures that interleave attention with state-space or linear-attention layers are quickly becoming the default recipe for frontier open-weight models. Models like Qwen and Nemotron all ship variants keeping a recurrent state alongside a KV cache. While this extra state makes hybrid models attractive at long context, it makes standard serving primitives—like prefix caching and speculative decoding—surprisingly challenging: every accepted draft step and every cache hit forces the runtime to decide, per request and per layer, whether to copy or roll back a state. This talk traces the journey we took in vLLM to shrink this overhead to a negligible fraction of the decode step, with Triton as the vehicle at every stage. Our starting point was a Python CPU loop that walked every request at every decode step to issue state copies — a design that forced device-to-host synchronizations and introduced a GPU bubble. Rewriting this loop as a single fused Triton kernel that consumes all per-request metadata directly on-device increased the decoding throughput by more than 20% on widely used models such as Qwen3.5 with speculative decoding and prefix caching enabled. We then further optimize the kernel: widening the copy body to uint64 transactions pushed HBM utilization to ~84% of peak on H100 and GB200, and tiling the copy in a 3D grid delivered a further 6× kernel throughput at small batches on GB200. The result is that prefix caching and speculative decoding, the two features that hybrid models were reputed to "not get for free" now run at effectively no overhead. We share the microbenchmark methodology, the end-to-end results and the design lessons.
Triton was designed around GPU programming, and much of its lowering stack reflects that history. Triton-CPU provides a portable CPU path, but on Arm, dot-product patterns without a dedicated lowering fall back to generic vector IR. This is sufficient for correctness, but it does not always produce code that makes good use of SME, SVE2, i8mm, or SDOT. The gap becomes especially visible in W4/W8 inference, where small kernels, memory traffic, and launch overhead all matter.
Our Arm path targets Apple M4 and SVE2-capable Linux AArch64 systems. During lowering, two-dimensional addressing and stride information from Triton loads are kept intact. Matmul and low-bit quantization patterns are then converted into loop-based microkernels instead of expanding a complete matrix tile into hundreds of independent vector operations. This produces smaller IR and keeps register pressure under control. Normalization, RoPE, and reduction kernels are tiled according to the vector width of the target processor.
The M4 path uses SME, while the Linux AArch64 path generates SVE2/i8mm and SDOT code through the compiler. Dispatch is gated by CPU features and the active SVE vector length, with the generic lowering retained as a fallback. ISA information is also included in the JIT cache key. For deployment, AOT compilation and a small C++ launcher allow the generated kernels to be called without the Python runtime.
Across the operators and shapes tested, the generated SVE2/i8mm kernels matched or outperformed optimized C/ACLE implementations. Model-level profiling also showed that these operator-level gains carried through to end-to-end quantized inference. The resulting design keeps Triton’s high-level programming model while organizing the generated code around the Arm hardware that executes it.
Kernel development is becoming a major bottleneck for rapidly evolving AI workloads, especially across heterogeneous accelerators. This poster presents KernelGen, a self-evolving AI kernel engineering system that automatically generates and optimizes Triton kernels from PyTorch operators and reference implementations.
KernelGen combines code generation, correctness verification, benchmarking, profiling, failure diagnosis, and iterative optimization in a closed loop, while continuously accumulating reusable optimization trajectories and hardware-specific knowledge.
To date, KernelGen has generated and optimized more than 600 kernels, including over 400 hardware-specialized implementations, and supports nine accelerator platforms. The system covers key workloads such as Attention, MoE, normalization, sampling, and matrix operations.
We share practical lessons on building AI agents for Triton kernel engineering, improving performance through execution feedback, and scaling high-performance kernel development across diverse hardware.
We present TritonPPM, a performance prediction model for GPU kernels written in Triton. TritonPPM can predict the running time of a kernel for given inputs with high accuracy without needing to actually compile or launch the kernel with those inputs. As such it can dramatically speed up traditional autotuning by statically reducing a combinatorially large search space to a small number of top candidates.
At the core of TritonPPM's architecture is a sophisticated static analysis of Triton source code that generates symbolic formulas for performance-relevant features (such as bytes accessed, FLOPs computed, and resources needed) in terms of the parameters of the kernel. Given inputs, these formulas are instantiated and fed to a simple tree-based regression model that predicts the running time.
The static analysis combines type analysis for tracking sizes of block-level values, scalar evolution analysis for averaging memory and compute costs over loops and grids, dependency analysis for estimating intra-block span, and liveness analysis for estimating register and shared-memory pressure and spill-induced local traffic. The generated features are not kernel-specific. As such the regression model is forced to learn the dynamics of the hardware.
We evaluate TritonPPM on >100 kernels drawn from Triton tutorials and TritonBench, ranking predicted running times against measurements for >900 input shapes and >250K input configurations. Our metrics are based on leave-one-kernel-out cross-validation. The smallest k such that top-k predictions contain a near-optimal configuration is 9.2 on average and 25 at p90. The slowdown relative to optimal for top-10 is 2.6% on average and 5.9% at p90, and is zero for 83% of input shapes. With a fixed budget of 10% of the configuration space, 91% of input shapes contain a near-optimal candidate and average slowdown is 0.9%. On average, static analysis takes only around 0.34 seconds per kernel, while inference takes around 52ms per 1000 inputs.
As AI systems scale beyond a single server, the rack is becoming the unit of computation. AMD Helios connects up to 72 AMD Instinct MI455X GPUs through a memory-semantic UALink-over-Ethernet fabric with single-hop, fixed-latency connectivity. Yet today’s GPU programming models still expose communication as host-orchestrated collective calls. Once execution reaches a collective, Triton’s compiler can no longer reason about communication alongside computation, limiting fusion, pipelining, and scheduling opportunities.
Hardware addressability alone is not enough for a kernel. UALink’s memory semantics naturally align with a symmetric-memory programming model, allowing remote memory operations to be expressed as ordinary Triton memory accesses rather than opaque library invocations. We present a communication abstraction that enables GPU-initiated communication from Triton, spanning GPUs within a node and across an entire rack.
Using this model, communication becomes another operation the Triton compiler can schedule alongside computation rather than a separate execution phase. We discuss the design decisions behind mapping the abstraction onto AMD’s rack-scale fabric, including topology-aware placement, communication-compute overlap, and the trade-offs between portability and hardware specialization (communication CUs, push vs. pull, fusion thresholds). Real examples from distributed AI workloads—including tensor parallelism and MoE communication—illustrate how the same Triton program scales naturally from node-local execution to rack-scale communication.
Finally, we examine what today’s Triton language enables—and what it does not. While current memory scopes are sufficient for existing GPU programming models, rack-scale execution exposes a missing abstraction between device-local and system-wide synchronization. We conclude by discussing how future Triton memory models could evolve to make rack-scale communication a language construct rather than an implementation detail.
PyTorch 2.0's compiler (PT2 Inductor, torch.compile) uses max-autotune-gemm to benchmark dozens of Triton configurations against alternative backends (e.g., ATen/cuBLAS), yielding significant performance gains for Triton GEMMs in production models. However, max-autotune-gemm incurs substantial host-side compilation time and memory overhead, historically 2–4× over baseline, because it must compile multiple Triton kernels and allocate extra buffers to simulate flushed caches. Furthermore, while Triton's key strength in Inductor is epilogue fusion (fusing GEMMs with surrounding pointwise operators), max-autotune-gemm selects the best config for the standalone GEMM without considering how fusion affects register pressure, shared memory usage, or occupancy, potentially leaving performance on the table.
We present two complementary solutions. First, Diode: an ML-based predictive modeling framework that replaces exhaustive benchmarking with intelligent config selection. Diode is a lightweight neural network trained on large-scale profiling data across heterogeneous GPU platforms (H100, H200, GB200, MI300). During compilation, it performs fast CPU inference—predicting optimal configs in milliseconds from a search space far larger than the ~20 hardcoded candidates, reducing compilation overhead to ~10% above baseline while matching or exceeding max-autotune-gemm kernel performance. Second, we leverage Meta Triton's autoWS (auto warp specialization) framework, which uses latency-driven scheduling and warp-group partitioning to push Triton's performance on Blackwell. Within torch.compile, we aim to extend this into fusion-aware config promotion: selecting GEMM configs that account for post-fusion register allocation, shared memory constraints, and occupancy, rather than optimizing the GEMM in isolation.
Together, these approaches enable faster compilation and better end-to-end kernel performance for Triton GEMMs in production PT2 workloads.
Case study 1: Analysis pass
How do you know whether a kernel is compute-bound or memory-bound before you ever run it? This talk walks through building a Triton extension that computes a kernel's arithmetic intensity at compile time, so you can identify the likely performance bottleneck early. We'll show how to use this information to evaluate an algorithm's efficiency and automatically prune the autotuning search space before a single kernel is launched. Along the way, we'll demonstrate how straightforward it is to build and load a Triton extension that pulls actionable information out of the compiler.
Case study 2: uTLX in production
Triton’s Plugin Extensions infrastructure was created in the past year because achieving peak performance on modern GPUs requires augmentation of the compiler and language: an example of this is TLX. Following up from last year’s TLX talk, we will demonstrate how we have reconstituted TLX through this new infrastructure as a Python wheel we call µTLX (“Micro-TLX”). We will also demonstrate real-world kernels that benefit from the distribution of these Triton augmentations without the need to fork Triton itself.
Agentic applications are becoming a major use case for LLMs, but they also create distinct challenges for inference systems. Unlike traditional chatbot workloads, agentic workloads repeatedly extend cached prefixes and generate text over long contexts. At the same time, model architectures continue to evolve, introducing mechanisms such as sparse and linear attention. GPU architectures are evolving as well: NVIDIA and AMD GPUs have different compute and memory hierarchies, and each hardware generation requires its own optimization strategy. Together, these trends create a complex optimization space for inference engines.
We present TokenSpeed-Kernel, the open-source kernel library behind the TokenSpeed inference engine. It addresses these challenges by separating kernel selection and hardware-specific implementations from model and runtime code. TokenSpeed-Kernel defines hardware-agnostic interfaces for attention, MoE, and other common LLM operators. Behind these interfaces, it brings together portable Triton implementations and highly optimized, hardware-specific kernels written in DSLs such as Gluon and CuTe DSL. Built-in validation, benchmarking, shape capture, and profiling make the optimization loop reproducible for both engineers and coding agents.
With this approach, TokenSpeed has demonstrated state-of-the-art kernel and end-to-end inference performance on both NVIDIA and AMD GPUs. Its modular and extensible design also provides a foundation for future model architectures and GPU generations.
Problem. System architects and simulation engineers need answers Triton's profilers can't give: how will a kernel behave on hardware — or a memory system — that doesn't exist yet? Every Triton profiler (Proton, FlashInfer, triton-distributed) measures one run on real hardware and emits a timeline. But a measured 40µs on today's GPU can't be replayed on a future accelerator or a pooled/disaggregated-memory fabric. Answering ""what if"" needs an execution graph — a dependency DAG of compute, communication, and memory a simulator can re-time — not a timeline. Today's only source, PyTorch's op-level execution-trace collection, records each fused Triton kernel as one opaque node — nothing to re-cost for new hardware.
Solution. We built a prototype that turns Triton's post-optimization IR (TTGIR) into Chakra, the open MLCommons execution-trace standard that simulators like ASTRA-sim already consume — reusing it, not inventing a format. Instead of op-level collection's single opaque node per kernel, it expands each kernel into its semantic phases (memory-bound loads, compute-bound MMA, reductions, writeback) with real dependencies and memory-access shapes from the IR — for a broad class of kernels, including hand-written ones. The output is schema-valid Chakra, ready for any Chakra feeder.
We propose extensions to make it a simulation tool: attaching Proton timing and feeding open-source ASTRA-sim and internal company simulators; re-pricing phases — via a cost model or higher-fidelity simulation — for hardware that doesn't exist yet; and adding communication nodes via triton-distributed. Together these enable what-if analysis across topologies and memory systems for future products.
A good performance engineer is one that knows how to make your code faster. A great one can tell when it's worth doing so. Software optimization balances memory against compute, latency versus throughput, total energy for instantaneous power. But the most fundamental tradeoff one must make is one of invested effort. Model intelligence is governed via well-studied scaling laws whose predictive power drives the allocation of staggering sums of money.
Performance ""scaling"" is much less mature and yet manages an even rarer resource: the attention of just a few GPU experts and compiler engineers around the world. Maximizing the value of their work is one of the most pressing problems for AI research today.
The release of Triton was a major step forward for scaling performance. For the first time, engineers could write their own GPU kernels with a syntax nearly as approachable as the PyTorch they were familiar with. With just a bit of effort, they could realize dramatic wins. Since then, a number of tools have sprung up promising to fill gaps in the ""effort-to-performance"" curve. Still, holes remain, particularly when it comes to ergonomics and correctness.
In this session, we discuss our process for taking high-level PyTorch code and iterating on it all the way down to the hardware. We'll cover the tools that let our engineers pull their Python abstractions into Triton's limited ""JIT"" environment, test advanced Gluon pipelines with the knowledge they are provably correct, and letting agents complement our humans by giving them actionable feedback at the SASS level that they wouldn't get from Nsight Compute. Our approach has allowed us to hit state-of-the-art MFU numbers for the many custom kernels in our stack, and we'd love to share our approach with the Triton community.
At Meta, recommendation systems power the experiences of billions of people, with hundreds of production models continuously evolving across heterogeneous GPU clusters. Achieving high GPU efficiency at this scale requires optimizing diverse and rapidly changing workloads across the full training stack, while keeping pace with fast production development cycles. To address these challenges, we developed GEO (Generative End-to-end Optimization), an AI-native optimization framework that coordinates optimization across computation kernels, memory, and parallelism. Our vision is to take a model as input and automatically produce a fully optimized, kernelized model that runs efficiently on distributed GPUs. This talk will share our experience developing GEO and leveraging Triton kernels to improve GPU performance at production scale.
This talk looks back at the last year of Triton development, from plugins and multi-CTA support to Gluon improvements and many smaller paper cuts.
I'll discuss why the project has been putting more emphasis on Gluon, some of the optimizations we’ve built with it, and how increasingly capable coding agents are changing the way we work. I'll also discuss how we can potentially bring agents into the optimization flow.
Jhalak Patel is a System Software Engineer at NVIDIA working on CUDA Tile and the Tile IR compiler. Previously, Jhalak worked on TensorRT, focusing on deep learning inference and quantization. This experience spans GPU compiler development and optimizing neural networks for efficient execution on NVIDIA GPUs.
Meta runs GPU workloads across a wide spectrum, and the right authoring strategy differs sharply along it. This talk surveys the authoring stacks, compilers and runtimes we use in production, and how GPU consumption really drives the optimization effort applied to each and thus the approach. We look at how coding agents are changing that calculus, and at the industry trends that follow: unavoidable fusion, a proliferation of GPU DSLs, and increasingly specialized silicon weighed against the pull of standards. Triton is the largest GPU kernel authoring surface at Meta, with thousands of custom kernels; we close on where it fits in our datacenter today and what it must address to remain the default tomorrow.
Leading end‑to‑end system modeling and simulation of large‑scale AI platforms. His work centers on translating real AI workloads into executable system models that expose performance, power, and cost tradeoffs across complete data‑center designs. He leads the definition and evolution of Murals, Marvell’s system‑level simulation environment, enabling architectural exploration and evidence‑based decision‑making before hardware deployment. His expertise lies in workload realism, system abstraction, and architectural validation, bridging early‑stage concepts to deployable AI infrastructure.
Dr. Burkhard Ringlein is a Research Staff Member in the AI Platform team of IBM Research, based in Zurich. He is an accomplished AI systems researcher and designs, builds, debugs, and optimizes practical systems for low-latency, high-throughput machine learning applications. Currently, he improves the platform- and performance-portability of generative AI applications and contributes to open-source projects like vLLM, Helion, and Triton.
