August 1,The 9th Meet AI Compiler technical salon, hosted by HyperAI, successfully concluded in Zhongguancun, Beijing.This salon was as popular as ever, with industry experts giving insightful presentations. It brought together technical leaders from leading companies and research institutions such as the Beijing Academy of Artificial Intelligence, the TileRT team, Tencent, Huawei Ascend, and Zhiyuan Innovation to share in-depth insights on cutting-edge technologies, underlying optimizations, open-source ecosystems, and application scenarios in the field of AI compilation. It provided a rich and informative technical feast for developers and professionals in attendance.

on site,Ma Lingxiao, a core member of the TileRT team, gave an in-depth presentation on the latest developments of TileRT, titled "Speed is Intelligence: Computational Exploration and Collaborative Design for Ultra-Low Latency Large Model Inference."From the evolution of AI compilers and runtime architecture to model-system co-design, as well as the integration results and practical cases with many leading companies and technology communities such as Z.AI, Xiaomi MIMO, and vLLM, the presentation details the software stack for large model computing with ultra-low latency.

HyperAI has compiled and summarized the speech without altering its original meaning. The following is a transcript of the speech.

Inference speed has become a core competitive advantage in high-value AI scenarios.

ChatGPT emerged in late 2022 and early 2023, showcasing astonishing capabilities. However, in retrospect, the early models primarily demonstrated applications in areas like chatting and companionship, with their actual value far lower than their computational costs. The output of large models only needed to keep pace with human reading speed. At that time, the focus was on the overall system throughput, not the model's output speed.

From the end of 2025 to the present, in less than a year, the emergence of coding agents like Claude Code has revealed more potential capabilities beneath the surface of large-scale models, such as genuinely helping people improve productivity. In this case, compared to the cost of computing power, it has actually generated higher practical value. Therefore,Recently, we have seen a surge in demand for computing power, which is a direct reflection of the need for high throughput.

Looking ahead based on this development trend, as model capabilities continue to grow, models will inevitably participate in more automated tasks, such as AI factories, quantitative decision-making, and real-time risk control projects. In this scenario, the model's output will be the model itself, thus creating an AI-automated workflow. At this stage,The faster the model runs, the higher its efficiency. Ultimate model speed will become a core factor determining productivity and user experience.

The latest industry news is also confirming the above speculation one after another:

* In late 2025, Nvidia acquired Grop for a huge sum to expand its high-speed inference capabilities;

* In February 2026, Anthropic launched Claude Fast, a high-speed version of the technology.

* In the same month, OpenAI, in collaboration with Cerebras, launched GPT-Codex-Spark;

* In May, Cerebras announced that it had achieved an inference speed of nearly 1,000 TPS for a 1T-level model through wafer-level chip technology.

* In addition, just two or three weeks ago, OpenAI partnered with Cerebras to enable its most advanced flagship model, GPT-5.6-Sol, to offer a high-speed inference service of 750 TPS.

Test-Time Scaling (TTS) is actually the main way to enhance model performance.Assuming a given time window, say 10 seconds, a slower model inference service at 50 tokens/s can only generate 500 tokens of thought. However, if the inference speed is increased from 50 tokens to 1000 tokens, it will generate 10,000 tokens of thought. Undoubtedly, within the same time window, faster inference will result in better model intelligence.

So, you might be wondering, there are many ways to implement TTS, such as Best-of-N, which can achieve stronger intelligence with just batch processing, so why pursue speed? This requires breaking down TTS.Considering it in terms of both depth and width,While TTS (Text-to-Speech) tasks can achieve their goals through batch processing, this model operates in a serial mode from the perspective of a long chain of thought or a series of agent execution links. This is unattainable with batch processing, and the depth of serial processing can only be achieved through extreme inference speed.

Let's look at an example from an SWE-bench trajectory, specifically an example from SWE-smith trajectory statistics. The actual execution trajectory shows that it averages about 30 model execution steps, generating 10,000 tokens. At a speed of 50 tokens/s, this would take approximately 200 seconds, or over 3 minutes. However, if the model speed is increased to 1000 tokens/s, it can complete the task in just 10 seconds. This will be a completely different experience.At normal speeds, people might be interrupted by the waiting time, but if it only takes a few seconds to complete, they will experience a completely real-time interactive process.

In addition, let's look at other scenarios. For example, high-frequency quantitative trading usually requires feedback in the range of about 10 milliseconds, real-time voice calls may require feedback in the range of hundreds of milliseconds, and financial risk control may also require feedback in the range of seconds. Therefore, the time feedback in these real-time interaction scenarios is roughly between a few seconds and 10 seconds. If these time thresholds cannot be met, the model cannot be truly implemented in these scenarios.

Therefore, we will push the reasoning speed to its limit.On the one hand, we hope to empower more application scenarios; on the other hand, we also hope to empower the generation of more powerful model intelligence.

Achieving maximum speed improvement through the collaborative approach of algorithms, systems, and hardware.

So, how do we maximize the decoding speed of a model? This is actually determined by three factors.The first is the performance of the hardware itself.Including computing power, bandwidth, and interconnectivity, it determines the upper limit of decoding speed;The second is the efficiency of the software system.In other words, how to maximize the performance provided by the hardware;Third is the algorithm.For example, by reducing computation and memory access through quantization, or by using methods such as speculative decoding, more tokens can be generated from a single forward computation.

Let's first look at system efficiency.Starting from first principles, there is still a significant gap between the actual speed of model inference and the theoretical upper limit provided by the hardware. This is because, in addition to the computation itself, there are fixed overheads during execution, such as kernel startup, scheduling, synchronization, and communication. As hardware performance continues to improve, these overheads become increasingly prominent, gradually becoming key bottlenecks for low-latency inference.

There are already many existing technologies for reducing these fixed overheads, such as CUDA Graph and PDL. CUDA Graph mainly reduces the overhead of kernel launch, while PDL allows two kernels to establish dependencies and pipelines in advance, but essentially it can only perform a one-hop overlap. For achieving ultimate performance, this is far from enough. Going further, we need to break the execution boundaries formed by the kernel itself and further decentralize the scheduling granularity.This allows computation, memory access, and communication, which were originally separated by different kernels, to overlap more fully.

In fact, kernel fusion is not a new concept; much exploration has been conducted since the DNN era. We have previously conducted a series of explorations on issues such as cross-operator scheduling, kernel generation, memory access, dynamic control flow, low-precision computation, and software pipelining, including Rammer, Roller, Welder, Cocktailer, Ladder, and PipeThreader.Gradually, a compilation capability has been developed, ranging from graph-level optimization to fine-grained execution at the tile level.It has been validated in various hardware and real-world scenarios.

However, these capabilities cannot be directly transferred to large model inference.Compared to traditional DNNs, the LLM era brings three new challenges:Improved hardware performance makes fixed overheads such as scheduling and communication more prominent; MoE routing, sparse computation, and speculative decoding mechanisms increase runtime dynamism; and the expansion of model size makes cross-card and cross-machine collaboration more common. Therefore, LLM-oriented compilation systems need to handle fine-grained execution, dynamic scheduling, and distributed collaboration simultaneously.

Faced with these challenges,The core idea of TileRT is to no longer regard the Kernel as the basic boundary of software scheduling, but to further break this boundary, decompose execution to a finer granularity, and re-orchestrate computation, memory access, and communication from a global perspective.

Let's first look at the scheduling process of the existing framework. For a large model, it is usually described as a computation graph and then deployed to the hardware for execution in an OP-by-OP, Kernel-by-Kernel manner. Each Kernel has its own startup, computation, and synchronization process, and the boundaries of the Kernel also limit the space for further overlap between different stages.

TileRT aims to break down precisely this boundary. By further breaking down computations within the kernel into finer-grained tile-level tasks, software can rearrange the execution order across existing kernel boundaries, allowing subsequent computations, memory accesses, and communications that already satisfy dependencies to begin earlier, forming a tighter pipeline. Essentially, we are not simply merging multiple kernels into a larger kernel, but rather breaking the kernel's limitations on the scheduling space, thereby achieving more overlap that was previously impossible.

TileRT uses real-world case studies to validate the feasibility of its technology.

TileRT first represents the model as a data flow graph and then generates a tile-level microkernel. It then performs fine-grained compilation and scheduling to generate the execution engine. This engine orchestrates computation, memory access, and communication, and maps tasks to different hardware units to fully utilize device performance.

Using this methodology, TileRT has already been validated in real-world production services. In May of this year, we partnered with the Zhipu GLM team to launch the GLM-5.1-HighSpeed high-speed inference service.It achieved a production-grade output speed of 400 tokens/s.One representative optimization is the Sparse Attention for GLM-5.1. Traditional Tensor Parallels typically have different GPUs execute the same computational logic, while TileRT splits it into different heterogeneous workers: one GPU handles the Sparse Indexer, Top-K selection, and routing, while the other seven GPUs handle computationally intensive tasks such as MLA and Attention. Communication, reduction, and synchronization are further integrated into the Tile-level pipeline. This allows different stages to adopt scaling methods more suited to their specific characteristics, reducing redundant computations and synchronization waits. Without MTP, the generation speed for short sequences is approximately 300 tokens/s, and for long sequences, approximately 200 tokens/s; with MTP-3 enabled and an average receive length of 3.2, short sequences can exceed 600 tokens/s, and long sequences approximately 400 tokens/s.

If we want even higher speeds, we also need to optimize the output efficiency of each byte of token.To optimize the token output per byte, there are two aspects to consider:On the one hand, we want to generate more tokens per step, so we can use MTP or more aggressive speculative decoding schemes like DFlash or DSpark; on the other hand, we can reduce the amount of data required per token by using some quantization methods to reduce the amount of data.

In June, we collaborated with the Xiaomi MiMo team to design the model and system based on this idea.In terms of quantization, the MoE Expert module of MiMo-V2.5-Pro constitutes the majority of the model parameters. Therefore, only the Expert module is quantized using FP4 QAT, while other modules maintain their original accuracy. Benchmarks show that the overall performance of the quantized model is close to that of the original FP8 model, while significantly reducing model size and memory access pressure. For speculative decoding, DFlash uses block-level parallel prediction, generating a set of candidate tokens in each forward pass, which are then uniformly verified by the main model, thereby reducing the serial overhead of traditional autoregressive drafting.

These models are used for collaborative optimization with the system.MiMo-V2.5-Pro-UltraSpeed ultimately achieved a generation speed of over 1000 tokens/s for 1T models on a single general-purpose 8-GPU node.More importantly, this result did not rely on wafer-level or other dedicated inference chips, demonstrating that through sufficient model-system collaboration, general-purpose GPUs can even surpass the extreme inference speed range previously achievable only by dedicated hardware. This result shows that as system performance gradually approaches the hardware's limits, further speed improvements are no longer just a matter for the inference engine itself, but require the joint evolution of model structure, compilation system, and hardware execution methods.

Of course, simply pursuing extreme decoding speed is not enough. A truly usable large-scale model service also requires OpenAI-compatible APIs, request scheduling, prefix caching, tool calls, and mature operational capabilities. If the entire service system had to be rebuilt just to integrate a high-speed decoding engine, the engineering costs would be prohibitively high. Therefore,We partnered with the vLLM community to allow TileRT to focus on low-latency decoding while leveraging the mature vLLM ecosystem.

The separation of Prefill and Decode provides a solid foundation for this combination. With Prefill and Decode decoupled, the Decode side can become a pluggable execution engine. In the joint TileRT and vLLM solution, Prefill, request scheduling, Chunked Prefill, Prefix Cache, and service APIs continue to be provided by native vLLM. Only requests with high single-user generation speed requirements enter the TileRT decoding pool. This preserves the full ecosystem capabilities of vLLM while introducing the ultra-low latency decoding performance of TileRT.

In its implementation, TileRT connects entirely through the vLLM's exposed Connector interface, requiring no modification or fork of vLLM, nor any intrusion into its internal Workers. The routing layer marks latency-sensitive requests, and the TileRT Connector only accepts these requests; other requests still follow the native vLLM processing path. The two decoding pools can share the same vLLM Prefill service. The state generated by Prefill is handed over to the corresponding decoding node through transport engines such as NIXL or Mooncake, and executed in parallel with subsequent Prefill processes.

Therefore, within the same deployment, latency-sensitive requests such as real-time agents and interactive coding can be routed to TileRT; regular requests targeting high concurrency and overall throughput continue to be handled by the vLLM native decoding engine. Both paths provide the same OpenAI-compatible API, and switching between services only requires adjusting the routing strategy.This enables TileRT and vLLM to form a truly coexisting heterogeneous inference service architecture, where each leverages its strengths.

The TileRT ecosystem is gradually growing.

Currently, TileRT supports multiple models and multiple ecosystem deployments:

At the end of last year, we released the prototype adapter for DeepSeek v3.2, which can achieve an inference speed of 500 TPS.

In May of this year, we and Zhipu launched a high-speed version of GLM-5.1, achieving inference services of 400 to 600 TPS;

In June of this year, we collaborated with Xiaomi MIMO to achieve a breakthrough of 1000 TPS for the first time for a 1T-level model, breaking the global inference speed record;

Subsequently, we partnered with MIND LAB to release the first platform that supports high-speed multi-LoRa inference, enabling real-time UI production within 5 seconds;

Finally, in collaboration with the vLLM community, we released a coexisting heterogeneous PD separate deployment solution, which requires zero modifications and supports ecosystem compatibility.

Finally, here are the latest developments in the Tile-AI community. The Tile-AI community is a software ecosystem we've redesigned based on Tile, specifically for large models and new hardware architectures, including the well-known TileLang. TileScale is a programming and compilation framework designed for distributed architectures, including in-band distributed core architectures.This also includes die-to-die interconnects, chip-to-chip interconnects, and multi-machine interconnects. We have unified and abstracted this multi-layered distributed structure in the description of Tile, and designed such a framework. TileRT is the inference engine for extreme speeds described above.

In addition, we recently released TileFoundry and TileOPs. TileFoundry is an AI-driven framework for automatically generating operators, and its output is TileOPs—a TileLang operator library for large models. At the same time, we are also about to release the TileSight performance analysis model, primarily for tile-level performance modeling, providing guidance for tile-level program performance optimization and architecture exploration analysis.