ML Drift: Google’s Next-Gen GPU Engine for On-Device AI Inference

Learn how Google ML Drift accelerates on-device AI inference with GPU optimization, LiteRT, cross-platform support, and next-generation edge AI.

ML Drift: Google’s Next-Gen GPU Engine for On-Device AI Inference

Imagine opening an AI-powered application on your phone and getting an instant response without sending every piece of data to a remote server.

A camera effect responds in real time. An AI model processes an image locally. A generative AI feature produces output without waiting for a round trip to the cloud.

That is the direction in which on-device AI is moving.

But there is a difficult engineering problem behind the experience: modern devices contain different GPU architectures, drivers and graphics APIs. Building fast AI inference that works reliably across this fragmented hardware environment is not easy.

Google AI Edge is addressing this challenge with ML Drift, an open-source GPU compute engine designed specifically for on-device AI and machine learning inference.

Google announced ML Drift on October 8, 2026, describing it as a cross-platform GPU engine that abstracts GPU and low-level API complexities across OpenGL ES, OpenCL, Metal and WebGPU. It serves as the GPU acceleration engine inside LiteRT while also being available as a standalone library. Google Developers Blog

Read the original Google Developers Blog announcement

So, what exactly is ML Drift, why does it matter, and what does it mean for developers building AI applications?

Let’s break it down.

What Is ML Drift?

ML Drift is Google’s open-source, cross-platform GPU compute engine for on-device AI and machine learning inference.

Its primary purpose is to make it easier to run AI workloads efficiently on GPUs across different types of hardware and platforms.

Instead of developers having to deal with the differences between multiple GPU architectures and APIs, ML Drift provides a more unified foundation.

Google says ML Drift supports GPU environments involving:

• OpenGL ES
• OpenCL
• Metal
• WebGPU

The technology is also the GPU acceleration engine inside LiteRT, Google’s on-device AI runtime, while developers can use ML Drift independently for custom graphics and inference runtimes. Google Developers Blog

In simple terms:

ML Drift attempts to make GPU-powered AI inference more portable, modern and efficient across edge devices.

This is important because edge AI is no longer limited to small image-classification models. Developers increasingly want to run computer vision, audio, spatial processing and generative AI directly on consumer hardware.

Why Does On-Device AI Need a New GPU Engine?

Running an AI model in a cloud data center is different from running it on millions of phones, laptops and other edge devices.

A cloud environment can use relatively predictable accelerator infrastructure.

Edge devices are much more complicated.

One smartphone may use a Qualcomm GPU, another may use an Arm Mali or Immortalis GPU, while computers can use different GPU technologies and drivers.

Developers therefore have to think about:

• GPU architecture
• Drivers
• Memory bandwidth
• Graphics APIs
• Model operators
• Tensor layouts
• Device power consumption
• Runtime compatibility

Google says the older TensorFlow Lite GPU delegate provided an important foundation for GPU acceleration, but modern workloads have expanded significantly. Today’s edge models can include real-time computer vision, audio, depth processing and high-parameter generative AI. Google Developers Blog

This creates a new requirement:

The software infrastructure must evolve alongside the models.

ML Drift and LiteRT

One of the most important parts of the announcement is ML Drift’s relationship with LiteRT.

ML Drift acts as the GPU acceleration engine within LiteRT.

That means developers using LiteRT can benefit from ML Drift’s GPU capabilities without necessarily building an entire GPU execution system themselves.

Google also describes a standalone option for developers who want lower-level control or need to integrate GPU computation into custom runtimes. Google Developers Blog

This creates two broad development paths:

Application developers

Developers building AI-powered applications can use the LiteRT ML Drift accelerator for features such as:

• On-device AI
• Computer vision
• Generative AI
• Real-time effects
• Local inference

GPU and systems engineers

Developers requiring more control can work directly with ML Drift’s standalone C++ APIs and custom compute graphs.

This distinction is useful because not every AI developer needs to understand GPU internals.

The Key Features of ML Drift

ML Drift is more than simply another GPU backend. Google highlights several architectural changes designed to support both traditional machine learning and modern generative AI.

1. Tensor Virtualization

One of the major changes is tensor virtualization.

Historically, GPU implementations could require logical tensors to be mapped directly to physical GPU resources. This could lead to backend-specific shader implementations.

ML Drift separates the logical representation of a tensor from its physical GPU allocation.

The system can resolve the required mapping during compiler initialization.

The practical benefit is significant:

Developers can work with a more unified shader model instead of maintaining completely different shader implementations for every backend. Google Developers Blog

For beginners, think of it like this:

Instead of designing a separate road system for every type of vehicle, you create a common road framework that can adapt to different vehicles.

2. Custom Operator Support

AI models frequently contain operations that aren’t perfectly covered by standard runtime operators.

ML Drift introduces a modern custom-operation framework with APIs that provide access to lower-level shader programming.

This gives developers greater flexibility when integrating specialized model components.

Google also says ML Drift includes an agentic SKILL.md guide intended to help coding agents author, register and verify custom shaders. Google Developers Blog

For advanced developers, this could make experimentation with custom GPU workloads considerably more approachable.

3. 5D Tensor Support

Another notable improvement is 5D tensor support.

The legacy TFLite GPU delegate was structurally designed around 4D tensors, which could create difficulties for models requiring higher-dimensional representations.

ML Drift’s LiteRT GPU accelerator supports 5D tensors directly.

Google specifically mentions workloads including:

• 3D convolutional networks
• Volumetric AI
• Spatiotemporal models
• YOLO 11n
• MobileViT v2
• Swin Transformer v2 Google Developers Blog

This matters because AI is increasingly moving beyond simple 2D images.

Video, depth, 3D data and other spatial workloads can require more complex tensor structures.

4. Better Performance for Classical ML Models

ML Drift isn’t designed only for large language models.

Google describes it as the successor to its legacy GPU backend and says existing classical workloads can receive performance improvements through more efficient kernel execution and reduced hardware-specific constraints. Google Developers Blog

That means existing computer-vision applications can potentially benefit from the newer architecture without completely changing their underlying AI use case.

5. Stage-Aware Optimization for Edge LLMs

Large language models create a different GPU challenge.

During inference, autoregressive LLMs have two important stages:

Prefill: processing the initial prompt/context.

Decode: generating tokens sequentially.

These stages have different computational characteristics.

Google explains that ML Drift can dynamically change kernels and layout configurations depending on which stage is active. During decoding, it uses a custom KV-cache layout and in-kernel activation quantization to reduce unnecessary memory movement. Google Developers Blog

Why does this matter?

Because LLM performance isn’t simply about raw GPU computing power.

Memory movement can become a major bottleneck.

Reducing unnecessary trips between memory and compute can help an edge device make better use of its available hardware.

ML Drift Supports More Than Smartphones

Although mobile and edge devices are a major focus, Google is also expanding ML Drift toward desktop environments.

The WebGPU implementation uses Dawn, Chromium’s WebGPU implementation, allowing the same codebase to run outside the browser.

Google says ML Drift can run on Windows and Linux using WebGPU, while also complementing native Metal support on macOS. Google Developers Blog

This could be useful for developers building:

• Local AI applications
• Desktop AI tools
• Coding assistants
• Local LLM runtimes
• AI development environments
• GPU-accelerated applications

The important idea is a common GPU abstraction across different environments.

Why Memory Efficiency Matters for Edge AI

When discussing AI performance, people often focus on tokens per second or inference latency.

But there is another constraint:

memory.

A model may be computationally capable of running on a device but still be impractical if it consumes too much memory.

Google reports that in its Gemma benchmarks, ML Drift achieved up to 12% lower memory overhead than other frameworks. Google Developers Blog

That does not mean every model or device will experience the same result. Benchmark outcomes depend on the workload, hardware, model configuration and runtime.

For developers, however, the broader lesson is clear:

Optimizing memory usage can be just as important as optimizing compute throughput.

Real-World Example: AI Camera Effects

Consider a smartphone camera application.

The application wants to apply an AI-powered style effect while the user is recording video.

The pipeline could involve:

1. Camera captures a frame.
2. The frame is transferred to the AI pipeline.
3. A model analyzes the image.
4. GPU computation generates the effect.
5. The result is displayed immediately.
6. The process repeats for subsequent frames.

If inference is too slow, the effect feels delayed.

If memory usage is excessive, other applications may compete for resources.

If GPU support varies significantly between devices, developers may face compatibility problems.

A unified GPU engine such as ML Drift is intended to simplify this type of cross-device optimization.

Google cites Snapchat as an example: Snap integrated LiteRT for GPU-based on-device inference, and Google says the ML Drift GPU accelerator produced a 30% model-latency improvement for ML-powered face and style-generator effects in Snapchat lenses. Google Developers Blog

This is a specific example rather than a universal performance guarantee.

ML Drift vs the Legacy TFLite GPU Delegate

One of the most important practical questions for existing developers is whether ML Drift replaces the older TensorFlow Lite GPU approach.

Google states that with ML Drift’s launch, the legacy TFLite GPU delegate will no longer receive new feature updates.

Google recommends moving to the LiteRT ML Drift GPU accelerator, which it says maintains backward compatibility for existing models while providing the newer acceleration architecture. Google Developers Blog

The transition can be viewed like this:

Legacy approach	ML Drift approach
TFLite GPU delegate	LiteRT ML Drift GPU accelerator
Older GPU architecture	Modernized GPU compute engine
Backend-specific constraints	More unified architecture
4D tensor limitation	5D tensor support
Traditional ML focus	Traditional ML + generative AI
Limited modernization	Custom GPU and shader capabilities

Developers should still test their actual models and devices rather than assuming identical performance after migration.

Which Platforms Does ML Drift Target?

ML Drift is designed as a cross-platform technology.

Google describes support across:

• Android
• iOS
• Web
• Desktop environments

The underlying GPU API abstraction covers OpenGL ES, OpenCL, Metal and WebGPU. Google Developers Blog

Google’s current emphasis remains production-grade inference on mobile and edge devices, while desktop support provides additional flexibility for development and local AI workloads.

ML Drift and Leading Hardware Partners

GPU software cannot be optimized in isolation from hardware.

Google says it worked with major silicon and IP partners during ML Drift’s development.

The announcement highlights optimization work involving:

Arm

Google describes co-optimized compute kernels and texture-cache locality for Mali and Immortalis GPUs.

Intel

ML Drift uses native WebGPU compute and Intel Xe Matrix Extensions for selected workloads on Intel Core Ultra processors with Xe3 graphics.

Qualcomm

Google says ML Drift uses optimized OpenCL kernels for Qualcomm Adreno GPUs to improve utilization of arithmetic and memory resources. Google Developers Blog

This hardware collaboration is important because edge AI has to operate within real-world constraints such as battery consumption, thermal limits and limited memory.

What Does ML Drift Mean for AI Developers?

The significance of ML Drift goes beyond one new Google technology.

It reflects a larger change in AI development.

AI is increasingly moving from:

Cloud-only inference → hybrid inference → local and edge inference.

For developers, that means thinking about more than model accuracy.

They also need to consider:

• Latency
• Privacy
• Memory
• Battery consumption
• Offline operation
• GPU compatibility
• Model size
• Hardware acceleration
• User experience

An application that responds instantly on-device can provide a very different experience from one that depends on a network connection for every operation.

How Developers Can Get Started With ML Drift

Google outlines two primary routes.

Option 1: Use LiteRT

Application developers can use the LiteRT ML Drift accelerator to take advantage of GPU acceleration without building everything from scratch.

This is likely the more approachable route for developers primarily interested in deploying models.

Option 2: Use ML Drift Directly

GPU and systems engineers can work with ML Drift’s standalone C++ APIs.

This approach provides more control for developers who need to:

• Construct custom GpuModel compute graphs
• Write custom shaders
• Integrate specialized GPU workloads
• Build custom inference runtimes

Google says ML Drift’s source code is open source under the Apache 2.0 license. Google Developers Blog

Google Developers Blog: ML Drift announcement

ML Drift source code on GitHub

Practical Tips Before Migrating to ML Drift

If you already have an AI application using GPU acceleration, don’t migrate simply because a new runtime exists.

Use a structured process.

1. Identify your current GPU backend

Document which GPU API and runtime your application currently uses.

2. Measure your baseline

Record:

• Model latency
• Memory consumption
• Startup time
• Throughput
• Battery impact where relevant

3. Test representative models

Don’t test only one small model.

Use the models that represent your real production workload.

4. Test multiple devices

Edge hardware varies considerably.

A model that performs well on one GPU may behave differently on another.

5. Examine memory usage

For LLMs and generative AI especially, memory behavior can be as important as raw compute performance.

6. Validate custom operators

If your model depends on custom operations, verify that they work correctly with the new runtime and determine whether custom shaders are beneficial.

7. Compare real user experience

A benchmark number is useful, but the final question is:

Does the application feel faster and more responsive to the user?

Why ML Drift Matters for the Future of Edge AI

The most interesting part of ML Drift isn’t simply that Google has introduced another GPU framework.

The bigger story is that AI inference is becoming a hardware-software optimization problem at the edge.

Modern AI models are becoming more capable while edge devices remain constrained by:

• Memory
• Power
• Thermal limits
• GPU architecture
• Storage
• Compute capacity

That creates pressure for better runtimes.

ML Drift’s architecture is designed around this reality by combining cross-platform GPU abstraction, tensor virtualization, custom operations, higher-dimensional tensor support and optimizations for generative AI workloads. Google Developers Blog

For developers, the result could be a simpler path toward applications where AI computation happens closer to the user.

Key Takeaways

• ML Drift is Google’s open-source GPU compute engine designed for on-device AI and machine learning inference.
• It abstracts GPU complexity across technologies including OpenGL ES, OpenCL, Metal and WebGPU. Google Developers Blog
• ML Drift serves as the GPU acceleration engine within LiteRT and is also available as a standalone library.
• Its architecture introduces capabilities such as tensor virtualization, custom operations and 5D tensor support.
• ML Drift is designed for both conventional ML workloads and newer generative AI and LLM workloads.
• Google recommends migrating from the legacy TFLite GPU delegate to the LiteRT ML Drift GPU accelerator as the older delegate moves out of new-feature development. Google Developers Blog

FAQ

1. What is ML Drift?

ML Drift is an open-source GPU compute engine from Google AI Edge designed to accelerate AI and machine learning inference on edge devices. It provides a cross-platform abstraction over GPU technologies such as OpenGL ES, OpenCL, Metal and WebGPU. Google Developers Blog

2. Is ML Drift part of LiteRT?

Yes. Google describes ML Drift as the core GPU acceleration engine within LiteRT. It can also be used independently through its standalone library and APIs. Google Developers Blog

3. Is ML Drift replacing the TensorFlow Lite GPU delegate?

Google says the legacy TFLite GPU delegate will no longer receive new feature updates and recommends transitioning to the LiteRT ML Drift GPU accelerator. Google Developers Blog

4. Can ML Drift run LLMs?

Yes. ML Drift includes optimizations specifically aimed at edge LLM inference, including different optimization strategies for the prefill and decode stages of autoregressive generation. Google Developers Blog

5. Is ML Drift open source?

Yes. Google states that ML Drift is open source under the Apache 2.0 license. Developers can access its source code and contribute to the project. Google Developers Blog

Conclusion

The arrival of ML Drift highlights how quickly on-device AI is evolving.

A few years ago, running a relatively small machine-learning model locally was already a useful capability. Today, developers are looking at increasingly sophisticated computer vision, generative AI and LLM workloads running directly on phones, browsers, desktops and other edge hardware.

That shift creates a difficult engineering challenge: AI software needs to take advantage of very different GPUs without forcing developers to rebuild their applications for every hardware environment.

ML Drift is Google’s answer to part of that challenge.

By providing a unified GPU compute foundation, integrating with LiteRT, supporting modern tensor structures and adding optimizations for generative AI workloads, ML Drift aims to make high-performance on-device AI inference more practical across platforms.

For developers already using the older TFLite GPU delegate, the announcement is particularly relevant because Google is directing new development toward the LiteRT ML Drift GPU accelerator.

The larger trend is even more important: the future of AI isn’t only about bigger models in the cloud. It is also about making increasingly capable models run efficiently where users are — on their own devices.