Google retrofits Pixel's Gemini Nano with faster token prediction
A new technique lets Gemini Nano v3 predict multiple tokens per pass without retraining the base model, cutting inference time and power draw on Pixel 9 and 10 devices.
How did Google speed up Gemini Nano on Pixel without retraining it?
Google Research retrofitted Multi-Token Prediction (MTP) onto existing, already-trained Gemini Nano v3 models, letting the model predict more than one token per forward pass. Because the base model is never modified, the technique avoids the cost of full retraining while still delivering measurably faster inference and lower energy consumption on Pixel 9 and 10 devices.
Why standard token generation is a bottleneck on phones
Language models ordinarily work autoregressively, producing a single token per forward pass. That sequential process leaves a phone's processor underutilized and hammers memory bandwidth, which slows AI features and drains the battery faster.
Mobile hardware compounds the problem. Unlike server infrastructure, a phone operates under a strict energy budget and fixed RAM ceiling, so every byte of model weight and every compute cycle carries extra cost.
The problem with traditional speculative decoding
The existing fix, speculative decoding, uses a small separate "drafter" model to propose a short run of candidate tokens and a large "verifier" model to confirm them in parallel. When candidates match what the verifier would have chosen, multiple tokens are accepted in a single pass.
But maintaining a standalone drafter model creates its own headaches. That secondary model competes for the same limited RAM and processing resources as the main model. Developers also face the burden of fine-tuning a new drafter for every task they want to accelerate.
What the frozen MTP approach does differently
Google's new architecture adds new components directly to the frozen production model rather than pairing it with an external drafter. The design builds on ideas from the EAGLE framework and Confident Adaptive Language Modeling (CALM), adapting them for the extreme constraints of edge computing.
The team describes the approach as a "late exit" strategy. By tapping into internal model representations rather than spinning up a separate network, the method avoids the memory and scheduling overhead of a standalone drafter while still generating candidate tokens for parallel verification.
Because the base model weights are never altered, the technique can be applied to existing production models as an out-of-the-box speedup, with no task-specific fine-tuning required from developers.
What this means for Pixel features and developers
The method has already rolled out to Pixel 9 and 10 series devices. Google says on-device features including AI Notification Summaries and Proofread now generate text faster and with lower energy draw as a result.
For developers building on-device AI, the bigger benefit may be architectural: they no longer need to maintain and ship a separate drafter model for each new application, removing a significant friction point from the on-device AI development process.
Google notes this work sits alongside a separate effort to accelerate Gemma 4 with MTP for server and developer-facing use cases. Today's announcement addresses a different problem space: squeezing the same class of speedup out of hardware that has no spare memory budget to absorb an extra model.