Home Artificial Intelligence On-device diffusion plugins for conditioned text-to-image technology – Google Analysis Weblog

On-device diffusion plugins for conditioned text-to-image technology – Google Analysis Weblog

On-device diffusion plugins for conditioned text-to-image technology – Google Analysis Weblog


In recent times, diffusion fashions have proven nice success in text-to-image technology, reaching excessive picture high quality, improved inference efficiency, and increasing our artistic inspiration. Nonetheless, it’s nonetheless difficult to effectively management the technology, particularly with situations which are tough to explain with textual content.

At present, we announce MediaPipe diffusion plugins, which allow controllable text-to-image technology to be run on-device. Increasing upon our prior work on GPU inference for on-device massive generative fashions, we introduce new low-cost options for controllable text-to-image technology that may be plugged into current diffusion fashions and their Low-Rank Adaptation (LoRA) variants.

Textual content-to-image technology with management plugins operating on-device.


With diffusion fashions, picture technology is modeled as an iterative denoising course of. Ranging from a noise picture, at every step, the diffusion mannequin regularly denoises the picture to disclose a picture of the goal idea. Analysis exhibits that leveraging language understanding by way of textual content prompts can significantly enhance picture technology. For text-to-image technology, the textual content embedding is linked to the mannequin by way of cross-attention layers. But, some info is tough to explain by textual content prompts, e.g., the place and pose of an object. To handle this drawback, researchers add further fashions into the diffusion to inject management info from a situation picture.

Frequent approaches for managed text-to-image technology embrace Plug-and-Play, ControlNet, and T2I Adapter. Plug-and-Play applies a extensively used denoising diffusion implicit mannequin (DDIM) inversion method that reverses the technology course of ranging from an enter picture to derive an preliminary noise enter, after which employs a duplicate of the diffusion mannequin (860M parameters for Steady Diffusion 1.5) to encode the situation from an enter picture. Plug-and-Play extracts spatial options with self-attention from the copied diffusion, and injects them into the text-to-image diffusion. ControlNet creates a trainable copy of the encoder of a diffusion mannequin, which connects by way of a convolution layer with zero-initialized parameters to encode conditioning info that’s conveyed to the decoder layers. Nevertheless, consequently, the dimensions is massive, half that of the diffusion mannequin (430M parameters for Steady Diffusion 1.5). T2I Adapter is a smaller community (77M parameters) and achieves comparable results in controllable technology. T2I Adapter solely takes the situation picture as enter, and its output is shared throughout all diffusion iterations. But, the adapter mannequin just isn’t designed for moveable gadgets.

The MediaPipe diffusion plugins

To make conditioned technology environment friendly, customizable, and scalable, we design the MediaPipe diffusion plugin as a separate community that’s:

  • Plugable: It may be simply linked to a pre-trained base mannequin.
  • Educated from scratch: It doesn’t use pre-trained weights from the bottom mannequin.
  • Moveable: It runs outdoors the bottom mannequin on cellular gadgets, with negligible value in comparison with the bottom mannequin inference.
Methodology    Parameter Dimension     Plugable     From Scratch     Moveable
Plug-and-Play    860M*     ✔️        
ControlNet    430M*     ✔️        
T2I Adapter    77M     ✔️     ✔️    
MediaPipe Plugin    6M     ✔️     ✔️     ✔️
Comparability of Plug-and-Play, ControlNet, T2I Adapter, and the MediaPipe diffusion plugin.
* The quantity varies relying on the particulars of the diffusion mannequin.

The MediaPipe diffusion plugin is a conveyable on-device mannequin for text-to-image technology. It extracts multiscale options from a conditioning picture, that are added to the encoder of a diffusion mannequin at corresponding ranges. When connecting to a text-to-image diffusion mannequin, the plugin mannequin can present an additional conditioning sign to the picture technology. We design the plugin community to be a light-weight mannequin with solely 6M parameters. It makes use of depth-wise convolutions and inverted bottlenecks from MobileNetv2 for quick inference on cellular gadgets.

Overview of the MediaPipe diffusion mannequin plugin. The plugin is a separate community, whose output may be plugged right into a pre-trained text-to-image technology mannequin. Options extracted by the plugin are utilized to the related downsampling layer of the diffusion mannequin (blue).

Not like ControlNet, we inject the identical management options in all diffusion iterations. That’s, we solely run the plugin as soon as for one picture technology, which saves computation. We illustrate some intermediate outcomes of a diffusion course of under. The management is efficient at each diffusion step and allows managed technology even at early steps. Extra iterations enhance the alignment of the picture with the textual content immediate and generate extra element.

Illustration of the technology course of utilizing the MediaPipe diffusion plugin.


On this work, we developed plugins for a diffusion-based text-to-image technology mannequin with MediaPipe Face Landmark, MediaPipe Holistic Landmark, depth maps, and Canny edge. For every activity, we choose about 100K photos from a web-scale image-text dataset, and compute management alerts utilizing corresponding MediaPipe options. We use refined captions from PaLI for coaching the plugins.

Face Landmark

The MediaPipe Face Landmarker activity computes 478 landmarks (with consideration) of a human face. We use the drawing utils in MediaPipe to render a face, together with face contour, mouth, eyes, eyebrows, and irises, with completely different colours. The next desk exhibits randomly generated samples by conditioning on face mesh and prompts. As a comparability, each ControlNet and Plugin can management text-to-image technology with given situations.

Face-landmark plugin for text-to-image technology, in contrast with ControlNet.

Holistic Landmark

MediaPipe Holistic Landmarker activity contains landmarks of physique pose, fingers, and face mesh. Under, we generate numerous stylized photos by conditioning on the holistic options.

Holistic-landmark plugin for text-to-image technology.


Depth-plugin for text-to-image technology.

Canny Edge

Canny-edge plugin for text-to-image technology.


We conduct a quantitative examine of the face landmark plugin to reveal the mannequin’s efficiency. The analysis dataset comprises 5K human photos. We evaluate the technology high quality as measured by the extensively used metrics, Fréchet Inception Distance (FID) and CLIP scores. The bottom mannequin is a pre-trained text-to-image diffusion mannequin. We use Steady Diffusion v1.5 right here.

As proven within the following desk, each ControlNet and the MediaPipe diffusion plugin produce a lot better pattern high quality than the bottom mannequin, by way of FID and CLIP scores. Not like ControlNet, which must run at each diffusion step, the MediaPipe plugin solely runs as soon as for every picture generated. We measured the efficiency of the three fashions on a server machine (with Nvidia V100 GPU) and a cell phone (Galaxy S23). On the server, we run all three fashions with 50 diffusion steps, and on cellular, we run 20 diffusion steps utilizing the MediaPipe picture technology app. In contrast with ControlNet, the MediaPipe plugin exhibits a transparent benefit in inference effectivity whereas preserving the pattern high quality.

Mannequin     FID↓     CLIP↑     Inference Time (s)
Nvidia V100     Galaxy S23
Base     10.32     0.26     5.0     11.5
Base + ControlNet     6.51     0.31     7.4 (+48%)     18.2 (+58.3%)
Base + MediaPipe Plugin     6.50     0.30     5.0 (+0.2%)     11.8 (+2.6%)
Quantitative comparability on FID, CLIP, and inference time.

We take a look at the efficiency of the plugin on a variety of cellular gadgets from mid-tier to high-end. We record the outcomes on some consultant gadgets within the following desk, protecting each Android and iOS.

System     Android     iOS
    Pixel 4     Pixel 6     Pixel 7     Galaxy S23     iPhone 12 Professional     iPhone 13 Professional
Time (ms)     128     68     50     48     73     63
Inference time (ms) of the plugin on completely different cellular gadgets.


On this work, we current MediaPipe, a conveyable plugin for conditioned text-to-image technology. It injects options extracted from a situation picture to a diffusion mannequin, and consequently controls the picture technology. Moveable plugins may be linked to pre-trained diffusion fashions operating on servers or gadgets. By operating text-to-image technology and plugins absolutely on-device, we allow extra versatile purposes of generative AI.


We’d wish to thank all crew members who contributed to this work: Raman Sarokin and Juhyun Lee for the GPU inference answer; Khanh LeViet, Chuo-Ling Chang, Andrei Kulik, and Matthias Grundmann for management. Particular because of Jiuqiang Tang, Joe Zou and Lu wang, who made this know-how and all of the demos operating on-device.



Please enter your comment!
Please enter your name here