Method
In this work, we introduce several optimizations to accelerate diffusion-based video editing:
- Architectural Enhancement: We enhance the denoising UNet by removing self-attention and cross-attention at the highest resolutions, which are the most computationally expensive layers. We also replace the standard autoencoder with a lightweight version called “Tiny AutoEncoder for Stable Diffusion” (TAESD). These changes result in the Mobile-Pix2Pix model, which is more efficient.
- Multimodal Guidance Distillation: We perform a novel distillation process that combines classifier-free guidance (CFG) with text and image inputs into a single forward pass per diffusion step, achieving a threefold increase in inference speed. Typically, CFG with several modalities requires at least three forward passes per step, but our method consolidates this into one, maintaining quality and control. This is done by modifying ResNet blocks to accept guidance scales as embeddings, allowing efficient processing. Details are shown in Figure 2 below.
- Adversarial Distillation: We optimize the number of diffusion steps to one through a novel adversarial distillation procedure. This method distills a multi-step model into a single-step model while preserving the controllability of edits. The student model, initialized from the teacher model, uses a discriminator to distinguish real from fake samples, enhancing its ability to produce high-quality, controlled edits. Details are shown in Figure 3 below.
These optimizations enable 12 frames per second video editing on mobile devices, marking a significant milestone towards real-time text-guided video editing on mobile platforms.
