VarKD: Knowledge Distillation for Visual Autoregressive Models

Abstract

Autoregressive (AR) image generation models are highly expressive but computationally intensive, motivating effective model compression. Knowledge distillation (KD) is a natural approach for model compression and has been widely studied in language modeling, yet its behavior in visual AR generation remains underexplored. In this work, we present the first systematic study of distillation strategies for AR image models. Our analysis shows that while standard distillation can yield meaningful gains, recent methods developed for language do not directly transfer to images: long decoding horizons and visual token ambiguity make teacher supervision unreliable especially under student-conditioned contexts. To address this, we propose VarKD, a distillation framework for visual autoregressive models that distills on student samples while selectively applying teacher supervision and reducing token-level ambiguity. Experiments on ImageNet across multiple AR backbones show that VarKD consistently outperforms prior distillation baselines, narrowing the gap to large-scale models.

Method Overview

We present VarKD, the first distillation framework tailored for visual autoregressive models. Our key contributions:

First systematic study of KD for visual AR models: We evaluate supervised KD, sequence-level KD, and on-policy distillation, identifying image-specific failure modes where long decoding horizons and token ambiguity make teacher supervision unreliable.
Mixed data–student rollouts: We condition on a ground-truth prefix of random length while letting the student generate the suffix, balancing on-policy exposure with stability.
Entropy-based selective supervision: We reweight the distillation loss using teacher predictive entropy, down-weighting ambiguous targets and truncating sequences once confidence drops below a threshold.
Codebook relaxation: We cluster visually similar tokens into a compressed vocabulary and compute the distillation loss in this compressed space, reducing sensitivity to token-level ambiguity.
Training-only — zero inference cost: All modifications operate exclusively at training time. Inference uses standard next-token autoregressive decoding with no additional overhead.

Visual Comparisons — LlamaGen

Same class samples across methods. VarKD produces images closer to the teacher with fewer artifacts than other distillation approaches.

Student

KD

GKD

VarKD (Ours)

Teacher

Student

KD

GKD

VarKD (Ours)

Teacher

Student

KD

GKD

VarKD (Ours)

Teacher

Visual Comparisons — ARPG

VarKD generalizes across architectures. Samples from ARPG-L student distilled with different methods.

Student

KD

GKD

VarKD (Ours)

Teacher

Student

KD

GKD

VarKD (Ours)

Teacher

Student

KD

GKD

VarKD (Ours)

Teacher

Conditional Student Sampling

VarKD uses mixed data–student rollouts: a ground-truth prefix anchors the beginning of each sequence, while the student generates the remaining suffix. This controls the distribution shift during training. Below we visualize how generation quality varies with different prefix lengths.

LlamaGen-B (next-token prediction)

Data (GT)

Prefix = 75%

Prefix = 50%

Prefix = 25%

Student (0%)

ARPG-L (random token order)

Data (GT)

Prefix = 75%

Prefix = 50%

Prefix = 25%

Student (0%)

Conditional student sampling with varying prefix lengths. Longer prefixes anchor generation closer to ground truth, while shorter prefixes expose the student to its own predictions — the regime where VarKD applies selective supervision.

Codebook Relaxation

Discrete visual token vocabularies often contain visually similar codes — the teacher's probability mass gets split across multiple tokens that look nearly identical. VarKD addresses this via codebook relaxation: we cluster similar tokens and compute the distillation loss in the compressed space, reducing noise from token-level ambiguity.

Original image

Tokenized (full codebook)

Compressed codebook (4× fewer tokens)

Codebook relaxation groups visually similar tokens. The compressed representation preserves image structure while reducing the token-level ambiguity that makes distillation targets noisy.

Quantitative Results

Class-conditional ImageNet generation. VarKD consistently achieves the best FID across architectures and scales.

Method	FID (↓)	IS (↑)	Prec. (↑)	Rec. (↑)
LlamaGen-B
Baseline	6.51	156.3	0.81	0.46
+ KD	4.92	195.6	0.84	0.45
+ SeqKD	5.08	197.9	0.84	0.44
+ GKD	4.94	194.7	0.85	0.44
+ VarKD (Ours)	4.58	203.3	0.85	0.46
LlamaGen-L
Baseline	3.07	156.0	0.83	0.52
+ KD	3.02	244.5	0.82	0.54
+ SeqKD	3.10	258.1	0.83	0.52
+ GKD	2.93	248.0	0.82	0.55
+ VarKD (Ours)	2.83	242.7	0.83	0.55
ARPG-L
Baseline	2.30	309.0	0.80	0.58
+ KD	2.21	293.1	0.80	0.58
+ SeqKD	2.22	310.2	0.80	0.58
+ GKD	2.19	307.8	0.81	0.59
+ VarKD (Ours)	2.15	301.3	0.80	0.59

BibTeX

@misc{peruzzo2026knowledgedistillationvisualautoregressive,
  title         = {Knowledge Distillation for Visual Autoregressive Models},
  author        = {Elia Peruzzo and Aritra Bhowmik and Guillaume Sautiere 
                   and Yuki M Asano and Amirhossein Habibian},
  year          = {2026},
  eprint        = {2606.06078},
  archivePrefix = {arXiv},
  primaryClass  = {cs.CV},
  url           = {https://arxiv.org/abs/2606.06078}
}

Knowledge Distillationfor Visual Autoregressive Models

Qualcomm AI Research

Abstract

Method Overview

Visual Comparisons — LlamaGen

Visual Comparisons — ARPG

Conditional Student Sampling

LlamaGen-B (next-token prediction)

ARPG-L (random token order)

Codebook Relaxation

Quantitative Results

BibTeX

Knowledge Distillation
for Visual Autoregressive Models