---

AutoQuant: Quantizing Qwen-3 0.6B¶

This tutorial walks through ff.autoquantize, FastForward's automated quantization workflow, applied to Qwen-3 0.6B model pulled from HuggingFace.

We will:

1. Load the floating-point model together with a calibration / evaluation dataset.
2. Establish a perplexity baseline on the FP model.
3. Run AutoQuant to generate a quantization-ready version of the model source code.
4. Configure quantizers for weights and activations.
5. Calibrate the quantizers and measure the resulting perplexity.

We start by downloading the model and tokenizer through the HuggingFace API.

In [1]:

import os

import fastforward as ff
import torch

from datasets import load_dataset
from transformers import AutoModelForCausalLM, AutoTokenizer, default_data_collator

# Any model from Qwen3 family can be used:
# - "Qwen/Qwen3-0.6B"
# - "Qwen/Qwen3-1.7B"
# - "Qwen/Qwen3-4B"
# - "Qwen/Qwen3-8B"
# - ...
model_name_or_path = os.environ.get("FF_AUTOQUANT_QWEN_MODEL", "Qwen/Qwen3-0.6B")

model_dtype = torch.float16
device = "cuda"

model = AutoModelForCausalLM.from_pretrained(
    pretrained_model_name_or_path=model_name_or_path,
    dtype=model_dtype,
    attn_implementation="eager",
    from_tf=False,
)
tokenizer = AutoTokenizer.from_pretrained(
    pretrained_model_name_or_path=model_name_or_path, attn_implementation="eager"
)

import os import fastforward as ff import torch from datasets import load_dataset from transformers import AutoModelForCausalLM, AutoTokenizer, default_data_collator # Any model from Qwen3 family can be used: # - "Qwen/Qwen3-0.6B" # - "Qwen/Qwen3-1.7B" # - "Qwen/Qwen3-4B" # - "Qwen/Qwen3-8B" # - ... model_name_or_path = os.environ.get("FF_AUTOQUANT_QWEN_MODEL", "Qwen/Qwen3-0.6B") model_dtype = torch.float16 device = "cuda" model = AutoModelForCausalLM.from_pretrained( pretrained_model_name_or_path=model_name_or_path, dtype=model_dtype, attn_implementation="eager", from_tf=False, ) tokenizer = AutoTokenizer.from_pretrained( pretrained_model_name_or_path=model_name_or_path, attn_implementation="eager" )

We will also need some data:

• A validation-set for evaluating perplexity before and after quantization.
• A calibration-set for calibrating the quantizers, i.e. running enough forward passes to estimate the quantizers ranges and set quantization parameters accordingly.

We use WikiText-2, where the training split feeds calibration, the validation split is used for evaluation.

In [2]:

from fastforward.testing.data import tokenize_dataset
from torch.utils.data import DataLoader

sequence_length = 1024
batch_size = 1

# Load Dataset
raw_validset = load_dataset("wikitext", "wikitext-2-v1", split="train")
raw_trainset = load_dataset("wikitext", "wikitext-2-v1", split="validation")

# Tokenize Dataset
tokenized_validset = tokenize_dataset(raw_validset, tokenizer, sequence_length)
tokenized_trainset = tokenize_dataset(raw_trainset, tokenizer, sequence_length)

# Create Dataloader
valid_loader = DataLoader(tokenized_validset, batch_size, collate_fn=default_data_collator)
train_loader = DataLoader(tokenized_trainset, batch_size, collate_fn=default_data_collator)

from fastforward.testing.data import tokenize_dataset from torch.utils.data import DataLoader sequence_length = 1024 batch_size = 1 # Load Dataset raw_validset = load_dataset("wikitext", "wikitext-2-v1", split="train") raw_trainset = load_dataset("wikitext", "wikitext-2-v1", split="validation") # Tokenize Dataset tokenized_validset = tokenize_dataset(raw_validset, tokenizer, sequence_length) tokenized_trainset = tokenize_dataset(raw_trainset, tokenizer, sequence_length) # Create Dataloader valid_loader = DataLoader(tokenized_validset, batch_size, collate_fn=default_data_collator) train_loader = DataLoader(tokenized_trainset, batch_size, collate_fn=default_data_collator)

Using the latest cached version of the dataset since wikitext couldn't be found on the Hugging Face Hub (offline mode is enabled).

Found the latest cached dataset configuration 'wikitext-2-v1' at /prj/corp/llm/lasvegas/llm-systems-scratch/cache/datasets/wikitext/wikitext-2-v1/0.0.0/b08601e04326c79dfdd32d625aee71d232d685c3 (last modified on Fri Aug 21 15:27:31 2026).

Using the latest cached version of the dataset since wikitext couldn't be found on the Hugging Face Hub (offline mode is enabled).

Found the latest cached dataset configuration 'wikitext-2-v1' at /prj/corp/llm/lasvegas/llm-systems-scratch/cache/datasets/wikitext/wikitext-2-v1/0.0.0/b08601e04326c79dfdd32d625aee71d232d685c3 (last modified on Fri Aug 21 15:27:31 2026).

[transformers] Token indices sequence length is longer than the specified maximum sequence length for this model (2549174 > 131072). Running this sequence through the model will result in indexing errors

[transformers] Token indices sequence length is longer than the specified maximum sequence length for this model (270673 > 131072). Running this sequence through the model will result in indexing errors

---

Evaluate the FP Model¶

Before quantizing anything, we measure the floating-point model's perplexity on the validation split. This is the reference number that the quantized model will be compared against.

In [3]:

from fastforward.testing.data import sliced_tqdm


def prepare_batch(batch: dict, device: torch.device):
    return {
        "input_ids": batch["input_ids"].to(device),
        "attention_mask": batch["attention_mask"].to(device),
        "labels": batch["labels"].to(torch.long).to(device),
    }


@torch.no_grad()
def evaluate_model(model, valid_loader, device, limit=None):
    model.eval()
    losses = []
    for batch in sliced_tqdm(valid_loader, limit):
        batch = prepare_batch(batch, device)
        outputs = model(**batch)
        losses.append(outputs.loss)

    eval_loss = torch.stack(losses).mean()
    perplexity = torch.exp(eval_loss)
    return float(perplexity)


model.to(device)
fp_task_results = evaluate_model(model, valid_loader, device, limit=8)
print(fp_task_results)

from fastforward.testing.data import sliced_tqdm def prepare_batch(batch: dict, device: torch.device): return { "input_ids": batch["input_ids"].to(device), "attention_mask": batch["attention_mask"].to(device), "labels": batch["labels"].to(torch.long).to(device), } @torch.no_grad() def evaluate_model(model, valid_loader, device, limit=None): model.eval() losses = [] for batch in sliced_tqdm(valid_loader, limit): batch = prepare_batch(batch, device) outputs = model(**batch) losses.append(outputs.loss) eval_loss = torch.stack(losses).mean() perplexity = torch.exp(eval_loss) return float(perplexity) model.to(device) fp_task_results = evaluate_model(model, valid_loader, device, limit=8) print(fp_task_results)

  0%|          | 0/8 [00:00<?, ?it/s]

 12%|█▎        | 1/8 [00:01<00:09,  1.43s/it]

 50%|█████     | 4/8 [00:01<00:01,  3.21it/s]

 75%|███████▌  | 6/8 [00:01<00:00,  4.95it/s]

100%|██████████| 8/8 [00:01<00:00,  6.78it/s]

100%|██████████| 8/8 [00:01<00:00,  4.39it/s]

28.213809967041016

---

AutoQuant: Generate Model Source¶

A single call to ff.autoquantize inspects the model and emits a Python file containing quantization-ready versions of every module it encounters. The generated code is a drop-in replacement: each module exposes the same interface as the original, but with explicit quantizer stubs ready to be configured.

The output is regular python source file; you can read it, edit it, and check it into version control like any other source file.

In [4]:

print(f"Autoquantize model {type(model).__name__}")

code = ff.autoquantize(
    model,
    output_path="_autoquantized_qwen.py",
    force_overwrite=True,
    auto_import=True,  # immediately import the generated code
)

print(f"Autoquantize model {type(model).__name__}") code = ff.autoquantize( model, output_path="_autoquantized_qwen.py", force_overwrite=True, auto_import=True, # immediately import the generated code )

Autoquantize model Qwen3ForCausalLM

---

Create the Quantization-Ready Model¶

With the generated module imported, ff.quantize_model rewrites the original model in place: every layer that has a quantized counterpart is swapped in. Functionally the model still behaves like the FP version because the quantizer stubs are inactive — they only start operating once we initialize them in the next step.

Note:

• To use autoquantized model, you must import the quantization-ready modules from the generated file.
• If you just run ff.autoquantize with auto_import=True, you don't need to explicitly import.

In [5]:

from _autoquantized_qwen import QuantizedQwen3ForCausalLM

# Transform the original model into a quantized-ready using the imported quantized-modules
ff.quantize_model(model, skip_quantized_modules=True)

from _autoquantized_qwen import QuantizedQwen3ForCausalLM # Transform the original model into a quantized-ready using the imported quantized-modules ff.quantize_model(model, skip_quantized_modules=True)

Out[5]:

QuantizedQwen3ForCausalLM(
  (model): QuantizedQwen3Model(
    (embed_tokens): QuantizedEmbedding(
      151936, 1024
      (weight_quantizer): QuantizerStub()
      (output_quantizer): QuantizerStub()
    )
    (layers): QuantizedModuleList(
      (0-27): 28 x QuantizedQwen3DecoderLayer(
        (self_attn): QuantizedQwen3Attention(
          (q_proj): QuantizedLinear(
            in_features=1024, out_features=2048, bias=False
            (input_quantizer): QuantizerStub()
            (weight_quantizer): QuantizerStub()
            (bias_quantizer): None
            (output_quantizer): QuantizerStub()
          )
          (k_proj): QuantizedLinear(
            in_features=1024, out_features=1024, bias=False
            (input_quantizer): QuantizerStub()
            (weight_quantizer): QuantizerStub()
            (bias_quantizer): None
            (output_quantizer): QuantizerStub()
          )
          (v_proj): QuantizedLinear(
            in_features=1024, out_features=1024, bias=False
            (input_quantizer): QuantizerStub()
            (weight_quantizer): QuantizerStub()
            (bias_quantizer): None
            (output_quantizer): QuantizerStub()
          )
          (o_proj): QuantizedLinear(
            in_features=2048, out_features=1024, bias=False
            (input_quantizer): QuantizerStub()
            (weight_quantizer): QuantizerStub()
            (bias_quantizer): None
            (output_quantizer): QuantizerStub()
          )
          (q_norm): QuantizedQwen3RMSNorm(
            (128,), eps=1e-06
            (quantizer_add): QuantizerStub()
            (quantizer_mul_1): QuantizerStub()
            (quantizer_mul_2): QuantizerStub()
            (quantizer_hidden_states): QuantizerStub()
            (quantizer_variance): QuantizerStub()
            (quantizer__tmp_3): QuantizerStub()
            (quantizer_self_variance_epsilon): QuantizerStub()
            (quantizer_self_weight): QuantizerStub()
          )
          (k_norm): QuantizedQwen3RMSNorm(
            (128,), eps=1e-06
            (quantizer_add): QuantizerStub()
            (quantizer_mul_1): QuantizerStub()
            (quantizer_mul_2): QuantizerStub()
            (quantizer_hidden_states): QuantizerStub()
            (quantizer_variance): QuantizerStub()
            (quantizer__tmp_3): QuantizerStub()
            (quantizer_self_variance_epsilon): QuantizerStub()
            (quantizer_self_weight): QuantizerStub()
          )
          (quantizer_apply_rotary_pos_emb_mul_1): QuantizerStub()
          (quantizer_apply_rotary_pos_emb_mul_2): QuantizerStub()
          (quantizer_apply_rotary_pos_emb_add_1): QuantizerStub()
          (quantizer_apply_rotary_pos_emb_mul_3): QuantizerStub()
          (quantizer_apply_rotary_pos_emb_mul_4): QuantizerStub()
          (quantizer_apply_rotary_pos_emb_add_2): QuantizerStub()
          (quantizer_apply_rotary_pos_emb_cos): QuantizerStub()
          (quantizer_apply_rotary_pos_emb_sin): QuantizerStub()
          (quantizer_apply_rotary_pos_emb__tmp_27): QuantizerStub()
          (quantizer_apply_rotary_pos_emb__tmp_30): QuantizerStub()
          (quantizer_apply_rotary_pos_emb_q): QuantizerStub()
          (quantizer_apply_rotary_pos_emb_k): QuantizerStub()
          (quantizer_apply_rotary_pos_emb_rotate_half_floor_divide_1): QuantizerStub()
          (quantizer_apply_rotary_pos_emb_rotate_half_floor_divide_2): QuantizerStub()
          (quantizer_apply_rotary_pos_emb_rotate_half_negative_1): QuantizerStub()
          (quantizer_apply_rotary_pos_emb_rotate_half_cat_1): QuantizerStub()
          (quantizer_apply_rotary_pos_emb_rotate_half_x_shape_minus_1_1): QuantizerStub()
          (quantizer_apply_rotary_pos_emb_rotate_half_x2_1): QuantizerStub()
          (quantizer_apply_rotary_pos_emb_rotate_half_floor_divide_3): QuantizerStub()
          (quantizer_apply_rotary_pos_emb_rotate_half_floor_divide_4): QuantizerStub()
          (quantizer_apply_rotary_pos_emb_rotate_half_negative_2): QuantizerStub()
          (quantizer_apply_rotary_pos_emb_rotate_half_cat_2): QuantizerStub()
          (quantizer_apply_rotary_pos_emb_rotate_half_x_shape_minus_1_2): QuantizerStub()
          (quantizer_apply_rotary_pos_emb_rotate_half_x2_2): QuantizerStub()
        )
        (mlp): QuantizedQwen3MLP(
          (gate_proj): QuantizedLinear(
            in_features=1024, out_features=3072, bias=False
            (input_quantizer): QuantizerStub()
            (weight_quantizer): QuantizerStub()
            (bias_quantizer): None
            (output_quantizer): QuantizerStub()
          )
          (up_proj): QuantizedLinear(
            in_features=1024, out_features=3072, bias=False
            (input_quantizer): QuantizerStub()
            (weight_quantizer): QuantizerStub()
            (bias_quantizer): None
            (output_quantizer): QuantizerStub()
          )
          (down_proj): QuantizedLinear(
            in_features=3072, out_features=1024, bias=False
            (input_quantizer): QuantizerStub()
            (weight_quantizer): QuantizerStub()
            (bias_quantizer): None
            (output_quantizer): QuantizerStub()
          )
          (act_fn): QuantizedSiLUActivation(
            (quantizer_silu): QuantizerStub()
            (quantizer_input): QuantizerStub()
          )
          (quantizer_mul): QuantizerStub()
          (quantizer__tmp_5): QuantizerStub()
          (quantizer__tmp_6): QuantizerStub()
        )
        (input_layernorm): QuantizedQwen3RMSNorm(
          (1024,), eps=1e-06
          (quantizer_add): QuantizerStub()
          (quantizer_mul_1): QuantizerStub()
          (quantizer_mul_2): QuantizerStub()
          (quantizer_hidden_states): QuantizerStub()
          (quantizer_variance): QuantizerStub()
          (quantizer__tmp_3): QuantizerStub()
          (quantizer_self_variance_epsilon): QuantizerStub()
          (quantizer_self_weight): QuantizerStub()
        )
        (post_attention_layernorm): QuantizedQwen3RMSNorm(
          (1024,), eps=1e-06
          (quantizer_add): QuantizerStub()
          (quantizer_mul_1): QuantizerStub()
          (quantizer_mul_2): QuantizerStub()
          (quantizer_hidden_states): QuantizerStub()
          (quantizer_variance): QuantizerStub()
          (quantizer__tmp_3): QuantizerStub()
          (quantizer_self_variance_epsilon): QuantizerStub()
          (quantizer_self_weight): QuantizerStub()
        )
        (quantizer_add_1): QuantizerStub()
        (quantizer_add_2): QuantizerStub()
        (quantizer_residual_1): QuantizerStub()
        (quantizer_hidden_states_1): QuantizerStub()
        (quantizer_residual_2): QuantizerStub()
        (quantizer_hidden_states_2): QuantizerStub()
      )
    )
    (norm): QuantizedQwen3RMSNorm(
      (1024,), eps=1e-06
      (quantizer_add): QuantizerStub()
      (quantizer_mul_1): QuantizerStub()
      (quantizer_mul_2): QuantizerStub()
      (quantizer_hidden_states): QuantizerStub()
      (quantizer_variance): QuantizerStub()
      (quantizer__tmp_3): QuantizerStub()
      (quantizer_self_variance_epsilon): QuantizerStub()
      (quantizer_self_weight): QuantizerStub()
    )
    (rotary_emb): QuantizedQwen3RotaryEmbedding(
      (quantizer_matmul): QuantizerStub()
      (quantizer_cat): QuantizerStub()
      (quantizer_mul_1): QuantizerStub()
      (quantizer_mul_2): QuantizerStub()
      (quantizer__tmp_16): QuantizerStub()
      (quantizer__tmp_17): QuantizerStub()
      (quantizer__tmp_19): QuantizerStub()
      (quantizer__tmp_20): QuantizerStub()
      (quantizer_self_attention_scaling): QuantizerStub()
    )
    (quantizer_add): QuantizerStub()
    (quantizer_past_seen_tokens): QuantizerStub()
    (quantizer_create_causal_mask_maybe_pad_block_sequence_ids_pad): QuantizerStub()
    (quantizer_create_causal_mask_maybe_pad_block_sequence_ids_block_sequence_ids): QuantizerStub()
    (quantizer_create_sliding_window_causal_mask_maybe_pad_block_sequence_ids_pad): QuantizerStub()
    (quantizer_create_sliding_window_causal_mask_maybe_pad_block_sequence_ids_block_sequence_ids): QuantizerStub()
  )
  (lm_head): QuantizedLinear(
    in_features=1024, out_features=151936, bias=False
    (input_quantizer): QuantizerStub()
    (weight_quantizer): QuantizerStub()
    (bias_quantizer): None
    (output_quantizer): QuantizerStub()
  )
)

In [6]:

# OPTIONAL: you can cast the model to QuantizedQwen3ForCausalLM to help LSP or IDE
from typing import cast

model: QuantizedQwen3ForCausalLM = cast(QuantizedQwen3ForCausalLM, model)

# OPTIONAL: you can cast the model to QuantizedQwen3ForCausalLM to help LSP or IDE from typing import cast model: QuantizedQwen3ForCausalLM = cast(QuantizedQwen3ForCausalLM, model)

---

Initialize the Quantizers¶

The model now contains placeholder quantizer stubs at every site where a quantizer could be inserted. We decide which stubs to activate, and with what configuration.

Quantizers are selected with ff.find_quantizers, which uses a glob-style pattern over fully-qualified module names. The [quantizer:...] filter narrows the match by quantizer kind (parameter vs activation) and target (weight, output, ...).

We group quantizers by role so each group can receive its own configuration:

• Linear weight quantizers in attention and MLP layers, plus lm_head and embed_tokens: 4-bit, per-channel.
• LayerNorm weight quantizers: 4-bit per-tensor (per-channel does not apply — the weight is 1D).
• Activation quantizers across the transformer blocks and residual paths: 16-bit, asymmetric.

Tweaking these settings is how the accuracy / efficiency trade-off is dialed in.

In [7]:

from fastforward.nn import LinearQuantizer

# Granularities
per_block_32 = ff.granularity.PerBlock(block_dims=1, block_sizes=32, per_channel_dims=0)
per_tensor = ff.granularity.PerTensor()

# Model weight quantizers: find and initialize
w_quants = ff.find_quantizers(model, "**/self_attn/**/[quantizer:parameter/weight]")
w_quants |= ff.find_quantizers(model, "**/mlp/**/[quantizer:parameter/weight]")
w_quants.initialize(LinearQuantizer, num_bits=4, granularity=per_block_32)
print(f"MLP and self-attention wegiht quantizers: {len(w_quants)}")

lmhead_w_quants = ff.find_quantizers(model, "**/lm_head/[quantizer:parameter/weight]")
lmhead_w_quants.initialize(LinearQuantizer, num_bits=4, granularity=per_block_32)
print(f"LM-Head wegiht quantizers: {len(lmhead_w_quants)}")

embed_w_quants = ff.find_quantizers(model, "**/embed_tokens/[quantizer:parameter/weight]")
embed_w_quants.initialize(LinearQuantizer, num_bits=4, granularity=per_block_32)
print(f"Embedding wegiht quantizers: {len(embed_w_quants)}")

from fastforward.nn import LinearQuantizer # Granularities per_block_32 = ff.granularity.PerBlock(block_dims=1, block_sizes=32, per_channel_dims=0) per_tensor = ff.granularity.PerTensor() # Model weight quantizers: find and initialize w_quants = ff.find_quantizers(model, "**/self_attn/**/[quantizer:parameter/weight]") w_quants |= ff.find_quantizers(model, "**/mlp/**/[quantizer:parameter/weight]") w_quants.initialize(LinearQuantizer, num_bits=4, granularity=per_block_32) print(f"MLP and self-attention wegiht quantizers: {len(w_quants)}") lmhead_w_quants = ff.find_quantizers(model, "**/lm_head/[quantizer:parameter/weight]") lmhead_w_quants.initialize(LinearQuantizer, num_bits=4, granularity=per_block_32) print(f"LM-Head wegiht quantizers: {len(lmhead_w_quants)}") embed_w_quants = ff.find_quantizers(model, "**/embed_tokens/[quantizer:parameter/weight]") embed_w_quants.initialize(LinearQuantizer, num_bits=4, granularity=per_block_32) print(f"Embedding wegiht quantizers: {len(embed_w_quants)}")

MLP and self-attention wegiht quantizers: 196
LM-Head wegiht quantizers: 1
Embedding wegiht quantizers: 1

---

Calibrate the Quantizers¶

Static quantizers like LinearQuantizer need to observe real activations to choose appropriate quantization ranges. We run a few batches from the training split through the model inside the ff.estimate_ranges context manager, which collects running min/max statistics and uses them to set each quantizer's range.

strict_quantization(False) context manager lets the model run even when some operations are not yet fully quantized.

In [8]:

model.to(device)

with ff.strict_quantization(False):
    with torch.no_grad(), ff.estimate_ranges(model, ff.range_setting.smoothed_minmax):
        for batch in sliced_tqdm(valid_loader, limit=4):
            model(**prepare_batch(batch, device))

model.to(device) with ff.strict_quantization(False): with torch.no_grad(), ff.estimate_ranges(model, ff.range_setting.smoothed_minmax): for batch in sliced_tqdm(valid_loader, limit=4): model(**prepare_batch(batch, device))

  0%|          | 0/4 [00:00<?, ?it/s]

 25%|██▌       | 1/4 [00:00<00:01,  2.59it/s]

 50%|█████     | 2/4 [00:00<00:00,  2.94it/s]

 75%|███████▌  | 3/4 [00:00<00:00,  3.08it/s]

100%|██████████| 4/4 [00:01<00:00,  3.15it/s]

100%|██████████| 4/4 [00:01<00:00,  3.06it/s]

---

Initialize Dynamic Quantizers¶

Dynamic quantizers do not need calibration, so we initialize them after we already calibrated all the static quantizers.

In [9]:

# Granularity
from fastforward.nn import DynamicLinearQuantizer

per_token = ff.granularity.PerChannel(channel_dim=1)

# Model activation quantizers: find and initialize
a_quants = ff.find_quantizers(model, "**/mlp/**/[quantizer:activation/output]")
a_quants |= ff.find_quantizers(model, "**/self_attn/**/[quantizer:activation/output]")
a_quants |= ff.find_quantizers(model, "**/input_layernorm/[quantizer:activation/output]")
a_quants |= ff.find_quantizers(model, "**/post_attention_layernorm/[quantizer:activation/output]")
a_quants |= ff.find_quantizers(model, "**/norm/[quantizer:activation/output]")
a_quants |= ff.find_quantizers(model, "**/attn_res_act_quantizer")
a_quants |= ff.find_quantizers(model, "**/mlp_res_act_quantizer")
a_quants |= ff.find_quantizers(model, "**/lm_head/[quantizer:activation/output]")
a_quants.initialize(DynamicLinearQuantizer, num_bits=8, symmetric=False, granularity=per_token)
print(f"Activation quantizers: {len(a_quants)}")

# Granularity from fastforward.nn import DynamicLinearQuantizer per_token = ff.granularity.PerChannel(channel_dim=1) # Model activation quantizers: find and initialize a_quants = ff.find_quantizers(model, "**/mlp/**/[quantizer:activation/output]") a_quants |= ff.find_quantizers(model, "**/self_attn/**/[quantizer:activation/output]") a_quants |= ff.find_quantizers(model, "**/input_layernorm/[quantizer:activation/output]") a_quants |= ff.find_quantizers(model, "**/post_attention_layernorm/[quantizer:activation/output]") a_quants |= ff.find_quantizers(model, "**/norm/[quantizer:activation/output]") a_quants |= ff.find_quantizers(model, "**/attn_res_act_quantizer") a_quants |= ff.find_quantizers(model, "**/mlp_res_act_quantizer") a_quants |= ff.find_quantizers(model, "**/lm_head/[quantizer:activation/output]") a_quants.initialize(DynamicLinearQuantizer, num_bits=8, symmetric=False, granularity=per_token) print(f"Activation quantizers: {len(a_quants)}")

Activation quantizers: 197

---

Evaluate the Quantized Model¶

Finally, we recompute perplexity on the validation split and compare against the FP baseline. The gap reflects the cost of the quantization configuration we picked above; tightening or loosening bit-widths and granularities is how we trade off accuracy against efficiency.

In [10]:

model.to(device)
with ff.strict_quantization(False):
    q_task_results = evaluate_model(model, valid_loader, limit=8, device=device)

print("Quantized Qwen performance:")
print(f"---> FP perplexity:    {fp_task_results}")
print(f"---> Quant perplexity: {q_task_results}")

model.to(device) with ff.strict_quantization(False): q_task_results = evaluate_model(model, valid_loader, limit=8, device=device) print("Quantized Qwen performance:") print(f"---> FP perplexity: {fp_task_results}") print(f"---> Quant perplexity: {q_task_results}")

  0%|          | 0/8 [00:00<?, ?it/s]

 12%|█▎        | 1/8 [00:00<00:02,  3.46it/s]

 25%|██▌       | 2/8 [00:00<00:01,  3.37it/s]

 38%|███▊      | 3/8 [00:00<00:01,  3.31it/s]

 50%|█████     | 4/8 [00:01<00:01,  3.30it/s]

 62%|██████▎   | 5/8 [00:01<00:00,  3.28it/s]

 75%|███████▌  | 6/8 [00:01<00:00,  3.26it/s]

 88%|████████▊ | 7/8 [00:02<00:00,  3.24it/s]

100%|██████████| 8/8 [00:02<00:00,  3.26it/s]

100%|██████████| 8/8 [00:02<00:00,  3.28it/s]

Quantized Qwen performance:
---> FP perplexity:    28.213809967041016
---> Quant perplexity: 33.55324935913086

---

Copyright (c) Qualcomm Technologies, Inc. and/or its subsidiaries. SPDX-License-Identifier: BSD-3-Clause-Clear