Yury Polyanskiy         Professor         (EECS)               Curriculum Vitae               Publications & Preprints               Research Group               Other

Quantization and tokenization in AI (mini course)

Class notes and materials (2025)

• This page contains some supplementary materials for a 4-lecture mini course taught by Y. Polyanskiy in Princeton ML School in 2025.
• Warning: The notes are handwritten and were not meant to be published, so they are very unpolished. Apologies for the rough presentation, missing references, and any errors.
• Lecture 1: Scalar Quantization.
  • Quantization Motivation: Classical A/D conversion. Quantization in LLMs. Tokenization of non-text modalities (VQGAN, SoundStream)
  • Scalar quantization. Uniform quantization: 6dB/bit
  • Dithering
  • Non-uniform quantization. FP4
  • Panter-Dite approximation
  • Uniform quantization of weights in LLMs. Non-MSE metric
  • GPTQ and LDLQ
• Lecture 2: Vector Quantization and Tokenization.
  • GPTQ and LDLQ
  • Hamming game
  • MXFP4
  • Lloyd-Max algorithm
  • VQVAE and VQGAN. Straight-through estimator. Commitment loss
  • Suboptimality and slow convergence of VQVAE
  • Information-theoretic lower bound on VQ
  • Derivation of Gaussian VQ formula
• Lecture 3: Vector Quantization II.
  • Random coding bound
  • Gaussian is hardest to quantize (random rotation)
  • Non-iid data, non-MSE metric: waterfilling solution
• Lecture 4: Quantization for Matrix Multiplication.
  • Setup
  • Naive quantization: worst case loss is O(n²)
  • Naive sketching: still O(n²)
  • Gaussian inner-product rate-distortion function
• Literature (very sketchy and incomplete):
  • Discrete Tokenization:
    • VQVAE paper https://arxiv.org/abs/1711.00937
    • VQGAN paper https://arxiv.org/pdf/2012.09841
    • Gumbel-Softmax quantization https://arxiv.org/pdf/1611.01144
    • Karpathy's code showing how to actually use Gumbel-Softmax for VQ GitHub
    • DALL-E1 paper https://arxiv.org/pdf/2102.12092 (uses Gumbel-Softmax for discrete tokenization)
    • Finite State Quantization (no STE or commitment loss magic): https://arxiv.org/abs/2309.15505
  • Weights-only quantization:
    • Optimal Brain Surgeon: https://ieeexplore.ieee.org/abstract/document/298572
    • GPTQ https://arxiv.org/abs/2210.17323
    • LDLQ https://arxiv.org/pdf/2307.13304 (same as QuIP)
  • Weights+Activation quantization:
    • Quantization for Matrix Multiplication (theory) https://arxiv.org/abs/2410.13780
    • NestQuant algo for quantizing weights+activations https://arxiv.org/abs/2502.09720
    • Random rotation idea: QuIP https://arxiv.org/abs/2307.13304 and QuaRot https://arxiv.org/abs/2404.00456