The core problem: everything must become tokens

Transformers still win across every modality at scale, and they speak tokens — discrete or continuous embeddings, each representing a semantic unit. Text subwords are roughly meaningful; a single pixel is not. So the central challenge of multimodality is finding the image/audio equivalent of the BPE tokenizer: convert non-text into tokens a transformer can digest (input), and generate them back out (output). This lecture is almost entirely about the input side.

CLIP — contrastive language-image pre-training (2021)

In the GPT-3 era, vision was still ImageNet/ResNet supervised learning. CLIP asked: can we leverage the web's noisy image-text pairs the way LMs leverage noisy text? The objective is simple — encode each image to a vector Iᵢ and each caption to Tᵢ, and make matched pairs align far more than mismatched ones (a 2N-way softmax classification over the batch):

Vision encoder: a ViT (beat ResNets) — chop the image into patches (14×14), treat each patch as a token, add positional embeddings, run a transformer, attention-pool to one vector. The best model is ViT-L/14 @ 336². Images are resized (short side to 336) and center-cropped — a heuristic that's fine for classification-centric work. Text encoder: a GPT-2-style transformer, using the EOS activation as the sequence vector. Data: 400M web pairs (CLIP, unreleased) / LAION-5B (OpenCLIP, which bootstrapped CLIP itself for filtering).

SigLIP — sigmoid loss (Google)

SigLIP replaces CLIP's multi-class softmax with a per-pair binary classification (aligned or not): diagonal entries are +1, off-diagonal −1, with a log_sigmoid loss. This decouples the loss from batch size — so small batches actually work better (CLIP degrades), and it parallelizes cleanly (each device holds some pairs and rotates text shards around to cover the off-diagonal blocks, like DDP). Trained on WebLI (~billion pairs, plus OCR'd text-image pairs, multilingual), it's far more efficient than CLIP, with ~32K as its critical batch size.

The template

Every VLM follows the same recipe — take a pre-trained vision encoder + a pre-trained LM and stitch them with an adapter, then mid/post-train (not from scratch):

The image vectors get mapped into the LM's token-embedding space and concatenated with text embeddings; the whole sequence runs through a standard transformer. Output is always text (these models ingest images but don't generate them).

LLaVA → OneVision: data and resolution

LLaVA trains in two stages: (1) freeze encoder + LM, train only the projector W (alignment — make image vectors look like token embeddings); (2) freeze the encoder, fine-tune W + LM on the synthesized image→text data. OneVision upgrades the parts and adds multi-image/video, via the key resolution trick:

Qwen-VL series: dynamic resolution & M-RoPE

Qwen2-VL made resolution truly dynamic (a big picture → ~11K tokens, a tiny equation → ~8), via AnyRes tiles + 2×2 compression to ~66 tokens/patch. It also introduced multimodal RoPE (M-RoPE): positions are now 3D (time, height, width), with RoPE computed per axis and concatenated. Qwen3-VL refines this:

• Interleaved M-RoPE frequencies — the naive block layout gave each axis only one frequency band (time=low, etc.); interleaving exposes every axis to all frequencies.
• Explicit video timestamps as tokens (vs implicit position) — you can refer to "what happened at 2 seconds."
• √-normalized per-token loss so long video examples don't dominate short image ones.
• DeepStack adapter — add the vision encoder's stacked embeddings directly into the LM residual stream (deep fusion vs treating the encoder as a black box), plus 256K context and SigLIP-2.

Note: the vision encoder is far smaller than the LM (often <1B params) — patch encoding is a local operation with little knowledge/reasoning; the capability lives in the LM.

Map images to discrete tokens, then just train an LM

VLMs only generate text. Chameleon (Meta, 2024) instead maps everything to discrete tokens so you can analyze and generate images uniformly (interleaved text+image). The image tokenizer is a VQ-VAE: encode an image to a continuous vector, round to the nearest of ~8K codebook codes, decode to reconstruct (trained on reconstruction loss); a 512² image → 1024 tokens. Then training is just standard LM training — no adapter, no separate encoder.

Golden rules & takeaways

• Everything becomes tokens. The hard part of multimodality is the non-text tokenizer — discrete (VQ-VAE) or continuous (CLIP-style embeddings).
• CLIP/SigLIP learn semantic image vectors from noisy web pairs; SigLIP's per-pair sigmoid decouples loss from batch size and parallelizes better.
• VLMs = encoder + adapter + LM, stitched and mid/post-trained, with a two/three-stage alignment-then-fine-tune recipe. Output is text.
• Resolution is dynamic (AnyRes tiling + compression) and position is 3D (M-RoPE); the vision encoder is small relative to the LM.
• Understanding ≠ generation. Small semantic vectors (CLIP) suffice for understanding; fine detail (diffusion) is needed for OCR/generation — there's no single universal encoder.
• Chameleon's all-discrete approach is elegant but lossy/unstable; frontier omni models are likely continuous encoders + diffusion (undisclosed). Weight modalities carefully — video is lower-density than text.

This rounds out the course: from tokenization and architecture through systems, scaling, data, alignment, reasoning — and now seeing.

References

Primary source is the lecture itself. Key works named:

1. Radford et al. (2021) — CLIP · Dosovitskiy et al. (2020) — ViT
2. Schuhmann et al. (2022) — LAION-5B / OpenCLIP
3. Zhai et al. (2023) — SigLIP
4. Liu et al. (2023) — LLaVA · Li et al. (2024) — LLaVA-OneVision
5. Bai et al. (2023) — Qwen-VL · Wang et al. (2024) — Qwen2-VL
6. Chameleon Team (2024) — Chameleon
7. van den Oord et al. (2017) — VQ-VAE