Companion reading

GPT-2 as the clean statement of the lecture's opening move: the next word is a target the data already contains, and one objective absorbs many tasks.

Read for the scale claim only: what happens when self-supervised text prediction meets essentially unlimited training material.

Start with reciprocal supervision, shared representations, and missing modalities before foundation-model scale obscures the original problem.

Use representation, alignment, fusion, translation, and co-learning as the grammar for everything that follows.

Canziani's lecture notes decompose the family into augmentation, backbone, distance, and anti-collapse losses — the same anatomy as this lecture's four questions. Continue into the regularised methods (15-2) for VICReg, BYOL, and the two collapse types.

Use it as a map of views, pairs, negatives, losses, and failure modes; return to the primary papers for mechanisms.

Work through InfoNCE as identification of the matching future latent among sampled alternatives.

The cleanest controlled account of augmentation, projection heads, temperature, and in-batch comparison.

Follow the move from two views of one image to image–text pairs. The demo scores one row of its matrix live — the one-word rival, the learned temperature, and the left/right question it cannot answer.

See why negatives are one anti-collapse mechanism rather than the definition of joint-embedding learning.

Separate invariance, variance, and covariance as three geometric pressures.

Self-supervised features strong enough to reuse without labels — the encoder the live demo interrogates patch by patch.

The same recipe scaled to 7B with Gram anchoring for stable dense features; its feature visualisations are the ones shown before the demo.

Study the isotropic-Gaussian target and SIGReg as an explicit anti-collapse proposal; keep the geometry claim separate from robotics.

The supplied 2026 paper: a current refinement that separates scale from distributional shape using sliced-Wasserstein sketches.

Use its controlled theory to understand when high-variance irrelevant detail favours latent-space prediction.

A sharper reconstruction critique, bounded by its theory and by the observation that masking changes the problem.

A masked, contrastive identification problem inside one audio stream.

Whisper is the corrective case: sequence-to-sequence learning at scale, not contrastive pretraining.

Compare audio–language alignment with both temporal audio prediction and transcription.

Ask what is gained, and assumed, when image-paired data become the hub for six modalities.

Trace Thinker–Talker, chunk-wise streaming, audio–video timestamps, causal speech rendering, and text–speech token-rate alignment.

ViT establishes the move the VLM chapter depends on: an image can enter a transformer as a sequence of patch embeddings.

LLaVA is the lecture's worked example of how image patches become tokens a language model reads: a projector maps CLIP patch features into the embedding space, trained by caption alignment and then visual instruction data.

Perceiver resampling and gated cross-attention into a frozen language model.

The Q-Former makes the bottleneck between frozen vision and language models explicit.

Update the architecture story through SigLIP-2, visual-token merging, DeepStack, multidimensional position, and explicit video timestamps.

The pretrained VLM (SigLIP encoder, SmolLM2 decoder) that the lecture's VLA starts from — read it before SmolVLA to see exactly what is taken frozen when a VLM is turned into a VLA and fine-tuned on LIBERO.

A controlled alternative to token-space prediction: continuous semantic targets and selective decoding, with explicit limits.

A practical digital action loop: generate code, render it, inspect the result with a VLM critic, and revise. Read the evaluation and latency limits alongside the improvement.

A bridge from multimodal language modelling to continuous sensor and state inputs; do not confuse embodied reasoning with low-level control.

Establish the large-scale behaviour-cloning substrate before the VLA label.

Understand action-as-token co-fine-tuning and distinguish semantic transfer from motor reliability.

The data substrate: scale through heterogeneous robots, plus normalisation and embodiment mismatch.

A useful non-VLM generalist-policy comparison: which gains come from data and policy pretraining?

Fused semantic and local visual features, a language backbone, action tokens, and a practical open fine-tuning route. The blue-cup / pink-cup and reattempt clips on the VLA chapter slides are OpenVLA.

Contrast discrete action tokens with a continuous action expert trained by flow matching — the mechanism SmolVLA inherits, and the folding footage in the deck.

The model the demo runs and fine-tunes: the frozen SmolVLM above with a ~100M flow-matching action expert — the architecture behind every training tier shown in the lecture.

Learn what useful ranking correspondence can establish without treating simulation as physical reality.

The central criticism: correct semantic intention can survive while physical execution fails. Read the benchmark limitations just as closely.

Read the programme as a diagnosis and proposal, not as experimental proof that a full architecture already works.

The image-level proof of concept for target prediction in representation space.

Extend feature prediction through time while keeping representation evidence separate from physical understanding.

Test whether the JEPA construction is modality-general by comparing it with wav2vec 2.0, CLAP, and Whisper.

Action-conditioned future-latent prediction with explicit Gaussian regulation; track short horizons and offline coverage.

The strongest theoretical claim, and a reminder to inspect Gaussian, dimensionality, optimum, and finite-sample assumptions.

Direct robotics evidence for image-goal planning; the demo uses its action-conditioned energy to score nine named commands and a 343-candidate grid, with the reversed-clip control. Read camera sensitivity, search cost, latency, and horizon limits alongside success.

A current attack on planning horizon through latent subgoals, still bounded by visual goals and selected manipulation settings.