Introduction
Most people meet artificial intelligence through chatbots. Behind them are large language models (LLMs), which generate text one token at a time. Yann LeCun, who led AI research at Meta, has argued that generating text or pixels is not enough for machines to understand the world. He proposed an alternative design called the Joint Embedding Predictive Architecture (JEPA).
This article explains what JEPA is, how it works, where it has been used, how to get started, and how it differs from LLMs and other self-supervised methods. It also covers the advantages and limitations of each approach.
Glossary
| Term | Meaning | |---|---| | Embedding | A list of numbers that represents the meaning of an input, such as an image patch or a word | | Latent space | The abstract space where embeddings live. Distances in this space reflect similarity | | Self-supervised learning | Training on unlabelled data by hiding or transforming part of the input and learning from the relationship between the parts | | Token | A small piece of text, such as a word or part of a word, that an LLM reads and generates | | ViT (Vision Transformer) | A transformer network that processes an image as a sequence of patches | | Representation collapse | A failure mode where the model outputs the same embedding for every input | | Stop-gradient / EMA target | Techniques that keep the target encoder from learning in a way that leads to collapse |
What Is JEPA?
JEPA is a self-supervised learning architecture. The model learns from the data itself, without human-written labels. It hides part of the input and learns to predict the hidden part.
The name describes the design:
- Joint embedding: both the visible input and the hidden target are converted into embeddings by encoders.
- Predictive: a predictor network takes the embedding of the visible part and predicts the embedding of the hidden part.
The key detail is that prediction happens in embedding space, not in pixel or token space. The model never has to reproduce every detail of the missing content. It only has to predict the abstract meaning of that content.
The Core Intuition
Imagine a photo of a dog running on a beach, with the lower-left corner covered by a black box. A generative model would try to paint the exact pixels under the box, including the sand grains and the shade of each wave. A JEPA model would instead learn that the hidden region contains a paw or a leg in motion. It predicts the concept, not the texture.
Most details in the world are irrelevant to understanding it. Predicting the exact ripple pattern of water wastes capacity. JEPA lets the model focus on what is predictable and important.
Three Ways to Learn From Unlabelled Data
JEPA is easier to understand when compared with the two main alternatives in self-supervised learning.
| Approach | What it predicts or compares | Main drawback | |---|---|---| | Generative (for example, masked autoencoders) | Redraws the hidden raw pixels or tokens | Spends effort on unpredictable detail such as noise and exact texture | | Contrastive (for example, SimCLR, CLIP-style training) | Pulls matching views together and pushes different examples apart | Needs many negative examples and hand-designed augmentations such as cropping and flipping | | JEPA | Predicts the hidden part's embedding from the visible part's embedding | Needs safeguards against collapse, such as an EMA target encoder or stop-gradient |
JEPA keeps the "predict the hidden part from the visible part" idea from generative models and the "compare in a summary space" idea from contrastive methods. It drops the pixel-level reconstruction of the first and the reliance on negative pairs and hand-made augmentations of the second.
How JEPA Works, Step by Step
A typical JEPA training loop has these parts:
- Split the input. Choose a visible context block and one or more hidden target blocks from the same image, video, or signal.
- Encode the context. A context encoder (usually a ViT) processes only the visible patches and turns them into embeddings.
- Encode the target. A target encoder turns the hidden block into embeddings. This encoder is usually updated slowly, as an exponential moving average (EMA) of the context encoder's weights, and gradients are not passed through it.
- Predict. A predictor network takes the context embeddings and the position of the hidden block, and predicts the target embeddings. Position tells the predictor which part it is guessing.
- Compare. A loss function measures the distance between the predicted and actual target embeddings. Training minimises that distance.
The slow-moving target encoder and the asymmetric design help prevent representation collapse. Without these safeguards, the model could make every embedding identical, which gives a perfect but useless prediction.
The following sketch shows the structure in simplified PyTorch-style pseudocode. It is illustrative and not a complete training script:
# Simplified JEPA training step (illustrative structure only)
context_emb = context_encoder(visible_patches) # encode what we can see
with torch.no_grad():
target_emb = target_encoder(hidden_patches) # slow-moving teacher, no gradients
predicted_emb = predictor(context_emb, hidden_positions) # predict hidden embeddings
loss = F.mse_loss(predicted_emb, target_emb) # compare in latent space
loss.backward()
optimizer.step()
update_ema(target_encoder, context_encoder, momentum=0.996) # slow update of the teacher
Notice what is missing: there is no decoder that turns embeddings back into pixels, and no word-by-word text generation.
A Numeric Example
Suppose the embedding space has 3 dimensions for simplicity. The predictor outputs [0.9, 0.1, 0.4] for a hidden patch, and the target encoder produces [1.0, 0.0, 0.5].
The difference vector is [-0.1, 0.1, -0.1]. Its squared length is 0.01 + 0.01 + 0.01 = 0.03, which is the squared-error loss for this example. Training pushes this number down over millions of examples. Real JEPA models use hundreds or thousands of dimensions, but the logic is the same.
JEPA Models in Practice
JEPA started as a principle and has since produced several concrete models.
| Model | Year | Domain | What it does | |---|---|---|---| | I-JEPA | 2023 | Images | Predicts the representations of masked image regions from visible regions, without hand-made augmentations | | V-JEPA | 2024 | Video | Learns from video by predicting masked spatiotemporal regions in embedding space | | V-JEPA 2 | 2025 | Video and robotics | A 1.2-billion-parameter world model that adds action-conditioned prediction for planning | | VL-JEPA | 2025 | Vision and language | Predicts continuous text embeddings rather than generating tokens | | LeJEPA | 2025 | Theory and training | A follow-up that puts JEPA-style training on firmer theoretical footing |
Example 1: Image Understanding with I-JEPA
Consider an image classifier that must recognise objects. A conventional approach needs many human-labelled images. I-JEPA is pretrained on unlabelled ImageNet images. It learns to predict what sits in masked blocks, such as "there is a wheel here" or "this region is part of a face."
Meta reports that a 632-million-parameter Vision Transformer was pretrained on 16 A100 GPUs in under 72 hours. With only 1% of ImageNet labels, the model reached state-of-the-art low-shot classification for its time, and comparable methods needed two to ten times more GPU-hours. This is useful when labelled data is expensive, such as in medical imaging.
Example 2: Video and Physical Reasoning
Suppose a video shows a ball rolling behind a wall and then appearing on the other side. A model that predicts in latent space can learn that the ball should still exist behind the wall and continue moving. If the video instead shows the ball vanishing or teleporting, the prediction error is high. This is the kind of "physical common sense" LeCun argues machines need.
Example 3: Robot Planning with V-JEPA 2
V-JEPA 2 was released in June 2025 as an open-source model with code and checkpoints. It is trained in two stages. The first stage uses more than a million hours of video with no human annotation. The second stage adds a small amount of robot data that pairs video with control actions.
The model can imagine the outcome of a planned movement in embedding space and choose the plan whose predicted future matches its goal. Meta reports zero-shot pick-and-place on robots in new labs, with 65% to 80% success on new objects when visual subgoals are provided. Planning in embedding space avoids rendering a full video of each option, which keeps it cheaper. Those success rates are not perfect, and the results are reported by Meta rather than independently verified.
Benchmarks Built Around the Idea
V-JEPA 2 was released with three benchmarks that test physical reasoning in video models:
- IntPhys 2 checks whether a model can tell physically plausible scenes from implausible ones.
- Minimal Video Pairs uses pairs of videos with small changes to reduce shortcut answers in video-language questions.
- CausalVQA tests cause and effect, including "what could happen next" questions.
Humans score between 85% and 95% on these tests, and current models, including V-JEPA 2, remain well below that level.
Getting Started with JEPA
JEPA has no hosted API and no account to sign up for. You run the models yourself with open-source code.
-
Install Python and PyTorch. Install PyTorch from pytorch.org, choosing the build that matches your hardware (CPU or CUDA GPU).
-
Clone the official repositories. Meta publishes the code under the facebookresearch GitHub organisation. The I-JEPA repository is at
github.com/facebookresearch/ijepa. Find the V-JEPA and V-JEPA 2 repositories on the same page.git clone https://github.com/facebookresearch/ijepa -
Install the requirements listed in each repository's README, usually with
pip install -r requirements.txt. -
Download pretrained weights. Each README links to checkpoints. V-JEPA 2 checkpoints are also listed on Hugging Face.
Hardware Expectations
- Pretraining from scratch needs several GPUs and a long training run.
- Feature extraction or fine-tuning a small model is practical on a single GPU.
- Small models can run on a CPU for feature extraction, but more slowly.
What You Get Out
JEPA produces embeddings, which are vectors of numbers, not text. To use the output for classification, search, or robotics, you train a small head on top of the embeddings. A JEPA model does not provide a chat interface on its own. For that, pair it with a language model.
What Is an LLM?
A large language model is a transformer neural network trained to predict the next token in a sequence of text. After training on large amounts of text, the model can answer questions, write code, summarise documents, and translate.
LLM training is generative. The model outputs a probability distribution over the next token. A token is sampled and appended to the input, and the loop repeats until the answer is complete.
Example: Next-Token Prediction
Given the input "The capital of France is", an LLM assigns high probability to " Paris". It appends that token and continues. The model is not looking up a fact table in the usual sense. It has learned statistical patterns from its training text that usually produce correct continuations.
JEPA vs LLM: Key Differences
| Aspect | JEPA | LLM | |---|---|---| | Core objective | Predict hidden content in embedding space | Predict the next token in a sequence | | Output | Embeddings, which are abstract vectors | Tokens, which are readable text | | Generation | Not generative by default, so it does not write sentences or images | Generative by design | | Typical input | Images, video, sensor data, and increasingly text | Mostly text, with growing multimodal support | | Learning signal | Self-supervised, by predicting masked parts of the input | Self-supervised, by predicting the next token | | Handling detail | Tends to ignore unpredictable low-level detail | Must model fine-grained surface detail to generate fluent text | | Main architecture | Encoders plus a predictor, usually ViT-based | Usually a decoder-only transformer | | Best known for | Representation learning, world models, planning | Conversation, writing, coding, reasoning in text | | Interpretability of output | Hard for humans to read directly | Easy, because the output is text | | Common deployment | Feature extractor inside a larger system | Standalone assistant or API |
The Central Difference
An LLM answers the question "what token comes next?" JEPA answers "what is the abstract state of the hidden part?" One produces language, the other builds an internal model of the world.
Text alone often misses physical intuition. A language model can describe a glass falling off a table, but it learns that from descriptions rather than from observing physics. JEPA-style models learn from video and sensor data, which is why many researchers see them as complementary to LLMs.
The line between the two is blurring. Vision-language models accept images as input, and VL-JEPA predicts text embeddings rather than tokens. Neither approach is strictly better, because they solve different problems. Many future systems may combine them: a JEPA-style encoder builds the understanding, and a language model explains it in words.
Advantages of JEPA
- No pixel-level reconstruction. The model does not spend capacity on textures and noise.
- No hand-made augmentations or negative pairs. I-JEPA avoids the view-augmentation pipelines that many contrastive methods depend on.
- Useful representations with fewer labels. Pretrained JEPA features can transfer to classification, detection, and depth estimation.
- Efficient training in some cases. Meta reports I-JEPA trains with fewer GPU-hours than comparable methods, depending on the task.
- Suited to world models. Latent prediction supports planning, because a system can test possible futures without rendering them.
- Works across modalities. The same principle applies to images, video, audio, and robotics signals.
Limitations of JEPA
- Not a chatbot. JEPA does not produce readable answers by itself, so it needs another component for dialogue.
- Harder to inspect. Embeddings are opaque vectors, so explaining a prediction takes extra tools.
- Early world models. Meta describes I-JEPA as a first step, and V-JEPA 2 predicts at a single time scale, which limits multi-step planning.
- Still short of human performance. On the physical reasoning benchmarks, current models remain well below human scores.
- Evaluation is less mature. LLMs have large benchmark ecosystems. JEPA benchmarks focus mainly on vision, video, and robotics tasks.
- Collapse must be managed. Training depends on safeguards such as EMA target encoders, which add design choices.
Advantages of LLMs
- Natural interface. Users interact through plain language.
- Broad knowledge and flexibility. One model can handle questions, code, translation, and summaries.
- Mature tooling. APIs, fine-tuning methods, and evaluation suites are widely available.
- Easy to verify output. A person can read the answer and judge it.
Limitations of LLMs
- Hallucinations. The model can produce fluent but false statements.
- Limited grounding in the physical world. Learning from text alone gives only indirect knowledge of physics and cause and effect.
- Cost at scale. Long generations need many sequential steps, which increases latency and expense.
- Sensitivity to phrasing. Small changes in a prompt can change the answer.
Common Misconceptions
- "JEPA is a chatbot." It is not. It learns representations and predictions, and it needs other components to hold a conversation.
- "JEPA needs an API key." It does not. You run the open-source models yourself.
- "JEPA replaces LLMs." Not necessarily. The two are often complementary.
- "JEPA is always better for images." It is strong for representation learning, but the best choice depends on the task, data, and compute budget.
Frequently Asked Questions
Is JEPA generative? Not in the usual sense. It predicts embeddings, not pixels or tokens, so it does not produce images or text directly.
Does JEPA need labelled data? No. Pretraining is self-supervised. A small labelled set is usually needed only for the final task, such as classification.
Does JEPA need negative examples? No. This is one of its main differences from contrastive methods.
Will JEPA replace LLMs? Current evidence suggests the two are complementary. JEPA-style models focus on perception and prediction, and LLMs focus on language.
Which Should You Use?
Use an LLM when the task is language-heavy: drafting text, answering questions, writing code, or summarising documents.
Use a JEPA-style model when the task is perception-heavy or needs an internal model of the world: image or video understanding with few labels, anomaly detection in sensor data, or planning for robots.
Many modern systems pair a perception encoder with a language model.
Key Takeaways
- JEPA predicts the embeddings of hidden parts of its input, so it learns abstract structure rather than pixels or tokens.
- Compared with other methods, it avoids pixel reconstruction, negative pairs, and hand-made augmentations, but it needs safeguards against collapse.
- Self-supervised training on unlabelled data is its main strength.
- I-JEPA, V-JEPA, V-JEPA 2, and VL-JEPA show the approach working for images, video, robotics, and vision-language tasks.
- LLMs are generative models that predict the next token, which makes them excellent at language but less direct at modelling the physical world.
- The two approaches are complementary. Future systems will probably combine world-modelling architectures like JEPA with the language abilities of LLMs.
References
- Assran, M., et al. (2023). Self-Supervised Learning from Images with a Joint-Embedding Predictive Architecture (I-JEPA). CVPR 2023. Meta AI announcement: ai.meta.com/blog/yann-lecun-ai-model-i-jepa
- Bardes, A., et al. (2024). Revisiting Feature Prediction for Learning Visual Representations from Video (V-JEPA). Meta AI.
- Meta AI (2025). V-JEPA 2 world model and benchmarks. ai.meta.com/blog/v-jepa-2-world-model-benchmarks
- LeCun, Y. (2022). A Path Towards Autonomous Machine Intelligence. Position paper.
- Official code: github.com/facebookresearch
Conclusion
JEPA represents a different path to machine intelligence. Instead of reconstructing the world in detail, it learns to predict what matters in an abstract space. LLMs have shown what generative language models can do, and JEPA offers a way to build the perception and planning abilities those models lack. Understanding both helps you choose the right tool and follow where the field is heading.