# Watching JEPA Write Code Live: What It Got Right, and the Check It Fooled

> RIAI Lab Notes #5: a live test of the JEPA code writer. Fast, readable, no LLM, and correct for its examples. But the examples didn't say what was meant.

- Author: Tai Pham Phu
- Published: 24 Oct 2026
- Tags: RIAI, AI Research, Self-Improving AI, JEPA, Program Synthesis
- URL: https://phamphutai.com/2026/10/24/riai-jepa-live-test/

Three examples went into the box:

```
'kiwi pear kiwi fig plum'     -> 'kiwi'
'sun moon moon dog moon'      -> 'moon'
'red red blue fig blue blue'  -> 'blue'
```

Then **Solve**. Seven seconds later, the screen showed a program, checked in the offline VM on all three examples:

```python
def f(p0):
    x1 = p0.split()
    x2 = max(x1, key=len)
    return x2
```

It passed every check. One more input was tried, `'cat dog cat'`, and it answered `'cat'`. Correct again.

Read the program, though. It doesn't find the **most common** word. It finds the **longest** word.

This is the fifth of my RIAI Lab Notes. [The last post](https://phamphutai.com/2026/10/24/riai-jepa-for-code/) introduced JEPA v2, a code writer that never predicts tokens: it scores possible Python statements in vector space and builds a program one statement at a time. On 12 exercises it got 8 right on the first try. This post is about the first live test: what it did right, which is a lot, and the lesson it taught in one screen.

---

## The live test

The tool is called `live_jepa`. You type examples, one per line, as *input -> wanted output*. (You can also type a task in words. Then Claude Haiku suggests three example inputs, the outputs are computed in the VM, and you confirm them.) When you press Solve, you watch the writer think.

![The RIAI live test window. On the left, three examples. On the right, the thinking steps: at each step, the partial programs it keeps and their values on example 1. At the bottom right, the program it found, checked in the VM, in 7.3 seconds: split the text, then take the longest word.](./riai-live-jepa.png)

Each step works the same way, and the screen shows all of it:

1. **Score.** JEPA v2 looks at the current state (every variable so far, on every example) and the goal (the wanted outputs), and scores every statement its grammar allows in vector space: *which of these leads toward the goal?*
2. **Run the best ones.** The top-scoring statements are run on the examples by the **safe calculator**, the small hand-written evaluator from the last post. At step 2 it ran the best **74** statements; at steps 3, 4 and 5, the best **192** each.
3. **Keep a few.** It keeps the partial programs that look most promising and shows their value on example 1. At step 2 you can see it trying `min(x1, key=len)` (giving `'fig'`), `x1[-1]` (giving `'plum'`) and `sorted(x1)`.
4. **Stop at the goal.** As soon as one variable equals the wanted output on **every** example, that program is a candidate. Shorter programs come first. The candidate is then checked for real in the offline VM.

Here, a 2-statement program reached the goal at step 2. The VM confirmed it on all three examples. Total time: **7.3 seconds**. No language model wrote any of the code.

As a demonstration of the mechanism, it's fast and readable, and you can see *why* it chose each step.

---

## What it did right

Before the catch, it's worth being clear about how much went right in those 7.3 seconds. Every one of these is visible on the screen.

**1. It wrote working code with no language model.** No next-token prediction anywhere. A small JEPA scored statements in vector space, a hand-written calculator ran them, and the result was a real Python function that ran correctly in the VM. That's the whole point of the direction: code that comes from a learned sense of "this step moves toward the goal", not from a model that has read the internet.

**2. It searched efficiently.** It didn't try every possible program. At each step it scored all the statements its grammar allows and ran only the best ones on the examples: 74 at step 2, then 192 at each later step, keeping just 6 partial programs each time. A 2-statement answer reached the goal at the first step where it could.

**3. It found a program that truly fits the data.** "Take the longest word" matches all three examples exactly. It isn't a lucky pile of statements that happens to print the right strings, like the "lucky" program from the last post. It's a clean, general rule that explains every example, and it kept giving the right answer on a new input it had never seen.

**4. It chose the simplest explanation.** Out of everything that fits, it took the shortest program. That's the same rule scientists use (prefer the simplest theory that explains the evidence), and in general it's a good defence against over-fitting: long, twisted programs that match the examples by accident are pushed to the back of the queue.

**5. Its reasoning is readable.** You can see every partial program it considered and kept, with its value on the first example: `min(x1, key=len)` gives `'fig'`, `x1[-1]` gives `'plum'`, and so on. When it's wrong, you can see exactly where and why. That's rare: most code writers, human or AI, don't show their rejected drafts.

**6. It was safe and checked.** Nothing ran outside the whitelist of the safe calculator while it was thinking. Only the finished program ran, and only inside the offline VM, which confirmed the outputs independently.

**7. It can write the "right" program too.** In the benchmark from the last post, with examples chosen to separate look-alikes, JEPA v2 wrote the real "most common word" program, counting words in a dictionary and taking the most frequent, in about half a second. The ability is there. What the live test exposed is what it does when the examples don't ask for it.

---

## Why it found the wrong program

The three examples read like "the most common word". *kiwi* appears twice, *moon* three times, *blue* three times.

But look at word lengths:

| Example | Most common word | Longest word |
|---|---|---|
| `kiwi pear kiwi fig plum` | kiwi (2 times) | kiwi, pear, plum (4 letters; `max` returns the first: **kiwi**) |
| `sun moon moon dog moon` | moon (3 times) | **moon** |
| `red red blue fig blue blue` | blue (3 times) | **blue** |

In every example, the most common word is also the longest word, or the first of the longest. Both programs fit all three examples perfectly.

And the writer is built to prefer the **shortest** program that reaches the goal. "Longest word" takes 2 statements. "Most common word" needs a dictionary and a loop to count, several statements more. So it found the short one first, and stopped.

The extra input didn't catch it either. In `'cat dog cat'`, *cat* and *dog* are the same length, so `max` returns the first, *cat*, which is also the most common word. A second coincidence.

Nothing here is a bug. Every part did its job: the scorer found a path to the goal, the calculator computed the right values, the VM confirmed the outputs. **The examples simply didn't say what was meant.**

---

## Why this matters more than one wrong answer

### Examples are the specification

When a task is given only as examples, the examples *are* the specification. A program that matches them is, by definition, correct, for those examples. If two different programs both match, the system can't know which one you meant. It picks by its own rule, here "shortest first", and it does so confidently.

That's not special to JEPA. Any system that writes programs from examples has this problem, and so do people. Give a new colleague three examples of a report and they may copy the wrong pattern too.

In the benchmark from the last post, I'd chosen the 3 examples for each exercise **to rule out programs that fit by luck**, and added a fresh input on top. That's why the "most common word" exercise there was solved correctly. The examples in the live test weren't chosen that way, and they produced exactly the failure those choices were designed to prevent.

### It matters most for the vision

My goal for RIAI is an agent that **writes its own Python code to develop new skills, and keeps developing, more and more.** In that loop, every new skill is kept because it passed its checks. Then the next skill builds on it.

The live test shows the weak spot of that loop. A skill can pass every check and still be the wrong skill. If it's kept, everything built on top of it inherits the mistake. "More and more" could quietly become "more and more wrong", with every check green.

This is the same lesson as [testing AI-written code](https://phamphutai.com/2026/10/22/tests-are-how-you-talk-to-the-ai/), from the other side: the tests *are* how you tell a code writer what you want, whether the writer is a language model, a search, or a JEPA.

---

## What would fix it

These are ideas for the next experiments, not results:

- **Look for disagreement before keeping a program.** If two candidate programs both pass the examples, there must be some input where they give different answers, here something like `'a bb a'` (most common `'a'`, longest `'bb'`). The system can search for such an input itself, using the same safe calculator, without running anything risky.
- **Then ask, or check.** Show that input and ask which answer is right. Or, when a task comes with a description in words, use the description to decide, and keep that pair as a new example.
- **Keep the distinguishing inputs with the skill.** A skill saved with the inputs that ruled out its look-alikes carries its own stronger test. Later skills that build on it inherit a sharper definition, not a blurrier one.
- **Report ambiguity instead of hiding it.** "I found a program, but another one fits your examples too" is a more honest answer than a green check.

The first of these fits naturally into the JEPA writer: it already keeps several partial programs at each step. Finding where the top candidates disagree is a small step from there.

---

## Temporary conclusion, updated

As of 24 October 2026, after one live session:

- **The mechanism works and is fast.** JEPA v2 wrote a clean, general 3-line program in 7.3 seconds, with no language model writing code, searched only the most promising statements, chose the simplest fit, and showed every step.
- **Correctness is limited by the examples, not the writer.** With ambiguous examples, it confidently returned a program that passed every check and wasn't the intended one.
- **For a self-developing agent, this is the risk that compounds.** Skills kept on weak checks would pass their mistakes on to every skill built on top.
- **So the next experiment adds disambiguation before a skill is kept:** find an input where the top candidates disagree, and resolve it.

One live session is a small sample, and the benchmark numbers from the last post still stand. But this screen taught something no score did: **a check that passes isn't the same as a skill that's right.**

---

## Key ideas

- **`live_jepa` lets you watch JEPA v2 write code:** score statements in vector space, run the best on the examples, keep a few, stop at the goal, check in the VM.
- **What it did right:** no LLM, an efficient search (74 to 192 statements run per step), a clean general rule that fits every example and a new input, the simplest explanation first, readable reasoning, and safe execution.
- **In the live test it found a program in 7.3 seconds** that passed every check, including one extra input.
- **It was the wrong program:** "longest word" instead of "most common word", because in every example the two gave the same answer.
- **Shortest-first chose the simpler program,** which is a sensible rule that only works when the examples rule out look-alikes.
- **Examples are the specification;** ambiguous examples give confident wrong answers, for any code writer.
- **For an agent that builds skills on skills, wrong-but-passing skills are the risk that compounds.**
- **Next: search for inputs where candidate programs disagree, before a skill is kept.**

*Related: [Next: Teaching an Agent to Imagine Code With JEPA](https://phamphutai.com/2026/10/24/riai-jepa-for-code/), and [Tests Are How You Talk to the AI](https://phamphutai.com/2026/10/22/tests-are-how-you-talk-to-the-ai/).*