A Terminal That Learns What Commands Do

“Hello! I am a terminal agent. I learned 35 commands by experimenting (3238 experiments).”

That’s how my terminal agent introduces itself in a short scripted demo. Then I ask it to count the vowels in a file called story.txt and write the number into vowels.txt.

It answers: “None of my commands can do that.” It offers to write a Python tool called count_vowels, and asks if it should. I say yes. Claude Haiku writes the code (more on that below), its test passes on the first attempt, and the tool becomes one of its commands. Then it plans in its head, finds a one-command plan, says what it expects that command to do, and asks again before running it. The file now says 11. Its own check says the goal is reached.

I ask it to do the same for notes.txt. Same tool, new file, answer 7.

It’s a tiny demo. But almost everything I care about in this project is packed into those few lines: an agent that learned what commands do by trying them, that plans before it acts, and that, when it hits a wall, grows a new command instead of giving up.

This is the second post in my RIAI Lab Notes. The first one gave the whole map. This one goes deep on the main line: the terminal agent.

Why the terminal

My idea is simple to say: the terminal does everything; it self-develops; it writes its own Python code to develop skills; and I expect it to keep developing, more and more.

Files, programs, data, other tools: nearly everything a computer can do is reachable by typing a command. If an agent truly knows what commands do, it doesn’t need a special connector for each job. And if it can write new commands for itself, its reach isn’t fixed by whoever built it.

Why “learn what commands do” instead of imitating

The obvious way to build a terminal agent today is to use a language model that has read millions of shell scripts. It already “knows” what sort does, because people wrote about it.

But copying answers isn’t the same as knowing how the world responds. A student who copies worked examples can do the examples. A student who knows what each move does can plan moves nobody showed them.

So this agent is built the other way round. It’s given a list of commands and is not told what any of them do. It has to find out the way a child finds out what a light switch does: press it, look, remember. What it builds is a world model, a little inner simulator that answers one question: if I run this command here, what will my files look like afterwards?

How it works

Here’s the loop piece by piece, straight from the code.

What it sees

The agent works in a folder called ~/work inside an isolated virtual machine. Its picture of the world, its state, is just the files in that folder: which folders and files exist, what each file contains, and whether it’s executable. Roughly what ls -R and cat would show you.

What it can press

It gets a “controller” of command templates with slots, like sort {A} > {B}, wc -l < {A} > {B}, tac {A} > {B} or sed -i 's/{T}/{U}/g' {A}. The letters are blanks it fills with file names or words. There are 35 of these to start with. A comment in the code puts it plainly: what they do is not given.

How it learns what a command does

For every template, the agent starts with a list of possible effects: “makes a new folder”, “copies A to B”, “B becomes the sorted lines of A”, “B becomes the number of lines of A”, “nothing changes”, and so on, with the slots in either role.

Every time it runs a command, it compares each possible effect with what really happened to the files. Effects that keep agreeing gain support. Effects that disagree lose it. Failures teach it too: separately, it tracks when a command fails, based on what the slots pointed at (a file, a folder, or nothing). Its prediction is the best-supported effect.

Two real examples, from the saved world model (world_model.json, as it stood after 3,360 experiments):

• sort {A} > {B}. The effect “B is the sorted lines of A” matched what happened 15 times out of 15. Its closest rival, “B is a copy of A”, matched 11 of 15. That rival isn’t silly: if a file is already in order, sorting it and copying it look exactly the same. Only the cases where the order changed could tell them apart.
• cp {A} {B}. “B is a copy of A” matched 4 of 5. The other time, the better fit was “the file lands inside B”, which is what cp does when B is a folder. So its best guess sits at 80%, below the bar it needs to trust a command.

Nobody told it that cp behaves differently when the target is a folder. Its own evidence is split, and its confidence shows it.

Curiosity: practise what you’re least sure of

Before any exam, the agent plays in random sandboxes: a few small text files, a folder with some files in it, sometimes repeated lines or numbers so that commands like uniq and sort -n can be told apart from a plain copy. It never practises on exam tasks.

Which command does it try next? The one it’s least sure about, plus a little randomness. A command with few tries, several equally good explanations, or a shaky best guess gets picked first. When a prediction turns out wrong, the screen literally says “surprise! the world model expected something else - it learns”.

From words to a target

A task arrives in plain English, such as “Write the lines of piano.txt sorted and without duplicates into uniq.txt.” The world model knows nothing about English, so a language model, Claude Haiku, translates the task once into a precise target state: a list of conditions like “file uniq.txt holds the sorted unique lines of piano.txt”.

Then goal memory takes over. It remembers the shape of each task it has seen, with the file names and words swapped out. A new task with the same shape gets its target state from memory, with zero calls to Haiku. By the end of these runs, goal memory held 53 task shapes.

Planning in imagination

With a target in hand, the agent searches for a sequence of commands inside its world model, without touching a single real file. It’s a beam search: try every command it trusts, with every sensible filling of the blanks, keep the 15 most promising imagined states, and go up to four commands deep. It only uses commands it’s confident about.

When an imagined state matches every condition of the target, that’s the plan. Then it runs the plan for real, one command at a time.

Check, learn, re-plan

After running, the agent checks the real files against its target. That’s its self-check. If a command did something it didn’t predict, that’s a surprise: it learns from it, updates the world model, and re-plans from where it now stands, with up to three attempts in total.

Separately, the host runs its own secret check for each exam task, which the agent never sees. That’s the score. It also catches the most dangerous mistake: the agent believing it’s done when it isn’t.

One small moment from the last run shows what “planning by world model” really means. The task said: create a folder honey, write “eagle” into honey/lemon.txt, then make a copy of that file named backup.txt. Its plan was:

mkdir honey ; echo eagle > honey/lemon.txt ; echo eagle > honey/backup.txt

No cp at all. It aimed at the end state, a backup file with the same content, and found a way there using commands it trusted. The host check passed. It’s not how a person would do it, but it’s exactly what the planner is built to do: reach the target state.

Growing its own skills

Some tasks can’t be done with any of its commands. “Average these numbers.” “Find the most common word.” “Sum the amount column of this CSV.”

When the goal translation says no known command can produce the result, the agent takes the path on the left of the diagram:

1. A Python tool is written for exactly that job: read one input file, write one output file.
2. It’s tested in the VM. If the test fails, the error goes back for a fix.
3. It’s registered as a new command, like python3 ~/tools/median.py {A} {B}.
4. Its effect is unknown until tried, like any other command. The agent practises it on copies of the kind of file it was made for, in a scratch folder, so the world model learns what it does and when it fails.
5. Then it plans with it, the same as with sort or grep.

I need to be exact here: today, Claude Haiku writes the tool code, not the agent. Haiku also flags that a new tool is needed when it translates the task. The agent gets the tool tested, adds it as a command, learns what it does and uses it. It doesn’t write the Python itself yet.

The first method had Haiku write the whole tool and a test in one go, with up to three tries. A later method writes each tool step by step: Haiku plans 3 to 5 small steps, and after each step the code so far is run on a real sample input and checked before the next step is added. Then a final test checks the whole tool, and if it fails, the error’s line number is used to find which step caused it.

In the step-by-step run, 13 of 15 coding tasks passed the host check. One task type, a CSV column average, failed both when its tool was first written and when it was reused.

There are now 11 tools in its toolbox: average, median, most_common_word, sum_csv_column, json_names_over_30, count_vowels, longest_line, csv_column_average, most_common_city, title_case and filter_evens.

Separately, there’s a coder in RIAI that builds small programs with no language model at all: it searches over small building blocks, guided by a world model it learned by running those blocks 88,712 times. It solved 13 of 13 short tasks, host-checked. That’s the direction for this agent: the “write it” step moving from a teacher model into the agent itself. The next post will cover it properly.

Results across seven runs

Each run continued from the world model, goal memory and tools of the run before, so the experiment count is a running total. Runs 1 to 4 used the original task set. From run 5, the exam switched to a harder set of 20 tasks: 9 that need newer commands (like tac, sort -u, paste), 4 coding tasks seen for the first time in that exam, 4 coding tasks of a type already seen in that exam (reuse), and 3 older task types.

What happened on the harder set:

• Run 5: 11 of 20. New-command tasks went 7 of 9. The coding tasks were a wipe-out: 0 of 4 new and 1 of 4 reuse. All four tools passed their own tests, yet the tasks still failed. The report screen from that run shows one of the new tools crashing with a traceback when it was practised on plain text files.
• Run 6: 18 of 20. Coding went to 4 of 4 new and 4 of 4 reuse. Two new-command tasks still failed: writing the lines that contain a word into a new file, and creating a nested folder path.
• Run 7: 20 of 20, with zero Haiku calls. Every target came from goal memory, and every coding task used a tool that already existed from run 6.

At the end of run 7, the report says: commands understood 35/39 (35 starting commands plus 4 tools), and prediction accuracy (recent) 98%, measured on the last 200 predictions before it learned from each one.

The self-check column matters to me as much as the score. In run 1, the agent twice said “done” when the host said no. In run 3 it did it once. That’s the dangerous kind of mistake, and it’s why the host check exists. In runs 3 and 4 there was also the opposite case: it said “not done” on a task the host passed. From run 5 on, the agent’s own judgement matched the hidden check on every task: 20/20 three runs in a row, and 15/15 in the coding run.

Honest limits

The numbers are real but small. Here’s what they don’t show.

• The exams are small. 18 to 25 tasks per run, and the coding exam had 15.
• Task types repeat across runs. The exam picks new file names and words each time, but the kinds of task come from a fixed list. Run 7 saw the same kinds as runs 5 and 6.
• Zero calls depends on goal memory. Goal memory matches tasks by the shape of the sentence. A task worded differently still needs Haiku: in the chat demo, my follow-up “Now do the same for notes.txt” needed one Haiku call to translate. So “0 calls” means “0 calls on task shapes it has seen before”, not “understands English on its own”.
• Run 7’s coding tasks didn’t involve writing code. The tools were written in run 6. Run 7 shows reuse with no help, not new code.
• Four commands aren’t “understood”. The report doesn’t list them, but reading the world model file, they look like these. Two are cat {A} and ls {A}: they print but don’t change any files, and an effect of “nothing changes” doesn’t count as understood by the rule in the code. The other two are cp {A} {B} (80% of 5) and mv {A} {B} (75% of 4). They’re below the 85% bar, and they haven’t been tried many times.
• Haiku still writes the code. The self-developing loop exists, but the creative step inside it is borrowed.
• The world is narrow. Only files in one folder, and only effects from a list I wrote. If a command does something that isn’t on that list, the agent can’t describe it, only notice that nothing fits.
• Some failures I can’t fully explain from the files. The CSV average task failed in the step-by-step run even though every step of its tool was judged correct. The logs show that, not why.

What’s next

The next step is to make the agent write more of its own code.

I’ve collected line-by-line traces of 61,611 programs running inside the sandbox, 982,893 traced lines in total, including the agent’s own 11 tools. The plan is to train a small code world model on them: a model that predicts what each line of code will do, the same way the terminal world model predicts what each command will do. With that, the agent could imagine running a program before it runs it, and check its own code in its head.

And then there’s my real expectation for this project: that it keeps developing, more and more. Every tool it gains becomes a command it can plan with. Every task shape it remembers costs nothing the next time. If that compounding is real, each new skill should make the next one a little easier to get.

Right now, that’s a hypothesis, not a result. Seven runs over one afternoon is a start, not proof. The way to test it is the same as everything else in RIAI: harder exams, new task types it has never seen, an honest outside check, and numbers I’m willing to publish even when they go down.

Key ideas

• The goal is a terminal that develops itself: it does everything from the command line, and grows new skills as Python tools, so it can keep developing.
• It learns what commands do instead of imitating: for each command it keeps possible effects and scores them against what really happens to the files, failures included.
• Curiosity picks what to practise: the command it’s least sure about goes first.
• It plans in imagination, then checks reality: search inside the world model, run for real, self-check, learn from surprises, re-plan.
• Language is used once, then remembered: Haiku turns a task into a target state, and goal memory reuses known task shapes with zero calls.
• New skills become new commands: a tested tool is just another button whose effect it has to learn. Today Haiku writes that code.
• The best run scored 20/20 with zero Haiku calls, with 35/39 commands understood and 98% recent prediction accuracy, but on a small exam of familiar task types.

Related: RIAI: Toward a Terminal Agent That Writes Its Own Skills, World Models: Can AI Simulate Reality?, and Tests Are How You Talk to the AI.