How I built a tutor that refuses to write your code

Making an AI tutor refuse to write your code is not a prompt instruction. It takes a grading contract that returns a verdict instead of prose, two model calls that must never be merged, and two execution paths so a free tier can exist without a card or a queue. Here is the whole build.

Two weeks ago I launched CodeTrain, an AI tutor with one rule: it never writes your code. You type every line, it plans the steps, runs what you wrote, and grades it. The launch post covers what it is and who it is for.

People probably assume the rule is a prompt. Something like “do not write code for the user,” pasted at the top of a system message, and done. That was the first version. It lasted about an hour.

What follows is what the rule actually cost to build: a grading contract instead of a chat reply, two separate model calls that must never be merged, two execution paths so the free tier costs almost nothing to run, and a short list of things that broke in front of real users.

A refusal is not prompted, its contracted

A model told to teach rather than solve will agree, and then help anyway the moment a learner struggles. The fix is to stop asking it for prose and require a decision instead. Every grading turn returns JSON with a fixed shape, a verdict of advance, retry or comment, and a boolean beside every criterion it checked.

Here is the failure mode nobody warns you about. Ask a model to teach rather than solve, and it agrees enthusiastically. Then the learner gets stuck, and the model helps. It writes “you could try something like this” and drops in four lines. Technically it did not actually solve the exercise. Practically the lesson is over.

The fix to this was to stop asking for prose and start requiring a decision. Every grading turn returns JSON with a fixed shape, or it is treated as a failure:

{
  "verdict": "advance" | "retry" | "comment",
  "feedback_md": "<concise Socratic markdown>",
  "checks": [{"label": "prints exactly Hello", "pass": true}],
  "learned": ["<short concept the learner demonstrated>"]
}

That single change did more for the product than any amount of instruction tuning in the prompt. A model writing prose has infinite room to be helpful in the wrong direction. A model that has to pick advance or retry, and list the criteria it checked with a boolean next to each, has to commit to a judgment about the learner’s code.

The checks array is the part learners actually respond to. Each item is one concrete criterion with pass or fail, and the prompt requires failing criteria to be included rather than quietly dropped. You see exactly which two of five things your code did, which is a very different experience from a paragraph explaining that you are on the right track. The demo lesson on the landing page is the clearest example I have of why that matters: three assertions, one of them about a side effect no return value can reveal.

Keep the two calls apart

Lesson authoring and grading are separate calls, with separate prompts, and merging them is the single worst thing you can do to a tutor like this. One call is cheaper and saves a round trip. It also lets the tutor write the answer into the next question, because the call composing that question has just watched the learner fail.

I know because I tried it. One call, full context, plan the next step and grade the last one at the same time. It saves a round trip and it is meaningfully cheaper. It also produces a tutor that could write the answer into the question. When the same call that just saw a struggling learner also composes the next step, the step it writes gets suspiciously specific. “Now add the except KeyError branch that returns an empty list” is just the solution with a task label on it.

Separated, the authoring call never sees the failure and cannot overfit to it. That constraint is now enforced in the codebase itself: control-plane bookkeeping stays in the router, the tutor engine stays synchronised with the agent’s copy of the prompt, and a captured-prompt test fails the build if the two drift apart.

Why does most of the work leave our infrastructure?

A tutor that grades code has to run code, and putting a container behind every Run button is why a lot of similar products have no free tier, or one with a queue in front of it. Python and JavaScript run in the learner’s own browser instead. The languages that genuinely need a real filesystem go to a capped server sandbox.

Python and JavaScript run entirely in the learner’s browser. Python goes through Pyodide in a dedicated worker, JavaScript runs in its own worker, and both stay pooled so the second Run does not pay startup again. A prewarm kicks off when the lesson opens, which usually finishes before anyone types their first line. Runs are capped at 20 seconds, which is generous for a teaching step and short enough that a runaway loop does not hang the tab.

On the free tier, the Python and JavaScript a learner writes never reaches a server, which is the only reason the tier can exist without a card and without a queue.

— free tier economics

That holds for the two languages most lessons use. bash, ruby, perl and php genuinely need a real interpreter and a filesystem, so those go to an isolated server sandbox with per-minute and per-day caps, on free accounts too. The browser runtimes cover the common case; the sandbox covers what a browser cannot honestly fake.

Keeping the common case off our machines is the same instinct behind why offline is the default everywhere else I build. Work that does not need to leave your machine should not leave it.

The measured cost tracks either way. Across a recent two-week window the managed model spend for every free-tier session on the platform came to about 25 cents total, and compute for running Python and JavaScript was zero, because it happened on the learner’s machine in a tab they already had open.

That sandbox is Alpine with BusyBox, a real shell and real files with no network, and it produced its own class of bug. The grader would demand a GNU coreutils flag that BusyBox does not implement, then fail a learner whose solution was correct. The prompt now names that constraint directly and instructs the grader to accept any working BusyBox-compatible approach.

Dockerfile lessons get static linting instead of a real build. Letting anonymous users build containers on my infrastructure is a speedrun to an incident writeup, and the lesson value was in the file, not the image.

Things that broke in front of real users

Four failures reached real learners: the model emitted JSON that would not parse, older output used a shape the UI could not render, the tutor asked permission to continue instead of advancing, and browser Python hit runtime limits the tutor itself had authored into the step. Each one is below, with the fix.

If you are going to skip ahead to anything here, this is the interesting stuff.

The model emitted JSON that was not JSON. Specifically, feedback containing a stray backslash, usually from a regex or a Windows path in the learner’s code, which invalidated an otherwise perfect verdict. The parser now repairs unescaped backslashes before parsing, and falls back to a retry carrying the raw text if it still cannot be read. A tutor that hard-fails because a learner typed \d is not a tutor that can function well.

Older model output used plain strings for checks instead of objects with a label and a boolean. Rather than break those, the coercion layer treats a bare string as a passing check. It is not elegant. It is the difference between rendering something useful and rendering an error.

The tutor asked permission to continue. Early versions ended a correct submission with “ready to move on?” Every single time, learners answered the question instead of coding, and the lesson turned into a conversation. Advancing is now the confirmation, stated as a hard rule in the prompt: when the submission is correct, advance, do not ask.

Browser Python is not Python. No subprocess, no real files, no network. A learner following an authored step involving subprocess was hitting a wall the tutor had built for them. The grader is now told to never fault the learner for a runtime limit, because that step should not have been written in the first place, and to guide toward a runnable approach or simply advance. I will be improving this behaviour into the future as well.

What happens to the code you write in repo mode?

The lesson holds every step and proposes one patch at the end, containing only the code you typed, against a branch you choose. Abandon the lesson and nothing touches your repository. The first design committed each step as it passed, which felt responsive and was wrong.

The free tier teaches on examples and any public repository you point it at. Paid repo mode runs against a real checkout on your machine, which raises the question I got wrong at first.

The first design applied each step as its own commit. It felt responsive and it was a mistake. Nobody wants seven commits titled “step 3” in their history, and a half-finished lesson left the working tree in a state the learner had to clean up.

Now the lesson holds everything and proposes one patch at the end, containing only the code you typed, against a branch you choose. You review it like any other diff. If you abandon the lesson, nothing touches your repository at all.

What I’m tuning next

The tutor is at its best on a focused concept in code it can read directly. Lesson scope is the dial still getting tuned in: a tight lesson on one module lands harder than one that ranges across a whole framework, so more of the authoring work is going into keeping lessons narrow by default.

Browser Python runs on Pyodide, whose package set is a subset of PyPI. The prompt-level guardrails already steer authored steps toward what the runtime can actually import, and widening that coverage, including routing more lesson types to the server sandbox where it makes sense, is the next piece of that work.

The grading loop is a model making a judgment, so the design choice was to make the judgment inspectable. Every criterion is listed with a pass or fail rather than one overall score, which means you can always see the basis for a verdict instead of blind faith in the model itself. It is the same problem a code review you cannot reproduce has, and the same answer: show the criteria, not just the conclusion. That visibility is also our own best feedback channel, since every reported step comes back with the checks attached, which is how the tuning above gets prioritised.

Try to break it

The free tier runs in your browser, ten lessons a month, no card and no trial clock: codetrain.ai. Point it at a public repository you did not write and ask for a lesson on the part you understand least. That is the fastest honest test of everything above.

If you talk it into writing your code outright, send me the screenshot. I will fix the loop, and probably frame the screenshot!

— Ethan L.