The bug your test cannot see

A function can return exactly the right answer and still corrupt the data you handed it. No return-value assertion can catch that, because the damage is not in the return value. Here is a four-line JavaScript function that does it, and what it costs three weeks later.

Here is a function. It has one job: give me the three highest scores.

function topThree(scores) {
  scores.sort((a, b) => b - a)
  return scores.slice(0, 3)
}

It works. topThree([72, 98, 64, 91, 87, 55]) returns [98, 91, 87], which is correct. It throws nothing. It has a test, and the test is green.

Now watch what happens when real code calls it.

const names  = ["Ada", "Bo", "Cy", "Di", "Eve", "Fay"]
const scores = [  72,   98,   64,   91,   87,    55  ]

const pair  = (n, i) => n + " " + scores[i]
const board = () => names.map(pair).join("   ")

console.log("before:   ", board())
const top = topThree(scores)
console.log("after:    ", board())

before:    Ada 72   Bo 98   Cy 64   Di 91   Eve 87   Fay 55
after:     Ada 98   Bo 91   Cy 87   Di 72   Eve 64   Fay 55

Ada scored 72. Ada now has 98. Every name in that list is sitting next to somebody else’s score, and the function that did it also returned exactly the right answer.

Why can’t a test catch this bug?

A test of the return value cannot observe a side effect, because the side effect is not in the return value. Array.prototype.sort sorts in place: it rearranges the array you handed it and returns that same array. A test that checks only what came back will pass while the caller’s data is quietly reordered.

The obvious reaction is that the test was lazy. Write a better one.

That reaction is wrong in a way I think is worth analyzing, because it is the difference between “we got sloppy” and “this class of bug needs a different kind of attention.”

So the topThree function shown above has two effects: the value it returns, and the change it makes to its argument. The original test only ever looked at the first one.

A test of the return value cannot observe a side effect. Not because it was written badly, but because a return value is not where the side effect resides.

You could write an assertion that catches it. expect(scores).toEqual([72, 98, 64, 91, 87, 55]) after the call would go red immediately. But you only write that assertion if you already suspected the function touched its argument, and if you already suspected that, you would have fixed the function instead.

That is what makes this expensive. A bug that throws tells you where it is. A bug that fails a test tells you where it is. This one returns the right answer, passes review, and shows up three weeks later as a support ticket about the wrong name on a leaderboard.

The near miss that catches experienced people

The fix is to sort a copy: [...scores].sort(). The near miss is [...scores.sort()], the same characters in a different order. With the spread on the outside, sort runs first, on the caller’s array, so the copy is taken after the damage is done. Both versions return the right answer.

Here is the fix in full:

return [...scores].sort((a, b) => b - a).slice(0, 3)

And here is the version I find genuinely interesting, because it looks like the fix and is not:

return [...scores.sort((a, b) => b - a)].slice(0, 3)

I have seen people who have been writing JS for a decade write that one, and its not a knowledge gap. It is that the two versions look identical at a glance and only one of them is about ordering of operations.

Why does lodash orderBy exist?

_.orderBy(scores, [], ['desc']) sorts a copy and hands that back. The array you passed in is never touched, and that guarantee is most of what a utility library is selling you when the language already has sort.

The naming does not help anyone. Nothing in these method names tells you which group they belong to, so it comes down to memory, and memory is exactly what fails at the worst times.

Method	Mutates your array	Returns
sort	Yes	the same array
reverse	Yes	the same array
splice	Yes	the removed items
push	Yes	the new length
pop	Yes	the removed item
map	No	a new array
filter	No	a new array
slice	No	a new array
concat	No	a new array
toSorted	No	a new array (ES2023)

Why I turned this into the demo on my landing page

I launched CodeTrain three weeks ago. It is an AI tutor with one rule: it never writes your code. You type every line, it plans the steps, runs what you wrote and grades it.

That rule is the whole product, and I wrote about why a tutor that refuses to write your code is harder to build than one that helps. The launch post has the rest of the background.

The thing I got wrong was assuming that if people heard about it, the ones who wanted it would sign up. So I did the whole distribution playbook. What I actually had was a funnel where people arrived, hit an email field, and had no way to find out whether the thing was any good before handing it over.

So the lesson above is now the demo, and it runs on the landing page with no account and no install.

It is a real lesson, not a video of one. Your code executes in a real Web Worker in your own tab. The checks are real assertions against what your function actually returned and what it did to its argument, including one case you cannot see, because a fix that only works on the numbers in front of you is not a fix. Break the slice and the check goes red. Delete the function and you get a real missing-entry error.

The one thing that is not live is the tutor’s replies. Those are written in advance, the page says so before you start, and every reply is labeled as such. A preview that implies a live model when there is not one is a lie about the product, and I would rather lose the visitor than start there.

The bit I nearly shipped backwards

The first version of the demo said the check did not catch the bug. I meant that as a statement about tests in general. On my own product page it reads as CodeTrain writing a check that misses bugs, which is the opposite of what the product is. The fix was to attribute the check correctly.

I did not see it until the fourth or fifth read.

The check in step one belongs to the codebase in the example, not to CodeTrain, and it is now labelled that way on screen for clarity. What CodeTrain checks is step two, and there are three assertions there including the one about the argument. The original test asked one question. The tutor asks three, and one of them is the question that matters.

Try it

The lesson is at codetrain.ai/#try. Two steps, about a minute, no account.

Two steps is a real lesson at the short end, incidentally. The tutor builds between two and six depending on how big the topic is. The difference in the real thing is that it reads whatever you actually wrote, so the questions come from your code instead of from a list I authored ahead of time. Ten lessons a month on the free tier, no card, and you can point it at any public repository and get a lesson built from the code that is really in it.

— Ethan L.