Specs and Tests Are Two Different Things

I landed my first assignment in 2010, and I was proud of it. The recruiter told me I was the first Ruby developer he had ever placed. The company wanted to stabilize its code with unit tests, and my job was to help by writing new ones.

It did not take long to understand what I was there for: to check, every time the code changed, what it really did. Not what it was meant to do, not what its author remembered of it. What it did.

That is where test frameworks caught my interest. Nobody can prove that a program works. What a framework offers instead is a way to describe an expected behaviour, scenario by scenario, and to show that the code behaves that way in each of them. When the scenarios cover every kind of case there is, the demonstration gets as close to a proof as we will ever get.

So all the confidence comes to rest on the test suite, on one condition: that the primitive the suite stands on is itself perfectly sound. In practice, that condition is never examined. The framework is the ground everything else is measured against, and the ground is not something one questions. It takes on a sacred air.

Yet a test framework is a program like any other. It can be large, and complex, sometimes more complex than the code it is asked to test. It has bugs, as every program has, and a bug in the wrong place does not fail loudly: it cascades into a false positive, and the suite goes green over code that is wrong. The reliability of that primitive is not a detail. It is the whole point.

The framework in my story was RSpec. Curiosity sent me into its source code, to demystify how it worked and how its domain-specific language was put together. I ran into a wall of complexity: metaprogramming on every floor, monkey patching, mutation all over the place. I came out knowing less about what my tests were resting on than when I went in.

A few years later, playing with Ruby's API, I started hacking, designing and implementing a test framework of my own. I called it Fix, and I built it the empirical way, by trial and error, showing it along the road at Ruby Kansai (slides) and at ParisRb (slides). The idea I wanted to put across was that a test framework should help keep two things apart:

1. specifications, which must be agnostic of any implementation of the code to be tested;
2. a function to test outside code, by confronting it with those specifications.

Put more simply: specs and tests are two different things, and the framework should not confuse them. A specification says what a subject must, should or may do, and nothing about how the subject is made. A test takes a subject and holds it against the specification. The first is a document; the second is a function. Keep them apart, and the primitive beneath the suite gets small enough to read in an afternoon, which is the only kind of primitive I am willing to trust.

That work kept me going for longer than I would have guessed. This week I published Fix 1.0, the first version I consider finished. In its wake, I updated my RSpec clone to version 2: the words of RSpec, running on the primitives of Fix.

To every Ruby developer out there: happy hacking.