RCOV C0 line coverage more generous than EMMA’s C1 line coverage

Coverage tests in Ruby (with rcov) are less strict than in Java (with EMMA), so watch out – 100% coverage is easy to attain but not as meaningful.

For Java code coverage, I like EMMA. Clover looks really nice and all, but seriously, $250 a seat? Yeesh. Talk about poor product pricing – I will never buy that product, nor seriously consider buying it, because of the outrageous price. I mean, a single license of IntelliJ is $1 cheaper. That’s like charging $300 for a better Ant. C’mon.

Anyway, EMMA is very conservative about coverage estimates. Consider this single line of Java code:

When you run it, the assignment will execute, as will the comparison, but the block containing throw will never be reached. So, the line is not fully covered. EMMA looks at bytecodes and maps those back to line numbers, and will mark this line as partially covered.

It’s maddening sometimes, but it’s correct; it’s your problem to figure out how to force all those darned IOExceptions that you know can’t ever happen, or to give up and let some apocalyptic error handling code not be covered. I used to shoot for 85-90% coverage, and just live with some uncovered wacky code when I couldn’t find a reasonable way to trigger hideous errors.

(There’s a little voice in my head that says, “Come on, you slacker, you should make it possible to dynamically break the database configuration and remount the root filesystem as read-only during the test suite so that the error handlers can all be 100% covered!” But that little voice never ships anything, he just sits around and writes more and more paranoid code — no Ariane 5 disaster on my watch! — and gets asymptotically closer to 100% coverage.)

Okay, now the Ruby version:

Same story, prettier code (cuz it’s Ruby; duh, of course it’s prettier): assignment gets executed, condition is executed, no exception is raised. Rcov says it’s 100% covered.

The difference is C0 coverage (Rcov) vs. C1 coverage (EMMA). EMMA only counts a line as 100% covered if all of the bytecodes compiled from it were executed. Rcov counts a line as 100% covered if execution visited that line at all.

(EMMA uses “basic blocks” instead of bytecodes, so in obscure failure cases the reported line coverage figure is even lower, but that doesn’t affect the conditions for what 100% line coverage requires.)

I’m not sure if it’s feasible to get C1 coverage on the regular Ruby VM, but it would be nice to have. In the meantime, just be aware that 100% C0 coverage is easy to attain with rcov, but doesn’t mean you’re done testing everything.

Of course, 100% C1 coverage doesn’t mean you’re done testing everything either, but it’s closer to meaning that than 100% C0 coverage is.

In fact, it’s good to over-test code that’s already covered once. On any given project in the real world, there is code that will succeed with some values and fail with others. (print x/y, for example.) Just testing with one set of values isn’t enough. I like to create an array of inputs and expected outputs and use Array.each to iterate over them and pass them into a block that calls the code under test and then compares the actual outputs to the expected outputs with assert_equal. It means that I’m following DRY (rather than cutting and pasting several lines of test code and then editing a teeny bit of each pasted chunk… eww) but still testing many different ways.

But, even with a C0 coverage tool, at least you know what hasn’t been touched at all, and you can use your big smart programmer brain to think of a few ways to torture that code. Just remember that you probably should be aiming for an unmeasurable way-over-100% imaginary coverage target, beating the heck out of your fanciest code with many, many different inputs and then moving on to the next chunk of uncovered code only when the last chunk is thoroughly proven to not suck.

Good testing!