Techniques for Exhaustively Testing a Rails App

I subscribe to dozens of tech blogs (including but not limited to Ruby and Rails), and although I’ve seen quite a lot of commentary about TDD and BDD, I’m not sold on either of them yet. TDD is interesting, but BDD seems like a waste of time. But I am completely sold on automated testing in general. I write lots and lots of tests and make sure using rcov that my tests cover all of the code.

Getting 100% coverage isn’t easy. In general it means you definitely can’t just write a single test case per method and declare victory when it passes. When the number of possible combinations of inputs (method arguments and/or mock objects) and expected outputs (return values, exceptions, and side effects) becomes large, then the potential for copy-and-paste errors in your test code becomes large, and legibility becomes an issue. This is the point at which I find the recent fascination with writing tests in a near-natural-language DSL to be a distraction. It’s orthogonal to the problem I’m dealing with, which is how to comprehensively test the code. In other words, the problem is not making a small number of tests more readable, but concisely expressing a large number of input/output combinations and a large number of different tests, and making it all readable.

I assume that there are others working on this problem too, so I’ll describe some of the things I’ve come up with, and hopefully if you have any good ideas you’ll post them in the comments section.

Test Data Iterators

If you express the set of inputs and outputs as a data structure, such as a Hash of input => expected ouput pairs, or an Array of [input, expected output] Arrays, you can easily do test_data.each do |input,output| ... end and drive tests that way.

(I considered pulling the data itself out of the test code and loading it from a YAML or CSV file or even including a data-only Ruby script, but that seemed like pointless added complexity. If I were working with a professional tester who didn’t want to touch Ruby code, I might change that to make it easier to dump from an Excel spreadsheet into a text file that the tests would use.)

One problem with this approach is that you may have a test_frobnicate method that invokes frobnicate dozens of times. This means that this one test can take a while to run if the tested method takes any amount of time, which is problematic if you just want to re-test that one input/output combo until it works. Your edit/test/debug loop should be ultra fast, and re-testing successful pairs gets in the way of testing the pair you’re not sure about.

More importantly, trying all those combinations in one test method means that if an assertion fails, the failure masks the outcome of the remaining combinations. If there are 25 combinations and combination #2 fails, then a Test::Unit::AssertionFailedError is thrown, and the other 23 of them go unchecked. That’s not ideal, since it can be useful in debugging if you can see that 24/25 failed vs. 1/25 failed.

To solve this, I changed the code so that it iterates over the test data inside the class definition, generating a new test method for each input/output pair:

That gives you a bunch of test methods that each run the test once with different input and expected output values. Autotest integrates well with this technique, since it will not re-run the test methods which previously passed until after you have fixed all of the ones which failed.

Test Setup Blocks

I also use a with_x_y_z do...end idiom for mock setup when possible. So a test might contain:

Since this idiom uses blocks, you can nest them to create complicated setup scenarios, pass arguments to the with_blah method to control the behavior of the mock objects, and so on. (You could do this with regular set_up_blah methods but I think this is easier to read.)

Calculated Outputs

Despite their shortcomings, I still use Fixtures (with the PreloadFixtures plugin) and I write unit tests for my models. That means that I can then assume that the models’ behavior is correct in other tests (provided that those unit tests pass). So I try not to put any data that’s already expressed in fixture files into my test method inputs or expected outputs. Instead, I ask for the fixture by its label (widgets(:doodad)) and then use its properties as needed in assertions within the current test. This reduces the size of the input/output data, which makes it easier to read and reduces the amount of effort required to add more combinations later.

Testing Views, Model Associations, and Model Validation

As Bruce Eckel says, If it’s not tested, it’s broken. Or as Peter Drucker said, “What gets measured gets managed.”

I test all of these. I’ve read the arguments of those who say that none of these are necessary, and I disagree. The arugment that testing associations and validations is tantamount to testing Rails itself is just incorrect. Rails’ own unit tests cannot possibly test the validations and associations expressed in every Rails application’s model classes, so Rails application developers still have to do this.

Associations

I can’t tell you how many times I’ve gotten has_one and belongs_to backwards and only found out when ActiveRecord told me that the column didn’t exist, because I told it to look at the table on the wrong side of the association. I feel stupid when I see that I made that mistake, but writing that test takes seconds and once I see the failure it’s easy to fix. Better to catch it and fix it now rather than have a user find it, right?

Validations

The same goes for model validations. Yes, I know that validates_format_of has been tested, but my regexes for username format and email addresses need testing too.

Views

Views should be tested only if you care what your users see.

Somewhere there is probably an organization with no users, where inputs matter but outputs can be anything. Maybe the goal of the project is to burn cash rather than to ship something good. They don’t need to test their views.

I agree that views should be dumbed down as much as possible using helpers, but that’s not enough to assure that they work. If you’re not testing views with automated tests, then how do you test them? With a browser? Every time you change anything, you’re going to exhaustively check every combination of inputs and outputs to make sure that all the dynamic goodies are in the right place? I doubt it. Obviously you do have to check views in a browser to make sure that purely visual aspects are correct, but there are things you can check automatically. You can do quite a lot with assert_select, and for client-side tests, Selenium can do pretty much anything. (It’s probably not worth the effort to have Selenium use the DOM to see that the browser applied CSS in exactly the way that you wanted it to, but if you wanted to, you could.)

I don’t know how to do something similar to coverage testing for views, but a good start is to just check for the effects of everything that the controller stored in instance variables. Maybe those should be displayed as-is in the resulting document, or maybe they should be truncated, or processed by a helper into a different representation.

In any case, you don’t have to test with the same set of inputs as the functional test for the controller; just do white box testing for the minimal set of combinations that should exercise all of the permutations of the view code (which should be a very small number). Testing the helpers separately can cut down on the number of permutations of inputs and outputs also. So if you have a helper that creates a navigation breadcrumb visual element from an array of strings, you could just test that in isolation with 0, 1, 2, and 3 elements, and then check for one version of breadcrumbs when looking for it in a view.

I’m particularly inspired by the claim that Sebastian Delmont of StreetEasy made in the May 31, 2006 episode of the Ruby on Rails Podcast about having no human approval step between a successful test suite run and deployment to production. That’s pretty darn bold, but if you stop and think about it, it is feasible with today’s tools to automate any “white glove test” that a real person would do before approving the release for deployment to production. I’m not that brave yet, but I like the idea of a condition that checks if the changed code has anything to do with a view (ERB, JavaScript, CSS, etc.), and if not, would just rubberstamp it and deploy it. Clearly a change that could require a manual test in multiple browsers wouldn’t be a wise thing to autodeploy, but something like fixing a bug deep in the code shouldn’t require a human to approve it, especially if the autodeploy condition depends on that chunk of code having 100% test coverage following the bug fix.

Your Tips?

I’m always looking for ways to test code with less effort on my part. Please let me know if you have ideas or suggestions.