Your Ruby tests are memory leaks

The title says it all. The only reason you haven't noticed, is that you probably aren't working on a large enough application, or have enough tests. But the fact is, Test::Unit leaks memory. Of course, that's to be expected if it is going to be able to report results. But that leak should be more or less linear.

That is not the case. So. To make this concrete, lets take a look at a test that exhibits the problem:

class LargeTest < Test::Unit::TestCase
def setup
  @large1 = ["foobar"] * 1000
  @large2 = ["fruxy"] * 1000
end

1000_000.times do |n|
  define_method :"test_abc#{n}" do
    assert true
  end
end
end

This is obviously fabricated. The important details are these: the setup method will create two semi large objects and assign them to instance variables. This is a common pattern in many test suites - you want to have the same objects created, so you assign them to instance variables. In most cases there is some tear down associated, but I rarely see teardown that includes assigning nil to the instance variables. Now, this will run one million tests, with one million setup calls. Not only that - the way Test::Unit works, it will actually create one million LargeTest instances. Each of those instances will have those two instance variables defined. Now, if you take a look at your test suites, you probably have less than one million tests all over. You also probably don't have that large objects all over the place. But remember, it's the object graph that counts. If you have a small object that refers to something else, the whole referral chain will be stopped from garbage collection.

... Or God forbid - if you have a closure somewhere inside of that stuff. Closures are a good way to leak lots of memory, if they aren't collected. The way the structures work, they refer to many things all over the place. Leaking closures will kill your application.

What's the solution? Well, the good one would be for test unit to change it's implementation of TestCase.run to remove all instance variables after teardown. Lacking that, something like this will do it:

class Test::Unit::TestCase
 NEEDED_INSTANCE_VARIABLES = %w(@loaded_fixtures @_assertion_wrapped @fixture_cache @test_passed @method_name)

 def teardown_instance_variables
   teardown_real
   instance_variables.each do |name|
     unless NEEDED_INSTANCE_VARIABLES.include?(name)
       instance_variable_set name, nil
     end
   end
 end

 def teardown_real; end
 alias teardown teardown_instance_variables

 def self.method_added(name)
   if name == :teardown && !@__inside
     alias_method :teardown_real, :teardown
     @__inside = true
     alias_method :teardown, :teardown_instance_variables
     @__inside = false
   end
 end
end

This code will make sure that all instance variables except for those that Test::Unit needs will be removed at teardown time. That means the instances will still be there, but no memory will be leaked for the things you're using. Much better, but at the end of the day, I feel that the approach Test::Unit uses is dangerous. At some point, this probably needs to be fixed for real.