onsdag, december 19, 2007

Your Ruby tests are memory leaks

The title says it all. The only reason you haven't noticed, is that you probably aren't working on a large enough application, or have enough tests. But the fact is, Test::Unit leaks memory. Of course, that's to be expected if it is going to be able to report results. But that leak should be more or less linear.

That is not the case. So. To make this concrete, lets take a look at a test that exhibits the problem:
class LargeTest < Test::Unit::TestCase
def setup
@large1 = ["foobar"] * 1000
@large2 = ["fruxy"] * 1000
end

1000_000.times do |n|
define_method :"test_abc#{n}" do
assert true
end
end
end
This is obviously fabricated. The important details are these: the setup method will create two semi large objects and assign them to instance variables. This is a common pattern in many test suites - you want to have the same objects created, so you assign them to instance variables. In most cases there is some tear down associated, but I rarely see teardown that includes assigning nil to the instance variables. Now, this will run one million tests, with one million setup calls. Not only that - the way Test::Unit works, it will actually create one million LargeTest instances. Each of those instances will have those two instance variables defined. Now, if you take a look at your test suites, you probably have less than one million tests all over. You also probably don't have that large objects all over the place. But remember, it's the object graph that counts. If you have a small object that refers to something else, the whole referral chain will be stopped from garbage collection.

... Or God forbid - if you have a closure somewhere inside of that stuff. Closures are a good way to leak lots of memory, if they aren't collected. The way the structures work, they refer to many things all over the place. Leaking closures will kill your application.

What's the solution? Well, the good one would be for test unit to change it's implementation of TestCase.run to remove all instance variables after teardown. Lacking that, something like this will do it:
class Test::Unit::TestCase
NEEDED_INSTANCE_VARIABLES = %w(@loaded_fixtures @_assertion_wrapped @fixture_cache @test_passed @method_name)

def teardown_instance_variables
teardown_real
instance_variables.each do |name|
unless NEEDED_INSTANCE_VARIABLES.include?(name)
instance_variable_set name, nil
end
end
end

def teardown_real; end
alias teardown teardown_instance_variables

def self.method_added(name)
if name == :teardown && !@__inside
alias_method :teardown_real, :teardown
@__inside = true
alias_method :teardown, :teardown_instance_variables
@__inside = false
end
end
end
This code will make sure that all instance variables except for those that Test::Unit needs will be removed at teardown time. That means the instances will still be there, but no memory will be leaked for the things you're using. Much better, but at the end of the day, I feel that the approach Test::Unit uses is dangerous. At some point, this probably needs to be fixed for real.

7 kommentarer:

djberg96 sa...

That's why I always assign my instance variables to nil in the teardown method. :)

But, yeah, if we could automate it, that would be good.

Christoph Sturm sa...

Oh god, test:unit has memory leaks. I hope you dont use your tests in a production enviroment :)

Nick Sieger sa...

Another reason to use RSpec?

Phillip Calçado sa...

@christoph

Join a 40-minutes-build-time project and then you'll be happy to get rid of leaks and waste in general ;)

Aslak Hellesøy sa...

Nick/Ola, do we actually handle this better in RSpec than in Test::Unit?

Nick Sieger sa...

@Aslak: It appears to be so. Maybe you can explain why?

Tammo Freese sa...

I think this is no memory leak in the sense of the word, but Test::Unit simply keeps around references to objects that it does not need anymore. After running a test, the test instance could immediately be discarded, since the result is stored in the TestResult.

The fix should better be placed where the instances are kept around: in TestSuite. With the following fix, 100_000 tests pass in roughly a minute on my machine (using Ruby 1.8.6):

class Test::Unit::TestSuite
def run(result, &progress_block)
yield(STARTED, name)
@tests.reverse!
while test = @tests.pop
test.run(result, &progress_block)
end
yield(FINISHED, name)
end
end