måndag, augusti 06, 2007

JRuby now JIT-compiles assertions

One of the major problems when running automated testing with JRuby is that all the standard Test::Unit assertions would never be JIT compiled, meaning that they would be quite slow. Actually, assertions seems to be very slow when running interpreted in JRuby. I have a small test case, courtesy of Michael Schubert:
require 'test/unit'
require 'benchmark'

class A < Test::Unit::TestCase
[10_000, 100_000].each do |n|
define_method "test_#{n}" do
puts "test_#{n}"
5.times do
puts Benchmark.measure{n.times{assert_equal true,true}}
end
end
end
end
This code will show quite nicely how large the overhead of asserts are, by using assert_equal. Now, the numbers for MRI for this benchmark looks like this:
Loaded suite test_assert
Started
test_10000
0.150000 0.000000 0.150000 ( 0.155817)
0.150000 0.000000 0.150000 ( 0.158376)
0.160000 0.000000 0.160000 ( 0.155575)
0.150000 0.000000 0.150000 ( 0.154380)
0.160000 0.000000 0.160000 ( 0.157737)
.test_100000
1.520000 0.010000 1.530000 ( 1.539325)
1.530000 0.000000 1.530000 ( 1.543889)
1.520000 0.010000 1.530000 ( 1.540376)
1.530000 0.000000 1.530000 ( 1.543742)
1.530000 0.010000 1.540000 ( 1.558292)
.
Finished in 8.509493 seconds.

2 tests, 550000 assertions, 0 failures, 0 errors
And for JRuby without compilation:
Loaded suite test_assert
Started
test_10000
1.408000 0.000000 1.408000 ( 1.408000)
0.582000 0.000000 0.582000 ( 0.582000)
0.425000 0.000000 0.425000 ( 0.426000)
0.419000 0.000000 0.419000 ( 0.418000)
0.466000 0.000000 0.466000 ( 0.467000)
.test_100000
4.189000 0.000000 4.189000 ( 4.190000)
4.196000 0.000000 4.196000 ( 4.196000)
4.139000 0.000000 4.139000 ( 4.139000)
4.165000 0.000000 4.165000 ( 4.165000)
4.162000 0.000000 4.162000 ( 4.162000)
.
Finished in 24.181 seconds.

2 tests, 550000 assertions, 0 failures, 0 errors
It's quite obvious that something is very wrong. We're about 2.5-3 times slower.

Now, the way the JRuby compiler works, we build it piece by piece and the JIT will try to compile a method that's used enough. If there is any node that can't be compiled it will fail and fall back on interpretation. In the case of assertions, all Test::Unit assertions use a small helper method called _wrap_assertion that looks like this:
def _wrap_assertion
@_assertion_wrapped ||= false
unless (@_assertion_wrapped)
@_assertion_wrapped = true
begin
add_assertion
return yield
ensure
@_assertion_wrapped = false
end
else
return yield
end
end
When I started out on this quest, there were two things in this method that doesn't compile. The first is the ||= construct, which I mentioned in an earlier blog post. The problem with it is that it requires that we can compile DefinedNode too, and that one is large. The second problem node is Ensure. After lots of work, I've finally managed to implement most of these safely, and falling back on interpretation when it's not safe. Without further ado, the numbers after compilation with these features added:
Loaded suite test_assert
Started
test_10000
0.996000 0.000000 0.996000 ( 1.013000)
0.415000 0.000000 0.415000 ( 0.415000)
0.110000 0.000000 0.110000 ( 0.110000)
0.099000 0.000000 0.099000 ( 0.100000)
0.109000 0.000000 0.109000 ( 0.109000)
.test_100000
1.012000 0.000000 1.012000 ( 1.012000)
1.008000 0.000000 1.008000 ( 1.000000)
1.017000 0.000000 1.017000 ( 1.017000)
1.039000 0.000000 1.039000 ( 1.039000)
1.024000 0.000000 1.024000 ( 1.024000)
.
Finished in 6.966 seconds.
So we're looking at over 4 times improvement in speed, and about 33% percent faster than MRI. Try your test cases; hopefully it will show up.

Inga kommentarer: