JRuby now JIT-compiles assertions

One of the major problems when running automated testing with JRuby is that all the standard Test::Unit assertions would never be JIT compiled, meaning that they would be quite slow. Actually, assertions seems to be very slow when running interpreted in JRuby. I have a small test case, courtesy of Michael Schubert:

require 'test/unit'
require 'benchmark'

class A < Test::Unit::TestCase
[10_000, 100_000].each do |n|
 define_method "test_#{n}" do
   puts "test_#{n}"
   5.times do
     puts Benchmark.measure{n.times{assert_equal true,true}}
   end
 end
end
end

This code will show quite nicely how large the overhead of asserts are, by using assert_equal. Now, the numbers for MRI for this benchmark looks like this:

Loaded suite test_assert
Started
test_10000
0.150000   0.000000   0.150000 (  0.155817)
0.150000   0.000000   0.150000 (  0.158376)
0.160000   0.000000   0.160000 (  0.155575)
0.150000   0.000000   0.150000 (  0.154380)
0.160000   0.000000   0.160000 (  0.157737)
.test_100000
1.520000   0.010000   1.530000 (  1.539325)
1.530000   0.000000   1.530000 (  1.543889)
1.520000   0.010000   1.530000 (  1.540376)
1.530000   0.000000   1.530000 (  1.543742)
1.530000   0.010000   1.540000 (  1.558292)
.
Finished in 8.509493 seconds.

2 tests, 550000 assertions, 0 failures, 0 errors

And for JRuby without compilation:

Loaded suite test_assert
Started
test_10000
1.408000   0.000000   1.408000 (  1.408000)
0.582000   0.000000   0.582000 (  0.582000)
0.425000   0.000000   0.425000 (  0.426000)
0.419000   0.000000   0.419000 (  0.418000)
0.466000   0.000000   0.466000 (  0.467000)
.test_100000
4.189000   0.000000   4.189000 (  4.190000)
4.196000   0.000000   4.196000 (  4.196000)
4.139000   0.000000   4.139000 (  4.139000)
4.165000   0.000000   4.165000 (  4.165000)
4.162000   0.000000   4.162000 (  4.162000)
.
Finished in 24.181 seconds.

2 tests, 550000 assertions, 0 failures, 0 errors

It's quite obvious that something is very wrong. We're about 2.5-3 times slower.

Now, the way the JRuby compiler works, we build it piece by piece and the JIT will try to compile a method that's used enough. If there is any node that can't be compiled it will fail and fall back on interpretation. In the case of assertions, all Test::Unit assertions use a small helper method called _wrap_assertion that looks like this:

def _wrap_assertion
 @_assertion_wrapped ||= false
 unless (@_assertion_wrapped)
   @_assertion_wrapped = true
   begin
     add_assertion
     return yield
   ensure
     @_assertion_wrapped = false
   end
 else
   return yield
 end
end

When I started out on this quest, there were two things in this method that doesn't compile. The first is the ||= construct, which I mentioned in an earlier blog post. The problem with it is that it requires that we can compile DefinedNode too, and that one is large. The second problem node is Ensure. After lots of work, I've finally managed to implement most of these safely, and falling back on interpretation when it's not safe. Without further ado, the numbers after compilation with these features added:

Loaded suite test_assert
Started
test_10000
  0.996000   0.000000   0.996000 (  1.013000)
  0.415000   0.000000   0.415000 (  0.415000)
  0.110000   0.000000   0.110000 (  0.110000)
  0.099000   0.000000   0.099000 (  0.100000)
  0.109000   0.000000   0.109000 (  0.109000)
.test_100000
  1.012000   0.000000   1.012000 (  1.012000)
  1.008000   0.000000   1.008000 (  1.000000)
  1.017000   0.000000   1.017000 (  1.017000)
  1.039000   0.000000   1.039000 (  1.039000)
  1.024000   0.000000   1.024000 (  1.024000)
.
Finished in 6.966 seconds.

So we're looking at over 4 times improvement in speed, and about 33% percent faster than MRI. Try your test cases; hopefully it will show up.