RailsConf Europe

I am a speaker at RailsConf Europe in Berlin. I'm going to talk about JRuby on Rails at ThoughtWorks. Hopefully many of you'll make it there! As far as I know, it seems the whole core JRuby team may be together for the first time ever, so it's going to be a great conference for JRuby.

See you in Berlin!

NetBeans Ruby support

It's very nice. Each time I get an update from Tor Norbye's blog with screenshots from the NetBeans Ruby support I see a new interesting feature. What I saw this time (here) was that the gotcha from my entry called "Ruby is hard" can now be detected by NetBeans. It's not really too hard, since NetBeans have the parse trees. But it's still a very nice feature to have in your editor.

But I ain't switching from Emacs yet...

Ruby is hard

I have been doing Ruby programming for several years now, and been closely involved with the JRuby implementation of it for about 20 months. I think I know the language pretty well, and I think I have for a long time. This has blinded me to something which I've just recently started to recognize. Ruby is really hard. It's hard in the same way LISP is hard. But there are also lots of gotchas and small things that can trip you up.

Ruby is a really nice language. It's very easy to learn and get started in, and you will be productive extremely quickly, but to actually master the language takes much time. I think this is underestimated in many circles. Ruby uses lots of language constructs which are extremely powerful, but they require you to shift your thinking - especially when coming from Java, which many current Ruby converts do.

As a small example of a gotcha, what does this code do:

class Foo
 attr_accessor :bar
  def initialize(value)
   bar = value
 end
end

puts Foo.new(42).bar

If your answer is that it prints 42, you're wrong. It will print nil. What's the necessary change to fix this? Introduce self:

class Foo
 attr_accessor :bar
 def initialize(value)
  self.bar = value
 end
end

puts Foo.new(42).bar

You could argue that this behavior is annoying. That it's bad. That it's wrong. But that's the way the language works, and I've seen this problem in many code bases - including my own - so it's worth keeping an eye open for it.

Why does it happen? Well, the problem is this, when the Ruby parser finds a word that begins with an underscore or a lower case letter, and where that word is followed by an equals sign, there is no way for the parser to know if this is a method call (the method foo=) or a variable assignment. The fall back of the parser is to assume it is a variable assignment and let you specify explicitly when it should be a VCall. So what happens in the first example is that a new local variable called "bar" will be introduced and assigned to.

JRuby presentation in Bergen

Next Wednesday (the 15th) I will present about JRuby in Bergen. I mentioned this in an earlier post. More information can be found here: http://www4.java.no/web/show.do?page=42;7&appmode=/showReply&articleid=5493.

JRuby now JIT-compiles assertions

One of the major problems when running automated testing with JRuby is that all the standard Test::Unit assertions would never be JIT compiled, meaning that they would be quite slow. Actually, assertions seems to be very slow when running interpreted in JRuby. I have a small test case, courtesy of Michael Schubert:

require 'test/unit'
require 'benchmark'

class A < Test::Unit::TestCase
[10_000, 100_000].each do |n|
 define_method "test_#{n}" do
   puts "test_#{n}"
   5.times do
     puts Benchmark.measure{n.times{assert_equal true,true}}
   end
 end
end
end

This code will show quite nicely how large the overhead of asserts are, by using assert_equal. Now, the numbers for MRI for this benchmark looks like this:

Loaded suite test_assert
Started
test_10000
0.150000   0.000000   0.150000 (  0.155817)
0.150000   0.000000   0.150000 (  0.158376)
0.160000   0.000000   0.160000 (  0.155575)
0.150000   0.000000   0.150000 (  0.154380)
0.160000   0.000000   0.160000 (  0.157737)
.test_100000
1.520000   0.010000   1.530000 (  1.539325)
1.530000   0.000000   1.530000 (  1.543889)
1.520000   0.010000   1.530000 (  1.540376)
1.530000   0.000000   1.530000 (  1.543742)
1.530000   0.010000   1.540000 (  1.558292)
.
Finished in 8.509493 seconds.

2 tests, 550000 assertions, 0 failures, 0 errors

And for JRuby without compilation:

Loaded suite test_assert
Started
test_10000
1.408000   0.000000   1.408000 (  1.408000)
0.582000   0.000000   0.582000 (  0.582000)
0.425000   0.000000   0.425000 (  0.426000)
0.419000   0.000000   0.419000 (  0.418000)
0.466000   0.000000   0.466000 (  0.467000)
.test_100000
4.189000   0.000000   4.189000 (  4.190000)
4.196000   0.000000   4.196000 (  4.196000)
4.139000   0.000000   4.139000 (  4.139000)
4.165000   0.000000   4.165000 (  4.165000)
4.162000   0.000000   4.162000 (  4.162000)
.
Finished in 24.181 seconds.

2 tests, 550000 assertions, 0 failures, 0 errors

It's quite obvious that something is very wrong. We're about 2.5-3 times slower.

Now, the way the JRuby compiler works, we build it piece by piece and the JIT will try to compile a method that's used enough. If there is any node that can't be compiled it will fail and fall back on interpretation. In the case of assertions, all Test::Unit assertions use a small helper method called _wrap_assertion that looks like this:

def _wrap_assertion
 @_assertion_wrapped ||= false
 unless (@_assertion_wrapped)
   @_assertion_wrapped = true
   begin
     add_assertion
     return yield
   ensure
     @_assertion_wrapped = false
   end
 else
   return yield
 end
end

When I started out on this quest, there were two things in this method that doesn't compile. The first is the ||= construct, which I mentioned in an earlier blog post. The problem with it is that it requires that we can compile DefinedNode too, and that one is large. The second problem node is Ensure. After lots of work, I've finally managed to implement most of these safely, and falling back on interpretation when it's not safe. Without further ado, the numbers after compilation with these features added:

Loaded suite test_assert
Started
test_10000
  0.996000   0.000000   0.996000 (  1.013000)
  0.415000   0.000000   0.415000 (  0.415000)
  0.110000   0.000000   0.110000 (  0.110000)
  0.099000   0.000000   0.099000 (  0.100000)
  0.109000   0.000000   0.109000 (  0.109000)
.test_100000
  1.012000   0.000000   1.012000 (  1.012000)
  1.008000   0.000000   1.008000 (  1.000000)
  1.017000   0.000000   1.017000 (  1.017000)
  1.039000   0.000000   1.039000 (  1.039000)
  1.024000   0.000000   1.024000 (  1.024000)
.
Finished in 6.966 seconds.

So we're looking at over 4 times improvement in speed, and about 33% percent faster than MRI. Try your test cases; hopefully it will show up.

JRuby presentation at javaBin in Norway

Next week I'm coming to Norway to do two presentations at the Java User Groups in Bergen and Oslo. I will visit Bergen on the 15th, and Oslo on the 16th. I will have 2x45 minutes to talk, so there will lots of time for interesting detours, and hopefully some discussion too. If you're in the area, do show up! More information about the Oslo evening can be found here: http://www4.java.no/web/show.do?page=42;7&appmode=/showReply&articleid=5447.

After Norway I'm going to Sweden. I'll be in Gothenburg for a few days, Malmö a few more, and then stop by in Stockholm the 23rd to 26th. I'm not going to present while there, though. It's purely recreational.

JRuby is now Java 5+

So, after some discussion on the #jruby IRC channel, the core team has decided to go with a 5+ strategy on trunk. The reasons for this is that almost everyone who commented on the issues advised us to move on, and the features of 5 is quite compelling. We are moving to annotations for describing Java method bindings, we will use enumerations for many values, and we will be able to use the native concurrency libraries. All of these are large wins.

Of course, we are not leaving 1.4 users completely behind. First of all, we provide a Retrotranslator/Retroweaver (we haven't decided which one yet) version, and will continue doing so. Secondly, the 1.0 branch will continue supporting Java 1.4.

This is the current state. First versions of the annotation based method bindings are already in trunk.

The pain of compiling try-catch

I've been spending some time trying to implement a compiler for the defined?-feature of Ruby. If you haven't seen it, be happy. It's quite annoying, and incredibly complicated to implement, since you basically need to create a small interpreter especially just for nodes existing within defined?. So why is defined? so important? Well, for one it's actually needed to implement the construct ||= correctly. And that is used everywhere, which means that not compiling it will severely impact our ability to compile code. Also, it just so happens that OpAsgnOrNode (as it's called), and EnsureNode, are the two nodes left to implement to be able to compile Test::Unit assert-methods, since the internal _wrap_assertion uses both ensure and ||=.

So, now you know why. Next, a quick intro to the compilation strategy of JRuby. Basically we try to compile each script and each method into one Java method. We try to use the stack as much as possible, since we in that way can link statements together correctly. And that's about it.

The problem enters when you need to handle exceptions in the emitted Java bytecode. This isn't a problem in the interpreter, since we explicitly return a value for each node, and the interpreter doesn't use the Java stack as much as the compiler does. We also want to be able to use finally blocks at places, especially to ensure that ensure can be compiled down, but also to make the implementation of defined? safe.

So what's the problem? Can't we just emit the catch-table and so on correctly? Well, yes, we can do that. But it doesn't work. Because of a very annoying feature of the JVM. Namely, when a catch-block is entered, the stack gets blown away. Completely. So if the Ruby code is in the middle of a long chained statement, everything will disappear. And what's worse, this will actually fail to load with a Verifier exception, saying "inconsistent stack height", since there will now be one code path with things on the stack, and one code path with no values on the stack, and the way JRuby works, these will end up at the same point later on. And the JVM doesn't allow that either.

This makes it incredibly hard to handle these constructs in bytecode, and frankly, right now I have no idea how to do it. My first approach was to actually create a new method for each try-catch or try-finally, and just have the code in there instead. The fine thing about that is that the surrounding stack will not be blown away since it's part of the invoking method, and not in the current activation frame. And that approach actually works fairly well. Until you want to refer to values from outside from the try or catch block. Then it breaks down.

So, right now I don't know what to do. We have no way of knowing at any specific place how low the stack is, so it's not possible to copy it somewhere, and then restore it in the catch block. That would be totally inefficient too. In fact, I have no idea how other implementations handle this. There's gotta be a trick to it.

I love XKCD

Every time my feed reports an update on XKCD, I get all happy. It's without a doubt the best online comic. The blend is perfect. Just take a look at todays:

How perfect isn't that?

New York

I landed in New York this morning, and I'll stay until Thursday. If anyone feels like taking a beer and discussing programming languages (or something else), don't hesitate to mail me or comment on this blog.

Languages of the future

Martin Fowler writes about one language, and neatly encapsulates what I think about the subject in the way I would have written it, if I were actually a good writer. Go read.

FOSCON slides

I have been asked for the slides of my FOSCON presentation. They can be downloaded in PDF format from here: http://ologix.com/JRubyWhirlwindTour.pdf.

The IronRuby scoop

As you almost certainly now at this time, IronRuby was released the beginning of this week (read more here, here and here). Now, this of course begs a few question. But first things first: note that Microsoft have decided to host the project on RubyForge, and will do so sometime next month. Not only that, it's actually a very open source license called MsPL: Microsoft Permissive License. If you look at the license, you will see that it's almost exactly a BSD license, except that the patent restrictions have been clarified. Obviously, I'm not a lawyer, but I can say that this license seems very nice and will allow good usage of IronRuby in many cases.

The code base is strictly limited right now. The things that actually work is most of the core language structure, the Array and String classes and .NET Interop. Of course, that is an extremely large achievement to have this working so well. Also, the compiler seems to generate very good code and judging from microbenchmarks, it has the possibility of having very good performance. It's important to remember that many of the more important corner cases aren't supported yet, and it is these that usually drags the performance of Ruby implementations down. I'm confident that Lam and company can handle this, though.

Overall, I would say that this is looking very good. I would recommend .NET people to take a look at it, and also try to contribute. Very soon, you will be able to contribute code back to everything except the core compiler and DLR.

Finally, the question on everyones mind: when will IronRuby be finished? I don't know, but I think it can happen with 6-12 months. The concerns raised two months ago have been resolved internally by Microsoft, which makes IronRuby a very real and important project.

Should JRuby support 1.4.2?

Right now we're trying to decide if JRuby should upgrade from Java 1.4.2 to Java 5. There are some compelling reasons for this, but I'm not a 100% sure it's a good idea. Any comments from my readers?

In practical terms, this will mean that JRuby 1.0 will continue to be supported on 1.4.2, but new development will only work on Java 5 or higher. There is talk about using retrotranslator for handling 1.4.2 compatibility in later versions.

So. Please, comments and opinions!

ThoughtWorks at OSCON

I am at OSCON with Roy Singham, the founder of ThoughtWorks. We're here as sponsors. If you are here too - Roy likes nothing more than an argument about anything related to a) the merits of various languages, b) Agile versus waterfall, c) the best cell-phone. Seriously, if there is something interested that should be big for custom-app dev in the enterprise, tell Roy or the other ten ThoughtWorkers here about it and we'll help make it happen. Come by our booth at the left side of the expo hall if you want to find us.

Really Radical Ruby

I had a very good time at FOSCON III with the Portland Ruby Brigade yesterday. There were lots of entertaining talks too. I would say that my "lightning talk" wasn't really a lightning talk at all. Unless you count the speed of my talking... I managed to race through all my 22 slides - all of them shock full with information - in the time alloted to me. Hopefully people learned something from it.

Chad Wathington demonstrated Mingle which also was very nice.

John Lam showed us a taste of IronRuby, and also talked some about the implementation particulars that made certain things faster in IronRuby than on MRI. Interesting stuff, but I'm looking forward to the full talk tomorrow.

Alan McKean from GemStone showcased GemStone/JRuby - still a work in progress though. For those of you who don't know what GemStone is, think extremely powerful object persistence. And they're building a new version for Ruby, on top of JRuby. Very cool.

I realized that there are a few points about JRuby that haven't been emphasized enough, though, so here is there executive summary bullet points:

• JRuby is totally Java compliant and runs on any Java.
• JRuby is 1.0
• JRuby supports Rails
• ThoughtWorks offers commercial support for JRuby
• JRuby performance is on par with the C implementation, on average.

OSCON: first tutorial day

Yesterday was the first tutorial day at OSCON. Due to some planning mistakes, I didn't get the correct conference pass, so I missed the first tutorial. After that was sorted out I proceeded to the second tutorial of the day: Advanced Techniques for Parsing, by Mark Dominus. Of course I knew that the code would be Perl, but that didn't disturb me so much, since I expected to see some advanced parsing techniques. This is where disappointment hit me. Maybe it was advanced Perl code used, but it was not in any way advanced parsing. The first 2 hours were spent implementing a recursive descent parser with 7 productions. After that, I decided that I wouldn't be learning anything from this presentation, and headed back to the hotel, which was good, since I got sick that afternoon and spent the rest of the evening slightly delirious in my bed.

But now I'm up and going again, sitting here waiting for the tutorial "Real World Grails" to begin. I'm looking forward to see how Grails is actually used, since the presentations I've seen on it usually just show scaffolding and simpler things.

I have also decided on the subject for tonights FOSCON, but the slides are not finished yet. And the topic I choose is kind of a cop out: JRuby Cavalcade is the title of the talk, and I will basically just run through loads of interesting and funny JRuby things until I run out of time or gets booed of stage. Hope to see you there!

Will JavaScript be the next big language?

I guess most of you have read the latest happenings in the whole NBL-story. Steve Yegge have created a Rhino on Rails; a JavaScript version of Rails. I have meant to write about this for a few weeks, but I felt I wanted to think it through before writing about it.

Of course, it's really cool that Rhino on Rails exists, and may someday be open sourced. On the other hand, the current version have no counterpart to ActiveRecord - instead it works against internal Google data systems. This may make very good sense from the pov of Google, but will make it hard to gain traction outside of Google when/if the open sourcing happens. At that point, a ActiveRecord version written using JDBC and Rails should happen. I'm personally looking forward to it, because such an undertaking would be able to take more advantage of JDBC than for example ActiveRecord-JDBC can do. ActiveRecord done "right" so to speak, but in JavaScript.

But is JavaScript the language of the future? What does it have that separates it from other current languages - most notably Ruby? Well, at first glance the big difference is in the amount of implementations. JavaScript have a multitude of implementations, while Ruby has three (Ruby, YARV? and JRuby), with many more in the works. JavaScript though, have a incredible amount of implementations, most of them bad.

It's got a standard. It's got full closures and a few more neat futures. And it has prototype object orientation; a very different object model compared to the standard of inheritance based object systems like Java, Ruby and Smalltalk.

On closer inspection, the prototype based object system is not really such a big deal. In fact, you already have it in Ruby. Just use Object.new and Kernel.clone. Not very practical in Ruby, but you can do it without any problem. Remember that the difference between dup and clone is that clone will actually copy the metaclass of the object, while dup just makes the duplicate use the same real class.

I see some advantages that JavaScript has over Ruby, that can actually count as real advantages. The spec is arguably one. Several good implementations is another. It's easier to make JavaScript have good performance. And maybe most important: most developers already know it to some degree or other.

But will it be the NBL in some form or another? No, I don't think so, for the simple reason that to many developers hate it. Not because of the language itself, but due to the connection with bad browser implementations. This is a debt that it will be very hard for JS to get away from, and I think it will tip the point against JavaScript compared any other language poised for the position of being a big language.

Further, I really don't believe in the idea that there will be a next big language. Call me strange, but just because we've had this cycle two or three times doesn't mean it will continue like that. I believe development is qualitively different now compared to 15 years ago. The challenges we meet are different and demand different tools. One single language will not cut it. I think the future lies in layered languages with different properties.

JRuby at FOSCON 2007

So, I will attend and present at FOSCON 2007, which is arranged by the Portland Ruby Brigade (PDX.rb), and the theme is Really Radical Ruby. It's on Tuesday, more information here. It seems to be an interesting event, so please show up!

I have not yet decided what I'm going to talk about. JRuby will be involved in some way of course. Anyone have any request about what I should talk about?

OSCON and other conferences

Next week I'll be at OSCON in Portland. I won't be speaking, which means I'll probably be able to enjoy other peoples presentations instead. Hopefully I'll get to meet lots of people too. Say hi if you see me!

I'll be presenting at RailsConfEU in September too, and that is gearing up to be a really interesting conference. RailsConf in Portland was awesome, and if the EU version can match a 10th of that energy, it's going to be wonderful.