JRuby now JIT-compiles assertions

One of the major problems when running automated testing with JRuby is that all the standard Test::Unit assertions would never be JIT compiled, meaning that they would be quite slow. Actually, assertions seems to be very slow when running interpreted in JRuby. I have a small test case, courtesy of Michael Schubert:

require 'test/unit'
require 'benchmark'

class A < Test::Unit::TestCase
[10_000, 100_000].each do |n|
 define_method "test_#{n}" do
   puts "test_#{n}"
   5.times do
     puts Benchmark.measure{n.times{assert_equal true,true}}
   end
 end
end
end

This code will show quite nicely how large the overhead of asserts are, by using assert_equal. Now, the numbers for MRI for this benchmark looks like this:

Loaded suite test_assert
Started
test_10000
0.150000   0.000000   0.150000 (  0.155817)
0.150000   0.000000   0.150000 (  0.158376)
0.160000   0.000000   0.160000 (  0.155575)
0.150000   0.000000   0.150000 (  0.154380)
0.160000   0.000000   0.160000 (  0.157737)
.test_100000
1.520000   0.010000   1.530000 (  1.539325)
1.530000   0.000000   1.530000 (  1.543889)
1.520000   0.010000   1.530000 (  1.540376)
1.530000   0.000000   1.530000 (  1.543742)
1.530000   0.010000   1.540000 (  1.558292)
.
Finished in 8.509493 seconds.

2 tests, 550000 assertions, 0 failures, 0 errors

And for JRuby without compilation:

Loaded suite test_assert
Started
test_10000
1.408000   0.000000   1.408000 (  1.408000)
0.582000   0.000000   0.582000 (  0.582000)
0.425000   0.000000   0.425000 (  0.426000)
0.419000   0.000000   0.419000 (  0.418000)
0.466000   0.000000   0.466000 (  0.467000)
.test_100000
4.189000   0.000000   4.189000 (  4.190000)
4.196000   0.000000   4.196000 (  4.196000)
4.139000   0.000000   4.139000 (  4.139000)
4.165000   0.000000   4.165000 (  4.165000)
4.162000   0.000000   4.162000 (  4.162000)
.
Finished in 24.181 seconds.

2 tests, 550000 assertions, 0 failures, 0 errors

It's quite obvious that something is very wrong. We're about 2.5-3 times slower.

Now, the way the JRuby compiler works, we build it piece by piece and the JIT will try to compile a method that's used enough. If there is any node that can't be compiled it will fail and fall back on interpretation. In the case of assertions, all Test::Unit assertions use a small helper method called _wrap_assertion that looks like this:

def _wrap_assertion
 @_assertion_wrapped ||= false
 unless (@_assertion_wrapped)
   @_assertion_wrapped = true
   begin
     add_assertion
     return yield
   ensure
     @_assertion_wrapped = false
   end
 else
   return yield
 end
end

When I started out on this quest, there were two things in this method that doesn't compile. The first is the ||= construct, which I mentioned in an earlier blog post. The problem with it is that it requires that we can compile DefinedNode too, and that one is large. The second problem node is Ensure. After lots of work, I've finally managed to implement most of these safely, and falling back on interpretation when it's not safe. Without further ado, the numbers after compilation with these features added:

Loaded suite test_assert
Started
test_10000
  0.996000   0.000000   0.996000 (  1.013000)
  0.415000   0.000000   0.415000 (  0.415000)
  0.110000   0.000000   0.110000 (  0.110000)
  0.099000   0.000000   0.099000 (  0.100000)
  0.109000   0.000000   0.109000 (  0.109000)
.test_100000
  1.012000   0.000000   1.012000 (  1.012000)
  1.008000   0.000000   1.008000 (  1.000000)
  1.017000   0.000000   1.017000 (  1.017000)
  1.039000   0.000000   1.039000 (  1.039000)
  1.024000   0.000000   1.024000 (  1.024000)
.
Finished in 6.966 seconds.

So we're looking at over 4 times improvement in speed, and about 33% percent faster than MRI. Try your test cases; hopefully it will show up.

JRuby presentation at javaBin in Norway

Next week I'm coming to Norway to do two presentations at the Java User Groups in Bergen and Oslo. I will visit Bergen on the 15th, and Oslo on the 16th. I will have 2x45 minutes to talk, so there will lots of time for interesting detours, and hopefully some discussion too. If you're in the area, do show up! More information about the Oslo evening can be found here: http://www4.java.no/web/show.do?page=42;7&appmode=/showReply&articleid=5447.

After Norway I'm going to Sweden. I'll be in Gothenburg for a few days, Malmö a few more, and then stop by in Stockholm the 23rd to 26th. I'm not going to present while there, though. It's purely recreational.

JRuby is now Java 5+

So, after some discussion on the #jruby IRC channel, the core team has decided to go with a 5+ strategy on trunk. The reasons for this is that almost everyone who commented on the issues advised us to move on, and the features of 5 is quite compelling. We are moving to annotations for describing Java method bindings, we will use enumerations for many values, and we will be able to use the native concurrency libraries. All of these are large wins.

Of course, we are not leaving 1.4 users completely behind. First of all, we provide a Retrotranslator/Retroweaver (we haven't decided which one yet) version, and will continue doing so. Secondly, the 1.0 branch will continue supporting Java 1.4.

This is the current state. First versions of the annotation based method bindings are already in trunk.

The pain of compiling try-catch

I've been spending some time trying to implement a compiler for the defined?-feature of Ruby. If you haven't seen it, be happy. It's quite annoying, and incredibly complicated to implement, since you basically need to create a small interpreter especially just for nodes existing within defined?. So why is defined? so important? Well, for one it's actually needed to implement the construct ||= correctly. And that is used everywhere, which means that not compiling it will severely impact our ability to compile code. Also, it just so happens that OpAsgnOrNode (as it's called), and EnsureNode, are the two nodes left to implement to be able to compile Test::Unit assert-methods, since the internal _wrap_assertion uses both ensure and ||=.

So, now you know why. Next, a quick intro to the compilation strategy of JRuby. Basically we try to compile each script and each method into one Java method. We try to use the stack as much as possible, since we in that way can link statements together correctly. And that's about it.

The problem enters when you need to handle exceptions in the emitted Java bytecode. This isn't a problem in the interpreter, since we explicitly return a value for each node, and the interpreter doesn't use the Java stack as much as the compiler does. We also want to be able to use finally blocks at places, especially to ensure that ensure can be compiled down, but also to make the implementation of defined? safe.

So what's the problem? Can't we just emit the catch-table and so on correctly? Well, yes, we can do that. But it doesn't work. Because of a very annoying feature of the JVM. Namely, when a catch-block is entered, the stack gets blown away. Completely. So if the Ruby code is in the middle of a long chained statement, everything will disappear. And what's worse, this will actually fail to load with a Verifier exception, saying "inconsistent stack height", since there will now be one code path with things on the stack, and one code path with no values on the stack, and the way JRuby works, these will end up at the same point later on. And the JVM doesn't allow that either.

This makes it incredibly hard to handle these constructs in bytecode, and frankly, right now I have no idea how to do it. My first approach was to actually create a new method for each try-catch or try-finally, and just have the code in there instead. The fine thing about that is that the surrounding stack will not be blown away since it's part of the invoking method, and not in the current activation frame. And that approach actually works fairly well. Until you want to refer to values from outside from the try or catch block. Then it breaks down.

So, right now I don't know what to do. We have no way of knowing at any specific place how low the stack is, so it's not possible to copy it somewhere, and then restore it in the catch block. That would be totally inefficient too. In fact, I have no idea how other implementations handle this. There's gotta be a trick to it.

I love XKCD

Every time my feed reports an update on XKCD, I get all happy. It's without a doubt the best online comic. The blend is perfect. Just take a look at todays:

How perfect isn't that?

New York

I landed in New York this morning, and I'll stay until Thursday. If anyone feels like taking a beer and discussing programming languages (or something else), don't hesitate to mail me or comment on this blog.

Languages of the future

Martin Fowler writes about one language, and neatly encapsulates what I think about the subject in the way I would have written it, if I were actually a good writer. Go read.

FOSCON slides

I have been asked for the slides of my FOSCON presentation. They can be downloaded in PDF format from here: http://ologix.com/JRubyWhirlwindTour.pdf.

The IronRuby scoop

As you almost certainly now at this time, IronRuby was released the beginning of this week (read more here, here and here). Now, this of course begs a few question. But first things first: note that Microsoft have decided to host the project on RubyForge, and will do so sometime next month. Not only that, it's actually a very open source license called MsPL: Microsoft Permissive License. If you look at the license, you will see that it's almost exactly a BSD license, except that the patent restrictions have been clarified. Obviously, I'm not a lawyer, but I can say that this license seems very nice and will allow good usage of IronRuby in many cases.

The code base is strictly limited right now. The things that actually work is most of the core language structure, the Array and String classes and .NET Interop. Of course, that is an extremely large achievement to have this working so well. Also, the compiler seems to generate very good code and judging from microbenchmarks, it has the possibility of having very good performance. It's important to remember that many of the more important corner cases aren't supported yet, and it is these that usually drags the performance of Ruby implementations down. I'm confident that Lam and company can handle this, though.

Overall, I would say that this is looking very good. I would recommend .NET people to take a look at it, and also try to contribute. Very soon, you will be able to contribute code back to everything except the core compiler and DLR.

Finally, the question on everyones mind: when will IronRuby be finished? I don't know, but I think it can happen with 6-12 months. The concerns raised two months ago have been resolved internally by Microsoft, which makes IronRuby a very real and important project.

Should JRuby support 1.4.2?

Right now we're trying to decide if JRuby should upgrade from Java 1.4.2 to Java 5. There are some compelling reasons for this, but I'm not a 100% sure it's a good idea. Any comments from my readers?

In practical terms, this will mean that JRuby 1.0 will continue to be supported on 1.4.2, but new development will only work on Java 5 or higher. There is talk about using retrotranslator for handling 1.4.2 compatibility in later versions.

So. Please, comments and opinions!

ThoughtWorks at OSCON

I am at OSCON with Roy Singham, the founder of ThoughtWorks. We're here as sponsors. If you are here too - Roy likes nothing more than an argument about anything related to a) the merits of various languages, b) Agile versus waterfall, c) the best cell-phone. Seriously, if there is something interested that should be big for custom-app dev in the enterprise, tell Roy or the other ten ThoughtWorkers here about it and we'll help make it happen. Come by our booth at the left side of the expo hall if you want to find us.

Really Radical Ruby

I had a very good time at FOSCON III with the Portland Ruby Brigade yesterday. There were lots of entertaining talks too. I would say that my "lightning talk" wasn't really a lightning talk at all. Unless you count the speed of my talking... I managed to race through all my 22 slides - all of them shock full with information - in the time alloted to me. Hopefully people learned something from it.

Chad Wathington demonstrated Mingle which also was very nice.

John Lam showed us a taste of IronRuby, and also talked some about the implementation particulars that made certain things faster in IronRuby than on MRI. Interesting stuff, but I'm looking forward to the full talk tomorrow.

Alan McKean from GemStone showcased GemStone/JRuby - still a work in progress though. For those of you who don't know what GemStone is, think extremely powerful object persistence. And they're building a new version for Ruby, on top of JRuby. Very cool.

I realized that there are a few points about JRuby that haven't been emphasized enough, though, so here is there executive summary bullet points:

• JRuby is totally Java compliant and runs on any Java.
• JRuby is 1.0
• JRuby supports Rails
• ThoughtWorks offers commercial support for JRuby
• JRuby performance is on par with the C implementation, on average.

OSCON: first tutorial day

Yesterday was the first tutorial day at OSCON. Due to some planning mistakes, I didn't get the correct conference pass, so I missed the first tutorial. After that was sorted out I proceeded to the second tutorial of the day: Advanced Techniques for Parsing, by Mark Dominus. Of course I knew that the code would be Perl, but that didn't disturb me so much, since I expected to see some advanced parsing techniques. This is where disappointment hit me. Maybe it was advanced Perl code used, but it was not in any way advanced parsing. The first 2 hours were spent implementing a recursive descent parser with 7 productions. After that, I decided that I wouldn't be learning anything from this presentation, and headed back to the hotel, which was good, since I got sick that afternoon and spent the rest of the evening slightly delirious in my bed.

But now I'm up and going again, sitting here waiting for the tutorial "Real World Grails" to begin. I'm looking forward to see how Grails is actually used, since the presentations I've seen on it usually just show scaffolding and simpler things.

I have also decided on the subject for tonights FOSCON, but the slides are not finished yet. And the topic I choose is kind of a cop out: JRuby Cavalcade is the title of the talk, and I will basically just run through loads of interesting and funny JRuby things until I run out of time or gets booed of stage. Hope to see you there!

Will JavaScript be the next big language?

I guess most of you have read the latest happenings in the whole NBL-story. Steve Yegge have created a Rhino on Rails; a JavaScript version of Rails. I have meant to write about this for a few weeks, but I felt I wanted to think it through before writing about it.

Of course, it's really cool that Rhino on Rails exists, and may someday be open sourced. On the other hand, the current version have no counterpart to ActiveRecord - instead it works against internal Google data systems. This may make very good sense from the pov of Google, but will make it hard to gain traction outside of Google when/if the open sourcing happens. At that point, a ActiveRecord version written using JDBC and Rails should happen. I'm personally looking forward to it, because such an undertaking would be able to take more advantage of JDBC than for example ActiveRecord-JDBC can do. ActiveRecord done "right" so to speak, but in JavaScript.

But is JavaScript the language of the future? What does it have that separates it from other current languages - most notably Ruby? Well, at first glance the big difference is in the amount of implementations. JavaScript have a multitude of implementations, while Ruby has three (Ruby, YARV? and JRuby), with many more in the works. JavaScript though, have a incredible amount of implementations, most of them bad.

It's got a standard. It's got full closures and a few more neat futures. And it has prototype object orientation; a very different object model compared to the standard of inheritance based object systems like Java, Ruby and Smalltalk.

On closer inspection, the prototype based object system is not really such a big deal. In fact, you already have it in Ruby. Just use Object.new and Kernel.clone. Not very practical in Ruby, but you can do it without any problem. Remember that the difference between dup and clone is that clone will actually copy the metaclass of the object, while dup just makes the duplicate use the same real class.

I see some advantages that JavaScript has over Ruby, that can actually count as real advantages. The spec is arguably one. Several good implementations is another. It's easier to make JavaScript have good performance. And maybe most important: most developers already know it to some degree or other.

But will it be the NBL in some form or another? No, I don't think so, for the simple reason that to many developers hate it. Not because of the language itself, but due to the connection with bad browser implementations. This is a debt that it will be very hard for JS to get away from, and I think it will tip the point against JavaScript compared any other language poised for the position of being a big language.

Further, I really don't believe in the idea that there will be a next big language. Call me strange, but just because we've had this cycle two or three times doesn't mean it will continue like that. I believe development is qualitively different now compared to 15 years ago. The challenges we meet are different and demand different tools. One single language will not cut it. I think the future lies in layered languages with different properties.

JRuby at FOSCON 2007

So, I will attend and present at FOSCON 2007, which is arranged by the Portland Ruby Brigade (PDX.rb), and the theme is Really Radical Ruby. It's on Tuesday, more information here. It seems to be an interesting event, so please show up!

I have not yet decided what I'm going to talk about. JRuby will be involved in some way of course. Anyone have any request about what I should talk about?

OSCON and other conferences

Next week I'll be at OSCON in Portland. I won't be speaking, which means I'll probably be able to enjoy other peoples presentations instead. Hopefully I'll get to meet lots of people too. Say hi if you see me!

I'll be presenting at RailsConfEU in September too, and that is gearing up to be a really interesting conference. RailsConf in Portland was awesome, and if the EU version can match a 10th of that energy, it's going to be wonderful.

Back to JRuby regular expressions

It seems that this issue comes up every third month. After all the work we have done, we realize that regular expressions need some real work again. Our current solution works quite well. We have imported JRegex into JRuby, and done a whole slew of modifications to it. It runs well, have no issues with to deep regular expressions (Javas engine uses a recursive algorithm, making it stack overflow for certain inputs. Certain very common inputs, in say ... Rails. *sigh*).

But JRegex is good. It's not perfect though. It's slightly slower than the Java engine, it doesn't support everything in the Java engine, and conversely, it supports some things that Java doesn't support. The major problem is that we don't have MRI compliant multibyte support, and the implementation of our engine is wildly different compared to MRI's engine, and Oniguruma.

At some point we will probably just bite the bullet and do a real port of Oniguruma. But until such time comes, I have extracted our current regular expression stuff, and put everything behind a common interface. What that means is that with the current trunk, you can actually choose which Regular Expression engine you want to use. You can even write your own and plug in. The interface is really small right now. At the moment we only have JRegex and Java, and the Java engine doesn't pass all tests (I think, I haven't tried, since that wasn't the point of this exercise.). Anyway; it means you can have Java Regular Expressions if you want them, right in your JRuby code. But only where you want them. So, you can regular which engine is used globally by doing one of these two:

jruby -J-Djruby.regexp=java your_ruby_script.rb
jruby -J-Djruby.regexp=jregex your_ruby_script.rb

The last is current the default, so it's not needed. In the future it may be possible that JRegex isn't the default though, but this options should still be there. But the more nice thing about this is also that you can use Java Regexps inline, even if you want to use JRegex for most expressions:

begin
 p(/\p{javaLowerCase}*/ =~ "abc")
 p $&
rescue => e
 p e
end

p(/\p{javaLowerCase}*/j =~ "abc")
p $&

Now, the first example will actually raise a RegexpError, because javaLowerCase is not a valid character class in JRegex. But not the small "j" I've added to the second Regexp literal! That expression works and will match exactly as you expected.

RSpec and RBehave runs on JRuby

I'm not sure if this is well known or not, so I've been meaning to write a quick notice about it. The short story is this: JRuby can run RSpec and RBehave. Why is this important? Well, you can write code that tests Java code using RSpec and RBehave, meaning that it will be possible to get much more readable tests, even for code living in Java land.

Even if your organization won't accept writing an application in Ruby, it would probably be easier to get the testing done in Ruby. And writing tests in an effective language means that you will either write more production code, or more tests. Either of those are a quite good outcome.

A quick example of this in action. To run this example, you need JRuby 1.0 or later, and the rspec gem:

require 'java'

describe java.util.ArrayList, " when first created" do
before(:each) do
  @list = java.util.ArrayList.new
end

it "should be empty" do
  @list.should be_empty
end

it "should be able to add an element" do
  @list.add "content"
end

it "should raise exception when getting anything" do
  lambda{ @list.get 0 }.should raise_error(java.lang.IndexOutOfBoundsException)
end
end

In this code the examples are not that realistic, but you can see that the RSpec code looks the same for Java code, as it does for Ruby code. Even the raise_error exception matcher works. You can run this:

jruby -S spec -fs arraylist_spec.rb

The RBehave test suite also runs, which means you can stop using JBehave now... =)

This is a perfect example of the intersection where JRuby's approach can be very powerful, utilizing the existing Ruby libraries to benefit your Java programming.

The results of JRuby compilation

If you are interested in what actually happens when JRuby compiles Ruby to Java bytecode, I have added some small utilities to help out with this. To compile a string:

require 'jruby'
JRuby::compile("puts 1,2,3")

If you are running with -J-Djruby.jit.enabled=false, you can also inspect the result of compiling a block:

require 'jruby'

JRuby::compile do
 puts "Hello World"
end

The results of both of these invocations will be an object of type JRuby::CompiledScript. It has four attributes: name, class_name, original_script and code. The original_script attribute is only available when compiling from a string. The code attribute contains a Java byte array, and as such is not so useful in itself. But you can use the inspect_bytecode method to get a string which describes the compiled class. So, to see how JRuby compiles a puts "Hello, World":

require 'jruby'

puts JRuby::compile(<<CODE).inspect_bytecode
  puts "Hello, World"
CODE

Once you know what happens, you can start contributing to the compiler! =)

JRuby Inside

Peter Cooper have opened a "sister" site to RubyInside, called JRubyInside. It seems very promising; the address is http://www.jrubyinside.com.

Closing over ZSuper

One of the features of Ruby which I sometimes like and sometimes hate, is ZSuper. (So called, because it differs from regular super in the AST.) ZSuper is the keyword super, with arguments and parenthesis, which will call the super method with the same arguments as the current invocation got. Of course, that's not all. For example, if you change the arguments, the changes will propagate to the super implementation. Not only if you change the object, but if you change the reference, which I found non intuitive the first time I found it.

That's all and well. The interesting thing happens when you close over the super call and return it as a Proc. I haven't seen anyone doing this, which I guess is why there seems to be a bug in the implementation. Look at this code and tell me what it prints:

class Base
def foo(*args)
 p [:Base, :foo, *args]
end
end

class Sub < Base
def foo(first, *args)
 super
 first = "changed"
 super
 proc { |*args| super }
end
end

Sub.new.foo("initial", "try", :four).call("args","to","block")

Notice that Base#foo will get called three times during this code. In Sub#foo we are changing the first argument to the new string "changed". As I told you before, the second super call will actually get "changed" as the first argument the second time. But what will happen after that? We first create a block that uses ZSuper. We send the block to proc, reifying the block into an instance of Proc, and returning that. Directly after returning the block, we call it with some arguments. Now, the way I expect this to work (and incidentally, that's the way JRuby works) is that the output should be something like this:

[:Base, :foo, "initial", "try", :four]
[:Base, :foo, "changed", "try", :four]
[:Base, :foo, "changed", "try", :four]

We see that the first argument changed from "initial" to "changed", but otherwise the result is the same; the closure is a real closure over everything in the frame and scope. I guess you've realized that the same isn't true for Ruby. Without further ado, this is the output from MRI 1.8.6:

[:Base, :foo, "initial", "try", :four]
[:Base, :foo, "changed", "try", :four]
[:Base, :foo, "changed", ["args", "to", "block"], false]

The first time I saw this, the words WTF passed through my mind. In fact, that still happens sometimes. What is happening here? Well, obviously, it seems as if the passing of arguments to the block somehow clobbers the part where MRI saves away the closure over passed arguments. I have no idea whatsoever what the false value comes from. Hmm. But now that I think about it (this is just a guess), but I believe it stands for the fact that the arguments should be splatted into one argument. (That's the one called args in the block). If it had been true, they should refer to different variables. I think there is some trickery like that involved in the splatting logic in MRI.

Anyway. Is this a bug or a feature? I can't see any way it could be used in an obvious way, and it runs counter to being understandable and unsurprising. Anyone who can give me a good example of where this is useful behavior?

A JRuby Rubinius machine

When I get bored with JRuby, I tend to go looking either at other languages or other language implementations. This happened a few days ago, and the result is what I will here document. Begin by creating a file called fib.rb:

def fib(n)
 if n < 2
   n
 else
   fib(n - 2) + fib(n - 1)
 end
end

p fib(15)

The next part requires that you have a recent version of Rubinius installed:

rbx compile fib.rb

This will generate fib.rbc. Next, take a recent JRuby version and run:

jruby -R fib.rbc

And presto, you should see 610 printed quite soon. This is JRuby executing Rubinius bytecode. I was quite happy about how it was to get this far with the functionality. Of course, JRuby doesn't support most bytecodes yet, only those needed to execute this small example, and similar things. We are also using JRuby's internals for this, which means that Rubinius MethodContext and such are not available.

Another interesting note is that running the iterative Fib algorithm like this with -J-server is actually 30% faster than MRI.

This approach is fun, and I have some other similar ideas I really want to look at. The best part about it though, is that I got the chance to look at the internals of Rubinius. I hope to have more time for it eventually. Another thing I really want to do some day is implement a Jubinius, which should be a full port of the Rubinius runtime, possibly excluding Subtend. I think it could be very nice to have the Smalltalk core of Rubinius working together with Java. Of course, I don't have any time for that, so we'll see what happens in a year or two. =) Maybe someone else does it.

Evil JRuby

After my last post I got several comments about evil.rb. Of course I had evil.rb in mind when doing some of it, but I also forgot to describe the two most evil methods of the JRuby module: runtime and reference. The runtime method will return the currently executing JRuby runtime as a Java Integration, meaning you can get access to almost anything you want with it. For example, if you want to take a look at the global CacheMap (used to cache method instances):

require 'jruby'
JRuby::runtime.cache_map

Whoops. And that's just the beginning. Are you interested in investigating the current call frame or activation frame (DynamicScope in JRuby):

require 'jruby'
p JRuby::runtime.current_context.current_frame
a = 1
p JRuby::runtime.current_context.current_scope

Of course, you can call all accessible (and some inaccessible) methods on these objects, just like if you were working with it from Java. Use the API's and take a look. You can change things without problem.

And that also brings us to one of the easiest examples of evil.rb, changing the frozen flag on a Ruby object. Well, with the reference method, that's easy:

require 'jruby'

str = "foo"
str.freeze

puts str.frozen?
JRuby::reference(str).setFrozen(false)
puts str.frozen?

JRuby::reference will return the same object sent in, wrapped in a Java Integration layer, meaning that you can inspect and modify it to your hearts like. In this way, you can get at the internals of JRuby in the same way you can using evil.rb for MRI. And I guess these features should mainly be used for looking and learning about the internals of JRuby.

So, have fun and don't be evil (overtly).

Some JRuby tricks

I have spent a few hours adding some useful features these last days. Nothing extraordinary, but things that might come in handy at one point or another. The problem with these features is that they are totally JRuby specific. That means you could probably implement them for MRI, but noone has done it. That means that if you want to use it, beware. Further, they exploit a few tricks in the JRuby implementation, meaning it can't be implemented in pure Ruby.

So, that was the disclaimer; now onto the fun stuff!

Breaking encapsulation (even more)
As you know, in Ruby everything is accessible in some form or another, and you can do almost everything with the metaprogramming facilities. Well, except for one small detail which I found out while working on the AR-JDBC database drivers.

We have some code there which needs to be separate for each database, and it just so happens that core ActiveRecord have already implemented them in a very good way. So, what do we do? Mix in them and remove the methods we don't want? No, because ActiveRecord adapters are classes, not modules, and you can't mix in classes. There is no way to get hold of a method and add that to an unrelated other class or module. Except if you're on JRuby, of course:

require 'jruby/ext'

class A
def foo
puts "A#foo"
end
def bar
puts "A#bar"
end
end

class B;end

class C;end

b = B.new
b.steal_method A, :foo
b.foo
B.new.foo rescue nil #will raise NoMethodError

C.steal_methods A, :foo, :bar
C.new.foo
C.new.bar

Of course, using this should be avoided at all costs. But it's interesting that such a powerful thing can be implemented using about 15 lines of Java code.

Introspection
JRuby parses Ruby code into an Abstract Syntax Tree. For a while now, the JRuby module have allowed you to parse a string and get the AST representation by executing:

require 'jruby'

JRuby.parse "puts 'hello'", 'filename.rb', false

This returns the Java AST representation directly, using the Java Integration features. That is old. What is new is that I have added pretty inspecting, a nice YAML format and some navigation features which makes it very easy to see exactly how the AST looks. Just do an inspect or to_yaml on an AST node and you will get the relevant information.

That is interesting. But what is even more nice is the ability to run and use arbitrary pieces of the AST (as long as they make sense together) and also run them:

require 'jruby'

ast_one = JRuby::ast_for("n = 1; n*(n+3)*(n+2)")
ast_two = JRuby::ast_for("n = 42; n*(n+1)*(n+2)")

p (ast_one.first.first + ast_two.first[1]).run
p (ast_two.first.first + ast_one.first[1]).run

As you can see, I take two fragments from different code, add them together and run them. You can also see that I'm using an alias for parse here, called ast_for. That makes much more sense when using the second parse feature, which we already know from ParseTree:

require 'jruby'

JRuby::ast_for do
puts "Hello"
end

Well, I guess that's all I wanted to show right now. These last small things I've added because I believe they will be highly useful for debugging JRuby code.

I also have some more ideas that I want to implement. I'll keep you posted about it.

ObjectSpace: to have or not to have

Among all the features of Ruby that JRuby supports, I would say that two things take the number one place as being really inconvenient. Threads are one; making the native threading of Java match the green threading semantics of Ruby is not fun, and it's not even possible for all edge cases. But that argument have been made several times by both me and Charles.

ObjectSpace now, that is another story. The problems with OS are many. But first, let's take a quick look at the most common usage of OS; iterating over classes:

ObjectSpace::each_object(Class) do |c|
  p c if c < Test::Unit::TestCase
end

This code is totally obvious; we iterate over all instances of Class in the system, and print an inspected version of them if the class is a subclass of Test::Unit::TestCase.

Before we take a closer look at this example, let's talk quickly about how MRI and JRuby implements this functionality. In fact, having this functionality in MRI is dead easy. It's actually very simple, and there are no performance problems of having it when it's not used. The trick is that MRI just walks the heap when iterating over ObjectSpace. Since MRI can inspect the heap and stack without problems, this means that nothing special needs to be done to support this behavior. (Note that this can never be safe when using a real threading system).

So, the other side of the story: how does JRuby implement it? Well, JRuby can't inspect the heap of course. So we need to keep a WeakReference to each instance of RubyObject ever created in the system. This is gross. We pay a huge penalty for managing all this stuff. Many of the larger performance benefits we have found the last year have revolved around having internal objects be smarter and not put themselves into ObjectSpace until necessary. One of my latest optimizations of regexp matching was simple to make MatchData lazy, so it only goes into OS when someone actually uses it. RDoc runs about 40% faster when ObjectSpace is turned off for JRuby.

So, is it worth it? In real life, when do you need the functionality of ObjectSpace? I've seen two places that use it in code I use every day. First, Rails uses it to find generators, and secondly, Test::Unit uses it to find instances of TestCase. But the fun thing is this; the above code is almost exactly what they do; they iterate over all classes in the system and checking if they inherit from a specific base class. Isn't that a quite gross implementation? Shouldn't it be possible to do something better? Euhm, yes:

module SubclassTracking
  def self.extended(klazz)
    (class <<klazz; self; end).send :attr_accessor, 
                                    :subclasses
    (class <<klazz; self; end).send :define_method, 
                                :inherited do |clzz|
      klazz.subclasses << clzz
      super
    end
    klazz.subclasses = []
  end
end

# Where Test::Unit::TestCase is defined
Test::Unit::TestCase.extend SubclassTracking

# Load all other classes

# To find all subclasses and test them:
Test::Unit::TestCase.subclasses

I would say that this code solves the problem more elegantly and useful than ObjectSpace. There are no performance degradation due to it, and it will only effect subclasses of the class you are interested in. What's the best benefit of this? You can use the -O flag when running JRuby, and your tests and rest of the code will run much faster and use less memory.

As a sidenote: I'm putting together a patch based on this to both Test::Unit and Rails. ObjectSpace is unnecessary for real code and the vision of JRuby is that you will explicitly have to turn it on to use it, instead of the other way around.

Anyone have any real world examples of things you need to do with ObjectSpace?

First two days of TSSJS

It has been two long days; not because I've been going to sessions all day long, but because I've reworked my presentations quite heavily. But now both the BOF and the TS are finished, and I think they went well. I had to keep the level to Ruby, JRuby and Rails introductionary material, though, since most developers here didn't seem to know what is possible with these technologies.

But it's been great; I've gotten good feedback and had some really interesting conversations with lots of people.

We have been doing the town each night, and I've found that I like Barcelona very much. Except for the food: this country doesn't seem to be good for vegetarians at all. Very annoying. I'm going for beer and wine instead of food the rest of the week. =)

One day left, though, and it's bound to be nice. Me and Martin are both on a developers panel about the state of programming languages in 2020; I have no idea what to say, and I'm thinking about just ad-libbing it. I know my own position in these questions fairly well, and the current Yegge-debate have made my opinions even more explicit.

But now it's time to see the town again.

Interviewed by AkitaOnRails

Yesterday I spent 2 hours chatting with Fabio Akita, of AkitaOnRails (the largest Rails blog in Brazil); the result is a long interview that was published today. It's got some good stuff, and some Ola-stuff, which you should recognize by now.

And I note that he calls me a workaholic; but he got this interview prepared in less than a day too, and also translated it into Portuguese.

You can find it at http://www.akitaonrails.com/pages/olabini.

First weeks at ThoughtWorks

I've finally started. I've finally moved to London. I've been working for two weeks at ThoughtWorks now, and it's been quite crazy. Everything is very nice and I'm having loads of fun. Of course, it's also lots of hard work, and I feel that I'm stretching my capacity considerably more than I ever did at Karolinska Institutet. That's great, and I feel that I'm really doing something real now. We have so many interesting things going on, and I wish I could tell you all about it.

What I can tell you is that I'm working quite much on Mingle, and I'm also spending time on other JRuby related issues. I've been planning on getting SQL Server and Oracle working as good as possible with AR-JDBC, and I've spent time on Derby performance. Hopefully I'll continue the database work this week, since especially SQL Server and Oracle is very important.

The most important work for this week is probably to prepare for TheServerSide in Barcelona. I still haven't had time to prepare my demos, so it's about time now. I hope to see many of you there.

In conclusion, my first weeks at ThoughtWorks have been awesome. I really like the pople, and everything is just neat. I like being able to walk to work and working in the very nice TW office on High Holborn. I'm very happy about it all.

Book update

As you know, I am writing on a book about JRuby on Rails. A few minutes ago I finished the first draft of chapter 14. That means that there are just 3 chapters and 3 appendixes left to write (chapter 1, 2 and 15). So the writing is going very well, but it's taking a heavy toll on me personally. I seriously don't recommend writing a book like this in your spare time, while at the same time switching employer, moving abroad and try to be a core developer in an open source project which is getting lots of attention.

So, in summary: it's going well, it still looks like it will be out in October, and I'm deadly tired.

JRuby 1.0

The JRuby community is pleased to announce the release of JRuby 1.0!

Homepage: http://www.jruby.org/
Download: http://dist.codehaus.org/jruby/

JRuby 1.0 is a major milestone for our project. Our main goal for 1.0 has been
Ruby compatibility. We feel this goal has been reached. When we see
companies like ThoughtWorks offering commercial support; we know this goal
has been reached. Please download JRuby and take it for a test drive. Try
running your Ruby and Ruby on Rails applications with it.

Give us feedback. Join our community. Help us continue to improve JRuby.

It is important to notice that JRuby 1.0 is not the end all of Ruby interpreters. It's not perfect. This is just the beginning. We now have a very good base to work from. This is were the real work begins. Join us. It will be a fun ride, and JRuby will just get better!

What's wrong with this code?

Today I will introduce to you a method from ActiveRecord. The method takes a parameter called type and that value can bu for example :primary_key, :string or :integer. Now, in the first line there is a call to native_database_types. Generally, that call returns a structure that looks somewhat like this:

def native_database_types #:nodoc:
  {
   :primary_key => "int(11) DEFAULT NULL auto_increment PRIMARY KEY",
   :string      => { :name => "varchar", :limit => 255 },
   :text        => { :name => "text" },
   :integer     => { :name => "int", :limit => 11 },
   :float       => { :name => "float" },
   :decimal     => { :name => "decimal" },
   :datetime    => { :name => "datetime" },
   :timestamp   => { :name => "datetime" },
   :time        => { :name => "time" },
   :date        => { :name => "date" },
   :binary      => { :name => "blob" },
   :boolean     => { :name => "tinyint", :limit => 1 }
 }
end

The method itself looks like this.

def type_to_sql(type, limit = nil, precision = nil, scale = nil) #:nodoc:
  native = native_database_types[type]
 column_type_sql = native.is_a?(Hash) ? native[:name] : native
 if type == :decimal # ignore limit, use precison and scale
    precision ||= native[:precision]
   scale ||= native[:scale]
   if precision
     if scale
       column_type_sql << "(#{precision},#{scale})"
     else
       column_type_sql << "(#{precision})"
     end
   else
     raise ArgumentError, "Error adding decimal column: precision cannot be empty if scale if specified" if scale
   end
   column_type_sql
 else
   limit ||= native[:limit]
   column_type_sql << "(#{limit})" if limit
   column_type_sql
 end
end

There is something very wrong with this implementation. Of course, there could exist many errors here, but what I'm thinking about right now is a violation of the usual way methods should work. And in effect, that problem with this method have caused ActiveRecord-JDBC to implement some very inefficient code to handle this method. And it gets called a lot in ActiveRecord. I'll get back later today with a pointer to what's wrong here, and I will also discuss some of what I've done in AR-JDBC to handle this situation. I hope for many suggestions here! =)

This is what's wrong

I must say, I got some really good responses to my post about what was wrong with the code I posted. Most of those responses concerned the design of the code, and I agree, this part of Rails could have been done much better. But what I was thinking about was actually a bug. And Lars Westegren (my former colleague) nailed it at the first try. Let me show two important excerpts from this code:

column_type_sql = native.is_a?(Hash) ? native[:name] : native

and here:

column_type_sql << "(#{limit})" if limit

Obviously, double left arrow is append, and for all cases where there is a limit, this append will change the String. This is one of the cases where it's kind of annoying that strings are mutable. If I cache away the values that native_database_types should return, then the next time anyone wants a string SQL type, that will generate VARCHAR(255)(255). The next time again, VARCHAR(255)(255)(255). And so one. So either I need to recreate the hash every time, or I need to do a deep clone of it every time. Neither of these options are very good, and it seems the deep clone option isn't fast enough, even when done in Java, so I decided to go with a hash literal instead. Was that the right choice? I don't know. It improves performance, but on the other hand it churns objects and creates new objects all the time. All of this because of some sloppy coding in Rails.

What's the lesson learned? Never modify your arguments, unless that is an explicit part of the contract for that method and part of the documentation.

Testing with JRuby on Rails and ActiveRecord-JDBC

This will be a highly uninflammatory blog post, as contrast to the last one. Specifically, there is a slight problem when running the command

jruby -S rake

in a a JRuby on Rails-application. This problem is caused by some hard coded values in the database Rake definitions for Rails. But don't despair, there is a simple solution to this. It's not as simple as it should be (invisible) but it's easy enough. Provided you have JRUBY_HOME set and your version of AR-JDBC is 0.3.1, execute this command from your Rails application root

cp $JRUBY_HOME/lib/ruby/gems/1.8/gems/ActiveRecord-JDBC-0.3.1/lib/tasks/jdbc_databases.rake lib/tasks

Since the hard coded values are hard to override, the jdbc_databases.rake file just hacks Rake to be able to redefine tasks and then redefines the core tasks. This shouldn't affect a bi-Ruby installation, since the overriding only happens on JRuby, not on MRI. If someone has a better way to do this, please tell me. =)

There can be only one, a tale about Ruby, IronRuby, MS and whatnot

(Updated: added a quote from John Lam about not being able to look at the MRI source code)

After RailsConf in Portland, there has flared up a discussion centered around IronRuby and Microsoft. We discussed many of these points in depth at the conference, and I'll elaborate some on my views on the issues in a bit.

But first I would like to talk some about the multitude of Ruby implementations springing up. I firmly believe that a language evolves in phases. The first phase, germination, is the period where a language needs one consistent implementation (or a spec). It's during this phase when most "alpha geek adoption" happens. Many important libraries are written, but most applications are not in the main economic center. Ruby have been in this phase for a long time, but the fact that new implementations are springing up left and right is a sure sign that Ruby is entering phase 2: implementation. For adoption to happen, there need to exist several competing implementations, all of them good. This is the evolutionary stage, where it's decided what kind of features an implementation should provide. Should we have green or native threads? Are all the features of the original implementation really that necessary? (Continuations, ObjectSpace). Is there cruft in the standard library that needs to be weeded out? (timeout.rb). All of these questions get answered when other people implement the language. The last phase, which I guess could be called adoption, is when the language have several working implementations, all good enough to deliver high end applications on, when many applications are written in the language, and there exists a plethora of libraries, systems and support for the language.

What this means is that for a language to be successful, there needs to exist competing implementations. They need to implement their features in different ways and make different choices during development. Otherwise, the language will die. (This is obviously not enough, since Smalltalk fulfilled this admirably and still never got widespread adoption.). But I still believe it's incredibly important for a language to evolve with many implementations, which is why I find Rubinius, JRuby, YARV and IronRuby to be extremely important projects for the welfare of Ruby. I want Ruby to be successful. I want Ruby to be the next major language for several reasons. But most importantly: I want Ruby to be a better language tomorrow, than it is today. The only way that's going to happen is by having lots of people implement the language.

So, that's enough of the introductory flame bait. This describes one half of why IronRuby is an important project, and why we can't let it fail. The other side of the coin is the same reason JRuby is important. .NET as a platform have some wildly useful features. There are many developers who swear by .NET for good reason. And what's more important, there are lots of large enterprises with such a vested interest in .NET, that they will never choose anything else. Now, for the welfare of all programmers in the world, I personally believe the world would be a better place if those .NET-environments also used Ruby. So that's the other coin of why IronRuby is important.

The most well read blog about the current Microsoft/Ruby controversy is Martin Fowlers article RubyMicrosoft. Go read it now, and then I'll just highlight the points I find most important.

First: John Lam is committed to creating a "compliant" Ruby implementation. I have no doubts that he can do it. But there are a few problems lurking.

For example, what is a compliant Ruby implementation? Since there exists no spec, and no comprehensive test suite, the only way to measure compliance is to check how close the behavior matches MRI. But this have some problems too. How do you check behavior? Well, you need to run applications. But how do you get so far as you can run applications?
What JRuby did was that we looked at MRI source code.

John Lam can not look at MRI source code. He cannot look at JRuby source code. He cannot look at Rubinius source code. If he does, he will be terminated.

So, the next best alternative: accepting patches from the community, which can look at Ruby source? Nope, no cigar. Microsoft is not really about Open Source yet. Their license allows us to look at their source code, and also to fork it and do what we want with it. But that's only half of what open source is about. The more important part is that you should be able to contribute back code without having to fork. You can't do that with IronRuby, since Microsoft is too scared about being sued for copyright infringement.

There was some doubt about Lam actually being banned from looking at MRI source code. This is the first quote that said it is so. It's from the discussion "Virtual classes and 'real' classes -- why?" on Ruby-core, this quote posted at 29/03/07:

Is this how things are actually implemented? (BTW I'm not lazy here - we cannot look for ourselves).

I am going to make a bold statement here. Under the current circumstances, I don't believe it's possible for John Lam and his team to create a Ruby implementation that runs Rails within at least 18 months. And frankly, that's not soon enough.

As I said above, I have all confidence that John can do great stuff if he has the right resources. But creating a Ruby implementation is hard enough while having all the benefits of the open source community.

The two points I want to make with this point is this: The Ruby community must damned well get serious about creating a good, complete specification and test suite. It's time to do it right now, and we need it. It's not a one-man job. The community needs to do it. (And yes, the two SoC projects are a very good start. But you still need to be able to run RSpec to take full advantage of them; and let's face it, the RSpec implementation uses many nice Ruby tricks.)

The second point is simpler: Microsoft needs to completely change how they handle Open Source. Their current strategy of trying to grow it into the organization will not work (at least not good enough). They need to turn around completely, reinvent themselves and make some really bold moves to be able to meet the new world. If they can't do this, they are as dead as Paul Graham claims.

Solving mounting problem on MacOS X

This is a highly specific post, but I thought I'd write about this so that if someone else have the same problem, they can try to solve it my way.

I got a new computer, Intel-based MBP last week. I upgraded it to latest OSX version (10.4.9 I believe), and immediately installed everything I needed. While ending my last job and returning the laptop associated with that post, I made backups to a 500GB USB LACIE hard drive. Very nice indeed, and half my life is now on that hard drive. (I have most of it on other places too, but not so easily accessible).

The first thing I managed to do was to shut off the hard drive without unmounting it correctly from OSX. I got the warning et all, but well, done is done. It was stupid, I know. Starting out with messing up everything. So what happened? Well, the next time I tried to get OSX to find my hard drive, by inputing the USB connection and switching on the power. Nothing happened. The drive would spin up, but no mount points or nice icons on the desktop. After some investigation I found that the Disk Manager HANGS when the LACIE is turned on and connected. I also found that when I switched the power off, my console says something about not being able to repair disk /dev/disk2s1. Interesting. After a few hours investigation on the Internet I despaired, and decided to try my own ingenuity instead.

I won't tell you about everything I did to find this solution. It would get severely boring very fast. So, here is the solution:

1. Attach the device and turn on the power.
2. Open up a terminal and create a new directory in /Volumes, (eg sudo mkdir /Volumes/LACIE2).
3. Mount the drive explicitly on this mount point (sudo mount -t msdos /dev/disk2s1 /Volumes/LACIE2).
4. Delete ALL .DS_Store files on the disk (cd /Volumes/LACIE2; find . -name 'DS_Store' -exec rm -rf \{\} \;).
5. Wait for a while. At this point you should have two LACIE drives on your Desktop, one fake and one real. Unmount the real one by dragging it to the trash can.
6. Turn of the external hard drive, or detach it.
7. Reboot.
8. Attach/Turn on the external hard drive.

This is a process that works for me, and my drive is now back to working mode. It seems that OSX stores some information in the DS_Store files somewhere on the disk that got corrupted for me. Hopefully this information can help someone else with the same problem.

Finally in London

Yesterday I landed in London, and let me tell you: it's been tough getting here. Packing and moving and arranging all takes lots of time and energy. And it's not helping being sick while pulling it off, either. But I'm finally here, and right now sitting at a hotel room in Holborn, close to Russel Square. Hopefully I'll be able to land an apartment soon too, and then I'll get back to my former speed, hopefully.

In the meantime, tomorrow is my first day at ThoughtWorks. It's bound to be interesting, but I don't expect any downtime. We have so many interesting things going, that it will be full speed ahead from day one.

Although I'm more tired than I should be, if someone feels like meeting up this or next week for a beer and talk JRuby, it would be fun.

Burning cycles: the next month

Getting back from RailsConf meant getting thrown into loads and loads of preparation work for London. Oh my. Since I'm moving in 8 days, and almost all packing needs to be done during the weekend (and I haven't begun yet), I don't think I'll be able to be that communicative. Further, I will have no access to a computer or cell phone from Wednesday to Monday, so if I don't answer email, don't get upset. =) That incidentally means I won't blog either.

June 4th will be my first day at ThoughtWorks and I'm incredibly excited by it. Hopefully I'll be able to blog some about my experiences, but I guess the first few days will be focused on getting setup and finding an apartment (If you know any nice place close to Camden Town that's available by the beginning of June, please do tell), and doing all manner of things.

If I'm unlucky I'll need to go to DC the week after that, but we'll see. Hopefully I can wrangle out of it, since Stella (my girlfriend) will arrive in London that weekend.

After that week I'll have a week of quiet and solitude, and then it's time for TheServerSide Java Symposium in Barcelona. I will in fact be involved in no less than three events during this conference. First, a technical session on JRuby, secondly a BoF about deploying JRuby on Rails applications, and third a panel discussion called "2020: A Developer's Odyssey Panel", which is bound to be interesting. I have no idea whatsoever about what I'll say yet, but Martin Fowler will be one of the panelists which makes it a certainty of fun. I'm looking forward to TSSJS for a few more reasons; seeing Barcelona, since that is said to be a lovely city, getting to say hi to Dr. Heinz Kabutz, meet up with Jonas Bonér from Terracotta and continue discussing how to utilize Terracotta and JRuby together. I'm looking very much forward to the whole event, in fact. If you are there, don't hesitate to say hi!

That gender thing

One of the foci on both JavaOne and RailsConf was the so called "gender problem"; that the current balance between men and women in the technological field is bad and that something needs to be done about it. I'm very happy about this getting awareness, but as my colleague Lars Westergren writes here (http://slashdot.org/~LarsWestergren/journal/172323) it seems like total hypocrisy when looking at the so called entertainment provided at these two events. We need to do better than this.

The ThoughtWorks story

As you know, I will be starting at ThoughtWorks Studios in June, and I've been to JavaOne and now RailsConf, head on head. There is something I really need to tell you right now, and something that I've felt very strongly this week and last week. As I said, I'll start at TW in two weeks. My interview process was basically like any interview process; I met lots of TW people, but I couldn't relate to them that much, because being part of the interview process made it very hard. My mind was focused on other things.

Martin Fowler said in one of his bliki entries (here), that people matter most, and: "I decided then that I wouldn't work with unpleasant people, however capable they might be.". And he then goes on and says that this was one of many reasons he actually joined TW. TW has a "no assholes" policy. I didn't actually get the reality of this (I mean seriously, would you believe that such a thing actually could be?), until RailsConf when I met something like 30-40 TWkers. I've spent much time with Roy, Chad Wathington, Cyndi Mitchell, Julian Boot, Alexey Verkhovsky, Desi McAdam and so many other people that I can't remember their names. But what I do remember is that I felt right at home with them; these are people I can relate to, I can discuss with them, I can agree or disagree with them but regardless I will have a good time. It feels very much like a family, and I feel so much at home going to TW. I think that this is the best thing that could ever happen me. I have no illusions about this; I know for a fact that most of these people are smarter in every way than me. That is totally fine, that means I can learn great things from them.

So, what I wanted to say is just that I feel great about this. I'm so excited, and it's going to be a great time; not only for me and TW, but for the Ruby and JRuby community.

Well, I just wanted all the TWkers I've met these days to know that I feel so great about this. I've never felt as welcome, and I've never found a place with such great people. ThoughtWorks is a radically different company, and it's the people who does that. ThoughtWorks is extremely different, in a very good way.

I will need to blog about the last day of RailsConf, and more overall feelings and stuff like that, but I'll do that when I get back to Sweden.

Some RailsConf impressions

So, I've spent lots of time talking to people about numerous things. We have had some really nice conversations about everything related to Ruby, JRuby and ThoughtWorks and there is an incredible buzz going on here. Yesterday me, Roy, Tim (Bray), Cyndi, Martin Fowler, Charles, Tom and Nick sat down over dinner with DHH and talked about JRuby, which was also very interesting.

All in all, it's very exciting, and I hope it will just continue to be. I'm looking forward to the rest of the day, and tomorrow is going to be a new story.

ThoughtWorks, Mingle, RubyWorks and JRuby

So this is something that's been brewing for a while now, and it's all very exciting and was announced this morning at the keynote by Cyndi Mitchell. The relevant points are Mingle (which I've talked some about already), RubyWorks which is an umbrella for the things the enterprise needs to make Ruby viable. The first product out of RubyWorks is an installable package which gives you the deployment story basically for free. It's quite awesame and ThoughtWorks will offer 24/7 support for it from June.

For me personally, the most important part is that ThoughtWorks will also offer JRuby support 24/7 from June. That's right, 24/7 JRuby support. Wow. ThoughtWorks does believe in JRuby, they think it's something really important, and we want everything to get this.

Find out more at http://rubyworks.thoughtworks.com.

David about Rails 2.0

The DHH keynote this year was really good for a few reasons. First of all, David was very quick in saying that Rails 2.0 will not be anything radical, nothing really new, just better. Also, he actually announced that he had changed his mind about HTTP authentication, and performance (and this is kinda amazing of him to say... =). But all in all, it was a good talk about what's coming next; an evolution of the current things, but still the same Rails, just with more REST support and other goodies.

JRuby RC2

The day before RailsConf started, me, Charles and Tom sat in the lobby of the Courtyard, coding away, fixing problems and pushing out something like 5 versions of 1.0RC2. But the final RC2 is really, really good. You should download it, test it, and if you're at RailsConf, report any bugs you find and get a cool t-shirt! JRuby is really making a huge buzz at RailsConf right now, and it's sort of overwhelming! I'm very happy about it, of course, and there is still some nice things happening. Yesterday, we got a small bug in truncating files fixed, which means exclusive file locking works now.

1.0 is looking really, really good. It should be out by the end of May, and it's going to be a smash.

RSpec 1.0

Yesterday, the RSpec team pushed out version 1.0 which I'm personally very happy to hear. RSpec is a really important project and we are seriously considering including RSpec in the base JRuby distribution. Congratulations on the 1.0 release!

Congrats to Ward Cunningham

I met Ward Cunningham for the first time this Thursday, but I've always been very impressed by the things he has done. And yesterday he got a new job at AboutUs, a wiki company founded in 2005, as CTO. Congratulations!

A scattering of posts from RailsConf

So, I've been really bad about blogging from RailsConf. I have basically not done it at all, and I'm going to change that right now. There are loads of interesting stuff happening here, and I plan to talk about several different things. I won't do it in one post, though. Instead, I'll post lots of small posts with specific intent, and maybe one or two overall posts.

But I really must say, the feeling of being here, among 1600 developers in a vibrant and very cool community feels great. I've met lots of nice people (to many to remember, in fact), and I've had many great discussions.

JavaOne day 4: the final friday

So, the day started out quite late (since I was way tired after last night...). First session was a very accomplished, though slightly shallow, comparison between doing an application with Java EE 5, Ruby on Rails and Grails. I didn't really learn anything new in this presentation except that Java EE 5 is even slicker with NetBeans, and that Grails is maddeningly, sickeningly, seriously slow when doing simple stuff like creating scaffolds. I can't understand why this is so, since it isn't much do it. Basically just create a file with a few rows of customization.

After that I paired up with Jon to look at Mingle again. Oh boy, the ones who is coming to RailsConf will get a treat, that's for sure.

The Jython session by Otmar Humbel was really good, and had a great example. Basically the whole demo sessions were done in one single application which he started at the beginning and never stopped. He just hotswapped the Jython code dynamically. Very nice. I think the most interesting part of that talk was in the Q&A when someone asked why you should use Jython instead of Groovy or JRuby. I'm sad to say that I had to literally run at that point, so I didn't hear the answer... But still, if someone was there and heard, please tell me. For me personally, it's very impressive that Jython got started in 97, and actually is still alive (due to a recent revival of course, but even so. I'm glad Charles gave up one of his talks to let the Jython guys in). It was also interesting to see the differences in philosophy between JRuby and Jython in terms of Java integration and things like that. Very good stuff.
'
The bytecode manipulation talk was interesting; it was nice to see what (and how) Terracotta does with ASM. The TopLink parts were nice, but really way to basic to be interesting. Charles stuff was good, of course. We know Charlie always does good things, don't we? =)

I had great fun at the Java Puzzlers this year. I'm happy about there not being many puzzles incorporating generics, because those are usually boring. The main problem that these guys always exploits, seems to be mostly in the boundary between Java Objects and primitives, and interactions between primitives. That's what you get when you try to create a language meant for both system development and application development. It just shows in all the seams. A good language should not have seams, unless they're necessary for the specific domain of that language.

Finally, Rob Harrops talk about exploiting JRuby to create DSL's was really good. It was great, and Rob is a very accomplished presenter. I had great fun. That said, I think that some of the stuff still went over the head of most Java developers in the audience.

And that seems to be the theme. Most things I enjoy, I think most other people didn't enjoy. Interesting, that.

So, now it's off for dinner at the Stinking Rose with the JRuby guys, Dion, Barry Burd and Jon.

JavaOne day 3

The third day of the conference started slow, due to my being dead tired from the day before. But I still managed to get to Charles and Tom's JRuby on Rails presentation. Obviously, this was a very good presentation and JRuby is very important. But you know all that, already, don't you?

I then went on to a session giving a technical overview over GlassFish v2. Now, this seemed interesting, but wasn't really so. There was some nice information about how clustering should be setup so you don't lose to much data if you have more than one instance on the same machine, but other than that... Nah.

After lunch, I went to the Advanced Groovy talk, expecting to see cool overridings of the MetaObjectProtocol, or how to implement your own AST shufflers. But no. What we instead got was some tidbits here and there, and explanation of how to add new methods to existing Java objects (and this is really gross, you have to wrap the usage of these methods in a block, and the methods are defined on a new class as static methods...) Anyway, on to the demos. I got a real feeling of Deja Vu, since I had seen this exact same demo last year... XMLRPC communication between two Groovy instances, then ActiveX usage of Excel, combining it with Swing. So, yeah. As someone else said "It's a really great Excel demo, but what has that got to do with Groovy?".

Next, I took it easy for a while, walked around and did some programming. Finally, it was time for Tor's and Martin's talk about Ruby Tooling in NetBeans. And boy, it was an outstanding talk if you're interested in compilers, type inference, the challenges of dynamic languages and other stuff like that. I loved it, but I think most of it went over the head of most people in the audience.

Then I was of to the Sun Certified Professionals party which is always nice. I met some cool people and Lars won good swag by being smart really fast. Good for him!

The After Dark bash was quite allright, but nothing extraordinary. And it got really tasteless when the showgirl in metal corselet came onto stage with a grinder, which she turned on herself, making sparks fly all over the place.

After that, we went to the JRubME talk, about JRuby on Symbian and Java ME. A great talk and a great project. In fact, it's really really cool, and if you're interested in ME, you should download the source and start hacking a way on it! The project needs contributors.

We went back to the After Dark bash finally, took some more beer, and then decided to move on. The quest was for Aquavit, and someone thought that the Starlight Room would be a good place for this, so we headed there just to find out they didn't have either Aquavit nor any good scotch. Oh well, it was fun anyway, and the view was fantastic. Characteristically, me, Tom and Charles got into a real interesting discussion about the subjectivity of self presentation, and representation of knowledge in terms of probabilities and deltas from ideal knowledge, and how this applies to onthologies. Neat stuff. But maybe the dance floor wasn't the best place to have this loud discussion...

After that we settled down and decided it would be fun to have a JRuby summit in Minnesota! So, we'll see if that is possible... Then I crawled home to my hotel. My feet are killing me today, but there are some really interesting presentations, so I've gotta go.

JavaOne day 2

So, my day didn't start out with the Oracle general session, which I'm quite happy with in retrospect. Instead the first talk I saw was "Quick and Easy Profiling with Integrated Tools", about NetBeans profiling support. Very useful stuff, actually, and 6.0 can do some amazing things.

After that, it was time for a packed Josh Bloch's Effect Java Reloaded, except it wasn't really reloaded that much yet. I was a little bit disappointed, since most of the stuff was the same as last year. Some parts about the Builder pattern, much info about do's and don'ts with generics (much of it having to do with type parameters and wildcards). The TypeRef pattern was interesting, but not something I will find very much use of, I think.

After lunch a went to a presentation which I really thought would give me something. "Ruby on Rails Meets the World of Enterprise Applications". Now, I really hoped this would include some information on the Enterprise problem with regard to Rails. I was very disappointed. The talk was basically about a 3-week application written in Rails that connected to SAP. So the enterprise in question was there because of SAP. That said, the presenter was good, it was just not the subject I had wished for. And by the way, what does this presentation do on JavaOne?

After that, I spent some time looking at Mingle issues with Jon. I decided to go to the upper Haights' area to shop, and then get back to my hotel room for some programming. As it turned out, I really didn't go to any more sessions yesterday. After programming, I went to a Swedish Java User Group meeting here (and what a turnout! it seems like there's over 200 Swedes at JavaOne this year.)

The plan was to get back to the BOF's after that point; there was especially one I really didn't want to miss (the Dynamic Languages BOF with Frank Cohen). But, I happened to get invited to the Google party, so I spent the rest of the evening there, meeting lots of interesting people. It was great fun.

And now I'm about to go in and see Charles and Tom do the JRuby on Rails presentation. More information later.

The RedMonk unconference at CommunityOne

So, as you know, RedMonk arranged an unconference at JavaOne/CommunityOne this year. I couldn't attend all sessions, but among the ones done were one about dynamic languages, and we got some really good discussions going. We haven't really finished those discussions yet, though, and it seems we will take it onto a mailing list, but it was a very good environment to start out.

I really hope RedMonk will be able to do this next year too. More info about it can be found here: http://redmonk.com/wiki/index.php/RedMonkUnconference.

JavaOne: The rest of day 1

I am sitting in the alumni lounge, waiting for the sessions of day 2 to begin. I was thinking about attending the Oracle General Session, but decided to take it easy for a while instead.

So, where were I yesterday? Well, the Technical General Session was really interesting. As I mentioned yesterday, Charles and Tor did a great demonstration that was very well received. We also got some more on the technical side of JavaFX. Very nice.

After that, I attended Web Algorithms. It was a very accomplished presentation, detailing a few important things you can use to make life in computing easier, beginning with swap-XOR and credit card validation, going through public key cryptography and looking at Google MapReduce. It was a good presentation, but nothing new in it for me though, which was sort of sad.

The next presentation I went to was about the next generation of Web support in Java. The Servlet 3.0 specification and so on. The presentation was quite vague and didn't say much new things, really. Ok, so next generation servlets will use annotations, and there will be support for Rest style things and better security and more non-idiotic defaults? Not surprising.

After that presentation I was feeling really tired, so I went back to my hotel and tried to rest some. That didn't go so well so I finished writing chapter 11 instead. Then it was back to Moscone center to see a BOF on web development in Java EE compared to Ruby on Rails, with tool support. This talk was quite disappointing; it wasn't that well executed, and it was very un-nuanced in detailing the good and bad parts of Ruby on Rails. It seems that when people do these comparisons they just talk about how easy it is to do CRUD-style applications, but seems to forget that Ruby on Rails can do harder things (http://studios.thoughtworks.com/) and that the benefits from the Ruby language in development productivity scales. If you compare doing a small-sized web app in Java and Rails to each other, you will get some kind of percentage. But if you do a comparison between a medium-sized app in Java and Rails, that percentage will be greater, since the Ruby languages scales development time in a totally different way from Java. Not to mention maintainability after the fact.

Last I went to Neal Gafters BOF on Closures in Java. I really like the way Neal does presentations, but we couldn't really get into the meat of stuff, since half the people on the BOF didn't know closures from their elbows. So most of the presentation was spent rehashing what they are and why they're needed in Java.

After that, I was beat. And now it's a new day, started with breakfast meeting with Roy Singham. Very interesting and entertaining. I'm more and more convinced that I'm going to love working for ThoughtWorks.

JRuby on the technical general session

Charles O Nutter and Tor Norbye just got on stage here, and showed of deploying Mephisto as a WAR-file, and then changing it to add text-to-speech functionality, with a Java library that generates an audio file. The total code they wrote on stage was about 10 lines of Ruby... The power of Java and Ruby together: it's beautiful.

I already knew what they would do, having helped fix many of the issues getting in the way for Mephisto, but it's still incredibly cool. Charles and Tor got several impromptu applauds from the audience too, so I'm pretty sure people think it's neat.

Right now, I'm starting to get curious why people continue to use the term "scripting language". It doesn't seem to fit either Ruby nor Groovy anymore. Oh well. Nitpicking.

Next up is Arun Gupta. He's talking about jMaki and Phobos, both very nice usages of JavaScript and other languages.

JavaOne: Keynote and Groovy session

I'm sitting in the general session hall, waiting for the first technical general session to begin. Yesterday was CommunityOne, where we had some interesting discussion about dynamic languages within the context of the RedMonk Unconference. Aside from that, the best parts of the day way announcing that Mingle runs on JRuby, and meeting up with all my soon-to-be fellow ThoughtWorkers. They're a great bunch of people, and we had good fun.

Today was the opening general session, where Rich Green announced some very interesting developments. Among these are the fact that the open sourcing of Java is now complete, that Sun is coopering quite heavily with UN to provide resources for education to areas of the world where this have long been a huge problem. But the most important announcements were about something called JavaFX. I can't really say I understand it completely yet, but it seems to be an effort to tackle Microsoft Silverlight, and also fix several deficiencies in Swing by providing JavaFX Script (which looks very much like F3. I'm not sure if it actually is F3 or something else.). This obviously begs the question why Sun finds it prudent to invent a new language, instead of using one of the many great efforts existing in the dynamic language communities for this problem. For example, both Groovy, Jython and JRuby have different versions of SwingBuilders, which allow you to rapidly create Swing interfaces with a specialized DSL for this.

Except for that, the general session and keynote was more or less like usual. Very flash, very markety, but still more technical than I imagine other conferences are. And it was fun to see Rich Green being compared to Steve Jobs...

The first technical session I went to was called "Cool Things You Can Do with the Groovy Dynamic Language", and was presented by Guillaume Laforge and Dierk König. It was a quite good session, but I can't get away from my general opinions about the Groovy language. So, before I say something about the presentation, I need to describe my feelings for Groovy. Remember, these are my personal opinions, and some are definitely based on feeling without any specific rationalization.

I would really like to like Groovy, but I can't. I've really tried, but I can't find the Groovy language to my liking. And believe me, for some circumstances, Groovy should be able to fill the gap between Ruby and Java better than JRuby, at least in some cases. The Java integration in JRuby is quite hard, dispatch and overloading makes calling Java code complicated (from an implementation point, not for the user). All of this would be much simpler with the Groovy approach. But no, I still can't get along with Groovy. The main reason, I believe, is the feeling I get from all code that the language Groovy have grown piecemal, adding stuff that's neat wherever. I'm not sure this is the actual way Groovy was designed, but it feels like a modern version of Perl. The syntax doesn't mesh, and there are numerous (small, but nonetheless there) inconsistencies in how things are handled; many of the things that the language provide for you is things that really shouldn't be part of the language, but part of a library instead.

So, from this point of view, the presentation walked through several new features of Groovy, and lots of things you can do with it. They talked a bit about the new support for annotations and the plans for generics. Some of it was quite cool; I like the builders (but I prefer Markaby to the XMLBuilder), and some of the features are quite handy. But looking back at my earlier feelings I still see a design process more focused on finding problems with Java, and duct taping them with Groovy. (Like this: in Groovy, all exceptions get handled automatically, making it look as if all exceptions are unchecked.) I have no problem with many of these features, but I don't think it's a good way to create a general purpose language.

I'll be back later with more info on the rest of the days sessions.

Mingle on JRuby

Here's the deal: ThoughtWorks first product offering is called Mingle. The kicker is, it will be deployed on JRuby. I have been involved with this for a while, getting everything in good working order, and I'm sure TW won't regret this decision. JRuby is important in their strategy, and Mingle, as the first commercial JRuby application is a huge deal.

You should go and read more about this exciting news at http://studios.thoughtworks.com/2007/5/7/mingle-to-run-on-jruby.

Announcing mongrel_jcluster

As I mentioned in my last post, that server/client thing wasn't the only thing I worked on during the flight. I also did a very useful hack of mongrel_cluster, and turned it into mongrel_jcluster. It's not very large changes, actually... Almost everything is the same, except that this mongrel plugin only runs in JRuby, and will start several mongrels in the same JVM.

jruby -S gem install mongrel_jcluster
cd your_rails_app
jruby -S mongrel_rails jcluster::configure -p 4000 -N 3 -e production -R 20202 -K thesecretkey
jruby -S mongrel_rails jcluster::start

The only difference in the configuration parameters are -R and -K which is the JRuby server port and key respectively. Now, after executing these commands, it will take some time for the mongrels to get online, so be patient (or monitor the progress in the log-files generated).

I need to warn you that this is still quite experimental and not guaranteed to work in any way. =) But it does for me.

Another thing, if you start Mongrel in production mode, the defualt Rails front-page will give you an error if you try to get at the properties. This is the expected behavior in production mode, though, and nothing wrong with JRuby.

JRuby server

I spent some time on the flight to SF hacking on an idea I've had for a while. The result was checked in yesterday in JRuby trunk, and the important files are bin/jrubysrv and bin/jrubycli. This is still quite experimental, and only guaranteed to work in *NIX environments right now.

So, what is it? Well, if you don't want the overhead of running one JVM for each JRuby process, these two commands allow you to start a JVM that listens on a specific localhost port, and then you can send JRuby commands to that port. You need a key as password, otherwise the process will fail. Right now, this isn't as high security as it should be, since the key is passed as a command line parameter. This is still not that big of a problem, since the most common scenario is to start a bunch of things, and then not start anything more. Let me illustrate what you can do with a very simple example.

Say that you have a Rails application, and Mongrel for JRuby installed. Go the the base directory of your Rails app and do this:

jrubysrv -p 20202 -k secretkey &
jrubycli -p 20202 -k secretkey -S mongrel_rails -p 4000
jrubycli -p 20202 -k secretkey -S mongrel_rails -p 4001
jrubycli -p 20202 -k secretkey -S mongrel_rails -p 4002
jrubycli -p 20202 -k secretkey -n

Now, if you are patient, in a while there will be three Mongrels running, listening on ports 4000, 4001 and 4002. And they will run inside the same JVM, but in different JRuby runtimes. Quite neat, and very simple. You may wonder about the last command, with the -n flag. That flag tells the JRuby server running at port 20202 to not start any more JRuby processes. In this way, you can start what you need, and then close it down. Due to problems with thread scheduling and safety, I haven't implemented a way to kill a single runtime. What you instead need to do is use jrubycli with the -t flag, which will kill the whole JRuby server.

Stay tuned for my next post, on what I've actually used this functionality to create. (If you have some slight amount of imagination, you should be able to guess from this blog post...)

But now, of to the races. Or CommunityOne in this case. See you there.

San Francisco and Portland

Tomorrow morning I'll leave Stockholm for San Francisco. I'll be arriving 5:ish PM at SFO and will stay at the Handlery Union Square Hotel. I will not present at either JavaOne or RailsConf, though, so I will have a fair amount of time for fun stuff. I will attend the RedMonk unconference at CommunityOne, participate in the DynLang event, and then attend regular JavaOne. If you see me (I'll be wearing my black hat...), please say hi!

Next Saturday me and my coworkers will move out of SF and take a car to Portland, where we plan to arrive on the following Tuesday or Wednesday, just in time for RailsConf. We leave from Portland early Monday after the conference.

Hope to see many of you there! It will be exciting, that's for sure. I will also try to report as much as possible from all events.