A JRuby Rubinius machine

When I get bored with JRuby, I tend to go looking either at other languages or other language implementations. This happened a few days ago, and the result is what I will here document. Begin by creating a file called fib.rb:

def fib(n)
 if n < 2
   n
 else
   fib(n - 2) + fib(n - 1)
 end
end

p fib(15)

The next part requires that you have a recent version of Rubinius installed:

rbx compile fib.rb

This will generate fib.rbc. Next, take a recent JRuby version and run:

jruby -R fib.rbc

And presto, you should see 610 printed quite soon. This is JRuby executing Rubinius bytecode. I was quite happy about how it was to get this far with the functionality. Of course, JRuby doesn't support most bytecodes yet, only those needed to execute this small example, and similar things. We are also using JRuby's internals for this, which means that Rubinius MethodContext and such are not available.

Another interesting note is that running the iterative Fib algorithm like this with -J-server is actually 30% faster than MRI.

This approach is fun, and I have some other similar ideas I really want to look at. The best part about it though, is that I got the chance to look at the internals of Rubinius. I hope to have more time for it eventually. Another thing I really want to do some day is implement a Jubinius, which should be a full port of the Rubinius runtime, possibly excluding Subtend. I think it could be very nice to have the Smalltalk core of Rubinius working together with Java. Of course, I don't have any time for that, so we'll see what happens in a year or two. =) Maybe someone else does it.

Evil JRuby

After my last post I got several comments about evil.rb. Of course I had evil.rb in mind when doing some of it, but I also forgot to describe the two most evil methods of the JRuby module: runtime and reference. The runtime method will return the currently executing JRuby runtime as a Java Integration, meaning you can get access to almost anything you want with it. For example, if you want to take a look at the global CacheMap (used to cache method instances):

require 'jruby'
JRuby::runtime.cache_map

Whoops. And that's just the beginning. Are you interested in investigating the current call frame or activation frame (DynamicScope in JRuby):

require 'jruby'
p JRuby::runtime.current_context.current_frame
a = 1
p JRuby::runtime.current_context.current_scope

Of course, you can call all accessible (and some inaccessible) methods on these objects, just like if you were working with it from Java. Use the API's and take a look. You can change things without problem.

And that also brings us to one of the easiest examples of evil.rb, changing the frozen flag on a Ruby object. Well, with the reference method, that's easy:

require 'jruby'

str = "foo"
str.freeze

puts str.frozen?
JRuby::reference(str).setFrozen(false)
puts str.frozen?

JRuby::reference will return the same object sent in, wrapped in a Java Integration layer, meaning that you can inspect and modify it to your hearts like. In this way, you can get at the internals of JRuby in the same way you can using evil.rb for MRI. And I guess these features should mainly be used for looking and learning about the internals of JRuby.

So, have fun and don't be evil (overtly).

Some JRuby tricks

I have spent a few hours adding some useful features these last days. Nothing extraordinary, but things that might come in handy at one point or another. The problem with these features is that they are totally JRuby specific. That means you could probably implement them for MRI, but noone has done it. That means that if you want to use it, beware. Further, they exploit a few tricks in the JRuby implementation, meaning it can't be implemented in pure Ruby.

So, that was the disclaimer; now onto the fun stuff!

Breaking encapsulation (even more)
As you know, in Ruby everything is accessible in some form or another, and you can do almost everything with the metaprogramming facilities. Well, except for one small detail which I found out while working on the AR-JDBC database drivers.

We have some code there which needs to be separate for each database, and it just so happens that core ActiveRecord have already implemented them in a very good way. So, what do we do? Mix in them and remove the methods we don't want? No, because ActiveRecord adapters are classes, not modules, and you can't mix in classes. There is no way to get hold of a method and add that to an unrelated other class or module. Except if you're on JRuby, of course:

require 'jruby/ext'

class A
def foo
puts "A#foo"
end
def bar
puts "A#bar"
end
end

class B;end

class C;end

b = B.new
b.steal_method A, :foo
b.foo
B.new.foo rescue nil #will raise NoMethodError

C.steal_methods A, :foo, :bar
C.new.foo
C.new.bar

Of course, using this should be avoided at all costs. But it's interesting that such a powerful thing can be implemented using about 15 lines of Java code.

Introspection
JRuby parses Ruby code into an Abstract Syntax Tree. For a while now, the JRuby module have allowed you to parse a string and get the AST representation by executing:

require 'jruby'

JRuby.parse "puts 'hello'", 'filename.rb', false

This returns the Java AST representation directly, using the Java Integration features. That is old. What is new is that I have added pretty inspecting, a nice YAML format and some navigation features which makes it very easy to see exactly how the AST looks. Just do an inspect or to_yaml on an AST node and you will get the relevant information.

That is interesting. But what is even more nice is the ability to run and use arbitrary pieces of the AST (as long as they make sense together) and also run them:

require 'jruby'

ast_one = JRuby::ast_for("n = 1; n*(n+3)*(n+2)")
ast_two = JRuby::ast_for("n = 42; n*(n+1)*(n+2)")

p (ast_one.first.first + ast_two.first[1]).run
p (ast_two.first.first + ast_one.first[1]).run

As you can see, I take two fragments from different code, add them together and run them. You can also see that I'm using an alias for parse here, called ast_for. That makes much more sense when using the second parse feature, which we already know from ParseTree:

require 'jruby'

JRuby::ast_for do
puts "Hello"
end

Well, I guess that's all I wanted to show right now. These last small things I've added because I believe they will be highly useful for debugging JRuby code.

I also have some more ideas that I want to implement. I'll keep you posted about it.

ObjectSpace: to have or not to have

Among all the features of Ruby that JRuby supports, I would say that two things take the number one place as being really inconvenient. Threads are one; making the native threading of Java match the green threading semantics of Ruby is not fun, and it's not even possible for all edge cases. But that argument have been made several times by both me and Charles.

ObjectSpace now, that is another story. The problems with OS are many. But first, let's take a quick look at the most common usage of OS; iterating over classes:

ObjectSpace::each_object(Class) do |c|
  p c if c < Test::Unit::TestCase
end

This code is totally obvious; we iterate over all instances of Class in the system, and print an inspected version of them if the class is a subclass of Test::Unit::TestCase.

Before we take a closer look at this example, let's talk quickly about how MRI and JRuby implements this functionality. In fact, having this functionality in MRI is dead easy. It's actually very simple, and there are no performance problems of having it when it's not used. The trick is that MRI just walks the heap when iterating over ObjectSpace. Since MRI can inspect the heap and stack without problems, this means that nothing special needs to be done to support this behavior. (Note that this can never be safe when using a real threading system).

So, the other side of the story: how does JRuby implement it? Well, JRuby can't inspect the heap of course. So we need to keep a WeakReference to each instance of RubyObject ever created in the system. This is gross. We pay a huge penalty for managing all this stuff. Many of the larger performance benefits we have found the last year have revolved around having internal objects be smarter and not put themselves into ObjectSpace until necessary. One of my latest optimizations of regexp matching was simple to make MatchData lazy, so it only goes into OS when someone actually uses it. RDoc runs about 40% faster when ObjectSpace is turned off for JRuby.

So, is it worth it? In real life, when do you need the functionality of ObjectSpace? I've seen two places that use it in code I use every day. First, Rails uses it to find generators, and secondly, Test::Unit uses it to find instances of TestCase. But the fun thing is this; the above code is almost exactly what they do; they iterate over all classes in the system and checking if they inherit from a specific base class. Isn't that a quite gross implementation? Shouldn't it be possible to do something better? Euhm, yes:

module SubclassTracking
  def self.extended(klazz)
    (class <<klazz; self; end).send :attr_accessor, 
                                    :subclasses
    (class <<klazz; self; end).send :define_method, 
                                :inherited do |clzz|
      klazz.subclasses << clzz
      super
    end
    klazz.subclasses = []
  end
end

# Where Test::Unit::TestCase is defined
Test::Unit::TestCase.extend SubclassTracking

# Load all other classes

# To find all subclasses and test them:
Test::Unit::TestCase.subclasses

I would say that this code solves the problem more elegantly and useful than ObjectSpace. There are no performance degradation due to it, and it will only effect subclasses of the class you are interested in. What's the best benefit of this? You can use the -O flag when running JRuby, and your tests and rest of the code will run much faster and use less memory.

As a sidenote: I'm putting together a patch based on this to both Test::Unit and Rails. ObjectSpace is unnecessary for real code and the vision of JRuby is that you will explicitly have to turn it on to use it, instead of the other way around.

Anyone have any real world examples of things you need to do with ObjectSpace?

First two days of TSSJS

It has been two long days; not because I've been going to sessions all day long, but because I've reworked my presentations quite heavily. But now both the BOF and the TS are finished, and I think they went well. I had to keep the level to Ruby, JRuby and Rails introductionary material, though, since most developers here didn't seem to know what is possible with these technologies.

But it's been great; I've gotten good feedback and had some really interesting conversations with lots of people.

We have been doing the town each night, and I've found that I like Barcelona very much. Except for the food: this country doesn't seem to be good for vegetarians at all. Very annoying. I'm going for beer and wine instead of food the rest of the week. =)

One day left, though, and it's bound to be nice. Me and Martin are both on a developers panel about the state of programming languages in 2020; I have no idea what to say, and I'm thinking about just ad-libbing it. I know my own position in these questions fairly well, and the current Yegge-debate have made my opinions even more explicit.

But now it's time to see the town again.

Interviewed by AkitaOnRails

Yesterday I spent 2 hours chatting with Fabio Akita, of AkitaOnRails (the largest Rails blog in Brazil); the result is a long interview that was published today. It's got some good stuff, and some Ola-stuff, which you should recognize by now.

And I note that he calls me a workaholic; but he got this interview prepared in less than a day too, and also translated it into Portuguese.

You can find it at http://www.akitaonrails.com/pages/olabini.

First weeks at ThoughtWorks

I've finally started. I've finally moved to London. I've been working for two weeks at ThoughtWorks now, and it's been quite crazy. Everything is very nice and I'm having loads of fun. Of course, it's also lots of hard work, and I feel that I'm stretching my capacity considerably more than I ever did at Karolinska Institutet. That's great, and I feel that I'm really doing something real now. We have so many interesting things going on, and I wish I could tell you all about it.

What I can tell you is that I'm working quite much on Mingle, and I'm also spending time on other JRuby related issues. I've been planning on getting SQL Server and Oracle working as good as possible with AR-JDBC, and I've spent time on Derby performance. Hopefully I'll continue the database work this week, since especially SQL Server and Oracle is very important.

The most important work for this week is probably to prepare for TheServerSide in Barcelona. I still haven't had time to prepare my demos, so it's about time now. I hope to see many of you there.

In conclusion, my first weeks at ThoughtWorks have been awesome. I really like the pople, and everything is just neat. I like being able to walk to work and working in the very nice TW office on High Holborn. I'm very happy about it all.

Book update

As you know, I am writing on a book about JRuby on Rails. A few minutes ago I finished the first draft of chapter 14. That means that there are just 3 chapters and 3 appendixes left to write (chapter 1, 2 and 15). So the writing is going very well, but it's taking a heavy toll on me personally. I seriously don't recommend writing a book like this in your spare time, while at the same time switching employer, moving abroad and try to be a core developer in an open source project which is getting lots of attention.

So, in summary: it's going well, it still looks like it will be out in October, and I'm deadly tired.

JRuby 1.0

The JRuby community is pleased to announce the release of JRuby 1.0!

Homepage: http://www.jruby.org/
Download: http://dist.codehaus.org/jruby/

JRuby 1.0 is a major milestone for our project. Our main goal for 1.0 has been
Ruby compatibility. We feel this goal has been reached. When we see
companies like ThoughtWorks offering commercial support; we know this goal
has been reached. Please download JRuby and take it for a test drive. Try
running your Ruby and Ruby on Rails applications with it.

Give us feedback. Join our community. Help us continue to improve JRuby.

It is important to notice that JRuby 1.0 is not the end all of Ruby interpreters. It's not perfect. This is just the beginning. We now have a very good base to work from. This is were the real work begins. Join us. It will be a fun ride, and JRuby will just get better!

What's wrong with this code?

Today I will introduce to you a method from ActiveRecord. The method takes a parameter called type and that value can bu for example :primary_key, :string or :integer. Now, in the first line there is a call to native_database_types. Generally, that call returns a structure that looks somewhat like this:

def native_database_types #:nodoc:
  {
   :primary_key => "int(11) DEFAULT NULL auto_increment PRIMARY KEY",
   :string      => { :name => "varchar", :limit => 255 },
   :text        => { :name => "text" },
   :integer     => { :name => "int", :limit => 11 },
   :float       => { :name => "float" },
   :decimal     => { :name => "decimal" },
   :datetime    => { :name => "datetime" },
   :timestamp   => { :name => "datetime" },
   :time        => { :name => "time" },
   :date        => { :name => "date" },
   :binary      => { :name => "blob" },
   :boolean     => { :name => "tinyint", :limit => 1 }
 }
end

The method itself looks like this.

def type_to_sql(type, limit = nil, precision = nil, scale = nil) #:nodoc:
  native = native_database_types[type]
 column_type_sql = native.is_a?(Hash) ? native[:name] : native
 if type == :decimal # ignore limit, use precison and scale
    precision ||= native[:precision]
   scale ||= native[:scale]
   if precision
     if scale
       column_type_sql << "(#{precision},#{scale})"
     else
       column_type_sql << "(#{precision})"
     end
   else
     raise ArgumentError, "Error adding decimal column: precision cannot be empty if scale if specified" if scale
   end
   column_type_sql
 else
   limit ||= native[:limit]
   column_type_sql << "(#{limit})" if limit
   column_type_sql
 end
end

There is something very wrong with this implementation. Of course, there could exist many errors here, but what I'm thinking about right now is a violation of the usual way methods should work. And in effect, that problem with this method have caused ActiveRecord-JDBC to implement some very inefficient code to handle this method. And it gets called a lot in ActiveRecord. I'll get back later today with a pointer to what's wrong here, and I will also discuss some of what I've done in AR-JDBC to handle this situation. I hope for many suggestions here! =)

This is what's wrong

I must say, I got some really good responses to my post about what was wrong with the code I posted. Most of those responses concerned the design of the code, and I agree, this part of Rails could have been done much better. But what I was thinking about was actually a bug. And Lars Westegren (my former colleague) nailed it at the first try. Let me show two important excerpts from this code:

column_type_sql = native.is_a?(Hash) ? native[:name] : native

and here:

column_type_sql << "(#{limit})" if limit

Obviously, double left arrow is append, and for all cases where there is a limit, this append will change the String. This is one of the cases where it's kind of annoying that strings are mutable. If I cache away the values that native_database_types should return, then the next time anyone wants a string SQL type, that will generate VARCHAR(255)(255). The next time again, VARCHAR(255)(255)(255). And so one. So either I need to recreate the hash every time, or I need to do a deep clone of it every time. Neither of these options are very good, and it seems the deep clone option isn't fast enough, even when done in Java, so I decided to go with a hash literal instead. Was that the right choice? I don't know. It improves performance, but on the other hand it churns objects and creates new objects all the time. All of this because of some sloppy coding in Rails.

What's the lesson learned? Never modify your arguments, unless that is an explicit part of the contract for that method and part of the documentation.

Testing with JRuby on Rails and ActiveRecord-JDBC

This will be a highly uninflammatory blog post, as contrast to the last one. Specifically, there is a slight problem when running the command

jruby -S rake

in a a JRuby on Rails-application. This problem is caused by some hard coded values in the database Rake definitions for Rails. But don't despair, there is a simple solution to this. It's not as simple as it should be (invisible) but it's easy enough. Provided you have JRUBY_HOME set and your version of AR-JDBC is 0.3.1, execute this command from your Rails application root

cp $JRUBY_HOME/lib/ruby/gems/1.8/gems/ActiveRecord-JDBC-0.3.1/lib/tasks/jdbc_databases.rake lib/tasks

Since the hard coded values are hard to override, the jdbc_databases.rake file just hacks Rake to be able to redefine tasks and then redefines the core tasks. This shouldn't affect a bi-Ruby installation, since the overriding only happens on JRuby, not on MRI. If someone has a better way to do this, please tell me. =)

There can be only one, a tale about Ruby, IronRuby, MS and whatnot

(Updated: added a quote from John Lam about not being able to look at the MRI source code)

After RailsConf in Portland, there has flared up a discussion centered around IronRuby and Microsoft. We discussed many of these points in depth at the conference, and I'll elaborate some on my views on the issues in a bit.

But first I would like to talk some about the multitude of Ruby implementations springing up. I firmly believe that a language evolves in phases. The first phase, germination, is the period where a language needs one consistent implementation (or a spec). It's during this phase when most "alpha geek adoption" happens. Many important libraries are written, but most applications are not in the main economic center. Ruby have been in this phase for a long time, but the fact that new implementations are springing up left and right is a sure sign that Ruby is entering phase 2: implementation. For adoption to happen, there need to exist several competing implementations, all of them good. This is the evolutionary stage, where it's decided what kind of features an implementation should provide. Should we have green or native threads? Are all the features of the original implementation really that necessary? (Continuations, ObjectSpace). Is there cruft in the standard library that needs to be weeded out? (timeout.rb). All of these questions get answered when other people implement the language. The last phase, which I guess could be called adoption, is when the language have several working implementations, all good enough to deliver high end applications on, when many applications are written in the language, and there exists a plethora of libraries, systems and support for the language.

What this means is that for a language to be successful, there needs to exist competing implementations. They need to implement their features in different ways and make different choices during development. Otherwise, the language will die. (This is obviously not enough, since Smalltalk fulfilled this admirably and still never got widespread adoption.). But I still believe it's incredibly important for a language to evolve with many implementations, which is why I find Rubinius, JRuby, YARV and IronRuby to be extremely important projects for the welfare of Ruby. I want Ruby to be successful. I want Ruby to be the next major language for several reasons. But most importantly: I want Ruby to be a better language tomorrow, than it is today. The only way that's going to happen is by having lots of people implement the language.

So, that's enough of the introductory flame bait. This describes one half of why IronRuby is an important project, and why we can't let it fail. The other side of the coin is the same reason JRuby is important. .NET as a platform have some wildly useful features. There are many developers who swear by .NET for good reason. And what's more important, there are lots of large enterprises with such a vested interest in .NET, that they will never choose anything else. Now, for the welfare of all programmers in the world, I personally believe the world would be a better place if those .NET-environments also used Ruby. So that's the other coin of why IronRuby is important.

The most well read blog about the current Microsoft/Ruby controversy is Martin Fowlers article RubyMicrosoft. Go read it now, and then I'll just highlight the points I find most important.

First: John Lam is committed to creating a "compliant" Ruby implementation. I have no doubts that he can do it. But there are a few problems lurking.

For example, what is a compliant Ruby implementation? Since there exists no spec, and no comprehensive test suite, the only way to measure compliance is to check how close the behavior matches MRI. But this have some problems too. How do you check behavior? Well, you need to run applications. But how do you get so far as you can run applications?
What JRuby did was that we looked at MRI source code.

John Lam can not look at MRI source code. He cannot look at JRuby source code. He cannot look at Rubinius source code. If he does, he will be terminated.

So, the next best alternative: accepting patches from the community, which can look at Ruby source? Nope, no cigar. Microsoft is not really about Open Source yet. Their license allows us to look at their source code, and also to fork it and do what we want with it. But that's only half of what open source is about. The more important part is that you should be able to contribute back code without having to fork. You can't do that with IronRuby, since Microsoft is too scared about being sued for copyright infringement.

There was some doubt about Lam actually being banned from looking at MRI source code. This is the first quote that said it is so. It's from the discussion "Virtual classes and 'real' classes -- why?" on Ruby-core, this quote posted at 29/03/07:

Is this how things are actually implemented? (BTW I'm not lazy here - we cannot look for ourselves).

I am going to make a bold statement here. Under the current circumstances, I don't believe it's possible for John Lam and his team to create a Ruby implementation that runs Rails within at least 18 months. And frankly, that's not soon enough.

As I said above, I have all confidence that John can do great stuff if he has the right resources. But creating a Ruby implementation is hard enough while having all the benefits of the open source community.

The two points I want to make with this point is this: The Ruby community must damned well get serious about creating a good, complete specification and test suite. It's time to do it right now, and we need it. It's not a one-man job. The community needs to do it. (And yes, the two SoC projects are a very good start. But you still need to be able to run RSpec to take full advantage of them; and let's face it, the RSpec implementation uses many nice Ruby tricks.)

The second point is simpler: Microsoft needs to completely change how they handle Open Source. Their current strategy of trying to grow it into the organization will not work (at least not good enough). They need to turn around completely, reinvent themselves and make some really bold moves to be able to meet the new world. If they can't do this, they are as dead as Paul Graham claims.

Solving mounting problem on MacOS X

This is a highly specific post, but I thought I'd write about this so that if someone else have the same problem, they can try to solve it my way.

I got a new computer, Intel-based MBP last week. I upgraded it to latest OSX version (10.4.9 I believe), and immediately installed everything I needed. While ending my last job and returning the laptop associated with that post, I made backups to a 500GB USB LACIE hard drive. Very nice indeed, and half my life is now on that hard drive. (I have most of it on other places too, but not so easily accessible).

The first thing I managed to do was to shut off the hard drive without unmounting it correctly from OSX. I got the warning et all, but well, done is done. It was stupid, I know. Starting out with messing up everything. So what happened? Well, the next time I tried to get OSX to find my hard drive, by inputing the USB connection and switching on the power. Nothing happened. The drive would spin up, but no mount points or nice icons on the desktop. After some investigation I found that the Disk Manager HANGS when the LACIE is turned on and connected. I also found that when I switched the power off, my console says something about not being able to repair disk /dev/disk2s1. Interesting. After a few hours investigation on the Internet I despaired, and decided to try my own ingenuity instead.

I won't tell you about everything I did to find this solution. It would get severely boring very fast. So, here is the solution:

1. Attach the device and turn on the power.
2. Open up a terminal and create a new directory in /Volumes, (eg sudo mkdir /Volumes/LACIE2).
3. Mount the drive explicitly on this mount point (sudo mount -t msdos /dev/disk2s1 /Volumes/LACIE2).
4. Delete ALL .DS_Store files on the disk (cd /Volumes/LACIE2; find . -name 'DS_Store' -exec rm -rf \{\} \;).
5. Wait for a while. At this point you should have two LACIE drives on your Desktop, one fake and one real. Unmount the real one by dragging it to the trash can.
6. Turn of the external hard drive, or detach it.
7. Reboot.
8. Attach/Turn on the external hard drive.

This is a process that works for me, and my drive is now back to working mode. It seems that OSX stores some information in the DS_Store files somewhere on the disk that got corrupted for me. Hopefully this information can help someone else with the same problem.

Finally in London

Yesterday I landed in London, and let me tell you: it's been tough getting here. Packing and moving and arranging all takes lots of time and energy. And it's not helping being sick while pulling it off, either. But I'm finally here, and right now sitting at a hotel room in Holborn, close to Russel Square. Hopefully I'll be able to land an apartment soon too, and then I'll get back to my former speed, hopefully.

In the meantime, tomorrow is my first day at ThoughtWorks. It's bound to be interesting, but I don't expect any downtime. We have so many interesting things going, that it will be full speed ahead from day one.

Although I'm more tired than I should be, if someone feels like meeting up this or next week for a beer and talk JRuby, it would be fun.

Burning cycles: the next month

Getting back from RailsConf meant getting thrown into loads and loads of preparation work for London. Oh my. Since I'm moving in 8 days, and almost all packing needs to be done during the weekend (and I haven't begun yet), I don't think I'll be able to be that communicative. Further, I will have no access to a computer or cell phone from Wednesday to Monday, so if I don't answer email, don't get upset. =) That incidentally means I won't blog either.

June 4th will be my first day at ThoughtWorks and I'm incredibly excited by it. Hopefully I'll be able to blog some about my experiences, but I guess the first few days will be focused on getting setup and finding an apartment (If you know any nice place close to Camden Town that's available by the beginning of June, please do tell), and doing all manner of things.

If I'm unlucky I'll need to go to DC the week after that, but we'll see. Hopefully I can wrangle out of it, since Stella (my girlfriend) will arrive in London that weekend.

After that week I'll have a week of quiet and solitude, and then it's time for TheServerSide Java Symposium in Barcelona. I will in fact be involved in no less than three events during this conference. First, a technical session on JRuby, secondly a BoF about deploying JRuby on Rails applications, and third a panel discussion called "2020: A Developer's Odyssey Panel", which is bound to be interesting. I have no idea whatsoever about what I'll say yet, but Martin Fowler will be one of the panelists which makes it a certainty of fun. I'm looking forward to TSSJS for a few more reasons; seeing Barcelona, since that is said to be a lovely city, getting to say hi to Dr. Heinz Kabutz, meet up with Jonas Bonér from Terracotta and continue discussing how to utilize Terracotta and JRuby together. I'm looking very much forward to the whole event, in fact. If you are there, don't hesitate to say hi!

That gender thing

One of the foci on both JavaOne and RailsConf was the so called "gender problem"; that the current balance between men and women in the technological field is bad and that something needs to be done about it. I'm very happy about this getting awareness, but as my colleague Lars Westergren writes here (http://slashdot.org/~LarsWestergren/journal/172323) it seems like total hypocrisy when looking at the so called entertainment provided at these two events. We need to do better than this.

The ThoughtWorks story

As you know, I will be starting at ThoughtWorks Studios in June, and I've been to JavaOne and now RailsConf, head on head. There is something I really need to tell you right now, and something that I've felt very strongly this week and last week. As I said, I'll start at TW in two weeks. My interview process was basically like any interview process; I met lots of TW people, but I couldn't relate to them that much, because being part of the interview process made it very hard. My mind was focused on other things.

Martin Fowler said in one of his bliki entries (here), that people matter most, and: "I decided then that I wouldn't work with unpleasant people, however capable they might be.". And he then goes on and says that this was one of many reasons he actually joined TW. TW has a "no assholes" policy. I didn't actually get the reality of this (I mean seriously, would you believe that such a thing actually could be?), until RailsConf when I met something like 30-40 TWkers. I've spent much time with Roy, Chad Wathington, Cyndi Mitchell, Julian Boot, Alexey Verkhovsky, Desi McAdam and so many other people that I can't remember their names. But what I do remember is that I felt right at home with them; these are people I can relate to, I can discuss with them, I can agree or disagree with them but regardless I will have a good time. It feels very much like a family, and I feel so much at home going to TW. I think that this is the best thing that could ever happen me. I have no illusions about this; I know for a fact that most of these people are smarter in every way than me. That is totally fine, that means I can learn great things from them.

So, what I wanted to say is just that I feel great about this. I'm so excited, and it's going to be a great time; not only for me and TW, but for the Ruby and JRuby community.

Well, I just wanted all the TWkers I've met these days to know that I feel so great about this. I've never felt as welcome, and I've never found a place with such great people. ThoughtWorks is a radically different company, and it's the people who does that. ThoughtWorks is extremely different, in a very good way.

I will need to blog about the last day of RailsConf, and more overall feelings and stuff like that, but I'll do that when I get back to Sweden.

Some RailsConf impressions

So, I've spent lots of time talking to people about numerous things. We have had some really nice conversations about everything related to Ruby, JRuby and ThoughtWorks and there is an incredible buzz going on here. Yesterday me, Roy, Tim (Bray), Cyndi, Martin Fowler, Charles, Tom and Nick sat down over dinner with DHH and talked about JRuby, which was also very interesting.

All in all, it's very exciting, and I hope it will just continue to be. I'm looking forward to the rest of the day, and tomorrow is going to be a new story.

ThoughtWorks, Mingle, RubyWorks and JRuby

So this is something that's been brewing for a while now, and it's all very exciting and was announced this morning at the keynote by Cyndi Mitchell. The relevant points are Mingle (which I've talked some about already), RubyWorks which is an umbrella for the things the enterprise needs to make Ruby viable. The first product out of RubyWorks is an installable package which gives you the deployment story basically for free. It's quite awesame and ThoughtWorks will offer 24/7 support for it from June.

For me personally, the most important part is that ThoughtWorks will also offer JRuby support 24/7 from June. That's right, 24/7 JRuby support. Wow. ThoughtWorks does believe in JRuby, they think it's something really important, and we want everything to get this.

Find out more at http://rubyworks.thoughtworks.com.

David about Rails 2.0

The DHH keynote this year was really good for a few reasons. First of all, David was very quick in saying that Rails 2.0 will not be anything radical, nothing really new, just better. Also, he actually announced that he had changed his mind about HTTP authentication, and performance (and this is kinda amazing of him to say... =). But all in all, it was a good talk about what's coming next; an evolution of the current things, but still the same Rails, just with more REST support and other goodies.

JRuby RC2

The day before RailsConf started, me, Charles and Tom sat in the lobby of the Courtyard, coding away, fixing problems and pushing out something like 5 versions of 1.0RC2. But the final RC2 is really, really good. You should download it, test it, and if you're at RailsConf, report any bugs you find and get a cool t-shirt! JRuby is really making a huge buzz at RailsConf right now, and it's sort of overwhelming! I'm very happy about it, of course, and there is still some nice things happening. Yesterday, we got a small bug in truncating files fixed, which means exclusive file locking works now.

1.0 is looking really, really good. It should be out by the end of May, and it's going to be a smash.

RSpec 1.0

Yesterday, the RSpec team pushed out version 1.0 which I'm personally very happy to hear. RSpec is a really important project and we are seriously considering including RSpec in the base JRuby distribution. Congratulations on the 1.0 release!

Congrats to Ward Cunningham

I met Ward Cunningham for the first time this Thursday, but I've always been very impressed by the things he has done. And yesterday he got a new job at AboutUs, a wiki company founded in 2005, as CTO. Congratulations!

A scattering of posts from RailsConf

So, I've been really bad about blogging from RailsConf. I have basically not done it at all, and I'm going to change that right now. There are loads of interesting stuff happening here, and I plan to talk about several different things. I won't do it in one post, though. Instead, I'll post lots of small posts with specific intent, and maybe one or two overall posts.

But I really must say, the feeling of being here, among 1600 developers in a vibrant and very cool community feels great. I've met lots of nice people (to many to remember, in fact), and I've had many great discussions.

JavaOne day 4: the final friday

So, the day started out quite late (since I was way tired after last night...). First session was a very accomplished, though slightly shallow, comparison between doing an application with Java EE 5, Ruby on Rails and Grails. I didn't really learn anything new in this presentation except that Java EE 5 is even slicker with NetBeans, and that Grails is maddeningly, sickeningly, seriously slow when doing simple stuff like creating scaffolds. I can't understand why this is so, since it isn't much do it. Basically just create a file with a few rows of customization.

After that I paired up with Jon to look at Mingle again. Oh boy, the ones who is coming to RailsConf will get a treat, that's for sure.

The Jython session by Otmar Humbel was really good, and had a great example. Basically the whole demo sessions were done in one single application which he started at the beginning and never stopped. He just hotswapped the Jython code dynamically. Very nice. I think the most interesting part of that talk was in the Q&A when someone asked why you should use Jython instead of Groovy or JRuby. I'm sad to say that I had to literally run at that point, so I didn't hear the answer... But still, if someone was there and heard, please tell me. For me personally, it's very impressive that Jython got started in 97, and actually is still alive (due to a recent revival of course, but even so. I'm glad Charles gave up one of his talks to let the Jython guys in). It was also interesting to see the differences in philosophy between JRuby and Jython in terms of Java integration and things like that. Very good stuff.
'
The bytecode manipulation talk was interesting; it was nice to see what (and how) Terracotta does with ASM. The TopLink parts were nice, but really way to basic to be interesting. Charles stuff was good, of course. We know Charlie always does good things, don't we? =)

I had great fun at the Java Puzzlers this year. I'm happy about there not being many puzzles incorporating generics, because those are usually boring. The main problem that these guys always exploits, seems to be mostly in the boundary between Java Objects and primitives, and interactions between primitives. That's what you get when you try to create a language meant for both system development and application development. It just shows in all the seams. A good language should not have seams, unless they're necessary for the specific domain of that language.

Finally, Rob Harrops talk about exploiting JRuby to create DSL's was really good. It was great, and Rob is a very accomplished presenter. I had great fun. That said, I think that some of the stuff still went over the head of most Java developers in the audience.

And that seems to be the theme. Most things I enjoy, I think most other people didn't enjoy. Interesting, that.

So, now it's off for dinner at the Stinking Rose with the JRuby guys, Dion, Barry Burd and Jon.

JavaOne day 3

The third day of the conference started slow, due to my being dead tired from the day before. But I still managed to get to Charles and Tom's JRuby on Rails presentation. Obviously, this was a very good presentation and JRuby is very important. But you know all that, already, don't you?

I then went on to a session giving a technical overview over GlassFish v2. Now, this seemed interesting, but wasn't really so. There was some nice information about how clustering should be setup so you don't lose to much data if you have more than one instance on the same machine, but other than that... Nah.

After lunch, I went to the Advanced Groovy talk, expecting to see cool overridings of the MetaObjectProtocol, or how to implement your own AST shufflers. But no. What we instead got was some tidbits here and there, and explanation of how to add new methods to existing Java objects (and this is really gross, you have to wrap the usage of these methods in a block, and the methods are defined on a new class as static methods...) Anyway, on to the demos. I got a real feeling of Deja Vu, since I had seen this exact same demo last year... XMLRPC communication between two Groovy instances, then ActiveX usage of Excel, combining it with Swing. So, yeah. As someone else said "It's a really great Excel demo, but what has that got to do with Groovy?".

Next, I took it easy for a while, walked around and did some programming. Finally, it was time for Tor's and Martin's talk about Ruby Tooling in NetBeans. And boy, it was an outstanding talk if you're interested in compilers, type inference, the challenges of dynamic languages and other stuff like that. I loved it, but I think most of it went over the head of most people in the audience.

Then I was of to the Sun Certified Professionals party which is always nice. I met some cool people and Lars won good swag by being smart really fast. Good for him!

The After Dark bash was quite allright, but nothing extraordinary. And it got really tasteless when the showgirl in metal corselet came onto stage with a grinder, which she turned on herself, making sparks fly all over the place.

After that, we went to the JRubME talk, about JRuby on Symbian and Java ME. A great talk and a great project. In fact, it's really really cool, and if you're interested in ME, you should download the source and start hacking a way on it! The project needs contributors.

We went back to the After Dark bash finally, took some more beer, and then decided to move on. The quest was for Aquavit, and someone thought that the Starlight Room would be a good place for this, so we headed there just to find out they didn't have either Aquavit nor any good scotch. Oh well, it was fun anyway, and the view was fantastic. Characteristically, me, Tom and Charles got into a real interesting discussion about the subjectivity of self presentation, and representation of knowledge in terms of probabilities and deltas from ideal knowledge, and how this applies to onthologies. Neat stuff. But maybe the dance floor wasn't the best place to have this loud discussion...

After that we settled down and decided it would be fun to have a JRuby summit in Minnesota! So, we'll see if that is possible... Then I crawled home to my hotel. My feet are killing me today, but there are some really interesting presentations, so I've gotta go.

JavaOne day 2

So, my day didn't start out with the Oracle general session, which I'm quite happy with in retrospect. Instead the first talk I saw was "Quick and Easy Profiling with Integrated Tools", about NetBeans profiling support. Very useful stuff, actually, and 6.0 can do some amazing things.

After that, it was time for a packed Josh Bloch's Effect Java Reloaded, except it wasn't really reloaded that much yet. I was a little bit disappointed, since most of the stuff was the same as last year. Some parts about the Builder pattern, much info about do's and don'ts with generics (much of it having to do with type parameters and wildcards). The TypeRef pattern was interesting, but not something I will find very much use of, I think.

After lunch a went to a presentation which I really thought would give me something. "Ruby on Rails Meets the World of Enterprise Applications". Now, I really hoped this would include some information on the Enterprise problem with regard to Rails. I was very disappointed. The talk was basically about a 3-week application written in Rails that connected to SAP. So the enterprise in question was there because of SAP. That said, the presenter was good, it was just not the subject I had wished for. And by the way, what does this presentation do on JavaOne?

After that, I spent some time looking at Mingle issues with Jon. I decided to go to the upper Haights' area to shop, and then get back to my hotel room for some programming. As it turned out, I really didn't go to any more sessions yesterday. After programming, I went to a Swedish Java User Group meeting here (and what a turnout! it seems like there's over 200 Swedes at JavaOne this year.)

The plan was to get back to the BOF's after that point; there was especially one I really didn't want to miss (the Dynamic Languages BOF with Frank Cohen). But, I happened to get invited to the Google party, so I spent the rest of the evening there, meeting lots of interesting people. It was great fun.

And now I'm about to go in and see Charles and Tom do the JRuby on Rails presentation. More information later.

The RedMonk unconference at CommunityOne

So, as you know, RedMonk arranged an unconference at JavaOne/CommunityOne this year. I couldn't attend all sessions, but among the ones done were one about dynamic languages, and we got some really good discussions going. We haven't really finished those discussions yet, though, and it seems we will take it onto a mailing list, but it was a very good environment to start out.

I really hope RedMonk will be able to do this next year too. More info about it can be found here: http://redmonk.com/wiki/index.php/RedMonkUnconference.

JavaOne: The rest of day 1

I am sitting in the alumni lounge, waiting for the sessions of day 2 to begin. I was thinking about attending the Oracle General Session, but decided to take it easy for a while instead.

So, where were I yesterday? Well, the Technical General Session was really interesting. As I mentioned yesterday, Charles and Tor did a great demonstration that was very well received. We also got some more on the technical side of JavaFX. Very nice.

After that, I attended Web Algorithms. It was a very accomplished presentation, detailing a few important things you can use to make life in computing easier, beginning with swap-XOR and credit card validation, going through public key cryptography and looking at Google MapReduce. It was a good presentation, but nothing new in it for me though, which was sort of sad.

The next presentation I went to was about the next generation of Web support in Java. The Servlet 3.0 specification and so on. The presentation was quite vague and didn't say much new things, really. Ok, so next generation servlets will use annotations, and there will be support for Rest style things and better security and more non-idiotic defaults? Not surprising.

After that presentation I was feeling really tired, so I went back to my hotel and tried to rest some. That didn't go so well so I finished writing chapter 11 instead. Then it was back to Moscone center to see a BOF on web development in Java EE compared to Ruby on Rails, with tool support. This talk was quite disappointing; it wasn't that well executed, and it was very un-nuanced in detailing the good and bad parts of Ruby on Rails. It seems that when people do these comparisons they just talk about how easy it is to do CRUD-style applications, but seems to forget that Ruby on Rails can do harder things (http://studios.thoughtworks.com/) and that the benefits from the Ruby language in development productivity scales. If you compare doing a small-sized web app in Java and Rails to each other, you will get some kind of percentage. But if you do a comparison between a medium-sized app in Java and Rails, that percentage will be greater, since the Ruby languages scales development time in a totally different way from Java. Not to mention maintainability after the fact.

Last I went to Neal Gafters BOF on Closures in Java. I really like the way Neal does presentations, but we couldn't really get into the meat of stuff, since half the people on the BOF didn't know closures from their elbows. So most of the presentation was spent rehashing what they are and why they're needed in Java.

After that, I was beat. And now it's a new day, started with breakfast meeting with Roy Singham. Very interesting and entertaining. I'm more and more convinced that I'm going to love working for ThoughtWorks.

JRuby on the technical general session

Charles O Nutter and Tor Norbye just got on stage here, and showed of deploying Mephisto as a WAR-file, and then changing it to add text-to-speech functionality, with a Java library that generates an audio file. The total code they wrote on stage was about 10 lines of Ruby... The power of Java and Ruby together: it's beautiful.

I already knew what they would do, having helped fix many of the issues getting in the way for Mephisto, but it's still incredibly cool. Charles and Tor got several impromptu applauds from the audience too, so I'm pretty sure people think it's neat.

Right now, I'm starting to get curious why people continue to use the term "scripting language". It doesn't seem to fit either Ruby nor Groovy anymore. Oh well. Nitpicking.

Next up is Arun Gupta. He's talking about jMaki and Phobos, both very nice usages of JavaScript and other languages.

JavaOne: Keynote and Groovy session

I'm sitting in the general session hall, waiting for the first technical general session to begin. Yesterday was CommunityOne, where we had some interesting discussion about dynamic languages within the context of the RedMonk Unconference. Aside from that, the best parts of the day way announcing that Mingle runs on JRuby, and meeting up with all my soon-to-be fellow ThoughtWorkers. They're a great bunch of people, and we had good fun.

Today was the opening general session, where Rich Green announced some very interesting developments. Among these are the fact that the open sourcing of Java is now complete, that Sun is coopering quite heavily with UN to provide resources for education to areas of the world where this have long been a huge problem. But the most important announcements were about something called JavaFX. I can't really say I understand it completely yet, but it seems to be an effort to tackle Microsoft Silverlight, and also fix several deficiencies in Swing by providing JavaFX Script (which looks very much like F3. I'm not sure if it actually is F3 or something else.). This obviously begs the question why Sun finds it prudent to invent a new language, instead of using one of the many great efforts existing in the dynamic language communities for this problem. For example, both Groovy, Jython and JRuby have different versions of SwingBuilders, which allow you to rapidly create Swing interfaces with a specialized DSL for this.

Except for that, the general session and keynote was more or less like usual. Very flash, very markety, but still more technical than I imagine other conferences are. And it was fun to see Rich Green being compared to Steve Jobs...

The first technical session I went to was called "Cool Things You Can Do with the Groovy Dynamic Language", and was presented by Guillaume Laforge and Dierk König. It was a quite good session, but I can't get away from my general opinions about the Groovy language. So, before I say something about the presentation, I need to describe my feelings for Groovy. Remember, these are my personal opinions, and some are definitely based on feeling without any specific rationalization.

I would really like to like Groovy, but I can't. I've really tried, but I can't find the Groovy language to my liking. And believe me, for some circumstances, Groovy should be able to fill the gap between Ruby and Java better than JRuby, at least in some cases. The Java integration in JRuby is quite hard, dispatch and overloading makes calling Java code complicated (from an implementation point, not for the user). All of this would be much simpler with the Groovy approach. But no, I still can't get along with Groovy. The main reason, I believe, is the feeling I get from all code that the language Groovy have grown piecemal, adding stuff that's neat wherever. I'm not sure this is the actual way Groovy was designed, but it feels like a modern version of Perl. The syntax doesn't mesh, and there are numerous (small, but nonetheless there) inconsistencies in how things are handled; many of the things that the language provide for you is things that really shouldn't be part of the language, but part of a library instead.

So, from this point of view, the presentation walked through several new features of Groovy, and lots of things you can do with it. They talked a bit about the new support for annotations and the plans for generics. Some of it was quite cool; I like the builders (but I prefer Markaby to the XMLBuilder), and some of the features are quite handy. But looking back at my earlier feelings I still see a design process more focused on finding problems with Java, and duct taping them with Groovy. (Like this: in Groovy, all exceptions get handled automatically, making it look as if all exceptions are unchecked.) I have no problem with many of these features, but I don't think it's a good way to create a general purpose language.

I'll be back later with more info on the rest of the days sessions.

Mingle on JRuby

Here's the deal: ThoughtWorks first product offering is called Mingle. The kicker is, it will be deployed on JRuby. I have been involved with this for a while, getting everything in good working order, and I'm sure TW won't regret this decision. JRuby is important in their strategy, and Mingle, as the first commercial JRuby application is a huge deal.

You should go and read more about this exciting news at http://studios.thoughtworks.com/2007/5/7/mingle-to-run-on-jruby.

Announcing mongrel_jcluster

As I mentioned in my last post, that server/client thing wasn't the only thing I worked on during the flight. I also did a very useful hack of mongrel_cluster, and turned it into mongrel_jcluster. It's not very large changes, actually... Almost everything is the same, except that this mongrel plugin only runs in JRuby, and will start several mongrels in the same JVM.

jruby -S gem install mongrel_jcluster
cd your_rails_app
jruby -S mongrel_rails jcluster::configure -p 4000 -N 3 -e production -R 20202 -K thesecretkey
jruby -S mongrel_rails jcluster::start

The only difference in the configuration parameters are -R and -K which is the JRuby server port and key respectively. Now, after executing these commands, it will take some time for the mongrels to get online, so be patient (or monitor the progress in the log-files generated).

I need to warn you that this is still quite experimental and not guaranteed to work in any way. =) But it does for me.

Another thing, if you start Mongrel in production mode, the defualt Rails front-page will give you an error if you try to get at the properties. This is the expected behavior in production mode, though, and nothing wrong with JRuby.

JRuby server

I spent some time on the flight to SF hacking on an idea I've had for a while. The result was checked in yesterday in JRuby trunk, and the important files are bin/jrubysrv and bin/jrubycli. This is still quite experimental, and only guaranteed to work in *NIX environments right now.

So, what is it? Well, if you don't want the overhead of running one JVM for each JRuby process, these two commands allow you to start a JVM that listens on a specific localhost port, and then you can send JRuby commands to that port. You need a key as password, otherwise the process will fail. Right now, this isn't as high security as it should be, since the key is passed as a command line parameter. This is still not that big of a problem, since the most common scenario is to start a bunch of things, and then not start anything more. Let me illustrate what you can do with a very simple example.

Say that you have a Rails application, and Mongrel for JRuby installed. Go the the base directory of your Rails app and do this:

jrubysrv -p 20202 -k secretkey &
jrubycli -p 20202 -k secretkey -S mongrel_rails -p 4000
jrubycli -p 20202 -k secretkey -S mongrel_rails -p 4001
jrubycli -p 20202 -k secretkey -S mongrel_rails -p 4002
jrubycli -p 20202 -k secretkey -n

Now, if you are patient, in a while there will be three Mongrels running, listening on ports 4000, 4001 and 4002. And they will run inside the same JVM, but in different JRuby runtimes. Quite neat, and very simple. You may wonder about the last command, with the -n flag. That flag tells the JRuby server running at port 20202 to not start any more JRuby processes. In this way, you can start what you need, and then close it down. Due to problems with thread scheduling and safety, I haven't implemented a way to kill a single runtime. What you instead need to do is use jrubycli with the -t flag, which will kill the whole JRuby server.

Stay tuned for my next post, on what I've actually used this functionality to create. (If you have some slight amount of imagination, you should be able to guess from this blog post...)

But now, of to the races. Or CommunityOne in this case. See you there.

San Francisco and Portland

Tomorrow morning I'll leave Stockholm for San Francisco. I'll be arriving 5:ish PM at SFO and will stay at the Handlery Union Square Hotel. I will not present at either JavaOne or RailsConf, though, so I will have a fair amount of time for fun stuff. I will attend the RedMonk unconference at CommunityOne, participate in the DynLang event, and then attend regular JavaOne. If you see me (I'll be wearing my black hat...), please say hi!

Next Saturday me and my coworkers will move out of SF and take a car to Portland, where we plan to arrive on the following Tuesday or Wednesday, just in time for RailsConf. We leave from Portland early Monday after the conference.

Hope to see many of you there! It will be exciting, that's for sure. I will also try to report as much as possible from all events.

JRuby at TheServerSide Java Symposium - Europe

Not sure if I've mentioned this, but I will represent JRuby at TheServerSide Java Symposium - Europe, in Barcelona, June 27-29. I will give one technical session on JRuby, and one BOF on how to deploy JRuby on Rails applications, both which should be quite exciting.

There are also a number of other nice presentations, so try to get to Barcelona!

Ye Zheng joins ThoughtWorks

Sorry if the title gives it away. I've been holding quiet about this for almost a month, but now it's official. Ye Zheng (more famous as dreamhead), who is one of the core developers of XRuby, have joined ThoughtWorks, and will begin work in China in May. He will be much involved in all things Ruby, and will also work on XRuby and JRuby, so we will probably be working close together. I have the highest respect for Ye Zheng and look forward to working with him. He single handedly rewrote most of the runtime in XRuby for the last release, which is not a small feat.

If you read Chinese, you can read more about it at his blog at http://dreamhead.blogbus.com/logs/5155305.html

JRuby 0.9.9 released

The JRuby team is happy to announce that 0.9.9 has now been released. This release have seen much focus on compatibility with Ruby 1.8, and also making the Java Integration features more solid. We are gearing up for a 1.0-release, and 0.9.9 should be pretty close to what you can expect from 1.0, so now is the time to start testing your applications on JRuby in earnest.

Homepage: http://www.jruby.org/
Download: http://dist.codehaus.org/jruby/

Some of the most important points of this release:

• Major compatibility and performance overhaul of String, Array, Hash
• Many YAML and Marshalling issues have been fixed
• Java Integration overhaul fixing many outstanding issues
• 180 Jira issues resolved
• Several more bottlenecks removed
• Rails applications like Mephisto and plugins like Goldberg are running without hitches
• Performance has improved by 40% over 0.9.8 based on YARV benchmarks

In response to all the things that needs doing before 1.0, we have decide to add two new committers to the team. Marcin Mielżyński and Bill Dortch have both done some incredibly nice things for the JRuby project, and we're very happy about them joining up as core developers.

In conjunction with 0.9.9, Rails-integration will see a new release, that improves many parts of it. Since we've actually started using Rails-integration in more places, we have also noticed things that need to be improved; the new release is very good.

ActiveRecord-JDBC 0.2.4 will also be released very soon. We have made some major changes, and much of the core have been rewritten. These changes make MySQL fully compatible with ActiveRecord, and we are working hard on improving support for other databases. Hopefully, the 0.2.4 release will mean Derby support is more or less complete too.

Practical JRuby on Rails

The time of waiting is over. I have hinted over and over that I've been doing some stuff that take lots of time. The first part of this was interviewing for ThoughtWorks, and I told you that a month back. And now I can finally tell you what's keeping me so busy, that I can't blog as much as I used to:

I am writing Practical JRuby on Rails: Web 2.0 projects for APress. I have been since late December, and the book is due to be published in October. But you can actually preorder the book today, from Amazon, here.

Me presenting JRuby

I totally forgot about this; 4-6 weeks ago, I did a presentation for the local Railsgroup here in Stockholm; since Dr Nic was there, I decided to improve it in English. And Dr Nic did us the service of taping the whole presentation; I was sick at the time, it's a short presentation about what's new in JRuby, but it could still be potentially interesting for some of you.

It's available at Google Video, here.

The State of JRuby

It's been sort of quiet on the JRuby front for a few weeks, so I thought I'd give a heads up on what things we're working on at the moment. At the moment, we're planning to release 0.9.9 sometime next week. At that point, several things we're working on right now should be fixed.

• There is a problem with deterministic threading in JRuby, which results in timeout errors sometimes escaping out of their rescue nodes. This is very serious for long running applications and Charles is hard at work providing better Thread semantics for us. The problem is that it's very hard to match MRI's green thread functionality with real native threads.
• Thomas is hard at work with one of his gigantic refactorings. This time the goal is to remove most of ThreadContext and make many parts of the call chain and block invocation chains easier to work with.
• Marcin have been working for some weeks now on a port of the hash implementation in MRI, and a corresponding rewrite of RubyHash. At the same time, he's hard at work rewriting RubyString to allow COW (copy-on-write) semantics. This could potentially improve performance and also make JRuby less memory intense.
• Earlier this week, we identified and fixed two very serious memory leaks. The second of these caused large leakage in all Rails applications using Cgi::Session for session data (which is almost all JRuby on Rails applications, except those running in Rails-integration). At the moment we don't see any leaking for Rails applications, so the situation looks good. (Both of these leaks were found with the help of Java monitoring tools, like JConsole, jmap and jhat. Very useful things.)
• I have been doing some serious YAML fixes these last days, with the result that our YAML situation is better than ever. It even handles recursive objects correctly. (Which is a pain in Java. In C it's easy, though.)

Overall, we're working very hard on the last, quirky compatibility issues right now; we have a few different, large Rails applications that we use to identify strange issues. And also, more and more people are trying their applications on JRuby, which means we get better (and more) bug reports. This is really great.

After 0.9.9 has been released, we're planning on finishing up remaining important bugs. I'm also very keen about getting more databases running really well with Rails. We will try to get Continous Integration set up for these too.

ActiveHibernate - Any takers?

This is a call for action. JRuby on Rails is getting more stable each day, and JRuby performance is consistently improving. This means that JRuby on Rails is well on the path of becoming a viable platform for several kinds of web development.

ActiveRecord-JDBC is central to this, and it's a good project and works very well. Within the limitations of ActiveRecord, of course. So, I'm thinking (and this is by all means not a new thought), that I would like to have the rest of Rails, but using Hibernate as backend. With JRuby approaching 1.0 fast, ActiveHibernate seems like a neat project. The only problem is time. So why is ActiveHibernate tempting? Well, for all those situations where ActiveRecord really doesn't fit; composite keys, complicated legacy systems. Or databases where you would like to use prepared statements for everything. Or get really nice speed.

What needs to be done? For it to be really useful, there are few points: First, a Hibernate extension that serializes and deserializes into RubyObjects. By doing it that way, there is no need to create Java bean classes. Secondly, provide all the useful help functionality around the Hibernate implementation, that AR users have gotten used to. This includes things like validations, automatic handling of updates and inserts, and is generally about doing a good Ruby interface around Hibernate. This also includes creating a good way of configuring Hibernate without having to resort to the XML. Maybe using YAML? Or a Ruby DSL? After that has been done, the final point is easy: get mind share and start doing applications with it! I for one know that I would like to use such a project!

Can your Ruby do this?

Recipe (should be baked with Java 6 for maximum pleasure):

1. Take one large Rails application with good Selenium test base
2. Convert database configuration to use JDBC
3. Start a selenium proxy with "jake test:selenium_proxy"
4. Start acceptance testing from another window with "jake test:acceptance"
5. In yet another window, write "jconsole"
6. Choose your application

What do you get?

You get this for free, just by running JRuby with Java 6. You can attach to any Java process at all. Remotely too. And get this kind of information. Can your Ruby do that?

Mongrel in JRuby

As I have told you before, we have been working on getting Mongrel working in JRuby off and on for a long while. One or two months ago, I got the Ragel definition ported correctly and the rest working. The plan is to cooperate with Zed and create a JRuby-native Mongrel gem out there, but until then I will describe the steps you need to take to get this working for yourself.

First of all, you need to install gem_plugin. The easiest way is to use gem:

jem install -y gem_plugin --no-rdoc --no-ri

After this is done, you can either download Mongrel-support from svn://rubyforge.org/var/svn/jruby-extras/trunk/mongrel-support, and build the extension manually, or you can download it directly here: http://opensource.ologix.com/http11.jar. If you decide to build it yourself, you first need a current version of JRuby from trunk. Check out mongrel-support from the Subversion repository and execute these commands:

cp $JRUBY_HOME/lib/jruby.jar lib
ant jar

After that you will have a http11.jar-file in lib. Regardless of how you get the http11.jar-file, place it in $JRUBY_HOME/lib/ruby/site_ruby/1.8/. Then you can proceed in a few different ways. You could copy the Mongrel-files from your MRI installation, or you could download Mongrel from version control. Regardless of how, you need to copy mongrel.rb, mutex_fix.rb and the mongrel-directory from Mongrel into $JRUBY_HOME/lib/ruby/site_ruby/1.8/. You can also copy bin/mongrel_rails into $JRUBY_HOME/bin and change the shebang to point to /usr/bin/env jruby. After this, you are good to go with JRuby and Mongrel.

On ActiveRecord-JDBC performance

I have been a bit concerned about the performance of our component that connects ActiveRecord with JDBC. Since ActiveRecord demands that every result of a select should be turned into a big array of hashes of strings to strings, I suspected we would be quite inefficient at this, and I wasn't sure I could put all my faith in JDBC either.

So, as a good developer, I decided to test this, with a very small microbenchmark, to see how bad the situation actually was.

Since I really wanted to check the raw database and unmarshalling performance, I decided to not use ActiveRecord classes, but do executions directly. The inner part of my benchmark execution looks like this:

conn.create_table :test_perf, :force => true do |t|
 t.column :one, :string
 t.column :two, :string
end
  
100.times do
 conn.insert("INSERT INTO test_perf(one, two) VALUES('one','two')")
end
  
1000.times do
 conn.select_all("SELECT * FROM test_perf")
end

conn.drop_table :test_perf

It is executed with a recent MySQL Community Edition 5 server, locally, with matching JDBC drivers. The MRI tests is run with 1.8.6, and both use ActiveRecord 1.15.3. ActiveRecord-JDBC is a prerelease of 0.2.4, available from trunk. My machine is an IBM Thinkpad T43p, running Debian. It's 32bit and Java 6.

The results were highly interesting. First, let's see the baseline: the Ruby results:

      user     system      total        real
 7.730000   0.020000   7.750000 (  8.531013)

Frankly, I wasn't that impressed with these numbers. I thought Ruby database performance was better. Oh well. The interesting part is the JRuby AR-JDBC results:

      user     system      total        real
 6.948000   0.000000   6.948000 (  6.948000)

WOW! We're actually faster in this quite interesting test. Not what I had expected at all, but very welcome news indeed. Note that there is still much block overhead in the interpreter, so the results are a little bit skewed in MRI's favour by this, too.

SuperRedCloth

RedCloth is cool. SuperRedCloth is cooler, faster, nicer, and all in all such a treat. And it works on JRuby now. At the time of writing, I haven't really wired together a gem of it yet, but I hope to get it done soon. Until then, if you want to try this, you can download the necessary jar-file from http://opensource.ologix.com/superredcloth_scan.jar, install the (very small) SuperRedCloth lib-file into your JRuby, add the superredcloth_scan.jar into your site_ruby for JRuby and everything should be set to go. Be warned, this requires a very recent trunk version of JRuby (something like newer than April 1:st).

SuperRedCloth is another of those nice Ragel based libraries. I'm finding Ragel such a joy to work with; just change the few C-thingies into Java-thingies and everything works exactly the same. Neat.

Some JRuby updates

So, there is prime time for JRuby now. So much, in fact, that I haven't had time to blog properly for a while. And won't have for a few days more. In the meantime, check what we've been busy with: http://headius.blogspot.com/2007/03/activerecord-100-performance-doubling.html.

Post Rails meetup

Last evening was very nice. We usually have a good time at the Rails meetups, hosted by Valtech. Yesterday it started out with Dr Nic William, who spoke about some metaprogramming techniques. Very neat, and the best part was his Magic Wiggly Lines. I'm not sure if it's genius or mental. But it's cool, at least; it's basically a const_missing hook, which makes it possible to use class names even if you spell them wrong. Hehe.

After that I spoke about new JRuby thing. I have a cold, so I took it pretty easy, and it was very free form with questions. There was many interesting questions, actually, and I think Nic taped it all...

After that, we drank beer and talked technology, which is always fun!

Rails meet in Stockholm

And it's time again for a Rails meetup in Stockholm, at Valtech. This Tuesday (20th) 6pm at Valtechs offices. It is usually very interesting and neat to meet other people involved in Rails and Ruby here in Stockholm, and it's usually very nice, all around.

I will have a short presentation on the new features in JRuby 0.9.8. I will provide a fast intro to JRuby, but this is not the introductionary presentation; I will also try a somewhat different presentation technique, so it is sure to be interesting. At least for me... =)

Show up! It's worth it. You need to reserve a place, though, which can be done at http://www.rails.se/rails/show/Railstr%C3%A4ff+20+Mars+2007

ThoughtWorks

I have resigned from my position at Karolinska Institutet. Instead, I will work for ThoughtWorks. I am extremely excited about this, obviously. I will start in London in the beginning of June. I'll stay in London for 6 months and then move to the San Francisco office.

As you may know, ThoughtWorks is doing lots of Ruby projects in the enterprise and are really helping spread Ruby to the masses. And that's what I will be working with. Ruby and JRuby for ThoughtWorks.

What's more, I will be doing development on the JRuby core at ThoughtWorks!

As an aside, I would like to mention two very interesting things that TW have announced very recently. First, CruiseControl.rb, which looks very nice, and secondly, Mingle. Both of these products have great potential.

I'm just so excited about this!

JRuby Regular Expressions

The Regular Expression support in JRuby is about to be revamped. I will here detail my plans for this work, and also some of the reasons for it. This post is as much for people interested in JRuby, as for the JRuby developers.

Stage 0: java.util.regex
JRuby has traditionally used java.util.regex. We stopped doing that March 11:th, 2007. The main reasons are because of the disconnect with MRI. Some of the operators work very differently, there are some problems with UTF-8, we can't support SJIS or EUC, nor posix-classes. And java.util.regex also uses a recursive implementation which means it can't handle certain large inputs. Further, we would like to be able to modify the implementation to work with the same stuff that backs the RubyString, to increase performance.

Stage 1: JRegex
Yesterday (March 11:th), I merged JRegex as the main regular expression engine for JRuby. The main reasons for this is twofold. First, we can change the implementation quite easily, and second, it is an iterative algorithm, which means it doesn't fail on the input that java.util.regex does. Since this caused problems with some Rails tests (and also in multipart handling in all libraries using cgi), I decided to merge this as a stopgap until the next incarnation of regex support.

Stage 2: REJ
In about 2 weeks, I hope to be able to merge REJ with JRuby. At the point of merging, it should be a better replacement than both JRegex and java.util.regex. REJ is a project I've started, which will be a direct port of the MRI 1.8.6 regular expression engine. The important thing about this is that the semantics for JRuby will match MRI very closely. We will be able to match UTF-8, SJIS and EUC regular expressions, and we are able to have the same quirks as MRI, even though people shouldn't depend on such quirks. In the process of writing REJ, I will also create a large suite of test cases for regular expressions, based on Henry Spencer's test files. I'll probably submit something initial to a separate repository very soon. If I get my wish, REJ is what will be the regular expression engine for JRuby 1.0.

Stage 3: Ojiguruma
After 1.0 has been released, I think it's time to make the Regexp engine in JRuby really extensible, and provide an interface from Ruby to change which engine to use. After that is done, I would be very interested in doing a port of Oniguruma to Java, which would give us far better multilanguage support, and also some interesting features. The reason I'm choosing to not do this right now is because Oniguruma is just too large.

Stage 4: (No official name yet)
Another engine that some in the JRuby/Ruby community has started working on is an engine which will be based on Ragel for parsing and a modified version of Thompson NFA and TCL-style backreferences for matching. It's an interesting project but it will take some time before it's usable.

Two more days in Kraków

I was very lazy on Saturday. Took it very easy, then went to a dinner with some of the presenters and organizers for SFI. And after that I gave my presentation, which I personally felt went very well. I had to adapt the technical level some, because it seemed most of the audience didn't know much about compilers. After that, Lukas held his presentation about Seaside, which also went well.

Then we celebrated. Then there was headache. And a look at the Cathedral. And a taxi driver who didn't understand one word of English. And he took me to the wrong terminal at the airport. But I finally got on the right flight and got home.

Two days in Kraków

So, I've been in Kraków for two days now. I will fly back on Sunday. It's a very interesting town. On one part, it's lovely. The buildings are absolutely amazing, and very beautiful. On the other hand, many of them are very run down, and the wear and tear is obvious all around. To me, the whole town seems sort of depressing, but on the other hand the people here are very up beat, and from some of the presentations I've seen, the technological future for Kraków looks very bright indeed.

I haven't really been able to see most of the presentations. Almost all have been in Polish, sadly enough. I did see two interesting ones yesterday. The first was Michael Foords talk about IronPython. This was very neat and I've talked some with him after that too. I'll get back to that. The other talk was about Google Ads, by Greg Badros. As always, hearing about how Google does things internally is always amazingly interesting. I would have enjoyed hearing more about the machine learning and NLP stuff they're doing, but obviously they can't discuss that too much.

Today I haven't been able to see any presentation, due to Polish. On the other hand me, Michael and Lukas Renggli have had some very interesting discussions both yesterday and today. Now, the company Michael's part of has the largest IronPython code base in the world, as far as I understand it. It's basically 90 000 LOC, where 20 000 is production code and 70 000 are testing code. That sounds about right... =)

Lukas Renggli is one of the core developers of Seaside, a framework which I'm quite fond of. What we three have in common is our interest in dynamic languages, so we had some very common ground to talk about. What's nice is that many of the things Michael talked about in his presentation is stuff we in JRuby also speak much about. In the same manner, it seems the IronPython guys have had basically the same problems we in JRuby has had. It seems there is some common ground to be found here, and possibly also a basis for conversations. I for one would find that very interesting, since I enjoy hearing about dynamic languages getting foot holds on statically typed virtual machines; that is just such a sweet spot.

Tomorrow evening is my talk. I'm going to be at an official dinner in the middle of the day, and then get over to the conference and give my presentation at 17:00. I think it's going to be interesting, and it will be an obvious counterpart to Michael's talk, since the reason for existing is so similar for IronPython and JRuby. It will be interesting to see how the audience react.

After that, I'll see Lukas talk about Seaside, and after that people are talking about having a party, so tomorrow will be quite intense. And finally, on Sunday I'm going back to Sweden. As far as I know now, I won't travel that much for at least a month now, which feels sort of nice. This week has been far to intense, and also have had some really great - but tiring - moments.

Tomorrow is today - or from London to Stockholm to Krakow

Part I: London

I got up at 5am, got dressed and packed and headed of to the airport. Finally got on the plane to London (thank god for online check-in) and tried to get some work done. That didn't go very well, but I got to read some instead. I'm rereading Ubik. Classic. Landed at Heathrow about 10 minutes behind schedule. Ran to the Piccadilly line, tried to talk with people in Sweden, but the connection was kinda bad. Arrived at Covent Garden, walked in brisk pace to the place of my meeting. Finally arrived, 5 minutes late, to find out there was no hurry. Sat down and had a pleasant chat with Dan North and Simon Stewart. Very nice. (Hey Dan, if you read this, get in touch when you're in Stockholm for ExpertZone. I know places with good beer.).

Then, Lunch. Haha. What a joke. I found a place that seemed to offer great vegetarian burgers, so I settled down for a quorn burger and some British chips. After 15 minutes waiting I got told the grill wouldn't light up. Lucky me. Instead, I had to make do with a Pret sandwich and a spicy vegetable Cornish pasty (someone told me it was the day of the Cornish patron saint yesterday, so I guess that fits). A few minutes walk on Oxford street was nice. I got photographed by some guys from an Austrian fashion magazine. Then back for the real meeting.

Cyndi Mitchell is a very formidable person. I was impressed. We had a 2 hour talk; very interesting. I will tell you all about it as soon as I can. But suffice to say, I was very happy when I left. Until I realized that I had exactly 2 hours until my flight taxed out from Heathrow. So, back to Covent Garden. Back on the underground. Which through some bad communication on the drivers part, almost ended badly when the train reached the end stop before Heathrow. Thanks to the people who told me to get of the train and switch to the other side. =) I finally got to Terminal 3, realized I already had a boarding card, found my way to the security check, got my boots scanned two times, and found the gate about 20 times before takeoff. Of course, as soon as the aircraft had taxed out, the flight management decided that we should stay 30 minutes on ground before finally taking off towards Copenhagen. It's a wonder I actually got there in time. So, in Copenhagen I walked 2 klicks to the transfer center, got a new boarding card and walked back to almost the exact place I started from, got on my flight to Stockholm and finally landed. Wow.

Airport coach to Stockholm city, and night bus home to my dear old mental hospital. When I finally got to bed I ached all over, and had been awake and active for 22 hours solid. Sleep was nice at that point. Especially since the day had been a success on all accounts.

Part II: Stockholm

I finally woke up, took care of mail and told everyone about London. Thought I'd take it easy, maybe practice on my presentation for Saturday, possibly start writing on the presentations for the CS course I'm going to speak for this April. But alas, after some mailing with Poland, I realized I've made a big mistake. I thought I was going to Kraków this Friday, and staying until Sunday. That's not entirely right. I'm actually going tomorrow. Surprise!

So.

Part III: Kraków

As mentioned above, I'll land in Kraków tomorrow (Wednesday) evening. This is good in several ways. I will have a chance to see the city, and I will have the opportunity to see some more presentations at the conference I'm at. There are many interesting things to be had; I'm sad half of them are in Polish. I guess I can look at the slides and have some fun trying the guess the meaning. I am going to see the talks about IronPython, Google Ads and Seaside. And, of course, if anyone is nearby and would like to take a beer and talk computers, programming languages, AI, or whatnot, please hit me a mail, or a comment on this post. My evenings will be mostly free it seems.

That is, unless someone knows any good clubs with nice music; especially electronic stuff and/or postpunk, new wave, darkwave or gothy things.

Anyway. Time to sleep.

JRuby 0.9.8 is here

The JRuby team is pleased to announce the release of JRuby 0.9.8.

Download at: http://dist.codehaus.org/jruby/

This release has some great improvements:

• Ruby on Rails support. We have been working hard on getting Rails own unit tests running and over 98% of them now run successfully. We feel things are running well enough to invite Ruby users to kick the tires and help root out any final issues.
• Ruby classes can extend concrete/abstract Java classes and override methods
• New Java primitive array syntax
• Reimplementation of String, Numeric classes, and Array to be more correct and performant
• Significant bottlenecks have been identified. In some cases IO is 6.5x faster than previous releases. Java included classes are significantly faster than in the past.
• 220 Jira issues resolved since last release

Special thanks to Marcin Mielżyński for his tireless work in rewriting a number of core classes to be much for correct and quick. His attention to detail has rooted out many corner cases.

The amount of IRC conversations, mailing list threads, bug reports, patches, and blog entries in the community has been a great help and our community is really making a huge difference in how fast JRuby is improving. The amount of progress is really staggering!

If you have ever thought that JRuby wasn't mature enough, I would like to contradict that now. With this release we are better than ever.

More information can be found at http://www.jruby.org.