First weeks at ThoughtWorks

I've finally started. I've finally moved to London. I've been working for two weeks at ThoughtWorks now, and it's been quite crazy. Everything is very nice and I'm having loads of fun. Of course, it's also lots of hard work, and I feel that I'm stretching my capacity considerably more than I ever did at Karolinska Institutet. That's great, and I feel that I'm really doing something real now. We have so many interesting things going on, and I wish I could tell you all about it.

What I can tell you is that I'm working quite much on Mingle, and I'm also spending time on other JRuby related issues. I've been planning on getting SQL Server and Oracle working as good as possible with AR-JDBC, and I've spent time on Derby performance. Hopefully I'll continue the database work this week, since especially SQL Server and Oracle is very important.

The most important work for this week is probably to prepare for TheServerSide in Barcelona. I still haven't had time to prepare my demos, so it's about time now. I hope to see many of you there.

In conclusion, my first weeks at ThoughtWorks have been awesome. I really like the pople, and everything is just neat. I like being able to walk to work and working in the very nice TW office on High Holborn. I'm very happy about it all.

Book update

As you know, I am writing on a book about JRuby on Rails. A few minutes ago I finished the first draft of chapter 14. That means that there are just 3 chapters and 3 appendixes left to write (chapter 1, 2 and 15). So the writing is going very well, but it's taking a heavy toll on me personally. I seriously don't recommend writing a book like this in your spare time, while at the same time switching employer, moving abroad and try to be a core developer in an open source project which is getting lots of attention.

So, in summary: it's going well, it still looks like it will be out in October, and I'm deadly tired.

JRuby 1.0

The JRuby community is pleased to announce the release of JRuby 1.0!

Homepage: http://www.jruby.org/
Download: http://dist.codehaus.org/jruby/

JRuby 1.0 is a major milestone for our project. Our main goal for 1.0 has been
Ruby compatibility. We feel this goal has been reached. When we see
companies like ThoughtWorks offering commercial support; we know this goal
has been reached. Please download JRuby and take it for a test drive. Try
running your Ruby and Ruby on Rails applications with it.

Give us feedback. Join our community. Help us continue to improve JRuby.

It is important to notice that JRuby 1.0 is not the end all of Ruby interpreters. It's not perfect. This is just the beginning. We now have a very good base to work from. This is were the real work begins. Join us. It will be a fun ride, and JRuby will just get better!

What's wrong with this code?

Today I will introduce to you a method from ActiveRecord. The method takes a parameter called type and that value can bu for example :primary_key, :string or :integer. Now, in the first line there is a call to native_database_types. Generally, that call returns a structure that looks somewhat like this:

def native_database_types #:nodoc:
  {
   :primary_key => "int(11) DEFAULT NULL auto_increment PRIMARY KEY",
   :string      => { :name => "varchar", :limit => 255 },
   :text        => { :name => "text" },
   :integer     => { :name => "int", :limit => 11 },
   :float       => { :name => "float" },
   :decimal     => { :name => "decimal" },
   :datetime    => { :name => "datetime" },
   :timestamp   => { :name => "datetime" },
   :time        => { :name => "time" },
   :date        => { :name => "date" },
   :binary      => { :name => "blob" },
   :boolean     => { :name => "tinyint", :limit => 1 }
 }
end

The method itself looks like this.

def type_to_sql(type, limit = nil, precision = nil, scale = nil) #:nodoc:
  native = native_database_types[type]
 column_type_sql = native.is_a?(Hash) ? native[:name] : native
 if type == :decimal # ignore limit, use precison and scale
    precision ||= native[:precision]
   scale ||= native[:scale]
   if precision
     if scale
       column_type_sql << "(#{precision},#{scale})"
     else
       column_type_sql << "(#{precision})"
     end
   else
     raise ArgumentError, "Error adding decimal column: precision cannot be empty if scale if specified" if scale
   end
   column_type_sql
 else
   limit ||= native[:limit]
   column_type_sql << "(#{limit})" if limit
   column_type_sql
 end
end

There is something very wrong with this implementation. Of course, there could exist many errors here, but what I'm thinking about right now is a violation of the usual way methods should work. And in effect, that problem with this method have caused ActiveRecord-JDBC to implement some very inefficient code to handle this method. And it gets called a lot in ActiveRecord. I'll get back later today with a pointer to what's wrong here, and I will also discuss some of what I've done in AR-JDBC to handle this situation. I hope for many suggestions here! =)

This is what's wrong

I must say, I got some really good responses to my post about what was wrong with the code I posted. Most of those responses concerned the design of the code, and I agree, this part of Rails could have been done much better. But what I was thinking about was actually a bug. And Lars Westegren (my former colleague) nailed it at the first try. Let me show two important excerpts from this code:

column_type_sql = native.is_a?(Hash) ? native[:name] : native

and here:

column_type_sql << "(#{limit})" if limit

Obviously, double left arrow is append, and for all cases where there is a limit, this append will change the String. This is one of the cases where it's kind of annoying that strings are mutable. If I cache away the values that native_database_types should return, then the next time anyone wants a string SQL type, that will generate VARCHAR(255)(255). The next time again, VARCHAR(255)(255)(255). And so one. So either I need to recreate the hash every time, or I need to do a deep clone of it every time. Neither of these options are very good, and it seems the deep clone option isn't fast enough, even when done in Java, so I decided to go with a hash literal instead. Was that the right choice? I don't know. It improves performance, but on the other hand it churns objects and creates new objects all the time. All of this because of some sloppy coding in Rails.

What's the lesson learned? Never modify your arguments, unless that is an explicit part of the contract for that method and part of the documentation.

Testing with JRuby on Rails and ActiveRecord-JDBC

This will be a highly uninflammatory blog post, as contrast to the last one. Specifically, there is a slight problem when running the command

jruby -S rake

in a a JRuby on Rails-application. This problem is caused by some hard coded values in the database Rake definitions for Rails. But don't despair, there is a simple solution to this. It's not as simple as it should be (invisible) but it's easy enough. Provided you have JRUBY_HOME set and your version of AR-JDBC is 0.3.1, execute this command from your Rails application root

cp $JRUBY_HOME/lib/ruby/gems/1.8/gems/ActiveRecord-JDBC-0.3.1/lib/tasks/jdbc_databases.rake lib/tasks

Since the hard coded values are hard to override, the jdbc_databases.rake file just hacks Rake to be able to redefine tasks and then redefines the core tasks. This shouldn't affect a bi-Ruby installation, since the overriding only happens on JRuby, not on MRI. If someone has a better way to do this, please tell me. =)

There can be only one, a tale about Ruby, IronRuby, MS and whatnot

(Updated: added a quote from John Lam about not being able to look at the MRI source code)

After RailsConf in Portland, there has flared up a discussion centered around IronRuby and Microsoft. We discussed many of these points in depth at the conference, and I'll elaborate some on my views on the issues in a bit.

But first I would like to talk some about the multitude of Ruby implementations springing up. I firmly believe that a language evolves in phases. The first phase, germination, is the period where a language needs one consistent implementation (or a spec). It's during this phase when most "alpha geek adoption" happens. Many important libraries are written, but most applications are not in the main economic center. Ruby have been in this phase for a long time, but the fact that new implementations are springing up left and right is a sure sign that Ruby is entering phase 2: implementation. For adoption to happen, there need to exist several competing implementations, all of them good. This is the evolutionary stage, where it's decided what kind of features an implementation should provide. Should we have green or native threads? Are all the features of the original implementation really that necessary? (Continuations, ObjectSpace). Is there cruft in the standard library that needs to be weeded out? (timeout.rb). All of these questions get answered when other people implement the language. The last phase, which I guess could be called adoption, is when the language have several working implementations, all good enough to deliver high end applications on, when many applications are written in the language, and there exists a plethora of libraries, systems and support for the language.

What this means is that for a language to be successful, there needs to exist competing implementations. They need to implement their features in different ways and make different choices during development. Otherwise, the language will die. (This is obviously not enough, since Smalltalk fulfilled this admirably and still never got widespread adoption.). But I still believe it's incredibly important for a language to evolve with many implementations, which is why I find Rubinius, JRuby, YARV and IronRuby to be extremely important projects for the welfare of Ruby. I want Ruby to be successful. I want Ruby to be the next major language for several reasons. But most importantly: I want Ruby to be a better language tomorrow, than it is today. The only way that's going to happen is by having lots of people implement the language.

So, that's enough of the introductory flame bait. This describes one half of why IronRuby is an important project, and why we can't let it fail. The other side of the coin is the same reason JRuby is important. .NET as a platform have some wildly useful features. There are many developers who swear by .NET for good reason. And what's more important, there are lots of large enterprises with such a vested interest in .NET, that they will never choose anything else. Now, for the welfare of all programmers in the world, I personally believe the world would be a better place if those .NET-environments also used Ruby. So that's the other coin of why IronRuby is important.

The most well read blog about the current Microsoft/Ruby controversy is Martin Fowlers article RubyMicrosoft. Go read it now, and then I'll just highlight the points I find most important.

First: John Lam is committed to creating a "compliant" Ruby implementation. I have no doubts that he can do it. But there are a few problems lurking.

For example, what is a compliant Ruby implementation? Since there exists no spec, and no comprehensive test suite, the only way to measure compliance is to check how close the behavior matches MRI. But this have some problems too. How do you check behavior? Well, you need to run applications. But how do you get so far as you can run applications?
What JRuby did was that we looked at MRI source code.

John Lam can not look at MRI source code. He cannot look at JRuby source code. He cannot look at Rubinius source code. If he does, he will be terminated.

So, the next best alternative: accepting patches from the community, which can look at Ruby source? Nope, no cigar. Microsoft is not really about Open Source yet. Their license allows us to look at their source code, and also to fork it and do what we want with it. But that's only half of what open source is about. The more important part is that you should be able to contribute back code without having to fork. You can't do that with IronRuby, since Microsoft is too scared about being sued for copyright infringement.

There was some doubt about Lam actually being banned from looking at MRI source code. This is the first quote that said it is so. It's from the discussion "Virtual classes and 'real' classes -- why?" on Ruby-core, this quote posted at 29/03/07:

Is this how things are actually implemented? (BTW I'm not lazy here - we cannot look for ourselves).

I am going to make a bold statement here. Under the current circumstances, I don't believe it's possible for John Lam and his team to create a Ruby implementation that runs Rails within at least 18 months. And frankly, that's not soon enough.

As I said above, I have all confidence that John can do great stuff if he has the right resources. But creating a Ruby implementation is hard enough while having all the benefits of the open source community.

The two points I want to make with this point is this: The Ruby community must damned well get serious about creating a good, complete specification and test suite. It's time to do it right now, and we need it. It's not a one-man job. The community needs to do it. (And yes, the two SoC projects are a very good start. But you still need to be able to run RSpec to take full advantage of them; and let's face it, the RSpec implementation uses many nice Ruby tricks.)

The second point is simpler: Microsoft needs to completely change how they handle Open Source. Their current strategy of trying to grow it into the organization will not work (at least not good enough). They need to turn around completely, reinvent themselves and make some really bold moves to be able to meet the new world. If they can't do this, they are as dead as Paul Graham claims.

Solving mounting problem on MacOS X

This is a highly specific post, but I thought I'd write about this so that if someone else have the same problem, they can try to solve it my way.

I got a new computer, Intel-based MBP last week. I upgraded it to latest OSX version (10.4.9 I believe), and immediately installed everything I needed. While ending my last job and returning the laptop associated with that post, I made backups to a 500GB USB LACIE hard drive. Very nice indeed, and half my life is now on that hard drive. (I have most of it on other places too, but not so easily accessible).

The first thing I managed to do was to shut off the hard drive without unmounting it correctly from OSX. I got the warning et all, but well, done is done. It was stupid, I know. Starting out with messing up everything. So what happened? Well, the next time I tried to get OSX to find my hard drive, by inputing the USB connection and switching on the power. Nothing happened. The drive would spin up, but no mount points or nice icons on the desktop. After some investigation I found that the Disk Manager HANGS when the LACIE is turned on and connected. I also found that when I switched the power off, my console says something about not being able to repair disk /dev/disk2s1. Interesting. After a few hours investigation on the Internet I despaired, and decided to try my own ingenuity instead.

I won't tell you about everything I did to find this solution. It would get severely boring very fast. So, here is the solution:

1. Attach the device and turn on the power.
2. Open up a terminal and create a new directory in /Volumes, (eg sudo mkdir /Volumes/LACIE2).
3. Mount the drive explicitly on this mount point (sudo mount -t msdos /dev/disk2s1 /Volumes/LACIE2).
4. Delete ALL .DS_Store files on the disk (cd /Volumes/LACIE2; find . -name 'DS_Store' -exec rm -rf \{\} \;).
5. Wait for a while. At this point you should have two LACIE drives on your Desktop, one fake and one real. Unmount the real one by dragging it to the trash can.
6. Turn of the external hard drive, or detach it.
7. Reboot.
8. Attach/Turn on the external hard drive.

This is a process that works for me, and my drive is now back to working mode. It seems that OSX stores some information in the DS_Store files somewhere on the disk that got corrupted for me. Hopefully this information can help someone else with the same problem.

Finally in London

Yesterday I landed in London, and let me tell you: it's been tough getting here. Packing and moving and arranging all takes lots of time and energy. And it's not helping being sick while pulling it off, either. But I'm finally here, and right now sitting at a hotel room in Holborn, close to Russel Square. Hopefully I'll be able to land an apartment soon too, and then I'll get back to my former speed, hopefully.

In the meantime, tomorrow is my first day at ThoughtWorks. It's bound to be interesting, but I don't expect any downtime. We have so many interesting things going, that it will be full speed ahead from day one.

Although I'm more tired than I should be, if someone feels like meeting up this or next week for a beer and talk JRuby, it would be fun.

Burning cycles: the next month

Getting back from RailsConf meant getting thrown into loads and loads of preparation work for London. Oh my. Since I'm moving in 8 days, and almost all packing needs to be done during the weekend (and I haven't begun yet), I don't think I'll be able to be that communicative. Further, I will have no access to a computer or cell phone from Wednesday to Monday, so if I don't answer email, don't get upset. =) That incidentally means I won't blog either.

June 4th will be my first day at ThoughtWorks and I'm incredibly excited by it. Hopefully I'll be able to blog some about my experiences, but I guess the first few days will be focused on getting setup and finding an apartment (If you know any nice place close to Camden Town that's available by the beginning of June, please do tell), and doing all manner of things.

If I'm unlucky I'll need to go to DC the week after that, but we'll see. Hopefully I can wrangle out of it, since Stella (my girlfriend) will arrive in London that weekend.

After that week I'll have a week of quiet and solitude, and then it's time for TheServerSide Java Symposium in Barcelona. I will in fact be involved in no less than three events during this conference. First, a technical session on JRuby, secondly a BoF about deploying JRuby on Rails applications, and third a panel discussion called "2020: A Developer's Odyssey Panel", which is bound to be interesting. I have no idea whatsoever about what I'll say yet, but Martin Fowler will be one of the panelists which makes it a certainty of fun. I'm looking forward to TSSJS for a few more reasons; seeing Barcelona, since that is said to be a lovely city, getting to say hi to Dr. Heinz Kabutz, meet up with Jonas Bonér from Terracotta and continue discussing how to utilize Terracotta and JRuby together. I'm looking very much forward to the whole event, in fact. If you are there, don't hesitate to say hi!

That gender thing

One of the foci on both JavaOne and RailsConf was the so called "gender problem"; that the current balance between men and women in the technological field is bad and that something needs to be done about it. I'm very happy about this getting awareness, but as my colleague Lars Westergren writes here (http://slashdot.org/~LarsWestergren/journal/172323) it seems like total hypocrisy when looking at the so called entertainment provided at these two events. We need to do better than this.

The ThoughtWorks story

As you know, I will be starting at ThoughtWorks Studios in June, and I've been to JavaOne and now RailsConf, head on head. There is something I really need to tell you right now, and something that I've felt very strongly this week and last week. As I said, I'll start at TW in two weeks. My interview process was basically like any interview process; I met lots of TW people, but I couldn't relate to them that much, because being part of the interview process made it very hard. My mind was focused on other things.

Martin Fowler said in one of his bliki entries (here), that people matter most, and: "I decided then that I wouldn't work with unpleasant people, however capable they might be.". And he then goes on and says that this was one of many reasons he actually joined TW. TW has a "no assholes" policy. I didn't actually get the reality of this (I mean seriously, would you believe that such a thing actually could be?), until RailsConf when I met something like 30-40 TWkers. I've spent much time with Roy, Chad Wathington, Cyndi Mitchell, Julian Boot, Alexey Verkhovsky, Desi McAdam and so many other people that I can't remember their names. But what I do remember is that I felt right at home with them; these are people I can relate to, I can discuss with them, I can agree or disagree with them but regardless I will have a good time. It feels very much like a family, and I feel so much at home going to TW. I think that this is the best thing that could ever happen me. I have no illusions about this; I know for a fact that most of these people are smarter in every way than me. That is totally fine, that means I can learn great things from them.

So, what I wanted to say is just that I feel great about this. I'm so excited, and it's going to be a great time; not only for me and TW, but for the Ruby and JRuby community.

Well, I just wanted all the TWkers I've met these days to know that I feel so great about this. I've never felt as welcome, and I've never found a place with such great people. ThoughtWorks is a radically different company, and it's the people who does that. ThoughtWorks is extremely different, in a very good way.

I will need to blog about the last day of RailsConf, and more overall feelings and stuff like that, but I'll do that when I get back to Sweden.

Some RailsConf impressions

So, I've spent lots of time talking to people about numerous things. We have had some really nice conversations about everything related to Ruby, JRuby and ThoughtWorks and there is an incredible buzz going on here. Yesterday me, Roy, Tim (Bray), Cyndi, Martin Fowler, Charles, Tom and Nick sat down over dinner with DHH and talked about JRuby, which was also very interesting.

All in all, it's very exciting, and I hope it will just continue to be. I'm looking forward to the rest of the day, and tomorrow is going to be a new story.

ThoughtWorks, Mingle, RubyWorks and JRuby

So this is something that's been brewing for a while now, and it's all very exciting and was announced this morning at the keynote by Cyndi Mitchell. The relevant points are Mingle (which I've talked some about already), RubyWorks which is an umbrella for the things the enterprise needs to make Ruby viable. The first product out of RubyWorks is an installable package which gives you the deployment story basically for free. It's quite awesame and ThoughtWorks will offer 24/7 support for it from June.

For me personally, the most important part is that ThoughtWorks will also offer JRuby support 24/7 from June. That's right, 24/7 JRuby support. Wow. ThoughtWorks does believe in JRuby, they think it's something really important, and we want everything to get this.

Find out more at http://rubyworks.thoughtworks.com.

David about Rails 2.0

The DHH keynote this year was really good for a few reasons. First of all, David was very quick in saying that Rails 2.0 will not be anything radical, nothing really new, just better. Also, he actually announced that he had changed his mind about HTTP authentication, and performance (and this is kinda amazing of him to say... =). But all in all, it was a good talk about what's coming next; an evolution of the current things, but still the same Rails, just with more REST support and other goodies.

JRuby RC2

The day before RailsConf started, me, Charles and Tom sat in the lobby of the Courtyard, coding away, fixing problems and pushing out something like 5 versions of 1.0RC2. But the final RC2 is really, really good. You should download it, test it, and if you're at RailsConf, report any bugs you find and get a cool t-shirt! JRuby is really making a huge buzz at RailsConf right now, and it's sort of overwhelming! I'm very happy about it, of course, and there is still some nice things happening. Yesterday, we got a small bug in truncating files fixed, which means exclusive file locking works now.

1.0 is looking really, really good. It should be out by the end of May, and it's going to be a smash.

RSpec 1.0

Yesterday, the RSpec team pushed out version 1.0 which I'm personally very happy to hear. RSpec is a really important project and we are seriously considering including RSpec in the base JRuby distribution. Congratulations on the 1.0 release!

Congrats to Ward Cunningham

I met Ward Cunningham for the first time this Thursday, but I've always been very impressed by the things he has done. And yesterday he got a new job at AboutUs, a wiki company founded in 2005, as CTO. Congratulations!

A scattering of posts from RailsConf

So, I've been really bad about blogging from RailsConf. I have basically not done it at all, and I'm going to change that right now. There are loads of interesting stuff happening here, and I plan to talk about several different things. I won't do it in one post, though. Instead, I'll post lots of small posts with specific intent, and maybe one or two overall posts.

But I really must say, the feeling of being here, among 1600 developers in a vibrant and very cool community feels great. I've met lots of nice people (to many to remember, in fact), and I've had many great discussions.

JavaOne day 4: the final friday

So, the day started out quite late (since I was way tired after last night...). First session was a very accomplished, though slightly shallow, comparison between doing an application with Java EE 5, Ruby on Rails and Grails. I didn't really learn anything new in this presentation except that Java EE 5 is even slicker with NetBeans, and that Grails is maddeningly, sickeningly, seriously slow when doing simple stuff like creating scaffolds. I can't understand why this is so, since it isn't much do it. Basically just create a file with a few rows of customization.

After that I paired up with Jon to look at Mingle again. Oh boy, the ones who is coming to RailsConf will get a treat, that's for sure.

The Jython session by Otmar Humbel was really good, and had a great example. Basically the whole demo sessions were done in one single application which he started at the beginning and never stopped. He just hotswapped the Jython code dynamically. Very nice. I think the most interesting part of that talk was in the Q&A when someone asked why you should use Jython instead of Groovy or JRuby. I'm sad to say that I had to literally run at that point, so I didn't hear the answer... But still, if someone was there and heard, please tell me. For me personally, it's very impressive that Jython got started in 97, and actually is still alive (due to a recent revival of course, but even so. I'm glad Charles gave up one of his talks to let the Jython guys in). It was also interesting to see the differences in philosophy between JRuby and Jython in terms of Java integration and things like that. Very good stuff.
'
The bytecode manipulation talk was interesting; it was nice to see what (and how) Terracotta does with ASM. The TopLink parts were nice, but really way to basic to be interesting. Charles stuff was good, of course. We know Charlie always does good things, don't we? =)

I had great fun at the Java Puzzlers this year. I'm happy about there not being many puzzles incorporating generics, because those are usually boring. The main problem that these guys always exploits, seems to be mostly in the boundary between Java Objects and primitives, and interactions between primitives. That's what you get when you try to create a language meant for both system development and application development. It just shows in all the seams. A good language should not have seams, unless they're necessary for the specific domain of that language.

Finally, Rob Harrops talk about exploiting JRuby to create DSL's was really good. It was great, and Rob is a very accomplished presenter. I had great fun. That said, I think that some of the stuff still went over the head of most Java developers in the audience.

And that seems to be the theme. Most things I enjoy, I think most other people didn't enjoy. Interesting, that.

So, now it's off for dinner at the Stinking Rose with the JRuby guys, Dion, Barry Burd and Jon.

JavaOne day 3

The third day of the conference started slow, due to my being dead tired from the day before. But I still managed to get to Charles and Tom's JRuby on Rails presentation. Obviously, this was a very good presentation and JRuby is very important. But you know all that, already, don't you?

I then went on to a session giving a technical overview over GlassFish v2. Now, this seemed interesting, but wasn't really so. There was some nice information about how clustering should be setup so you don't lose to much data if you have more than one instance on the same machine, but other than that... Nah.

After lunch, I went to the Advanced Groovy talk, expecting to see cool overridings of the MetaObjectProtocol, or how to implement your own AST shufflers. But no. What we instead got was some tidbits here and there, and explanation of how to add new methods to existing Java objects (and this is really gross, you have to wrap the usage of these methods in a block, and the methods are defined on a new class as static methods...) Anyway, on to the demos. I got a real feeling of Deja Vu, since I had seen this exact same demo last year... XMLRPC communication between two Groovy instances, then ActiveX usage of Excel, combining it with Swing. So, yeah. As someone else said "It's a really great Excel demo, but what has that got to do with Groovy?".

Next, I took it easy for a while, walked around and did some programming. Finally, it was time for Tor's and Martin's talk about Ruby Tooling in NetBeans. And boy, it was an outstanding talk if you're interested in compilers, type inference, the challenges of dynamic languages and other stuff like that. I loved it, but I think most of it went over the head of most people in the audience.

Then I was of to the Sun Certified Professionals party which is always nice. I met some cool people and Lars won good swag by being smart really fast. Good for him!

The After Dark bash was quite allright, but nothing extraordinary. And it got really tasteless when the showgirl in metal corselet came onto stage with a grinder, which she turned on herself, making sparks fly all over the place.

After that, we went to the JRubME talk, about JRuby on Symbian and Java ME. A great talk and a great project. In fact, it's really really cool, and if you're interested in ME, you should download the source and start hacking a way on it! The project needs contributors.

We went back to the After Dark bash finally, took some more beer, and then decided to move on. The quest was for Aquavit, and someone thought that the Starlight Room would be a good place for this, so we headed there just to find out they didn't have either Aquavit nor any good scotch. Oh well, it was fun anyway, and the view was fantastic. Characteristically, me, Tom and Charles got into a real interesting discussion about the subjectivity of self presentation, and representation of knowledge in terms of probabilities and deltas from ideal knowledge, and how this applies to onthologies. Neat stuff. But maybe the dance floor wasn't the best place to have this loud discussion...

After that we settled down and decided it would be fun to have a JRuby summit in Minnesota! So, we'll see if that is possible... Then I crawled home to my hotel. My feet are killing me today, but there are some really interesting presentations, so I've gotta go.

JavaOne day 2

So, my day didn't start out with the Oracle general session, which I'm quite happy with in retrospect. Instead the first talk I saw was "Quick and Easy Profiling with Integrated Tools", about NetBeans profiling support. Very useful stuff, actually, and 6.0 can do some amazing things.

After that, it was time for a packed Josh Bloch's Effect Java Reloaded, except it wasn't really reloaded that much yet. I was a little bit disappointed, since most of the stuff was the same as last year. Some parts about the Builder pattern, much info about do's and don'ts with generics (much of it having to do with type parameters and wildcards). The TypeRef pattern was interesting, but not something I will find very much use of, I think.

After lunch a went to a presentation which I really thought would give me something. "Ruby on Rails Meets the World of Enterprise Applications". Now, I really hoped this would include some information on the Enterprise problem with regard to Rails. I was very disappointed. The talk was basically about a 3-week application written in Rails that connected to SAP. So the enterprise in question was there because of SAP. That said, the presenter was good, it was just not the subject I had wished for. And by the way, what does this presentation do on JavaOne?

After that, I spent some time looking at Mingle issues with Jon. I decided to go to the upper Haights' area to shop, and then get back to my hotel room for some programming. As it turned out, I really didn't go to any more sessions yesterday. After programming, I went to a Swedish Java User Group meeting here (and what a turnout! it seems like there's over 200 Swedes at JavaOne this year.)

The plan was to get back to the BOF's after that point; there was especially one I really didn't want to miss (the Dynamic Languages BOF with Frank Cohen). But, I happened to get invited to the Google party, so I spent the rest of the evening there, meeting lots of interesting people. It was great fun.

And now I'm about to go in and see Charles and Tom do the JRuby on Rails presentation. More information later.

The RedMonk unconference at CommunityOne

So, as you know, RedMonk arranged an unconference at JavaOne/CommunityOne this year. I couldn't attend all sessions, but among the ones done were one about dynamic languages, and we got some really good discussions going. We haven't really finished those discussions yet, though, and it seems we will take it onto a mailing list, but it was a very good environment to start out.

I really hope RedMonk will be able to do this next year too. More info about it can be found here: http://redmonk.com/wiki/index.php/RedMonkUnconference.

JavaOne: The rest of day 1

I am sitting in the alumni lounge, waiting for the sessions of day 2 to begin. I was thinking about attending the Oracle General Session, but decided to take it easy for a while instead.

So, where were I yesterday? Well, the Technical General Session was really interesting. As I mentioned yesterday, Charles and Tor did a great demonstration that was very well received. We also got some more on the technical side of JavaFX. Very nice.

After that, I attended Web Algorithms. It was a very accomplished presentation, detailing a few important things you can use to make life in computing easier, beginning with swap-XOR and credit card validation, going through public key cryptography and looking at Google MapReduce. It was a good presentation, but nothing new in it for me though, which was sort of sad.

The next presentation I went to was about the next generation of Web support in Java. The Servlet 3.0 specification and so on. The presentation was quite vague and didn't say much new things, really. Ok, so next generation servlets will use annotations, and there will be support for Rest style things and better security and more non-idiotic defaults? Not surprising.

After that presentation I was feeling really tired, so I went back to my hotel and tried to rest some. That didn't go so well so I finished writing chapter 11 instead. Then it was back to Moscone center to see a BOF on web development in Java EE compared to Ruby on Rails, with tool support. This talk was quite disappointing; it wasn't that well executed, and it was very un-nuanced in detailing the good and bad parts of Ruby on Rails. It seems that when people do these comparisons they just talk about how easy it is to do CRUD-style applications, but seems to forget that Ruby on Rails can do harder things (http://studios.thoughtworks.com/) and that the benefits from the Ruby language in development productivity scales. If you compare doing a small-sized web app in Java and Rails to each other, you will get some kind of percentage. But if you do a comparison between a medium-sized app in Java and Rails, that percentage will be greater, since the Ruby languages scales development time in a totally different way from Java. Not to mention maintainability after the fact.

Last I went to Neal Gafters BOF on Closures in Java. I really like the way Neal does presentations, but we couldn't really get into the meat of stuff, since half the people on the BOF didn't know closures from their elbows. So most of the presentation was spent rehashing what they are and why they're needed in Java.

After that, I was beat. And now it's a new day, started with breakfast meeting with Roy Singham. Very interesting and entertaining. I'm more and more convinced that I'm going to love working for ThoughtWorks.

JRuby on the technical general session

Charles O Nutter and Tor Norbye just got on stage here, and showed of deploying Mephisto as a WAR-file, and then changing it to add text-to-speech functionality, with a Java library that generates an audio file. The total code they wrote on stage was about 10 lines of Ruby... The power of Java and Ruby together: it's beautiful.

I already knew what they would do, having helped fix many of the issues getting in the way for Mephisto, but it's still incredibly cool. Charles and Tor got several impromptu applauds from the audience too, so I'm pretty sure people think it's neat.

Right now, I'm starting to get curious why people continue to use the term "scripting language". It doesn't seem to fit either Ruby nor Groovy anymore. Oh well. Nitpicking.

Next up is Arun Gupta. He's talking about jMaki and Phobos, both very nice usages of JavaScript and other languages.

JavaOne: Keynote and Groovy session

I'm sitting in the general session hall, waiting for the first technical general session to begin. Yesterday was CommunityOne, where we had some interesting discussion about dynamic languages within the context of the RedMonk Unconference. Aside from that, the best parts of the day way announcing that Mingle runs on JRuby, and meeting up with all my soon-to-be fellow ThoughtWorkers. They're a great bunch of people, and we had good fun.

Today was the opening general session, where Rich Green announced some very interesting developments. Among these are the fact that the open sourcing of Java is now complete, that Sun is coopering quite heavily with UN to provide resources for education to areas of the world where this have long been a huge problem. But the most important announcements were about something called JavaFX. I can't really say I understand it completely yet, but it seems to be an effort to tackle Microsoft Silverlight, and also fix several deficiencies in Swing by providing JavaFX Script (which looks very much like F3. I'm not sure if it actually is F3 or something else.). This obviously begs the question why Sun finds it prudent to invent a new language, instead of using one of the many great efforts existing in the dynamic language communities for this problem. For example, both Groovy, Jython and JRuby have different versions of SwingBuilders, which allow you to rapidly create Swing interfaces with a specialized DSL for this.

Except for that, the general session and keynote was more or less like usual. Very flash, very markety, but still more technical than I imagine other conferences are. And it was fun to see Rich Green being compared to Steve Jobs...

The first technical session I went to was called "Cool Things You Can Do with the Groovy Dynamic Language", and was presented by Guillaume Laforge and Dierk König. It was a quite good session, but I can't get away from my general opinions about the Groovy language. So, before I say something about the presentation, I need to describe my feelings for Groovy. Remember, these are my personal opinions, and some are definitely based on feeling without any specific rationalization.

I would really like to like Groovy, but I can't. I've really tried, but I can't find the Groovy language to my liking. And believe me, for some circumstances, Groovy should be able to fill the gap between Ruby and Java better than JRuby, at least in some cases. The Java integration in JRuby is quite hard, dispatch and overloading makes calling Java code complicated (from an implementation point, not for the user). All of this would be much simpler with the Groovy approach. But no, I still can't get along with Groovy. The main reason, I believe, is the feeling I get from all code that the language Groovy have grown piecemal, adding stuff that's neat wherever. I'm not sure this is the actual way Groovy was designed, but it feels like a modern version of Perl. The syntax doesn't mesh, and there are numerous (small, but nonetheless there) inconsistencies in how things are handled; many of the things that the language provide for you is things that really shouldn't be part of the language, but part of a library instead.

So, from this point of view, the presentation walked through several new features of Groovy, and lots of things you can do with it. They talked a bit about the new support for annotations and the plans for generics. Some of it was quite cool; I like the builders (but I prefer Markaby to the XMLBuilder), and some of the features are quite handy. But looking back at my earlier feelings I still see a design process more focused on finding problems with Java, and duct taping them with Groovy. (Like this: in Groovy, all exceptions get handled automatically, making it look as if all exceptions are unchecked.) I have no problem with many of these features, but I don't think it's a good way to create a general purpose language.

I'll be back later with more info on the rest of the days sessions.

Mingle on JRuby

Here's the deal: ThoughtWorks first product offering is called Mingle. The kicker is, it will be deployed on JRuby. I have been involved with this for a while, getting everything in good working order, and I'm sure TW won't regret this decision. JRuby is important in their strategy, and Mingle, as the first commercial JRuby application is a huge deal.

You should go and read more about this exciting news at http://studios.thoughtworks.com/2007/5/7/mingle-to-run-on-jruby.

Announcing mongrel_jcluster

As I mentioned in my last post, that server/client thing wasn't the only thing I worked on during the flight. I also did a very useful hack of mongrel_cluster, and turned it into mongrel_jcluster. It's not very large changes, actually... Almost everything is the same, except that this mongrel plugin only runs in JRuby, and will start several mongrels in the same JVM.

jruby -S gem install mongrel_jcluster
cd your_rails_app
jruby -S mongrel_rails jcluster::configure -p 4000 -N 3 -e production -R 20202 -K thesecretkey
jruby -S mongrel_rails jcluster::start

The only difference in the configuration parameters are -R and -K which is the JRuby server port and key respectively. Now, after executing these commands, it will take some time for the mongrels to get online, so be patient (or monitor the progress in the log-files generated).

I need to warn you that this is still quite experimental and not guaranteed to work in any way. =) But it does for me.

Another thing, if you start Mongrel in production mode, the defualt Rails front-page will give you an error if you try to get at the properties. This is the expected behavior in production mode, though, and nothing wrong with JRuby.

JRuby server

I spent some time on the flight to SF hacking on an idea I've had for a while. The result was checked in yesterday in JRuby trunk, and the important files are bin/jrubysrv and bin/jrubycli. This is still quite experimental, and only guaranteed to work in *NIX environments right now.

So, what is it? Well, if you don't want the overhead of running one JVM for each JRuby process, these two commands allow you to start a JVM that listens on a specific localhost port, and then you can send JRuby commands to that port. You need a key as password, otherwise the process will fail. Right now, this isn't as high security as it should be, since the key is passed as a command line parameter. This is still not that big of a problem, since the most common scenario is to start a bunch of things, and then not start anything more. Let me illustrate what you can do with a very simple example.

Say that you have a Rails application, and Mongrel for JRuby installed. Go the the base directory of your Rails app and do this:

jrubysrv -p 20202 -k secretkey &
jrubycli -p 20202 -k secretkey -S mongrel_rails -p 4000
jrubycli -p 20202 -k secretkey -S mongrel_rails -p 4001
jrubycli -p 20202 -k secretkey -S mongrel_rails -p 4002
jrubycli -p 20202 -k secretkey -n

Now, if you are patient, in a while there will be three Mongrels running, listening on ports 4000, 4001 and 4002. And they will run inside the same JVM, but in different JRuby runtimes. Quite neat, and very simple. You may wonder about the last command, with the -n flag. That flag tells the JRuby server running at port 20202 to not start any more JRuby processes. In this way, you can start what you need, and then close it down. Due to problems with thread scheduling and safety, I haven't implemented a way to kill a single runtime. What you instead need to do is use jrubycli with the -t flag, which will kill the whole JRuby server.

Stay tuned for my next post, on what I've actually used this functionality to create. (If you have some slight amount of imagination, you should be able to guess from this blog post...)

But now, of to the races. Or CommunityOne in this case. See you there.

San Francisco and Portland

Tomorrow morning I'll leave Stockholm for San Francisco. I'll be arriving 5:ish PM at SFO and will stay at the Handlery Union Square Hotel. I will not present at either JavaOne or RailsConf, though, so I will have a fair amount of time for fun stuff. I will attend the RedMonk unconference at CommunityOne, participate in the DynLang event, and then attend regular JavaOne. If you see me (I'll be wearing my black hat...), please say hi!

Next Saturday me and my coworkers will move out of SF and take a car to Portland, where we plan to arrive on the following Tuesday or Wednesday, just in time for RailsConf. We leave from Portland early Monday after the conference.

Hope to see many of you there! It will be exciting, that's for sure. I will also try to report as much as possible from all events.

JRuby at TheServerSide Java Symposium - Europe

Not sure if I've mentioned this, but I will represent JRuby at TheServerSide Java Symposium - Europe, in Barcelona, June 27-29. I will give one technical session on JRuby, and one BOF on how to deploy JRuby on Rails applications, both which should be quite exciting.

There are also a number of other nice presentations, so try to get to Barcelona!

Ye Zheng joins ThoughtWorks

Sorry if the title gives it away. I've been holding quiet about this for almost a month, but now it's official. Ye Zheng (more famous as dreamhead), who is one of the core developers of XRuby, have joined ThoughtWorks, and will begin work in China in May. He will be much involved in all things Ruby, and will also work on XRuby and JRuby, so we will probably be working close together. I have the highest respect for Ye Zheng and look forward to working with him. He single handedly rewrote most of the runtime in XRuby for the last release, which is not a small feat.

If you read Chinese, you can read more about it at his blog at http://dreamhead.blogbus.com/logs/5155305.html

JRuby 0.9.9 released

The JRuby team is happy to announce that 0.9.9 has now been released. This release have seen much focus on compatibility with Ruby 1.8, and also making the Java Integration features more solid. We are gearing up for a 1.0-release, and 0.9.9 should be pretty close to what you can expect from 1.0, so now is the time to start testing your applications on JRuby in earnest.

Homepage: http://www.jruby.org/
Download: http://dist.codehaus.org/jruby/

Some of the most important points of this release:

• Major compatibility and performance overhaul of String, Array, Hash
• Many YAML and Marshalling issues have been fixed
• Java Integration overhaul fixing many outstanding issues
• 180 Jira issues resolved
• Several more bottlenecks removed
• Rails applications like Mephisto and plugins like Goldberg are running without hitches
• Performance has improved by 40% over 0.9.8 based on YARV benchmarks

In response to all the things that needs doing before 1.0, we have decide to add two new committers to the team. Marcin Mielżyński and Bill Dortch have both done some incredibly nice things for the JRuby project, and we're very happy about them joining up as core developers.

In conjunction with 0.9.9, Rails-integration will see a new release, that improves many parts of it. Since we've actually started using Rails-integration in more places, we have also noticed things that need to be improved; the new release is very good.

ActiveRecord-JDBC 0.2.4 will also be released very soon. We have made some major changes, and much of the core have been rewritten. These changes make MySQL fully compatible with ActiveRecord, and we are working hard on improving support for other databases. Hopefully, the 0.2.4 release will mean Derby support is more or less complete too.

Practical JRuby on Rails

The time of waiting is over. I have hinted over and over that I've been doing some stuff that take lots of time. The first part of this was interviewing for ThoughtWorks, and I told you that a month back. And now I can finally tell you what's keeping me so busy, that I can't blog as much as I used to:

I am writing Practical JRuby on Rails: Web 2.0 projects for APress. I have been since late December, and the book is due to be published in October. But you can actually preorder the book today, from Amazon, here.

Me presenting JRuby

I totally forgot about this; 4-6 weeks ago, I did a presentation for the local Railsgroup here in Stockholm; since Dr Nic was there, I decided to improve it in English. And Dr Nic did us the service of taping the whole presentation; I was sick at the time, it's a short presentation about what's new in JRuby, but it could still be potentially interesting for some of you.

It's available at Google Video, here.

The State of JRuby

It's been sort of quiet on the JRuby front for a few weeks, so I thought I'd give a heads up on what things we're working on at the moment. At the moment, we're planning to release 0.9.9 sometime next week. At that point, several things we're working on right now should be fixed.

• There is a problem with deterministic threading in JRuby, which results in timeout errors sometimes escaping out of their rescue nodes. This is very serious for long running applications and Charles is hard at work providing better Thread semantics for us. The problem is that it's very hard to match MRI's green thread functionality with real native threads.
• Thomas is hard at work with one of his gigantic refactorings. This time the goal is to remove most of ThreadContext and make many parts of the call chain and block invocation chains easier to work with.
• Marcin have been working for some weeks now on a port of the hash implementation in MRI, and a corresponding rewrite of RubyHash. At the same time, he's hard at work rewriting RubyString to allow COW (copy-on-write) semantics. This could potentially improve performance and also make JRuby less memory intense.
• Earlier this week, we identified and fixed two very serious memory leaks. The second of these caused large leakage in all Rails applications using Cgi::Session for session data (which is almost all JRuby on Rails applications, except those running in Rails-integration). At the moment we don't see any leaking for Rails applications, so the situation looks good. (Both of these leaks were found with the help of Java monitoring tools, like JConsole, jmap and jhat. Very useful things.)
• I have been doing some serious YAML fixes these last days, with the result that our YAML situation is better than ever. It even handles recursive objects correctly. (Which is a pain in Java. In C it's easy, though.)

Overall, we're working very hard on the last, quirky compatibility issues right now; we have a few different, large Rails applications that we use to identify strange issues. And also, more and more people are trying their applications on JRuby, which means we get better (and more) bug reports. This is really great.

After 0.9.9 has been released, we're planning on finishing up remaining important bugs. I'm also very keen about getting more databases running really well with Rails. We will try to get Continous Integration set up for these too.

ActiveHibernate - Any takers?

This is a call for action. JRuby on Rails is getting more stable each day, and JRuby performance is consistently improving. This means that JRuby on Rails is well on the path of becoming a viable platform for several kinds of web development.

ActiveRecord-JDBC is central to this, and it's a good project and works very well. Within the limitations of ActiveRecord, of course. So, I'm thinking (and this is by all means not a new thought), that I would like to have the rest of Rails, but using Hibernate as backend. With JRuby approaching 1.0 fast, ActiveHibernate seems like a neat project. The only problem is time. So why is ActiveHibernate tempting? Well, for all those situations where ActiveRecord really doesn't fit; composite keys, complicated legacy systems. Or databases where you would like to use prepared statements for everything. Or get really nice speed.

What needs to be done? For it to be really useful, there are few points: First, a Hibernate extension that serializes and deserializes into RubyObjects. By doing it that way, there is no need to create Java bean classes. Secondly, provide all the useful help functionality around the Hibernate implementation, that AR users have gotten used to. This includes things like validations, automatic handling of updates and inserts, and is generally about doing a good Ruby interface around Hibernate. This also includes creating a good way of configuring Hibernate without having to resort to the XML. Maybe using YAML? Or a Ruby DSL? After that has been done, the final point is easy: get mind share and start doing applications with it! I for one know that I would like to use such a project!

Can your Ruby do this?

Recipe (should be baked with Java 6 for maximum pleasure):

1. Take one large Rails application with good Selenium test base
2. Convert database configuration to use JDBC
3. Start a selenium proxy with "jake test:selenium_proxy"
4. Start acceptance testing from another window with "jake test:acceptance"
5. In yet another window, write "jconsole"
6. Choose your application

What do you get?

You get this for free, just by running JRuby with Java 6. You can attach to any Java process at all. Remotely too. And get this kind of information. Can your Ruby do that?

Mongrel in JRuby

As I have told you before, we have been working on getting Mongrel working in JRuby off and on for a long while. One or two months ago, I got the Ragel definition ported correctly and the rest working. The plan is to cooperate with Zed and create a JRuby-native Mongrel gem out there, but until then I will describe the steps you need to take to get this working for yourself.

First of all, you need to install gem_plugin. The easiest way is to use gem:

jem install -y gem_plugin --no-rdoc --no-ri

After this is done, you can either download Mongrel-support from svn://rubyforge.org/var/svn/jruby-extras/trunk/mongrel-support, and build the extension manually, or you can download it directly here: http://opensource.ologix.com/http11.jar. If you decide to build it yourself, you first need a current version of JRuby from trunk. Check out mongrel-support from the Subversion repository and execute these commands:

cp $JRUBY_HOME/lib/jruby.jar lib
ant jar

After that you will have a http11.jar-file in lib. Regardless of how you get the http11.jar-file, place it in $JRUBY_HOME/lib/ruby/site_ruby/1.8/. Then you can proceed in a few different ways. You could copy the Mongrel-files from your MRI installation, or you could download Mongrel from version control. Regardless of how, you need to copy mongrel.rb, mutex_fix.rb and the mongrel-directory from Mongrel into $JRUBY_HOME/lib/ruby/site_ruby/1.8/. You can also copy bin/mongrel_rails into $JRUBY_HOME/bin and change the shebang to point to /usr/bin/env jruby. After this, you are good to go with JRuby and Mongrel.

On ActiveRecord-JDBC performance

I have been a bit concerned about the performance of our component that connects ActiveRecord with JDBC. Since ActiveRecord demands that every result of a select should be turned into a big array of hashes of strings to strings, I suspected we would be quite inefficient at this, and I wasn't sure I could put all my faith in JDBC either.

So, as a good developer, I decided to test this, with a very small microbenchmark, to see how bad the situation actually was.

Since I really wanted to check the raw database and unmarshalling performance, I decided to not use ActiveRecord classes, but do executions directly. The inner part of my benchmark execution looks like this:

conn.create_table :test_perf, :force => true do |t|
 t.column :one, :string
 t.column :two, :string
end
  
100.times do
 conn.insert("INSERT INTO test_perf(one, two) VALUES('one','two')")
end
  
1000.times do
 conn.select_all("SELECT * FROM test_perf")
end

conn.drop_table :test_perf

It is executed with a recent MySQL Community Edition 5 server, locally, with matching JDBC drivers. The MRI tests is run with 1.8.6, and both use ActiveRecord 1.15.3. ActiveRecord-JDBC is a prerelease of 0.2.4, available from trunk. My machine is an IBM Thinkpad T43p, running Debian. It's 32bit and Java 6.

The results were highly interesting. First, let's see the baseline: the Ruby results:

      user     system      total        real
 7.730000   0.020000   7.750000 (  8.531013)

Frankly, I wasn't that impressed with these numbers. I thought Ruby database performance was better. Oh well. The interesting part is the JRuby AR-JDBC results:

      user     system      total        real
 6.948000   0.000000   6.948000 (  6.948000)

WOW! We're actually faster in this quite interesting test. Not what I had expected at all, but very welcome news indeed. Note that there is still much block overhead in the interpreter, so the results are a little bit skewed in MRI's favour by this, too.

SuperRedCloth

RedCloth is cool. SuperRedCloth is cooler, faster, nicer, and all in all such a treat. And it works on JRuby now. At the time of writing, I haven't really wired together a gem of it yet, but I hope to get it done soon. Until then, if you want to try this, you can download the necessary jar-file from http://opensource.ologix.com/superredcloth_scan.jar, install the (very small) SuperRedCloth lib-file into your JRuby, add the superredcloth_scan.jar into your site_ruby for JRuby and everything should be set to go. Be warned, this requires a very recent trunk version of JRuby (something like newer than April 1:st).

SuperRedCloth is another of those nice Ragel based libraries. I'm finding Ragel such a joy to work with; just change the few C-thingies into Java-thingies and everything works exactly the same. Neat.

Some JRuby updates

So, there is prime time for JRuby now. So much, in fact, that I haven't had time to blog properly for a while. And won't have for a few days more. In the meantime, check what we've been busy with: http://headius.blogspot.com/2007/03/activerecord-100-performance-doubling.html.

Post Rails meetup

Last evening was very nice. We usually have a good time at the Rails meetups, hosted by Valtech. Yesterday it started out with Dr Nic William, who spoke about some metaprogramming techniques. Very neat, and the best part was his Magic Wiggly Lines. I'm not sure if it's genius or mental. But it's cool, at least; it's basically a const_missing hook, which makes it possible to use class names even if you spell them wrong. Hehe.

After that I spoke about new JRuby thing. I have a cold, so I took it pretty easy, and it was very free form with questions. There was many interesting questions, actually, and I think Nic taped it all...

After that, we drank beer and talked technology, which is always fun!

Rails meet in Stockholm

And it's time again for a Rails meetup in Stockholm, at Valtech. This Tuesday (20th) 6pm at Valtechs offices. It is usually very interesting and neat to meet other people involved in Rails and Ruby here in Stockholm, and it's usually very nice, all around.

I will have a short presentation on the new features in JRuby 0.9.8. I will provide a fast intro to JRuby, but this is not the introductionary presentation; I will also try a somewhat different presentation technique, so it is sure to be interesting. At least for me... =)

Show up! It's worth it. You need to reserve a place, though, which can be done at http://www.rails.se/rails/show/Railstr%C3%A4ff+20+Mars+2007

ThoughtWorks

I have resigned from my position at Karolinska Institutet. Instead, I will work for ThoughtWorks. I am extremely excited about this, obviously. I will start in London in the beginning of June. I'll stay in London for 6 months and then move to the San Francisco office.

As you may know, ThoughtWorks is doing lots of Ruby projects in the enterprise and are really helping spread Ruby to the masses. And that's what I will be working with. Ruby and JRuby for ThoughtWorks.

What's more, I will be doing development on the JRuby core at ThoughtWorks!

As an aside, I would like to mention two very interesting things that TW have announced very recently. First, CruiseControl.rb, which looks very nice, and secondly, Mingle. Both of these products have great potential.

I'm just so excited about this!

JRuby Regular Expressions

The Regular Expression support in JRuby is about to be revamped. I will here detail my plans for this work, and also some of the reasons for it. This post is as much for people interested in JRuby, as for the JRuby developers.

Stage 0: java.util.regex
JRuby has traditionally used java.util.regex. We stopped doing that March 11:th, 2007. The main reasons are because of the disconnect with MRI. Some of the operators work very differently, there are some problems with UTF-8, we can't support SJIS or EUC, nor posix-classes. And java.util.regex also uses a recursive implementation which means it can't handle certain large inputs. Further, we would like to be able to modify the implementation to work with the same stuff that backs the RubyString, to increase performance.

Stage 1: JRegex
Yesterday (March 11:th), I merged JRegex as the main regular expression engine for JRuby. The main reasons for this is twofold. First, we can change the implementation quite easily, and second, it is an iterative algorithm, which means it doesn't fail on the input that java.util.regex does. Since this caused problems with some Rails tests (and also in multipart handling in all libraries using cgi), I decided to merge this as a stopgap until the next incarnation of regex support.

Stage 2: REJ
In about 2 weeks, I hope to be able to merge REJ with JRuby. At the point of merging, it should be a better replacement than both JRegex and java.util.regex. REJ is a project I've started, which will be a direct port of the MRI 1.8.6 regular expression engine. The important thing about this is that the semantics for JRuby will match MRI very closely. We will be able to match UTF-8, SJIS and EUC regular expressions, and we are able to have the same quirks as MRI, even though people shouldn't depend on such quirks. In the process of writing REJ, I will also create a large suite of test cases for regular expressions, based on Henry Spencer's test files. I'll probably submit something initial to a separate repository very soon. If I get my wish, REJ is what will be the regular expression engine for JRuby 1.0.

Stage 3: Ojiguruma
After 1.0 has been released, I think it's time to make the Regexp engine in JRuby really extensible, and provide an interface from Ruby to change which engine to use. After that is done, I would be very interested in doing a port of Oniguruma to Java, which would give us far better multilanguage support, and also some interesting features. The reason I'm choosing to not do this right now is because Oniguruma is just too large.

Stage 4: (No official name yet)
Another engine that some in the JRuby/Ruby community has started working on is an engine which will be based on Ragel for parsing and a modified version of Thompson NFA and TCL-style backreferences for matching. It's an interesting project but it will take some time before it's usable.

Two more days in Kraków

I was very lazy on Saturday. Took it very easy, then went to a dinner with some of the presenters and organizers for SFI. And after that I gave my presentation, which I personally felt went very well. I had to adapt the technical level some, because it seemed most of the audience didn't know much about compilers. After that, Lukas held his presentation about Seaside, which also went well.

Then we celebrated. Then there was headache. And a look at the Cathedral. And a taxi driver who didn't understand one word of English. And he took me to the wrong terminal at the airport. But I finally got on the right flight and got home.

Two days in Kraków

So, I've been in Kraków for two days now. I will fly back on Sunday. It's a very interesting town. On one part, it's lovely. The buildings are absolutely amazing, and very beautiful. On the other hand, many of them are very run down, and the wear and tear is obvious all around. To me, the whole town seems sort of depressing, but on the other hand the people here are very up beat, and from some of the presentations I've seen, the technological future for Kraków looks very bright indeed.

I haven't really been able to see most of the presentations. Almost all have been in Polish, sadly enough. I did see two interesting ones yesterday. The first was Michael Foords talk about IronPython. This was very neat and I've talked some with him after that too. I'll get back to that. The other talk was about Google Ads, by Greg Badros. As always, hearing about how Google does things internally is always amazingly interesting. I would have enjoyed hearing more about the machine learning and NLP stuff they're doing, but obviously they can't discuss that too much.

Today I haven't been able to see any presentation, due to Polish. On the other hand me, Michael and Lukas Renggli have had some very interesting discussions both yesterday and today. Now, the company Michael's part of has the largest IronPython code base in the world, as far as I understand it. It's basically 90 000 LOC, where 20 000 is production code and 70 000 are testing code. That sounds about right... =)

Lukas Renggli is one of the core developers of Seaside, a framework which I'm quite fond of. What we three have in common is our interest in dynamic languages, so we had some very common ground to talk about. What's nice is that many of the things Michael talked about in his presentation is stuff we in JRuby also speak much about. In the same manner, it seems the IronPython guys have had basically the same problems we in JRuby has had. It seems there is some common ground to be found here, and possibly also a basis for conversations. I for one would find that very interesting, since I enjoy hearing about dynamic languages getting foot holds on statically typed virtual machines; that is just such a sweet spot.

Tomorrow evening is my talk. I'm going to be at an official dinner in the middle of the day, and then get over to the conference and give my presentation at 17:00. I think it's going to be interesting, and it will be an obvious counterpart to Michael's talk, since the reason for existing is so similar for IronPython and JRuby. It will be interesting to see how the audience react.

After that, I'll see Lukas talk about Seaside, and after that people are talking about having a party, so tomorrow will be quite intense. And finally, on Sunday I'm going back to Sweden. As far as I know now, I won't travel that much for at least a month now, which feels sort of nice. This week has been far to intense, and also have had some really great - but tiring - moments.

Tomorrow is today - or from London to Stockholm to Krakow

Part I: London

I got up at 5am, got dressed and packed and headed of to the airport. Finally got on the plane to London (thank god for online check-in) and tried to get some work done. That didn't go very well, but I got to read some instead. I'm rereading Ubik. Classic. Landed at Heathrow about 10 minutes behind schedule. Ran to the Piccadilly line, tried to talk with people in Sweden, but the connection was kinda bad. Arrived at Covent Garden, walked in brisk pace to the place of my meeting. Finally arrived, 5 minutes late, to find out there was no hurry. Sat down and had a pleasant chat with Dan North and Simon Stewart. Very nice. (Hey Dan, if you read this, get in touch when you're in Stockholm for ExpertZone. I know places with good beer.).

Then, Lunch. Haha. What a joke. I found a place that seemed to offer great vegetarian burgers, so I settled down for a quorn burger and some British chips. After 15 minutes waiting I got told the grill wouldn't light up. Lucky me. Instead, I had to make do with a Pret sandwich and a spicy vegetable Cornish pasty (someone told me it was the day of the Cornish patron saint yesterday, so I guess that fits). A few minutes walk on Oxford street was nice. I got photographed by some guys from an Austrian fashion magazine. Then back for the real meeting.

Cyndi Mitchell is a very formidable person. I was impressed. We had a 2 hour talk; very interesting. I will tell you all about it as soon as I can. But suffice to say, I was very happy when I left. Until I realized that I had exactly 2 hours until my flight taxed out from Heathrow. So, back to Covent Garden. Back on the underground. Which through some bad communication on the drivers part, almost ended badly when the train reached the end stop before Heathrow. Thanks to the people who told me to get of the train and switch to the other side. =) I finally got to Terminal 3, realized I already had a boarding card, found my way to the security check, got my boots scanned two times, and found the gate about 20 times before takeoff. Of course, as soon as the aircraft had taxed out, the flight management decided that we should stay 30 minutes on ground before finally taking off towards Copenhagen. It's a wonder I actually got there in time. So, in Copenhagen I walked 2 klicks to the transfer center, got a new boarding card and walked back to almost the exact place I started from, got on my flight to Stockholm and finally landed. Wow.

Airport coach to Stockholm city, and night bus home to my dear old mental hospital. When I finally got to bed I ached all over, and had been awake and active for 22 hours solid. Sleep was nice at that point. Especially since the day had been a success on all accounts.

Part II: Stockholm

I finally woke up, took care of mail and told everyone about London. Thought I'd take it easy, maybe practice on my presentation for Saturday, possibly start writing on the presentations for the CS course I'm going to speak for this April. But alas, after some mailing with Poland, I realized I've made a big mistake. I thought I was going to Kraków this Friday, and staying until Sunday. That's not entirely right. I'm actually going tomorrow. Surprise!

So.

Part III: Kraków

As mentioned above, I'll land in Kraków tomorrow (Wednesday) evening. This is good in several ways. I will have a chance to see the city, and I will have the opportunity to see some more presentations at the conference I'm at. There are many interesting things to be had; I'm sad half of them are in Polish. I guess I can look at the slides and have some fun trying the guess the meaning. I am going to see the talks about IronPython, Google Ads and Seaside. And, of course, if anyone is nearby and would like to take a beer and talk computers, programming languages, AI, or whatnot, please hit me a mail, or a comment on this post. My evenings will be mostly free it seems.

That is, unless someone knows any good clubs with nice music; especially electronic stuff and/or postpunk, new wave, darkwave or gothy things.

Anyway. Time to sleep.

JRuby 0.9.8 is here

The JRuby team is pleased to announce the release of JRuby 0.9.8.

Download at: http://dist.codehaus.org/jruby/

This release has some great improvements:

• Ruby on Rails support. We have been working hard on getting Rails own unit tests running and over 98% of them now run successfully. We feel things are running well enough to invite Ruby users to kick the tires and help root out any final issues.
• Ruby classes can extend concrete/abstract Java classes and override methods
• New Java primitive array syntax
• Reimplementation of String, Numeric classes, and Array to be more correct and performant
• Significant bottlenecks have been identified. In some cases IO is 6.5x faster than previous releases. Java included classes are significantly faster than in the past.
• 220 Jira issues resolved since last release

Special thanks to Marcin Mielżyński for his tireless work in rewriting a number of core classes to be much for correct and quick. His attention to detail has rooted out many corner cases.

The amount of IRC conversations, mailing list threads, bug reports, patches, and blog entries in the community has been a great help and our community is really making a huge difference in how fast JRuby is improving. The amount of progress is really staggering!

If you have ever thought that JRuby wasn't mature enough, I would like to contradict that now. With this release we are better than ever.

More information can be found at http://www.jruby.org.

The world is spinning

If you didn't know that, the title tells it all. Of course, that's old news.

I haven't really been able to blog as much as I wish I could have, lately. There are reasons for this, of course. Two very exciting reasons, in fact. If everything pans out, I will be able to write about it in 2 to 3 weeks time.

In other news, we are gearing up for another JRuby release. This one will be a biggie. Many nice things will be in place, and it will set the record for both new features and bug fixes. I think no one will be disappointed by it, actually.

I have a few presentations lined up too. The closest to now will be in exactly two weeks. I will speak at the Academic Computer Science Festival in Craków, Poland. If you're somewhere close, by all means offer to show me the city. =) I will land March 9:th and fly out again March 11:th. My presentation will be at 17:00 March 10:th, CET. If you would like more information, it can be found at the festivals homepage, here. The presentation will be in English (since I don't speak Polish, obviously), and it will be slightly more technical than the usual JRuby presentations. There will probably be some detail about our runtime, interpreter, parser and lexer, and hopefully I'll get some info in about our YARV and Java bytecode compiler efforts. This will be very exciting to talk about, I'm kinda salivating just thinking about it. =)

Another, more long term presentation, has just been decided. I will attend TheServerSide Java Symposium Europe, in Barcelona, from June 27:th to June 29:th, and talk about JRuby from the perspective of a Java developer, and what it can do for you. Hopefully I will have time to see the city too.

I will update with more information when possible.

The new base

I have for a time argued that the JVM should become more like an Operating System, and Java the language for OS development. Other languages should run on top of the JVM for running applications. It seems I'm not alone in this line of thought. Robert Varttinen wrote some about it here. To me it seems like a compelling future. But it seems the next logical step for enterprise applications would be further virtualization. I would for example like my beans implemented in Ruby. But not only that, I would want all the business logic to be OS agnostic. I would like my Ruby logic to live on top of a JVM J2EE server, but that logic should be able to move, transparently, to a .NET-server and provide the same business logic at that place. What would be even better is if I didn't have to deploy it manually to all places, but the logic would just move to the places where it's needed. Will we see that anytime soon?

RailsConf galore

So, it's decided. Me and two colleagues from Karolinska Institutet will attend RailsConf in Portland, OR. I'm looking very much forward to it, especially since we are also going to JavaOne, and we plan to drive from San Francisco to Portland.

Hopefully I'll get to meet many of the people on the Ruby scene that I've only had mail contact with until now. See you there!

Ragel performance

I did some performance testing on the old and new Resolver implementation. The testing have some stupid tests that exercise bad parts of both implementations (like longest match, where it can't be decided what type something is until we have to backtrack about 20 characters). I placed these 24 strings in an array, and pounded on it with an instance of the ResolverImpl that is used in exactly the same way on all scalar values in an YAML document. The objective is to find out if the value is an implicit type or not. So basically, we give it a String, and get back a tag URI. So it's not like I'm parsing a language or anything. I'm just doing some recognizing here.

The old implementation was based on a Map> where the first letter of the string to resolve was used as an index to find a list of patterns to try sequentially. This worked fine, and made it extensible. But not very fast. This is the baseline. For 24 different strings, iterated 100 000 times for 2 400 000 resolves it takes 7879ms. That's OK, but not great.

Now, the new Ragel implementation is dead simple. It's just a translation of the regexps in the aforementioned Pattern's into a state machine. At EOF out actions (%/ for people in the Ragel knowhow), I execute an action that sets a local variable to a tag, and at the end of the resolve method returns that tag. Dead simple, and not exercising the full strength of Ragel, of course.
So, for the same number of resolves, this ResolverImpl takes 1288ms. That's 611% improvement in speed. Ain't it nice to have a friend such as Ragel? And the best part is, for harder tasks, these improvements would be even larger.
Finite State Machines are your friends. All your base are belongs to us.

Results of jvYAMLb

Well, the YAML-based loading is in JRuby trunk. On the way, some parts of the codebase got seriously simplified. Very nice. The final result, with regard to performance, is about 20-30% on speed. But the important gain is in memory usage. The new implementation takes only about one fourth of the memory the original used. So that's great.

Regarding the Resolver, as I mentioned in the last post, it required a different approach, since regular JvYAML uses regular expressions to recognize implicit tags. Since that approach isn't good with byte arrays, I decided to use Ragel to generate a recognizer. That approach was very successful. As soon as I got that working it was the obvious approach. Ragel is good. Ragel is great. Ragel is wonderful. I will use the same approach for regular JvYAML to get away from all those Java regexps.

So, next step will be to do the same conversion of the emitter. Of course, at that point performance isn't that important. It's more about memory usage and the need to get away from another external dependency in JRuby.

Faster YAML with byte processing

As noted in my last post, I have started work on converting JvYAML into JvYAMLb. Right now I have finished the work on the Scanner and the Parser, and it's looking quite good. The numbers I reported in the last post for regular JvYAML performance was wrong though. We're looking at about 7.8s to 10.0s for scanning that 3.5MB gemspec file. (And that's only the scanning, not file IO). But with the Scanner converted to use bytes and ByteList, the same processing takes 2.8s. That's a substantial difference. But it doesn't end with that.

As I said I also converted the Parser. It doesn't do any String processing at all, so I didn't expect either a speedup or slowdown except for that from the Scanner. But... Before, parsing the gemspec took 18.515s, but after, it runs in 4s. That's a dramatic speedup, and I don't really know where it comes from. Unless the earlier implementation generated so much more garbage, and used more memory, that it was noticeable in speed. Anyway, this looks good for JRuby YAML processing, since I expect big reductions in complexity in the callpath and generation of objects after the YAML processor is byted all the way through.

But tomorrow it's time to work on the Resolver, and that's going to be hard. Optimally, it would be nice to have a byte-based Regexp engine. And maybe that would be something for JRuby too, know? Our Regular Expressions must be dead slow now that they have to convert to strings all the time.

Announcing JvYAMLb, a fork

The conversion to using byte-arrays as the basis of our String work in JRuby has led me to realize that JvYAML just doesn't cut it anymore. The performance wasn't good to begin with, and it's even worse having to convert EVERY SINGLE STRING read into bytes. That's no good. As an example why something needs to be done I'm going to describe the transformations that happen to data in JRuby if executing this code:

YAML.load_file "gems.yml"

First, the file is opened, and wrapped inside a RandomAccessFile. Then data is read from it by YAML. Reading will proceed like this:
1. Bytes are read through the RAF, hopefully in chunks.
2. Those bytes are wrapped in a RubyString so they can be returned from the IO#read method.
3. An IOReader wraps that RubyIO object, gets the RubyString and converts it from bytes into a String, and this String gets converted into a char array.
4. That char array is returned to the YAML Scanner.
5. The chars from the char array is collected in a StringBuffer, and saved in various Strings as token values.
6. The parser, resolver and constructor work on these Strings in various ways.
7. The JRubyConstructor takes these Strings and creates RubyString objects from them and in the process converting the String back to a byte array.

Is there any doubt that this process is slow? Well, it hasn't been that big of a problem until now, since we are doing so well on performance in other parts of the system.

So, the radical decision is to rewrite JvYAML, making it more SYCK-compliant, working with InputStreams and byte-arrays, and in the process get away from several of the steps above. So that's what I'm going to do. I hereby create JvYAMLb. It will only be a part of the JRuby codebase, but it will be reasonably separate, so it can be extracted for other purposes. I will not stop work on regular JvYAML, but will maintain both projects.

Since the objective of this new project is blazing speed, I will post some numbers on this now and again. But first I will show you the speed of the regular system. JvYAML's Scanner can scan an old gem source index (about 3.5MB) of 435654 tokens in about 1654ms. This is the baseline I'm going to use to test performance, and I'll post more on this as soon as the byte-based Scanner is ready to try out.

Bytes bites. Or maybe not.

Well, the byte arrays are in, for good and evil. We had to wrap them in a counterpart to StringBuffer, but backed by byte[] instead, since all that explicit allocation and deallocation was way unperformant.

Of course, we aren't seeing any performance benefits from this right now. The problem is that there is still many places that use IRubyObject#toString to get at the contents. That operation is very expensive right now, so gem installs are slower, for example. But we have good hopes on improving the situation, and many parts of the codebase have become much clearer without the need to do String-to-byte[] and byte[]-to-String all over the place.

Fractured blogging

My blogging in the future will be a little bit fractured (or more fractured, some might say), since I have been invited to write at Inside Java for APress. The address is http://java.apress.com and I do recommend that you subscribe to the feed if you're interested in Java or associated technologies. My first posting was about the two closure proposals for Java 7, and I will try to focus my posting there to be more Java specific, mostly in article format. But if I write anything I deem to be exceptionally good, I promise to link from here to there.

Go subscribe now!

Serial JRuby

Things are really moving along faster than ever in JRuby land. It's so fun! As my last entry told you, Hpricot is now available for JRuby (and Java) people. I need to share a few lines from the logs of yesterday evenings conversation at #jruby:

<headius> seeya ola!
* shellac does some xsl-ing, plays on the wii, 
  then finds ola got HPRICOT working in that time
<shellac> I'm wasting my life

Some would say that what I do with JRuby is a waste of life... Well, we'll see about that.

Anyway, what's happened in JRuby world since last week? First, and most important, Charles has changed our RubyString implementation. It used to be backed by either a Java String or a StringBuffer. The problem with both of these is that Ruby has a tendency to use Strings as byte buckets. And our code was riddled with encoding and decoding into and out of byte arrays. So Charles took the big step, converted RubyString to use a byte-array instead, and fixed all the bugs that he found by doing that. The result is a happier codebase, less encoding and possibly faster Zlib and IO operations. That's big.

Tom is working on removing visibility and refactoring scopes. That could have huge impact too.

This Sunday I merged and fixed some code that allow Ruby code to inherit from Java classes and override methods there, and this overriding will be seen if an instance is sent back to Java. I'm planning on using this for some interesting tricks with Java ContentHandler's, and this functionality is really, really, really important. But it's also complex, since it requires generating bytecode at runtime. Fun, but hard. But now it's in trunk, and it's time to find the bugs in it and fix them.

I also need you to go read what Jonas Bonér has done with JRuby and OpenTerracotta. I could describe it here, but Jonas does a good job of it himself. So go there: http://jonasboner.com/2007/02/05/clustering-jruby-with-open-terracotta/. Very cool stuff, indeed!

So, the future is coming faster each day. JRuby will still conquer the world!

Hpricot goodness

This is just so cool, I cannot contain it. For those of you who haven't heard about Hpricot, it is one of why the lucky stiff's incredibly cool tools (which he probably will use to take over the world any day now...). It's HTML parsing goodness, very flexible, with the goal of being able to parse (and fix) everything that Firefox handles.

"So what?" you're probably asking... Well, Hpricot uses Ragel and some C code to achieve blinding speed. This means JRuby can't run it. Or I should say couldn't run it:

orpheus:~/workspace/jruby> jruby bin/gem install hpricot --source http://code.whytheluckystiff.net
Bulk updating Gem source index for: http://code.whytheluckystiff.net
Select which gem to install for your platform (java)
1. hpricot 0.5.110 (jruby)
2. hpricot 0.5.110 (mswin32)
3. hpricot 0.5.110 (ruby)
4. hpricot 0.5 (ruby)
5. hpricot 0.5 (mswin32)
6. hpricot 0.5.0 (ruby)
7. hpricot 0.5.0 (mswin32)
8. hpricot 0.4.99 (ruby)
9. hpricot 0.4.99 (mswin32)
10. hpricot 0.4.92 (ruby)
11. hpricot 0.4.92 (mswin32)
12. Skip this gem
13. Cancel installation
> 1
Successfully installed hpricot-0.5.110-jruby
Installing ri documentation for hpricot-0.5.110-jruby...
Installing RDoc documentation for hpricot-0.5.110-jruby...

That's right, Hpricot is now more promiscuous than any other gem with native parts.
What can you do with it? Well, I'm just going to point you to _why's own description of it. All he says at http://code.whytheluckystiff.net/hpricot/ will work fine in JRuby!

How did this come to be? Well, me and _why did some joint hacking, which was helped along by the fact that Adrian Thurston (the genius behind Ragel) recently added Java support to it. So, basically, most of the Ragel definition is exactly the same for both the C and the Java versions. The native code has been factored out, and both versions are buildable with rake from _why's code repository.

This is important. Don't think anything else. This strategy will, and can, be used for other gems with native parts. It's just a question of time.