Concurrent DSLs

With all the current talk of DSLs and concurrency, what I find lacking is discussions about how to combine the two. Of course, domain specific languages are incredibly important - they create a logical separation between the implementors of the business logic, and the people implementing the actual implementation of the DSL. Does it seem like a strange idea to want many DSLs to be able to run parallel to each other? I would imagine that in most cases a DSL that describes business rules and business logic is sequential in the particulars, but that there are also larger concurrency possibilities. This should be totally invisible for the business rule implementor in most cases - the runtime system should be able to run everything as efficient as possible.

A natural way of looking at DSLs is as declarative languages. In many cases that's the way you write them (just look at the ActiveRecord API. it looks very declarative - it just happens to be implemented using imperative primitives). Now, if the language is truly declarative it should be side effect free. In the end, that isn't a real goal, but if it would be possible to clearly understand which parts of the rules are using side effects, the rest of the implementation should be able to run totally concurrently.

These kinds of things should be possible to implement in any language with sane multi threading/multi processing. That said, I wouldn't want the task of doing it with Java's concurrency primitives if I can help it. So what kind of tools would be helpful? Possibly Erlang of course, since it's already functional and that makes the identification of side effects much easier. Another possible alternative seems to be Gambit Scheme and Termite.

Anyone else thinking about these issues? Is there any research going on that would shed some light on it? And further, what's the next step? Why haven't this question already been discussed? It seems to be well time for it now.

Should Ruby have optional typing and compiler directives?

I'm starting to miss a few Common Lisp special forms more and more. To be more specific, I miss the family of declare/declaim/proclaim. I think a version of them could actually be extremely useful for Ruby. The current approach to String encodings in 1.9.1 uses a "pragma" inside of a comment on the first line of a Ruby file, like this:

# coding=utf-8

This works fine for coding, I guess, but I would like to have something more general - something that can be used inside of code too. As a typical example of the kind of thing I would love to be able to do, here is an example of a Common Lisp implementation of fib:

(defun fib (n)
(cond
  ((or (eql n 0) (eql n 1)) n)
  (t (+
      (fib (- n 1))
      (fib (- n 2))))))

As you can see, it's the standard implementation. But, after this implementation is finished, I can continue by adding declarations:

(defun fib (n)
(declare (integer n))
(cond
  ((or (eql n 0) (eql n 1)) n)
  (t (+
      (fib (- n 1))
      (fib (- n 2))))))

This will help the compiler by telling that the n-argument is always an integer. You can also add more information to the declaration:

(defun fib (n)
(declare (integer n)
         (optimize (safety 0)
                   (speed 3)))
(cond
  ((or (eql n 0) (eql n 1)) n)
  (t (+
      (fib (- n 1))
      (fib (- n 2))))))

Now we're looking at some directives about how the compiler prioritize between safety and speed. You can also add directives for how much debug information should be retained or if the compiler should improve speed or size. All of this is interesting, but if I compile it with SBCL I'll get a few warnings. I've specified I want a high level of speed and low safety, but there are many optimizations that SBCL still can't do, so it warns about them. In most cases it's because there is no way to see if the result of the fib-operations return numbers, floats or rationals. If fib is seen as a bottleneck, the way to fix this is to add declarations inline the code in the style of this: (the numeric (- n 1)) and so on. This will allow the compiler to generate the best code possible.

Now, these kinds of changes are quite intrusive, and they aren't really something you want all over your code base. That's why you can do these declarations selectively, at only the places where it makes sense from a performance perspective to do it. By doing it correctly, Common Lisp code can be compiled to be incredibly efficient and fast, but very clean since in most cases you don't add the type declarations at all.

This is also why Common Lisp is very good for rapid prototyping. You can get something working really quickly, but you can also change the code to be efficient later on with quite small changes.

I guess you all wonder what this has got to do with Ruby. Or maybe you don't. Anyway - I want this in Ruby, or something equivalent to it. Not necessarily the exactly same thing, but something which in a standard way can add type declarations and also other things that can be interesting to know from a compiler perspective. It needs to be a keyword with specific syntax so the information is available before runtime. It should probably be used for the encoding declarations too, so it should be possible to use either at top level, or within class declarations , or method definitions. What do I want to be able to do? Well, type declarations is the first one. The second is compiler directives that can help improve code more than is possible right now. A typical example could look like this, maybe:

declare encoding: utf-8

def test_one(first, second, *rest)
 declare type: [Fixnum first, Enumerable second, Array rest]
 declare return: Fixnum
 declare compiler: [no-bindings, no-eval, no-closures]

 puts "hello"
 return 1+1
end

This code includes several things that would be very useful to know and would allow compilers like JRuby, XRuby, YARV, Rubinius and IronRuby to produce much better code. If we know that the arguments should always be specific types we can declare this and hopefully get better results. The return value is also usually the same type and declaring this can make code that uses this method more efficient. Finally, the compiler declarations will help with some of the pain points for currently compiling things efficiently. If no bindings, or evals or closures are used, the code generated will be much better.

This kind of information would be very useful not only for the compiler but also for tool support. If you know something should always be a specific type, you can add that as a declaration and things will automatically function better.

I know that this is kind of un-Ruby like, but I really believe there is great value to be had by adding this kind of declarations.

JavaScript 2

After all discussions about the NBL and all surrounding issues, I realized I haven't actually looked at what JavaScript 2/ECMAScript 4 will contain. I decided to do some research. What I found was mostly very old (2 years or so), but very enticing. If only half of it comes true, JS2 will be a power house. Of course, the C inspired JS syntax is still kind of repugnant, and it's hard to see any kind of general purpose macro facility working on it. Pattern matching is the only reasonable way of getting macros in there, me thinks, and that just doesn't pack the same power.

But still, there are seriously nice things there:

• Classes (the classic kind), packages and namespaces
• Destructuring
• Generators and iterators
• A real numeric tower
• Optional static typing

The end result is that JS2 will look much more like Java, but pack greater power in many ways. I wouldn't say that I agree about JS2 being the NBL, but we're still talking some nice innovation here.

ThoughtWorks Immersion

I've just come back from Bangalore, India where I spent two weeks with intense learning, much socializing and incredible amounts of fun. In short, I've attended ThoughtWorks Immersion. I found it to be a singular experience, and something that will be immensely helpful during my career at ThoughtWorks.

So what is Immersion? Basically, it's two weeks of learning that all ThoughtWorkers go through. The aim is not really to teach technical concepts or specific ways of working, but more to establish a common vocabulary, get everyone to understand the history of ThoughtWorks, and also give insight into all parts of the organization. How everything works and how we do things. A large part also is about what makes ThoughtWorks different from other companies, both in the way we do things but also the why of it all. The values TWers should share and what makes TW a unique place to work.

I feel I've learned incredibly much. The two weeks away from regular work was hard but definitely worth it. I wish all companies had something like this - but I guess that this is one of the ways that make TW a very different place.

The learning side was important, but the social aspect probably even more so. I now have 13 new friends within ThoughtWorks, spread all over the world. We had a good time together. Also, Bill Kimmel and Naresh Jain turned out to be outstanding teachers and very nice people to boot.

Of course, the stay in Bangalore got even more productive when Roy Singham, Chad Wathington, Cyndi Mitchell and a few other people arrived there. I spent some time with them, planning and scheming.

Meeting Håkan Råberg and seeing the cool stuff he's working on (still secret) was also very nice.

All in all, two weeks well spent.

Upcoming

Next week I'll be in Berlin for RailsConf EU. I'll be speaking about JRuby at ThoughtWorks, and hopefully I'll meet lots of you there.

ThoughtWorks is actually massively involved in this RailsConf, and I'm hoping for the same kind of energy I observed at RailsConf in Portland.

See you next week!

On the road again...

Bangalore, India - here I come!

My presentation from JavaBin

If someone is interested in the presentation I did at javaBin in Bergen and Oslo, it can be found here.

An introduction to categories of type systems

Since the current world is moving away from languages in the classical imperative paradigm, it's more and more important to understand the fundamental type differences between programming languages. I've seen over and over that this is still something people are confused by. This post won't give you all you need - for that I recommend Programming Language Pragmatics by Michael L. Scott, a very good book.

Right now, I just wanted to minimize the confusion that abounds surrounding two ways of categorizing programming languages. Namely strong versus weak typing and dynamic versus static typing.

The first you need to know is that these two typings are independent of each other, meaning that there are four different types of languages.

First, strong vs weak: A strongly typed language is a language where a value always have the same type, and you need to apply explicit conversions to turn a value into another type. Java is a strongly typed language. Conversely, C is a weakly typed language.

Secondly, dynamic vs static: A static language can usually be recognized by the presence of a compiler. This is not the full story, though - there are compilers for Lisp and Smalltalk, which are dynamic. Static typing basically means that the type of every variable is known at compile time. This is usually handled by either static type declarations or type inference. This is why Scala is actually statically typed, but looks like a dynamic language in many cases. C, C++, Java and most mainstream languages are statically typed. Visual Basic, JavaScript, Lisp, Ruby, Smalltalk and most "scripting" languages are dynamically typed.

See, that's not too hard, is it? So, when I say that Ruby is a strongly, dynamically typed language, you know what that means?

C is a actually an interesting beast to classify. It's the only weakly, statically typed language I can think of right now. Anyone has any more examples?

To find out more, read the book above, or look up "Type systems" on Wikipedia.

Plans - Oslo and Sweden

I'm right now in Bergen, but I'm flying to Oslo later today - I'll basically land and head directly to the javaBin meeting. Afterwards I'm up for beers or something, if someone wants to.

On Friday I'm heading for Sweden. For purely recreational purposes. I'll be in Gothenburg, Malmö and Stockholm for a week or so, so if someone feels like saying hi, do tell. I always like good discussions about programming languages. And other things as well, in fact.

Comprehensions in Ruby

Now, this is totally awesome. Jay found a way to implement generic comprehensions in Ruby. In twenty lines of code. It's really quite obvious when you look at it. An example:

(1..100).find_all &it % 2 == 0
%w'sdfgsdfg foo bazar bara'.sort_by(&its.length).map &it.reverse.capitalize

%w'sdfgsdfg foo bazar bara'.map &:to_sym
%w'sdfgsdfg foo bazar bara'.map &it.to_sym

Now, the last two lines are what I like the most. The Symbol to_proc tric is widely used, but I actually think the comprehension version is more readable. It's even better if replacing "map " with "collect".

This is actually real fun. You can link how many methods you like - you can send arguments to the methods, you can send blocks to them, you can link and nest however you want. It apples for Arrays, Sets, Hashes - everything you can send a block to can use this, so it's not limited to collections or anything like that.

I think it's really, really cool, and it should be part of Facets, ActiveSupport, hell, the Ruby core library. I want it there.

Now, the only, only, only quibble I have with it... He choose to call it "The Methodphitamine". You should actually use 'require "methodphitamine"'. It's a gem. I love it. But I hate the name. So, read more in his blog, here: http://jicksta.com/articles/2007/08/04/the-methodphitamine.

New double-operators in Ruby

So, I had a great time at the London RUG tonight, and Jay Phillips managed to show me two really neat Ruby tricks. The first one is in this post, and the next one is about the second.

Now, Ruby has a limited amount of operator overloading, but that is in most cases limited by what is predefined. There is actually a category of operators that are not available be the regular syntax, but that can still be created. I'm talking about almost all the regular single operators followed by either a plus sign or a minus sign.

Right now, I don't have a perfect example of where this is useful, but I guess someone will come up with it. You can use it in all cases where you want to be able to use stuff like binary ++, binary --, /-, *+, *-, %-, %+, %^, and so forth.

This trick makes use of the parsing of unary operators combined with binary operators. So, for example, this code:

module FixProxy; end

class String
 def +@
   extend FixProxy
 end
 alias __old_plus +
 def +(val)
   if val.kind_of?(FixProxy)
     __old_plus(" BAR " << val)
   else
     __old_plus(val)
  end
 end
end

puts("Foo" ++ "Baz")

will actually output "Foo BAR Baz".

I'm pretty certain someone kind find a good use for it.

London RUG tonight

Anyone showing up for the London RUG tonight? Seems like it will be interesting: Ruby2Ruby and ParseTree, and also Jay Phillips about Adhearsion (which is really cool).

I will not be presenting, so don't worry about that. =)

Hope to see you there.

ActiveHibernate is happening

So, it seems someone stepped up to the challenge and started working on ActiveHibernate from me initial idea. The code got pushed online a while back and I have finally had the time to look at it. Over all it looks really nice.

Of course, it's very early times. I've submitted some patches making the configuration aspect easier and more AcitveRecord like - but it's going to be important to have access to all the features of Hibernate too. When in the balance, I don't think we should aim for total ActiveRecord equivalence.

Read more here and find the Google Code project here.

I would encourage everyone interested in a merger between Rails and Hibernate to take a look at this. Now is the time to come with advice on how it should work, what it should do and how it should look like.

RailsConf Europe

I am a speaker at RailsConf Europe in Berlin. I'm going to talk about JRuby on Rails at ThoughtWorks. Hopefully many of you'll make it there! As far as I know, it seems the whole core JRuby team may be together for the first time ever, so it's going to be a great conference for JRuby.

See you in Berlin!

NetBeans Ruby support

It's very nice. Each time I get an update from Tor Norbye's blog with screenshots from the NetBeans Ruby support I see a new interesting feature. What I saw this time (here) was that the gotcha from my entry called "Ruby is hard" can now be detected by NetBeans. It's not really too hard, since NetBeans have the parse trees. But it's still a very nice feature to have in your editor.

But I ain't switching from Emacs yet...

Ruby is hard

I have been doing Ruby programming for several years now, and been closely involved with the JRuby implementation of it for about 20 months. I think I know the language pretty well, and I think I have for a long time. This has blinded me to something which I've just recently started to recognize. Ruby is really hard. It's hard in the same way LISP is hard. But there are also lots of gotchas and small things that can trip you up.

Ruby is a really nice language. It's very easy to learn and get started in, and you will be productive extremely quickly, but to actually master the language takes much time. I think this is underestimated in many circles. Ruby uses lots of language constructs which are extremely powerful, but they require you to shift your thinking - especially when coming from Java, which many current Ruby converts do.

As a small example of a gotcha, what does this code do:

class Foo
 attr_accessor :bar
  def initialize(value)
   bar = value
 end
end

puts Foo.new(42).bar

If your answer is that it prints 42, you're wrong. It will print nil. What's the necessary change to fix this? Introduce self:

class Foo
 attr_accessor :bar
 def initialize(value)
  self.bar = value
 end
end

puts Foo.new(42).bar

You could argue that this behavior is annoying. That it's bad. That it's wrong. But that's the way the language works, and I've seen this problem in many code bases - including my own - so it's worth keeping an eye open for it.

Why does it happen? Well, the problem is this, when the Ruby parser finds a word that begins with an underscore or a lower case letter, and where that word is followed by an equals sign, there is no way for the parser to know if this is a method call (the method foo=) or a variable assignment. The fall back of the parser is to assume it is a variable assignment and let you specify explicitly when it should be a VCall. So what happens in the first example is that a new local variable called "bar" will be introduced and assigned to.

JRuby presentation in Bergen

Next Wednesday (the 15th) I will present about JRuby in Bergen. I mentioned this in an earlier post. More information can be found here: http://www4.java.no/web/show.do?page=42;7&appmode=/showReply&articleid=5493.

JRuby now JIT-compiles assertions

One of the major problems when running automated testing with JRuby is that all the standard Test::Unit assertions would never be JIT compiled, meaning that they would be quite slow. Actually, assertions seems to be very slow when running interpreted in JRuby. I have a small test case, courtesy of Michael Schubert:

require 'test/unit'
require 'benchmark'

class A < Test::Unit::TestCase
[10_000, 100_000].each do |n|
 define_method "test_#{n}" do
   puts "test_#{n}"
   5.times do
     puts Benchmark.measure{n.times{assert_equal true,true}}
   end
 end
end
end

This code will show quite nicely how large the overhead of asserts are, by using assert_equal. Now, the numbers for MRI for this benchmark looks like this:

Loaded suite test_assert
Started
test_10000
0.150000   0.000000   0.150000 (  0.155817)
0.150000   0.000000   0.150000 (  0.158376)
0.160000   0.000000   0.160000 (  0.155575)
0.150000   0.000000   0.150000 (  0.154380)
0.160000   0.000000   0.160000 (  0.157737)
.test_100000
1.520000   0.010000   1.530000 (  1.539325)
1.530000   0.000000   1.530000 (  1.543889)
1.520000   0.010000   1.530000 (  1.540376)
1.530000   0.000000   1.530000 (  1.543742)
1.530000   0.010000   1.540000 (  1.558292)
.
Finished in 8.509493 seconds.

2 tests, 550000 assertions, 0 failures, 0 errors

And for JRuby without compilation:

Loaded suite test_assert
Started
test_10000
1.408000   0.000000   1.408000 (  1.408000)
0.582000   0.000000   0.582000 (  0.582000)
0.425000   0.000000   0.425000 (  0.426000)
0.419000   0.000000   0.419000 (  0.418000)
0.466000   0.000000   0.466000 (  0.467000)
.test_100000
4.189000   0.000000   4.189000 (  4.190000)
4.196000   0.000000   4.196000 (  4.196000)
4.139000   0.000000   4.139000 (  4.139000)
4.165000   0.000000   4.165000 (  4.165000)
4.162000   0.000000   4.162000 (  4.162000)
.
Finished in 24.181 seconds.

2 tests, 550000 assertions, 0 failures, 0 errors

It's quite obvious that something is very wrong. We're about 2.5-3 times slower.

Now, the way the JRuby compiler works, we build it piece by piece and the JIT will try to compile a method that's used enough. If there is any node that can't be compiled it will fail and fall back on interpretation. In the case of assertions, all Test::Unit assertions use a small helper method called _wrap_assertion that looks like this:

def _wrap_assertion
 @_assertion_wrapped ||= false
 unless (@_assertion_wrapped)
   @_assertion_wrapped = true
   begin
     add_assertion
     return yield
   ensure
     @_assertion_wrapped = false
   end
 else
   return yield
 end
end

When I started out on this quest, there were two things in this method that doesn't compile. The first is the ||= construct, which I mentioned in an earlier blog post. The problem with it is that it requires that we can compile DefinedNode too, and that one is large. The second problem node is Ensure. After lots of work, I've finally managed to implement most of these safely, and falling back on interpretation when it's not safe. Without further ado, the numbers after compilation with these features added:

Loaded suite test_assert
Started
test_10000
  0.996000   0.000000   0.996000 (  1.013000)
  0.415000   0.000000   0.415000 (  0.415000)
  0.110000   0.000000   0.110000 (  0.110000)
  0.099000   0.000000   0.099000 (  0.100000)
  0.109000   0.000000   0.109000 (  0.109000)
.test_100000
  1.012000   0.000000   1.012000 (  1.012000)
  1.008000   0.000000   1.008000 (  1.000000)
  1.017000   0.000000   1.017000 (  1.017000)
  1.039000   0.000000   1.039000 (  1.039000)
  1.024000   0.000000   1.024000 (  1.024000)
.
Finished in 6.966 seconds.

So we're looking at over 4 times improvement in speed, and about 33% percent faster than MRI. Try your test cases; hopefully it will show up.

JRuby presentation at javaBin in Norway

Next week I'm coming to Norway to do two presentations at the Java User Groups in Bergen and Oslo. I will visit Bergen on the 15th, and Oslo on the 16th. I will have 2x45 minutes to talk, so there will lots of time for interesting detours, and hopefully some discussion too. If you're in the area, do show up! More information about the Oslo evening can be found here: http://www4.java.no/web/show.do?page=42;7&appmode=/showReply&articleid=5447.

After Norway I'm going to Sweden. I'll be in Gothenburg for a few days, Malmö a few more, and then stop by in Stockholm the 23rd to 26th. I'm not going to present while there, though. It's purely recreational.

JRuby is now Java 5+

So, after some discussion on the #jruby IRC channel, the core team has decided to go with a 5+ strategy on trunk. The reasons for this is that almost everyone who commented on the issues advised us to move on, and the features of 5 is quite compelling. We are moving to annotations for describing Java method bindings, we will use enumerations for many values, and we will be able to use the native concurrency libraries. All of these are large wins.

Of course, we are not leaving 1.4 users completely behind. First of all, we provide a Retrotranslator/Retroweaver (we haven't decided which one yet) version, and will continue doing so. Secondly, the 1.0 branch will continue supporting Java 1.4.

This is the current state. First versions of the annotation based method bindings are already in trunk.

The pain of compiling try-catch

I've been spending some time trying to implement a compiler for the defined?-feature of Ruby. If you haven't seen it, be happy. It's quite annoying, and incredibly complicated to implement, since you basically need to create a small interpreter especially just for nodes existing within defined?. So why is defined? so important? Well, for one it's actually needed to implement the construct ||= correctly. And that is used everywhere, which means that not compiling it will severely impact our ability to compile code. Also, it just so happens that OpAsgnOrNode (as it's called), and EnsureNode, are the two nodes left to implement to be able to compile Test::Unit assert-methods, since the internal _wrap_assertion uses both ensure and ||=.

So, now you know why. Next, a quick intro to the compilation strategy of JRuby. Basically we try to compile each script and each method into one Java method. We try to use the stack as much as possible, since we in that way can link statements together correctly. And that's about it.

The problem enters when you need to handle exceptions in the emitted Java bytecode. This isn't a problem in the interpreter, since we explicitly return a value for each node, and the interpreter doesn't use the Java stack as much as the compiler does. We also want to be able to use finally blocks at places, especially to ensure that ensure can be compiled down, but also to make the implementation of defined? safe.

So what's the problem? Can't we just emit the catch-table and so on correctly? Well, yes, we can do that. But it doesn't work. Because of a very annoying feature of the JVM. Namely, when a catch-block is entered, the stack gets blown away. Completely. So if the Ruby code is in the middle of a long chained statement, everything will disappear. And what's worse, this will actually fail to load with a Verifier exception, saying "inconsistent stack height", since there will now be one code path with things on the stack, and one code path with no values on the stack, and the way JRuby works, these will end up at the same point later on. And the JVM doesn't allow that either.

This makes it incredibly hard to handle these constructs in bytecode, and frankly, right now I have no idea how to do it. My first approach was to actually create a new method for each try-catch or try-finally, and just have the code in there instead. The fine thing about that is that the surrounding stack will not be blown away since it's part of the invoking method, and not in the current activation frame. And that approach actually works fairly well. Until you want to refer to values from outside from the try or catch block. Then it breaks down.

So, right now I don't know what to do. We have no way of knowing at any specific place how low the stack is, so it's not possible to copy it somewhere, and then restore it in the catch block. That would be totally inefficient too. In fact, I have no idea how other implementations handle this. There's gotta be a trick to it.

I love XKCD

Every time my feed reports an update on XKCD, I get all happy. It's without a doubt the best online comic. The blend is perfect. Just take a look at todays:

How perfect isn't that?

New York

I landed in New York this morning, and I'll stay until Thursday. If anyone feels like taking a beer and discussing programming languages (or something else), don't hesitate to mail me or comment on this blog.

Languages of the future

Martin Fowler writes about one language, and neatly encapsulates what I think about the subject in the way I would have written it, if I were actually a good writer. Go read.

FOSCON slides

I have been asked for the slides of my FOSCON presentation. They can be downloaded in PDF format from here: http://ologix.com/JRubyWhirlwindTour.pdf.

The IronRuby scoop

As you almost certainly now at this time, IronRuby was released the beginning of this week (read more here, here and here). Now, this of course begs a few question. But first things first: note that Microsoft have decided to host the project on RubyForge, and will do so sometime next month. Not only that, it's actually a very open source license called MsPL: Microsoft Permissive License. If you look at the license, you will see that it's almost exactly a BSD license, except that the patent restrictions have been clarified. Obviously, I'm not a lawyer, but I can say that this license seems very nice and will allow good usage of IronRuby in many cases.

The code base is strictly limited right now. The things that actually work is most of the core language structure, the Array and String classes and .NET Interop. Of course, that is an extremely large achievement to have this working so well. Also, the compiler seems to generate very good code and judging from microbenchmarks, it has the possibility of having very good performance. It's important to remember that many of the more important corner cases aren't supported yet, and it is these that usually drags the performance of Ruby implementations down. I'm confident that Lam and company can handle this, though.

Overall, I would say that this is looking very good. I would recommend .NET people to take a look at it, and also try to contribute. Very soon, you will be able to contribute code back to everything except the core compiler and DLR.

Finally, the question on everyones mind: when will IronRuby be finished? I don't know, but I think it can happen with 6-12 months. The concerns raised two months ago have been resolved internally by Microsoft, which makes IronRuby a very real and important project.

Should JRuby support 1.4.2?

Right now we're trying to decide if JRuby should upgrade from Java 1.4.2 to Java 5. There are some compelling reasons for this, but I'm not a 100% sure it's a good idea. Any comments from my readers?

In practical terms, this will mean that JRuby 1.0 will continue to be supported on 1.4.2, but new development will only work on Java 5 or higher. There is talk about using retrotranslator for handling 1.4.2 compatibility in later versions.

So. Please, comments and opinions!

ThoughtWorks at OSCON

I am at OSCON with Roy Singham, the founder of ThoughtWorks. We're here as sponsors. If you are here too - Roy likes nothing more than an argument about anything related to a) the merits of various languages, b) Agile versus waterfall, c) the best cell-phone. Seriously, if there is something interested that should be big for custom-app dev in the enterprise, tell Roy or the other ten ThoughtWorkers here about it and we'll help make it happen. Come by our booth at the left side of the expo hall if you want to find us.

Really Radical Ruby

I had a very good time at FOSCON III with the Portland Ruby Brigade yesterday. There were lots of entertaining talks too. I would say that my "lightning talk" wasn't really a lightning talk at all. Unless you count the speed of my talking... I managed to race through all my 22 slides - all of them shock full with information - in the time alloted to me. Hopefully people learned something from it.

Chad Wathington demonstrated Mingle which also was very nice.

John Lam showed us a taste of IronRuby, and also talked some about the implementation particulars that made certain things faster in IronRuby than on MRI. Interesting stuff, but I'm looking forward to the full talk tomorrow.

Alan McKean from GemStone showcased GemStone/JRuby - still a work in progress though. For those of you who don't know what GemStone is, think extremely powerful object persistence. And they're building a new version for Ruby, on top of JRuby. Very cool.

I realized that there are a few points about JRuby that haven't been emphasized enough, though, so here is there executive summary bullet points:

• JRuby is totally Java compliant and runs on any Java.
• JRuby is 1.0
• JRuby supports Rails
• ThoughtWorks offers commercial support for JRuby
• JRuby performance is on par with the C implementation, on average.

OSCON: first tutorial day

Yesterday was the first tutorial day at OSCON. Due to some planning mistakes, I didn't get the correct conference pass, so I missed the first tutorial. After that was sorted out I proceeded to the second tutorial of the day: Advanced Techniques for Parsing, by Mark Dominus. Of course I knew that the code would be Perl, but that didn't disturb me so much, since I expected to see some advanced parsing techniques. This is where disappointment hit me. Maybe it was advanced Perl code used, but it was not in any way advanced parsing. The first 2 hours were spent implementing a recursive descent parser with 7 productions. After that, I decided that I wouldn't be learning anything from this presentation, and headed back to the hotel, which was good, since I got sick that afternoon and spent the rest of the evening slightly delirious in my bed.

But now I'm up and going again, sitting here waiting for the tutorial "Real World Grails" to begin. I'm looking forward to see how Grails is actually used, since the presentations I've seen on it usually just show scaffolding and simpler things.

I have also decided on the subject for tonights FOSCON, but the slides are not finished yet. And the topic I choose is kind of a cop out: JRuby Cavalcade is the title of the talk, and I will basically just run through loads of interesting and funny JRuby things until I run out of time or gets booed of stage. Hope to see you there!

Will JavaScript be the next big language?

I guess most of you have read the latest happenings in the whole NBL-story. Steve Yegge have created a Rhino on Rails; a JavaScript version of Rails. I have meant to write about this for a few weeks, but I felt I wanted to think it through before writing about it.

Of course, it's really cool that Rhino on Rails exists, and may someday be open sourced. On the other hand, the current version have no counterpart to ActiveRecord - instead it works against internal Google data systems. This may make very good sense from the pov of Google, but will make it hard to gain traction outside of Google when/if the open sourcing happens. At that point, a ActiveRecord version written using JDBC and Rails should happen. I'm personally looking forward to it, because such an undertaking would be able to take more advantage of JDBC than for example ActiveRecord-JDBC can do. ActiveRecord done "right" so to speak, but in JavaScript.

But is JavaScript the language of the future? What does it have that separates it from other current languages - most notably Ruby? Well, at first glance the big difference is in the amount of implementations. JavaScript have a multitude of implementations, while Ruby has three (Ruby, YARV? and JRuby), with many more in the works. JavaScript though, have a incredible amount of implementations, most of them bad.

It's got a standard. It's got full closures and a few more neat futures. And it has prototype object orientation; a very different object model compared to the standard of inheritance based object systems like Java, Ruby and Smalltalk.

On closer inspection, the prototype based object system is not really such a big deal. In fact, you already have it in Ruby. Just use Object.new and Kernel.clone. Not very practical in Ruby, but you can do it without any problem. Remember that the difference between dup and clone is that clone will actually copy the metaclass of the object, while dup just makes the duplicate use the same real class.

I see some advantages that JavaScript has over Ruby, that can actually count as real advantages. The spec is arguably one. Several good implementations is another. It's easier to make JavaScript have good performance. And maybe most important: most developers already know it to some degree or other.

But will it be the NBL in some form or another? No, I don't think so, for the simple reason that to many developers hate it. Not because of the language itself, but due to the connection with bad browser implementations. This is a debt that it will be very hard for JS to get away from, and I think it will tip the point against JavaScript compared any other language poised for the position of being a big language.

Further, I really don't believe in the idea that there will be a next big language. Call me strange, but just because we've had this cycle two or three times doesn't mean it will continue like that. I believe development is qualitively different now compared to 15 years ago. The challenges we meet are different and demand different tools. One single language will not cut it. I think the future lies in layered languages with different properties.

JRuby at FOSCON 2007

So, I will attend and present at FOSCON 2007, which is arranged by the Portland Ruby Brigade (PDX.rb), and the theme is Really Radical Ruby. It's on Tuesday, more information here. It seems to be an interesting event, so please show up!

I have not yet decided what I'm going to talk about. JRuby will be involved in some way of course. Anyone have any request about what I should talk about?

OSCON and other conferences

Next week I'll be at OSCON in Portland. I won't be speaking, which means I'll probably be able to enjoy other peoples presentations instead. Hopefully I'll get to meet lots of people too. Say hi if you see me!

I'll be presenting at RailsConfEU in September too, and that is gearing up to be a really interesting conference. RailsConf in Portland was awesome, and if the EU version can match a 10th of that energy, it's going to be wonderful.

Back to JRuby regular expressions

It seems that this issue comes up every third month. After all the work we have done, we realize that regular expressions need some real work again. Our current solution works quite well. We have imported JRegex into JRuby, and done a whole slew of modifications to it. It runs well, have no issues with to deep regular expressions (Javas engine uses a recursive algorithm, making it stack overflow for certain inputs. Certain very common inputs, in say ... Rails. *sigh*).

But JRegex is good. It's not perfect though. It's slightly slower than the Java engine, it doesn't support everything in the Java engine, and conversely, it supports some things that Java doesn't support. The major problem is that we don't have MRI compliant multibyte support, and the implementation of our engine is wildly different compared to MRI's engine, and Oniguruma.

At some point we will probably just bite the bullet and do a real port of Oniguruma. But until such time comes, I have extracted our current regular expression stuff, and put everything behind a common interface. What that means is that with the current trunk, you can actually choose which Regular Expression engine you want to use. You can even write your own and plug in. The interface is really small right now. At the moment we only have JRegex and Java, and the Java engine doesn't pass all tests (I think, I haven't tried, since that wasn't the point of this exercise.). Anyway; it means you can have Java Regular Expressions if you want them, right in your JRuby code. But only where you want them. So, you can regular which engine is used globally by doing one of these two:

jruby -J-Djruby.regexp=java your_ruby_script.rb
jruby -J-Djruby.regexp=jregex your_ruby_script.rb

The last is current the default, so it's not needed. In the future it may be possible that JRegex isn't the default though, but this options should still be there. But the more nice thing about this is also that you can use Java Regexps inline, even if you want to use JRegex for most expressions:

begin
 p(/\p{javaLowerCase}*/ =~ "abc")
 p $&
rescue => e
 p e
end

p(/\p{javaLowerCase}*/j =~ "abc")
p $&

Now, the first example will actually raise a RegexpError, because javaLowerCase is not a valid character class in JRegex. But not the small "j" I've added to the second Regexp literal! That expression works and will match exactly as you expected.

RSpec and RBehave runs on JRuby

I'm not sure if this is well known or not, so I've been meaning to write a quick notice about it. The short story is this: JRuby can run RSpec and RBehave. Why is this important? Well, you can write code that tests Java code using RSpec and RBehave, meaning that it will be possible to get much more readable tests, even for code living in Java land.

Even if your organization won't accept writing an application in Ruby, it would probably be easier to get the testing done in Ruby. And writing tests in an effective language means that you will either write more production code, or more tests. Either of those are a quite good outcome.

A quick example of this in action. To run this example, you need JRuby 1.0 or later, and the rspec gem:

require 'java'

describe java.util.ArrayList, " when first created" do
before(:each) do
  @list = java.util.ArrayList.new
end

it "should be empty" do
  @list.should be_empty
end

it "should be able to add an element" do
  @list.add "content"
end

it "should raise exception when getting anything" do
  lambda{ @list.get 0 }.should raise_error(java.lang.IndexOutOfBoundsException)
end
end

In this code the examples are not that realistic, but you can see that the RSpec code looks the same for Java code, as it does for Ruby code. Even the raise_error exception matcher works. You can run this:

jruby -S spec -fs arraylist_spec.rb

The RBehave test suite also runs, which means you can stop using JBehave now... =)

This is a perfect example of the intersection where JRuby's approach can be very powerful, utilizing the existing Ruby libraries to benefit your Java programming.

The results of JRuby compilation

If you are interested in what actually happens when JRuby compiles Ruby to Java bytecode, I have added some small utilities to help out with this. To compile a string:

require 'jruby'
JRuby::compile("puts 1,2,3")

If you are running with -J-Djruby.jit.enabled=false, you can also inspect the result of compiling a block:

require 'jruby'

JRuby::compile do
 puts "Hello World"
end

The results of both of these invocations will be an object of type JRuby::CompiledScript. It has four attributes: name, class_name, original_script and code. The original_script attribute is only available when compiling from a string. The code attribute contains a Java byte array, and as such is not so useful in itself. But you can use the inspect_bytecode method to get a string which describes the compiled class. So, to see how JRuby compiles a puts "Hello, World":

require 'jruby'

puts JRuby::compile(<<CODE).inspect_bytecode
  puts "Hello, World"
CODE

Once you know what happens, you can start contributing to the compiler! =)

JRuby Inside

Peter Cooper have opened a "sister" site to RubyInside, called JRubyInside. It seems very promising; the address is http://www.jrubyinside.com.

Closing over ZSuper

One of the features of Ruby which I sometimes like and sometimes hate, is ZSuper. (So called, because it differs from regular super in the AST.) ZSuper is the keyword super, with arguments and parenthesis, which will call the super method with the same arguments as the current invocation got. Of course, that's not all. For example, if you change the arguments, the changes will propagate to the super implementation. Not only if you change the object, but if you change the reference, which I found non intuitive the first time I found it.

That's all and well. The interesting thing happens when you close over the super call and return it as a Proc. I haven't seen anyone doing this, which I guess is why there seems to be a bug in the implementation. Look at this code and tell me what it prints:

class Base
def foo(*args)
 p [:Base, :foo, *args]
end
end

class Sub < Base
def foo(first, *args)
 super
 first = "changed"
 super
 proc { |*args| super }
end
end

Sub.new.foo("initial", "try", :four).call("args","to","block")

Notice that Base#foo will get called three times during this code. In Sub#foo we are changing the first argument to the new string "changed". As I told you before, the second super call will actually get "changed" as the first argument the second time. But what will happen after that? We first create a block that uses ZSuper. We send the block to proc, reifying the block into an instance of Proc, and returning that. Directly after returning the block, we call it with some arguments. Now, the way I expect this to work (and incidentally, that's the way JRuby works) is that the output should be something like this:

[:Base, :foo, "initial", "try", :four]
[:Base, :foo, "changed", "try", :four]
[:Base, :foo, "changed", "try", :four]

We see that the first argument changed from "initial" to "changed", but otherwise the result is the same; the closure is a real closure over everything in the frame and scope. I guess you've realized that the same isn't true for Ruby. Without further ado, this is the output from MRI 1.8.6:

[:Base, :foo, "initial", "try", :four]
[:Base, :foo, "changed", "try", :four]
[:Base, :foo, "changed", ["args", "to", "block"], false]

The first time I saw this, the words WTF passed through my mind. In fact, that still happens sometimes. What is happening here? Well, obviously, it seems as if the passing of arguments to the block somehow clobbers the part where MRI saves away the closure over passed arguments. I have no idea whatsoever what the false value comes from. Hmm. But now that I think about it (this is just a guess), but I believe it stands for the fact that the arguments should be splatted into one argument. (That's the one called args in the block). If it had been true, they should refer to different variables. I think there is some trickery like that involved in the splatting logic in MRI.

Anyway. Is this a bug or a feature? I can't see any way it could be used in an obvious way, and it runs counter to being understandable and unsurprising. Anyone who can give me a good example of where this is useful behavior?

A JRuby Rubinius machine

When I get bored with JRuby, I tend to go looking either at other languages or other language implementations. This happened a few days ago, and the result is what I will here document. Begin by creating a file called fib.rb:

def fib(n)
 if n < 2
   n
 else
   fib(n - 2) + fib(n - 1)
 end
end

p fib(15)

The next part requires that you have a recent version of Rubinius installed:

rbx compile fib.rb

This will generate fib.rbc. Next, take a recent JRuby version and run:

jruby -R fib.rbc

And presto, you should see 610 printed quite soon. This is JRuby executing Rubinius bytecode. I was quite happy about how it was to get this far with the functionality. Of course, JRuby doesn't support most bytecodes yet, only those needed to execute this small example, and similar things. We are also using JRuby's internals for this, which means that Rubinius MethodContext and such are not available.

Another interesting note is that running the iterative Fib algorithm like this with -J-server is actually 30% faster than MRI.

This approach is fun, and I have some other similar ideas I really want to look at. The best part about it though, is that I got the chance to look at the internals of Rubinius. I hope to have more time for it eventually. Another thing I really want to do some day is implement a Jubinius, which should be a full port of the Rubinius runtime, possibly excluding Subtend. I think it could be very nice to have the Smalltalk core of Rubinius working together with Java. Of course, I don't have any time for that, so we'll see what happens in a year or two. =) Maybe someone else does it.

Evil JRuby

After my last post I got several comments about evil.rb. Of course I had evil.rb in mind when doing some of it, but I also forgot to describe the two most evil methods of the JRuby module: runtime and reference. The runtime method will return the currently executing JRuby runtime as a Java Integration, meaning you can get access to almost anything you want with it. For example, if you want to take a look at the global CacheMap (used to cache method instances):

require 'jruby'
JRuby::runtime.cache_map

Whoops. And that's just the beginning. Are you interested in investigating the current call frame or activation frame (DynamicScope in JRuby):

require 'jruby'
p JRuby::runtime.current_context.current_frame
a = 1
p JRuby::runtime.current_context.current_scope

Of course, you can call all accessible (and some inaccessible) methods on these objects, just like if you were working with it from Java. Use the API's and take a look. You can change things without problem.

And that also brings us to one of the easiest examples of evil.rb, changing the frozen flag on a Ruby object. Well, with the reference method, that's easy:

require 'jruby'

str = "foo"
str.freeze

puts str.frozen?
JRuby::reference(str).setFrozen(false)
puts str.frozen?

JRuby::reference will return the same object sent in, wrapped in a Java Integration layer, meaning that you can inspect and modify it to your hearts like. In this way, you can get at the internals of JRuby in the same way you can using evil.rb for MRI. And I guess these features should mainly be used for looking and learning about the internals of JRuby.

So, have fun and don't be evil (overtly).

Some JRuby tricks

I have spent a few hours adding some useful features these last days. Nothing extraordinary, but things that might come in handy at one point or another. The problem with these features is that they are totally JRuby specific. That means you could probably implement them for MRI, but noone has done it. That means that if you want to use it, beware. Further, they exploit a few tricks in the JRuby implementation, meaning it can't be implemented in pure Ruby.

So, that was the disclaimer; now onto the fun stuff!

Breaking encapsulation (even more)
As you know, in Ruby everything is accessible in some form or another, and you can do almost everything with the metaprogramming facilities. Well, except for one small detail which I found out while working on the AR-JDBC database drivers.

We have some code there which needs to be separate for each database, and it just so happens that core ActiveRecord have already implemented them in a very good way. So, what do we do? Mix in them and remove the methods we don't want? No, because ActiveRecord adapters are classes, not modules, and you can't mix in classes. There is no way to get hold of a method and add that to an unrelated other class or module. Except if you're on JRuby, of course:

require 'jruby/ext'

class A
def foo
puts "A#foo"
end
def bar
puts "A#bar"
end
end

class B;end

class C;end

b = B.new
b.steal_method A, :foo
b.foo
B.new.foo rescue nil #will raise NoMethodError

C.steal_methods A, :foo, :bar
C.new.foo
C.new.bar

Of course, using this should be avoided at all costs. But it's interesting that such a powerful thing can be implemented using about 15 lines of Java code.

Introspection
JRuby parses Ruby code into an Abstract Syntax Tree. For a while now, the JRuby module have allowed you to parse a string and get the AST representation by executing:

require 'jruby'

JRuby.parse "puts 'hello'", 'filename.rb', false

This returns the Java AST representation directly, using the Java Integration features. That is old. What is new is that I have added pretty inspecting, a nice YAML format and some navigation features which makes it very easy to see exactly how the AST looks. Just do an inspect or to_yaml on an AST node and you will get the relevant information.

That is interesting. But what is even more nice is the ability to run and use arbitrary pieces of the AST (as long as they make sense together) and also run them:

require 'jruby'

ast_one = JRuby::ast_for("n = 1; n*(n+3)*(n+2)")
ast_two = JRuby::ast_for("n = 42; n*(n+1)*(n+2)")

p (ast_one.first.first + ast_two.first[1]).run
p (ast_two.first.first + ast_one.first[1]).run

As you can see, I take two fragments from different code, add them together and run them. You can also see that I'm using an alias for parse here, called ast_for. That makes much more sense when using the second parse feature, which we already know from ParseTree:

require 'jruby'

JRuby::ast_for do
puts "Hello"
end

Well, I guess that's all I wanted to show right now. These last small things I've added because I believe they will be highly useful for debugging JRuby code.

I also have some more ideas that I want to implement. I'll keep you posted about it.

ObjectSpace: to have or not to have

Among all the features of Ruby that JRuby supports, I would say that two things take the number one place as being really inconvenient. Threads are one; making the native threading of Java match the green threading semantics of Ruby is not fun, and it's not even possible for all edge cases. But that argument have been made several times by both me and Charles.

ObjectSpace now, that is another story. The problems with OS are many. But first, let's take a quick look at the most common usage of OS; iterating over classes:

ObjectSpace::each_object(Class) do |c|
  p c if c < Test::Unit::TestCase
end

This code is totally obvious; we iterate over all instances of Class in the system, and print an inspected version of them if the class is a subclass of Test::Unit::TestCase.

Before we take a closer look at this example, let's talk quickly about how MRI and JRuby implements this functionality. In fact, having this functionality in MRI is dead easy. It's actually very simple, and there are no performance problems of having it when it's not used. The trick is that MRI just walks the heap when iterating over ObjectSpace. Since MRI can inspect the heap and stack without problems, this means that nothing special needs to be done to support this behavior. (Note that this can never be safe when using a real threading system).

So, the other side of the story: how does JRuby implement it? Well, JRuby can't inspect the heap of course. So we need to keep a WeakReference to each instance of RubyObject ever created in the system. This is gross. We pay a huge penalty for managing all this stuff. Many of the larger performance benefits we have found the last year have revolved around having internal objects be smarter and not put themselves into ObjectSpace until necessary. One of my latest optimizations of regexp matching was simple to make MatchData lazy, so it only goes into OS when someone actually uses it. RDoc runs about 40% faster when ObjectSpace is turned off for JRuby.

So, is it worth it? In real life, when do you need the functionality of ObjectSpace? I've seen two places that use it in code I use every day. First, Rails uses it to find generators, and secondly, Test::Unit uses it to find instances of TestCase. But the fun thing is this; the above code is almost exactly what they do; they iterate over all classes in the system and checking if they inherit from a specific base class. Isn't that a quite gross implementation? Shouldn't it be possible to do something better? Euhm, yes:

module SubclassTracking
  def self.extended(klazz)
    (class <<klazz; self; end).send :attr_accessor, 
                                    :subclasses
    (class <<klazz; self; end).send :define_method, 
                                :inherited do |clzz|
      klazz.subclasses << clzz
      super
    end
    klazz.subclasses = []
  end
end

# Where Test::Unit::TestCase is defined
Test::Unit::TestCase.extend SubclassTracking

# Load all other classes

# To find all subclasses and test them:
Test::Unit::TestCase.subclasses

I would say that this code solves the problem more elegantly and useful than ObjectSpace. There are no performance degradation due to it, and it will only effect subclasses of the class you are interested in. What's the best benefit of this? You can use the -O flag when running JRuby, and your tests and rest of the code will run much faster and use less memory.

As a sidenote: I'm putting together a patch based on this to both Test::Unit and Rails. ObjectSpace is unnecessary for real code and the vision of JRuby is that you will explicitly have to turn it on to use it, instead of the other way around.

Anyone have any real world examples of things you need to do with ObjectSpace?

First two days of TSSJS

It has been two long days; not because I've been going to sessions all day long, but because I've reworked my presentations quite heavily. But now both the BOF and the TS are finished, and I think they went well. I had to keep the level to Ruby, JRuby and Rails introductionary material, though, since most developers here didn't seem to know what is possible with these technologies.

But it's been great; I've gotten good feedback and had some really interesting conversations with lots of people.

We have been doing the town each night, and I've found that I like Barcelona very much. Except for the food: this country doesn't seem to be good for vegetarians at all. Very annoying. I'm going for beer and wine instead of food the rest of the week. =)

One day left, though, and it's bound to be nice. Me and Martin are both on a developers panel about the state of programming languages in 2020; I have no idea what to say, and I'm thinking about just ad-libbing it. I know my own position in these questions fairly well, and the current Yegge-debate have made my opinions even more explicit.

But now it's time to see the town again.

Interviewed by AkitaOnRails

Yesterday I spent 2 hours chatting with Fabio Akita, of AkitaOnRails (the largest Rails blog in Brazil); the result is a long interview that was published today. It's got some good stuff, and some Ola-stuff, which you should recognize by now.

And I note that he calls me a workaholic; but he got this interview prepared in less than a day too, and also translated it into Portuguese.

You can find it at http://www.akitaonrails.com/pages/olabini.

First weeks at ThoughtWorks

I've finally started. I've finally moved to London. I've been working for two weeks at ThoughtWorks now, and it's been quite crazy. Everything is very nice and I'm having loads of fun. Of course, it's also lots of hard work, and I feel that I'm stretching my capacity considerably more than I ever did at Karolinska Institutet. That's great, and I feel that I'm really doing something real now. We have so many interesting things going on, and I wish I could tell you all about it.

What I can tell you is that I'm working quite much on Mingle, and I'm also spending time on other JRuby related issues. I've been planning on getting SQL Server and Oracle working as good as possible with AR-JDBC, and I've spent time on Derby performance. Hopefully I'll continue the database work this week, since especially SQL Server and Oracle is very important.

The most important work for this week is probably to prepare for TheServerSide in Barcelona. I still haven't had time to prepare my demos, so it's about time now. I hope to see many of you there.

In conclusion, my first weeks at ThoughtWorks have been awesome. I really like the pople, and everything is just neat. I like being able to walk to work and working in the very nice TW office on High Holborn. I'm very happy about it all.

Book update

As you know, I am writing on a book about JRuby on Rails. A few minutes ago I finished the first draft of chapter 14. That means that there are just 3 chapters and 3 appendixes left to write (chapter 1, 2 and 15). So the writing is going very well, but it's taking a heavy toll on me personally. I seriously don't recommend writing a book like this in your spare time, while at the same time switching employer, moving abroad and try to be a core developer in an open source project which is getting lots of attention.

So, in summary: it's going well, it still looks like it will be out in October, and I'm deadly tired.

JRuby 1.0

The JRuby community is pleased to announce the release of JRuby 1.0!

Homepage: http://www.jruby.org/
Download: http://dist.codehaus.org/jruby/

JRuby 1.0 is a major milestone for our project. Our main goal for 1.0 has been
Ruby compatibility. We feel this goal has been reached. When we see
companies like ThoughtWorks offering commercial support; we know this goal
has been reached. Please download JRuby and take it for a test drive. Try
running your Ruby and Ruby on Rails applications with it.

Give us feedback. Join our community. Help us continue to improve JRuby.

It is important to notice that JRuby 1.0 is not the end all of Ruby interpreters. It's not perfect. This is just the beginning. We now have a very good base to work from. This is were the real work begins. Join us. It will be a fun ride, and JRuby will just get better!

What's wrong with this code?

Today I will introduce to you a method from ActiveRecord. The method takes a parameter called type and that value can bu for example :primary_key, :string or :integer. Now, in the first line there is a call to native_database_types. Generally, that call returns a structure that looks somewhat like this:

def native_database_types #:nodoc:
  {
   :primary_key => "int(11) DEFAULT NULL auto_increment PRIMARY KEY",
   :string      => { :name => "varchar", :limit => 255 },
   :text        => { :name => "text" },
   :integer     => { :name => "int", :limit => 11 },
   :float       => { :name => "float" },
   :decimal     => { :name => "decimal" },
   :datetime    => { :name => "datetime" },
   :timestamp   => { :name => "datetime" },
   :time        => { :name => "time" },
   :date        => { :name => "date" },
   :binary      => { :name => "blob" },
   :boolean     => { :name => "tinyint", :limit => 1 }
 }
end

The method itself looks like this.

def type_to_sql(type, limit = nil, precision = nil, scale = nil) #:nodoc:
  native = native_database_types[type]
 column_type_sql = native.is_a?(Hash) ? native[:name] : native
 if type == :decimal # ignore limit, use precison and scale
    precision ||= native[:precision]
   scale ||= native[:scale]
   if precision
     if scale
       column_type_sql << "(#{precision},#{scale})"
     else
       column_type_sql << "(#{precision})"
     end
   else
     raise ArgumentError, "Error adding decimal column: precision cannot be empty if scale if specified" if scale
   end
   column_type_sql
 else
   limit ||= native[:limit]
   column_type_sql << "(#{limit})" if limit
   column_type_sql
 end
end

There is something very wrong with this implementation. Of course, there could exist many errors here, but what I'm thinking about right now is a violation of the usual way methods should work. And in effect, that problem with this method have caused ActiveRecord-JDBC to implement some very inefficient code to handle this method. And it gets called a lot in ActiveRecord. I'll get back later today with a pointer to what's wrong here, and I will also discuss some of what I've done in AR-JDBC to handle this situation. I hope for many suggestions here! =)

This is what's wrong

I must say, I got some really good responses to my post about what was wrong with the code I posted. Most of those responses concerned the design of the code, and I agree, this part of Rails could have been done much better. But what I was thinking about was actually a bug. And Lars Westegren (my former colleague) nailed it at the first try. Let me show two important excerpts from this code:

column_type_sql = native.is_a?(Hash) ? native[:name] : native

and here:

column_type_sql << "(#{limit})" if limit

Obviously, double left arrow is append, and for all cases where there is a limit, this append will change the String. This is one of the cases where it's kind of annoying that strings are mutable. If I cache away the values that native_database_types should return, then the next time anyone wants a string SQL type, that will generate VARCHAR(255)(255). The next time again, VARCHAR(255)(255)(255). And so one. So either I need to recreate the hash every time, or I need to do a deep clone of it every time. Neither of these options are very good, and it seems the deep clone option isn't fast enough, even when done in Java, so I decided to go with a hash literal instead. Was that the right choice? I don't know. It improves performance, but on the other hand it churns objects and creates new objects all the time. All of this because of some sloppy coding in Rails.

What's the lesson learned? Never modify your arguments, unless that is an explicit part of the contract for that method and part of the documentation.

Testing with JRuby on Rails and ActiveRecord-JDBC

This will be a highly uninflammatory blog post, as contrast to the last one. Specifically, there is a slight problem when running the command

jruby -S rake

in a a JRuby on Rails-application. This problem is caused by some hard coded values in the database Rake definitions for Rails. But don't despair, there is a simple solution to this. It's not as simple as it should be (invisible) but it's easy enough. Provided you have JRUBY_HOME set and your version of AR-JDBC is 0.3.1, execute this command from your Rails application root

cp $JRUBY_HOME/lib/ruby/gems/1.8/gems/ActiveRecord-JDBC-0.3.1/lib/tasks/jdbc_databases.rake lib/tasks

Since the hard coded values are hard to override, the jdbc_databases.rake file just hacks Rake to be able to redefine tasks and then redefines the core tasks. This shouldn't affect a bi-Ruby installation, since the overriding only happens on JRuby, not on MRI. If someone has a better way to do this, please tell me. =)

There can be only one, a tale about Ruby, IronRuby, MS and whatnot

(Updated: added a quote from John Lam about not being able to look at the MRI source code)

After RailsConf in Portland, there has flared up a discussion centered around IronRuby and Microsoft. We discussed many of these points in depth at the conference, and I'll elaborate some on my views on the issues in a bit.

But first I would like to talk some about the multitude of Ruby implementations springing up. I firmly believe that a language evolves in phases. The first phase, germination, is the period where a language needs one consistent implementation (or a spec). It's during this phase when most "alpha geek adoption" happens. Many important libraries are written, but most applications are not in the main economic center. Ruby have been in this phase for a long time, but the fact that new implementations are springing up left and right is a sure sign that Ruby is entering phase 2: implementation. For adoption to happen, there need to exist several competing implementations, all of them good. This is the evolutionary stage, where it's decided what kind of features an implementation should provide. Should we have green or native threads? Are all the features of the original implementation really that necessary? (Continuations, ObjectSpace). Is there cruft in the standard library that needs to be weeded out? (timeout.rb). All of these questions get answered when other people implement the language. The last phase, which I guess could be called adoption, is when the language have several working implementations, all good enough to deliver high end applications on, when many applications are written in the language, and there exists a plethora of libraries, systems and support for the language.

What this means is that for a language to be successful, there needs to exist competing implementations. They need to implement their features in different ways and make different choices during development. Otherwise, the language will die. (This is obviously not enough, since Smalltalk fulfilled this admirably and still never got widespread adoption.). But I still believe it's incredibly important for a language to evolve with many implementations, which is why I find Rubinius, JRuby, YARV and IronRuby to be extremely important projects for the welfare of Ruby. I want Ruby to be successful. I want Ruby to be the next major language for several reasons. But most importantly: I want Ruby to be a better language tomorrow, than it is today. The only way that's going to happen is by having lots of people implement the language.

So, that's enough of the introductory flame bait. This describes one half of why IronRuby is an important project, and why we can't let it fail. The other side of the coin is the same reason JRuby is important. .NET as a platform have some wildly useful features. There are many developers who swear by .NET for good reason. And what's more important, there are lots of large enterprises with such a vested interest in .NET, that they will never choose anything else. Now, for the welfare of all programmers in the world, I personally believe the world would be a better place if those .NET-environments also used Ruby. So that's the other coin of why IronRuby is important.

The most well read blog about the current Microsoft/Ruby controversy is Martin Fowlers article RubyMicrosoft. Go read it now, and then I'll just highlight the points I find most important.

First: John Lam is committed to creating a "compliant" Ruby implementation. I have no doubts that he can do it. But there are a few problems lurking.

For example, what is a compliant Ruby implementation? Since there exists no spec, and no comprehensive test suite, the only way to measure compliance is to check how close the behavior matches MRI. But this have some problems too. How do you check behavior? Well, you need to run applications. But how do you get so far as you can run applications?
What JRuby did was that we looked at MRI source code.

John Lam can not look at MRI source code. He cannot look at JRuby source code. He cannot look at Rubinius source code. If he does, he will be terminated.

So, the next best alternative: accepting patches from the community, which can look at Ruby source? Nope, no cigar. Microsoft is not really about Open Source yet. Their license allows us to look at their source code, and also to fork it and do what we want with it. But that's only half of what open source is about. The more important part is that you should be able to contribute back code without having to fork. You can't do that with IronRuby, since Microsoft is too scared about being sued for copyright infringement.

There was some doubt about Lam actually being banned from looking at MRI source code. This is the first quote that said it is so. It's from the discussion "Virtual classes and 'real' classes -- why?" on Ruby-core, this quote posted at 29/03/07:

Is this how things are actually implemented? (BTW I'm not lazy here - we cannot look for ourselves).

I am going to make a bold statement here. Under the current circumstances, I don't believe it's possible for John Lam and his team to create a Ruby implementation that runs Rails within at least 18 months. And frankly, that's not soon enough.

As I said above, I have all confidence that John can do great stuff if he has the right resources. But creating a Ruby implementation is hard enough while having all the benefits of the open source community.

The two points I want to make with this point is this: The Ruby community must damned well get serious about creating a good, complete specification and test suite. It's time to do it right now, and we need it. It's not a one-man job. The community needs to do it. (And yes, the two SoC projects are a very good start. But you still need to be able to run RSpec to take full advantage of them; and let's face it, the RSpec implementation uses many nice Ruby tricks.)

The second point is simpler: Microsoft needs to completely change how they handle Open Source. Their current strategy of trying to grow it into the organization will not work (at least not good enough). They need to turn around completely, reinvent themselves and make some really bold moves to be able to meet the new world. If they can't do this, they are as dead as Paul Graham claims.

Solving mounting problem on MacOS X

This is a highly specific post, but I thought I'd write about this so that if someone else have the same problem, they can try to solve it my way.

I got a new computer, Intel-based MBP last week. I upgraded it to latest OSX version (10.4.9 I believe), and immediately installed everything I needed. While ending my last job and returning the laptop associated with that post, I made backups to a 500GB USB LACIE hard drive. Very nice indeed, and half my life is now on that hard drive. (I have most of it on other places too, but not so easily accessible).

The first thing I managed to do was to shut off the hard drive without unmounting it correctly from OSX. I got the warning et all, but well, done is done. It was stupid, I know. Starting out with messing up everything. So what happened? Well, the next time I tried to get OSX to find my hard drive, by inputing the USB connection and switching on the power. Nothing happened. The drive would spin up, but no mount points or nice icons on the desktop. After some investigation I found that the Disk Manager HANGS when the LACIE is turned on and connected. I also found that when I switched the power off, my console says something about not being able to repair disk /dev/disk2s1. Interesting. After a few hours investigation on the Internet I despaired, and decided to try my own ingenuity instead.

I won't tell you about everything I did to find this solution. It would get severely boring very fast. So, here is the solution:

1. Attach the device and turn on the power.
2. Open up a terminal and create a new directory in /Volumes, (eg sudo mkdir /Volumes/LACIE2).
3. Mount the drive explicitly on this mount point (sudo mount -t msdos /dev/disk2s1 /Volumes/LACIE2).
4. Delete ALL .DS_Store files on the disk (cd /Volumes/LACIE2; find . -name 'DS_Store' -exec rm -rf \{\} \;).
5. Wait for a while. At this point you should have two LACIE drives on your Desktop, one fake and one real. Unmount the real one by dragging it to the trash can.
6. Turn of the external hard drive, or detach it.
7. Reboot.
8. Attach/Turn on the external hard drive.

This is a process that works for me, and my drive is now back to working mode. It seems that OSX stores some information in the DS_Store files somewhere on the disk that got corrupted for me. Hopefully this information can help someone else with the same problem.

Finally in London

Yesterday I landed in London, and let me tell you: it's been tough getting here. Packing and moving and arranging all takes lots of time and energy. And it's not helping being sick while pulling it off, either. But I'm finally here, and right now sitting at a hotel room in Holborn, close to Russel Square. Hopefully I'll be able to land an apartment soon too, and then I'll get back to my former speed, hopefully.

In the meantime, tomorrow is my first day at ThoughtWorks. It's bound to be interesting, but I don't expect any downtime. We have so many interesting things going, that it will be full speed ahead from day one.

Although I'm more tired than I should be, if someone feels like meeting up this or next week for a beer and talk JRuby, it would be fun.

Burning cycles: the next month

Getting back from RailsConf meant getting thrown into loads and loads of preparation work for London. Oh my. Since I'm moving in 8 days, and almost all packing needs to be done during the weekend (and I haven't begun yet), I don't think I'll be able to be that communicative. Further, I will have no access to a computer or cell phone from Wednesday to Monday, so if I don't answer email, don't get upset. =) That incidentally means I won't blog either.

June 4th will be my first day at ThoughtWorks and I'm incredibly excited by it. Hopefully I'll be able to blog some about my experiences, but I guess the first few days will be focused on getting setup and finding an apartment (If you know any nice place close to Camden Town that's available by the beginning of June, please do tell), and doing all manner of things.

If I'm unlucky I'll need to go to DC the week after that, but we'll see. Hopefully I can wrangle out of it, since Stella (my girlfriend) will arrive in London that weekend.

After that week I'll have a week of quiet and solitude, and then it's time for TheServerSide Java Symposium in Barcelona. I will in fact be involved in no less than three events during this conference. First, a technical session on JRuby, secondly a BoF about deploying JRuby on Rails applications, and third a panel discussion called "2020: A Developer's Odyssey Panel", which is bound to be interesting. I have no idea whatsoever about what I'll say yet, but Martin Fowler will be one of the panelists which makes it a certainty of fun. I'm looking forward to TSSJS for a few more reasons; seeing Barcelona, since that is said to be a lovely city, getting to say hi to Dr. Heinz Kabutz, meet up with Jonas Bonér from Terracotta and continue discussing how to utilize Terracotta and JRuby together. I'm looking very much forward to the whole event, in fact. If you are there, don't hesitate to say hi!

That gender thing

One of the foci on both JavaOne and RailsConf was the so called "gender problem"; that the current balance between men and women in the technological field is bad and that something needs to be done about it. I'm very happy about this getting awareness, but as my colleague Lars Westergren writes here (http://slashdot.org/~LarsWestergren/journal/172323) it seems like total hypocrisy when looking at the so called entertainment provided at these two events. We need to do better than this.