The JRuby Tutorial #4: Writing Java extensions for JRuby

There are many reasons to write a Java extensions for JRuby. Maybe your favorite Ruby library hasn't been ported to JRuby yet, or you want to directly interface with some Java code without going through JRuby's Java interface. Maybe you need the speed from doing calculations in Java, or you just want to add missing functionality. Whatever the reason, writing extensions for JRuby can be tricky if you don't know how the internals of JRuby work. The purpose of this tutorial is to show how to build a simple extension the exercises many parts of the Ruby language and how to implement this with Java.

The example will be a module called Sequence with one class inside it called Sequence. Whenever I create something as a Java extension, I usually write functional Ruby code for doing it first, to get the structure of the code straight in my head. So, without further ado, here is the Sequence module:

module Sequence
 def self.fibonacci(to=20)
   Sequence.new(1,1,1..to)
 end

 def self.lucas(to=20)
   Sequence.new(1,3,1..to)
 end

 class Sequence
   include Enumerable
   attr_reader :n1,:n2,:range
   def initialize(n1,n2,range)
     @n1, @n2, @range = n1,n2,range
     regenerate
   end
   %w(n1 n2 range).each do |n|
     define_method(n) do |v|
       send("#{n}=",v)
       regenerate
     end
   end
   def regenerate
     @value = []
     v1, v2 = @n1, @n2
     @value << v1 if @range === 1
     @value << v2 if @range === 2
     3.upto(@range.last) do |i|
       v1, v2 = v2, v1+v2
       @value << v2 if @range === i
     end
     nil
   end
   def [](ix)
     @range = ix..(@range.last) if ix < @range.first
     @range = (@range.first)..(ix+1) if ix > @range.last
     regenerate
     @value[ix-@range.first]
   end
   def each(&b)
     @value.each(&b)
   end
   def to_a
     @value
   end
   def to_s
     @value.to_s
   end
   def inspect
     "#<Sequence::Sequence n1=#@n1 n2=#@n2 range=#@range value=#{@value.inspect}>"
   end
 end
end

Interfacing with the JRuby runtime

There are a few different ways to write extensions for JRuby. The difference isn't big from a functional viewpoint, but there is a definite gap in usability. I call the two major ways to implement an extension the MetaClass way, and the MRI way. The MetaClass subclasses the Java class that represent a Ruby class, called RubyClass, and implements some meta information methods and classes. The MRI way, in contrast, just creates the Ruby class in code, and adds methods to it in some static initializer. This tutorial will use the MRI way for two reasons; first, it's easier and doesn't require so many files and classes, and second, when porting MRI C extensions, the MetaClass way doesn't map very well to how MRI does things.

Project setup

To make the extension building as simple as possible, it helps to follow a few conventions. First of all, I'm going to call the extension "fib". I want my potential users to be able to require 'fib' and get all the good Sequence-functionality. To achieve this there are two things to keep in mind. First, the jar-file should be called fib.jar and put somewhere in JRuby's load path. Secondly, there should be a class called FibService that implements the BasicLibraryService
interface. For our purposes, FibService.java will contain all functionality, but in a realistic situation is makes sense to extract the functionality and let the library loader just set up the
environment. The skeleton for my FibService.java will look like this:

import java.io.IOException;
      
import org.jruby.IRuby;

import org.jruby.runtime.load.BasicLibraryService;

public class FibService implements BasicLibraryService {
   public boolean basicLoad(IRuby runtime) throws IOException {
       return true;
   }
}

At this point the only imports needed are for IRuby, which is the main interface for the JRuby runtime, and the BasicLibraryService which provides the basicLoad method. The return value specifies if the service was loaded correctly or not.

Basic structure

I will start by adding the basic structure for our code; the Sequence module and class:

import java.io.IOException;
      
import org.jruby.IRuby;
import org.jruby.RubyClass;
import org.jruby.RubyModule;

import org.jruby.runtime.builtin.IRubyObject;

import org.jruby.runtime.load.BasicLibraryService;

public class FibService implements BasicLibraryService {
   public boolean basicLoad(IRuby runtime) throws IOException {
       RubyModule mSequence = runtime.defineModule("Sequence");
       RubyClass cSequence = mSequence.defineClassUnder("Sequence",runtime.getObject());
       cSequence.includeModule(runtime.getModule("Enumerable"));
       cSequence.attr_reader(new IRubyObject[]{runtime.newSymbol("n1"),
                                               runtime.newSymbol("n2"),
                                               runtime.newSymbol("range")});
       return true;
   }
}

What this code does is to establish the Sequence module at the top level, and then define the Sequence class inside this module. We need to specify a super class for it, and this is what the
runtime.getObject()-call is about. Basically it's a shortcut for writing runtime.getClass("Object"). After we have defined the class, make it include Enumerable, and then create attribute readers for the 3 instance variables. Despite the name, newSymbol doesn't necessarily create a new symbol; it returns an existing if there is one.

Singleton methods

We're going to create the singleton factory methods before actually creating the implementation for the class. The new class looks like this:

import java.io.IOException;
      
import org.jruby.IRuby;
import org.jruby.RubyClass;
import org.jruby.RubyFixnum;
import org.jruby.RubyModule;
import org.jruby.RubyNumeric;

import org.jruby.runtime.CallbackFactory;
import org.jruby.runtime.builtin.IRubyObject;
import org.jruby.runtime.load.BasicLibraryService;

public class FibService implements BasicLibraryService {
   public boolean basicLoad(IRuby runtime) throws IOException {
       RubyModule mSequence = runtime.defineModule("Sequence");
       RubyClass cSequence = mSequence.defineClassUnder("Sequence",runtime.getObject());
       cSequence.includeModule(runtime.getModule("Enumerable"));
       cSequence.attr_reader(new IRubyObject[]{runtime.newSymbol("n1"),
                                               runtime.newSymbol("n2"),
                                               runtime.newSymbol("range")});

       CallbackFactory fibService_cb = runtime.callbackFactory(FibService.class);
       mSequence.defineSingletonMethod("fibonacci",fibService_cb.getOptSingletonMethod("fibonacci"));
       mSequence.defineSingletonMethod("lucas",fibService_cb.getOptSingletonMethod("lucas"));

       return true;
   }

   private static IRubyObject seq(int a1, int a2, RubyModule module, IRubyObject[] args) {
       IRuby runtime = module.getRuntime();
       int to = 20;
       if(module.checkArgumentCount(args,0,1) == 1) {
           to = RubyNumeric.fix2int(args[0]);
       }
       IRubyObject[] seqArgs = new IRubyObject[3];
       seqArgs[0] = runtime.newFixnum(a1);
       seqArgs[1] = runtime.newFixnum(a2);
       seqArgs[2] = runtime.getClass("Range").callMethod("new",
               new IRubyObject[]{RubyFixnum.one(runtime),runtime.newFixnum(to)});
       return module.getClass("Sequence").callMethod("new",seqArgs);
   }

   public static IRubyObject fibonacci(IRubyObject recv, IRubyObject[] args) {
       return seq(1,1,(RubyModule)recv,args);
   }

   public static IRubyObject lucas(IRubyObject recv, IRubyObject[] args) {
       return seq(1,3,(RubyModule)recv,args);
   }
}

This code contains a number of new things. First of all, our singleton methods needs implementations. Since we don't need any data associated for these methods, static Java-methods suffice for implementation. A CallbackFactory is used to get a reflection handle at the methods. I use the method call getOptSingletonMethod on the CallbackFactory; this is because the one parameter to the two methods are optional, so the callback factory will look for a static method with signature IRubyObject name(IRubyObject, IRubyObject[]). We'll later see how we
can specify explicit types for method arguments. The recv argument is a specialty for static methods. Usually when working with Ruby instances from Java code, you will have a handle to the runtime implicit in the self, but this isn't possible for static methods. The recv parameter is the instance of RubyModule/RubyClass that the method is called on. In our case this is a handy way of getting hold of the Sequence-module.

All IRubyObject's have checkArgumentCount which is a simple utility method for methods with optional arguments. Basically, it takes an array, the minimum and maximum argument count, and throws a Ruby exception if it isn't correct. It also returns the actual argument count (which is the same as args.length right now). Note, if porting C Ruby code, that this two numeric parameters to checkArgumentCount is NOT the same as rb_scan_args where for example "12" means one required and two optional parameters. The equivalent with checkArgumentCount would be checkArgumentCount(args,1,3).

RubyNumeric has a few utility methods, where fix2int is one of the more useful. It basically allows us translate a Ruby integer into the Java corresponding type.

The most common types have shortcut creation methods in IRuby, and newFixnum is one of these. To create a new Range we have to get a reference to the class and call new on it, though.

The Sequence class

Here comes the meat of it all. This is the final version of the Java source:

import java.io.IOException;

import java.util.ArrayList;
import java.util.List;
import java.util.Iterator;
      
import org.jruby.IRuby;
import org.jruby.RubyArray;
import org.jruby.RubyClass;
import org.jruby.RubyFixnum;
import org.jruby.RubyModule;
import org.jruby.RubyNumeric;
import org.jruby.RubyObject;
import org.jruby.RubyRange;

import org.jruby.runtime.CallbackFactory;
import org.jruby.runtime.builtin.IRubyObject;
import org.jruby.runtime.load.BasicLibraryService;

public class FibService implements BasicLibraryService {
   public boolean basicLoad(IRuby runtime) throws IOException {
       RubyModule mSequence = runtime.defineModule("Sequence");
       RubyClass cSequence = mSequence.defineClassUnder("Sequence",runtime.getObject());
       cSequence.includeModule(runtime.getModule("Enumerable"));
       cSequence.attr_reader(new IRubyObject[]{runtime.newSymbol("n1"),
                                               runtime.newSymbol("n2"),
                                               runtime.newSymbol("range")});

       CallbackFactory fibService_cb = runtime.callbackFactory(FibService.class);
       mSequence.defineSingletonMethod("fibonacci",fibService_cb.getOptSingletonMethod("fibonacci"));
       mSequence.defineSingletonMethod("lucas",fibService_cb.getOptSingletonMethod("lucas"));

       CallbackFactory seq_cb = runtime.callbackFactory(Sequence.class);
       cSequence.defineSingletonMethod("new",seq_cb.getOptSingletonMethod("newInstance"));
       cSequence.defineMethod("initialize",seq_cb.getMethod("initialize",RubyFixnum.class,RubyFixnum.class,RubyRange.class));
       cSequence.defineMethod("n1=",seq_cb.getMethod("set_n1",RubyFixnum.class));
       cSequence.defineMethod("n2=",seq_cb.getMethod("set_n2",RubyFixnum.class));
       cSequence.defineMethod("range=",seq_cb.getMethod("set_range",RubyRange.class));
       cSequence.defineMethod("[]",seq_cb.getMethod("arr_ix",RubyFixnum.class));
       cSequence.defineMethod("each",seq_cb.getMethod("each"));
       cSequence.defineMethod("to_a",seq_cb.getMethod("to_a"));
       cSequence.defineMethod("to_s",seq_cb.getMethod("to_s"));
       cSequence.defineMethod("inspect",seq_cb.getMethod("inspect"));

       return true;
   }

   private static IRubyObject seq(int a1, int a2, RubyModule module, IRubyObject[] args) {
       IRuby runtime = module.getRuntime();
       int to = 20;
       if(module.checkArgumentCount(args,0,1) == 1) {
           to = RubyNumeric.fix2int(args[0]);
       }
       IRubyObject[] seqArgs = new IRubyObject[3];
       seqArgs[0] = runtime.newFixnum(a1);
       seqArgs[1] = runtime.newFixnum(a2);
       seqArgs[2] = runtime.getClass("Range").callMethod("new",
               new IRubyObject[]{RubyFixnum.one(runtime),runtime.newFixnum(to)});
       return module.getClass("Sequence").callMethod("new",seqArgs);
   }

   public static IRubyObject fibonacci(IRubyObject recv, IRubyObject[] args) {
       return seq(1,1,(RubyModule)recv,args);
   }

   public static IRubyObject lucas(IRubyObject recv, IRubyObject[] args) {
       return seq(1,3,(RubyModule)recv,args);
   }

   public static class Sequence extends RubyObject {
       public static IRubyObject newInstance(IRubyObject recv, IRubyObject[] args) {
           Sequence result = new Sequence(recv.getRuntime(), (RubyClass)recv);
           result.callInit(args);
           return result;
       }

       public Sequence(IRuby runtime, RubyClass type) {
           super(runtime,type);
       }

       public IRubyObject initialize(RubyFixnum n1, RubyFixnum n2, RubyRange range) {
           setInstanceVariable("@n1",n1);
           setInstanceVariable("@n2",n2);
           setInstanceVariable("@range",range);
           regenerate();
           return this;
       }

       public IRubyObject set_n1(RubyFixnum n1) {
           setInstanceVariable("@n1",n1);
           regenerate();
           return n1;
       }

       public IRubyObject set_n2(RubyFixnum n2) {
           setInstanceVariable("@n2",n2);
           regenerate();
           return n2;
       }

       public IRubyObject set_range(RubyRange range) {
           setInstanceVariable("@range",range);
           regenerate();
           return range;
       }

       private void regenerate() {
           List v = new ArrayList();
           int v1 = RubyNumeric.fix2int(getInstanceVariable("@n1"));
           int v2 = RubyNumeric.fix2int(getInstanceVariable("@n2"));
           IRubyObject r = getInstanceVariable("@range");
           if(r.callMethod("===",getRuntime().newFixnum(1)).isTrue()) {
               v.add(getRuntime().newFixnum(v1));
           }
           if(r.callMethod("===",getRuntime().newFixnum(2)).isTrue()) {
               v.add(getRuntime().newFixnum(v2));
           }
           int l = RubyNumeric.fix2int(r.callMethod("last"));
           for(int i=3;i<=l;i++) {
               int tmp = v1;
               v1 = v2;
               v2 = tmp + v1;
               if(r.callMethod("===",getRuntime().newFixnum(i)).isTrue()) {
                   v.add(getRuntime().newFixnum(v2));
               }
           }
           setInstanceVariable("@value",getRuntime().newArray(v));
       }

       public IRubyObject arr_ix(RubyFixnum ix) {
           int index = RubyNumeric.fix2int(ix);
           if(index < RubyNumeric.fix2int(getInstanceVariable("@range").callMethod("first"))) {
               setInstanceVariable("@range",getRuntime().getClass("Range").callMethod("new",
                  new IRubyObject[]{ix,getInstanceVariable("@range").callMethod("last")}));
           }
           if(index > RubyNumeric.fix2int(getInstanceVariable("@range").callMethod("last"))) {
               setInstanceVariable("@range",getRuntime().getClass("Range").callMethod("new",
                  new IRubyObject[]{getInstanceVariable("@range").callMethod("first"), getRuntime().newFixnum(index+1)}));
           }
           regenerate();
           return getInstanceVariable("@value").callMethod("[]",
                         getRuntime().newFixnum(index -
                         RubyNumeric.fix2int(getInstanceVariable("@range").callMethod("first"))));
       }

       public IRubyObject each() {
           Iterator iter = ((RubyArray)getInstanceVariable("@value")).getList().iterator();
           while(iter.hasNext()) {
               getRuntime().getCurrentContext().yield((IRubyObject)iter.next());
           }
           return getRuntime().getNil();
       }

       public IRubyObject to_a() {
           return getInstanceVariable("@value");
       }

       public IRubyObject to_s() {
           return getInstanceVariable("@value").callMethod("to_s");
       }

       public IRubyObject inspect() {
           StringBuffer sb = new StringBuffer("#<Sequence::Sequence n1=");
           sb.append(getInstanceVariable("@n1").toString());
           sb.append(" n2=");
           sb.append(getInstanceVariable("@n2").toString());
           sb.append(" range=");
           sb.append(getInstanceVariable("@range").toString());
           sb.append(" value=");
           sb.append(getInstanceVariable("@value").callMethod("inspect").toString());
           sb.append(">");
           return getRuntime().newString(sb.toString());
       }
   }
}

Compiling this and placing it in fib.jar on your load path will allow JRuby to use the code as if it was Ruby. Try it out.

Now, let's take the code in pieces. First of all, the initialization code defines the methods available and gives them a reflected implementation through CallbackFactory. We create a static inner class to hold the actualy implementation of the class. This isn't strictly necessary in this case, since we haven't associated any external state with the object, but it makes for cleaner separation and easier to understand code. Note that we need to have our own
new-implementation. This is one of the drawbacks with the MRI technique. When using MetaClasses you can define an allocateObject-method that automatically get's used by the runtime. Most of CallbackFactory's different getMethods-variants are used. This display how to have a fixed number of arguments with specific classes.

The initialize method just sets the instance variables and then call the method regenerate. Note that this isn't a Ruby method anymore. I didn't feel it was necessary to expose it, and using Java call semantics makes this slightly more efficient. Apart from that, there is nothing really strange in this code. I use the fact that you can create a new Ruby array from a list to make the regeneration of @value easier. But in most cases this is purely translated Ruby to JRuby-code. The only point where something strange is happening is in fact in the each-method. Handling blocks with JRuby in Java isn't always practical, so I tend to find it easier to refactor the Ruby code into something that calls yield specifically, by itself.

Conclusion

Implementing a Java extension for JRuby can be tricky, but the hard part is mostly to know what services are available where. By having the JRuby source code available it's easy to get a peek into the internals and find out more about those things that are problematic. Taking a look at how the core classes are implemented often give some hints on how continue, too. For example, RubyZlib, RubyYAML, RubyOpenSSL, RubyStringIO and RubyEnumerable are all mostly written in this style, and there are various examples of the different styles available.

If you need the speed or if it's more practical to implement the functionality in Java, I would say that writing an extension is fairly easy once you get started. The important thing to remember is to be sure what the interface should be, and implement everything else outside of JRuby, demarcating the interface from the implementation.

ResourceBundle pain.

I just found out something very disturbing and annoying. Java Property-files (and ResourceBundles in extension) will only load files in ISO-8859-1 format. Of course, you can use Unicode escape codes in the files, but that is not very convenient. Java is, over all, really good at internationalization, but this is one sore spot. And it doesn't seem to go away in Java 6 either. Isn't it highly ironic that ResourceBundles can't use regular UTF-8-files?

OpenSSL in JRuby

This weekend I started work on OpenSSL in JRuby. It's a pretty big undertaking, and it's actually worse than I suspected from the beginning, so I'm going to tell a bit about my endeavours and have far I've gotten right now. First of all, if someone is interested in the work, I have set up a branch for this, since it will take time and be very big. The branch is called svn://svn.codehaus.org/jruby/branches/openssl.

The approach
I have investigated several variants of implementing OpenSSL in JRuby. One that seemed easy was to find a JNI-library that wraps OpenSSL in Java. But it seems there are no full scale versions implemented anywhere. I also checked around other script-on-JVM-languages to see how they had solved it, but I didn't find anything close to OpenSSL functionality. Which left me with the approach I've decided to try out: implement as OpenSSL compatible Java code as possible with JCE and JSSE. I'm convinced this is doable, but right now it feels quite hard.

The progress
So, how far have I gotten these last days? Pretty far, but not far enough. I'm basing the work on MRI's test suite for OpenSSL. From those I have test_digest.rb and test_cipher.rb passing all tests. This doesn't sound like much, but especially test_cipher was a pain to get running.

The plan from hereon is to get the utils.rb-file to load, which means implementing OpenSSL::PKey::RSA and OpenSSL::PKey::DSA, getting the basics of OpenSSL::X509 in place and also find a way to fix OpenSSL::ASN1. Oh well, I've got loads of time for this. Or not. =)

The problem
The real problem when implementing this, is the fact that Ruby's OpenSSL support is... Well, how shall I put it? Thin, you might say. It's basically a a wrapper around the C-library, which means that the disconnect when implementing this functionality with JCE is quite large. Just translating OpenSSL cipher names to the JCE equivalent is a challenge. But the big problem with the ciphers was initiating the key and IV (initialization vector). I have tried all the PBE solutions available, including the versions in BouncyCastle ending with "-OPENSSL". No luck.

The problem is that Ruby uses the function called EVP_BytesToKey, which, according to the documentation implements PKCS#5 1.5 with a few tricks up its sleeve. Not very nice. In the end I had to implement my own version of this to generate keys. And since I had to look like mad for this information, I will here give you the implementation to this function in Java. Just use the return value to initialize your own SecretKey-implementation and instantiate an IvParameterSpec and you should be set to go: (note, I release this into the public domain. And note, this is just quick, ported code to show the concept.)

    public byte[][] EVP_BytesToKey(int key_len, int iv_len, MessageDigest md, byte[] salt, byte[] data, int count) {
       byte[][] both = new byte[2][];
       byte[] key = new byte[key_len];
       int key_ix = 0;
       byte[]  iv = new byte[iv_len];
       int iv_ix = 0;
       both[0] = key;
       both[1] = iv;
       byte[] md_buf = null;
       int nkey = key_len;
       int niv = iv_len;
       int i = 0;
       if(data == null) {
           return both;
       }
       int addmd = 0;
       for(;;) {
           md.reset();
           if(addmd++ > 0) {
               md.update(md_buf);
           }
           md.update(data);
           if(null != salt) {
               md.update(salt,0,8);
           }
           md_buf = md.digest();
           for(i=1;i<count;i++) {
               md.reset();
               md.update(md_buf);
               md_buf = md.digest();
           }
           i=0;
           if(nkey > 0) {
               for(;;) {
                   if(nkey == 0) break;
                   if(i == md_buf.length) break;
                   key[key_ix++] = md_buf[i];
                   nkey--;
                   i++;
               }
           }
           if(niv > 0 && i != md_buf.length) {
               for(;;) {
                   if(niv == 0) break;
                   if(i == md_buf.length) break;
                   iv[iv_ix++] = md_buf[i];
                   niv--;
                   i++;
               }
           }
           if(nkey == 0 && niv == 0) {
               break;
           }
       }
       for(i=0;i<md_buf.length;i++) {
           md_buf[i] = 0;
       }
       return both;
   }

JRuby presentation in Malmö

It is now final that I will be presenting JRuby at JavaForum in Malmö, the 30:th October. There is more information available at www.javaforum.se.

Effectiveness of automated refactoring.

The last few weeks have seen some discussion regarding refactoring tools for dynamic languages. The basic questions are if it is possible, and if so, how effective it would be. More information about the debate in question can be found here, here and here. I'm not really going into the fray here; I just wanted to provide my hypothesis on something tangential to the issue.

The interesting point is something Cedric said in his blog:

And without this, who wants an IDE that performs a correct refactoring "most of the time"?

The underlying assumption here is that there can actually exist a refactoring tool that works all the time. So, my question is this: "Can a Refactoring tool be 100% effective, (where effectiveness is defined as completely fulfilling the refactoring preconditions, postconditions and invariants and without introducing dangers or errors in the code)."

My hypothesis (and there will be no rigorous proof of my position) is based on one axiom:

Automated refactoring is a subset of the Church-Turing halting problem.

I have no direct proof for this position, but it seems intuitive to me, that for a refactoring to always be completely correct, you would need to know things about the program that isn't always entirely possible to predict only from code. In this case, test runs would be necessary, and in those cases the halting problem enters. Now, for a strongly, statically typed language you would have to go to some effort to actually produce a program that couldn't be safely refactored, but the possibility is still there.

One commenter on one of the blog entries above said that a precondition for 100% refactoring of Java would be that you didn't use reflection and other meta-tricks. But the problem is, to avoid the halting problem you would have to remove enough features of Java to make it into a Turing-incomplete language. And by then it wouldn't be usable for general purpose programming.

I see no way out of this dilemma, unless my axiom is wrong. But if it is correct, there can never ever be a 100% effective refactoring tool.

JRuby import

OK, for those of you who thought that importing a class into the current namespace by assigning a constant, here is a small implementation (based on include_class), that lets you use import like you do it in Java:

 require 'java'

class Object
  def import(name)
    unless String === name
      name = name.java_class.inspect
    end
    class_name = name.match(/((.*)\.)?([^\.]*)/)[3]
    clazz = self.kind_of?(Module) ? self : self.class
    unless clazz.const_defined?(class_name)
      if (respond_to?(:class_eval, true))
        class_eval("#{class_name} = #{name}")
      else
        eval("#{class_name} = #{name}")
      end
    end
  end
end

import java.util.TreeMap

x = TreeMap.new
x['foo'] = 'bar'
puts x

JRuby progress

I thought that I should post a little notice about what's happening with JRuby right now. Most of this is available for you if you subscribe to FishEye for JRuby (http://fisheye.codehaus.org/changelog/~rss/jruby/rss.xml).

There are a few very nice innovations going on. The most important is probably that Java-integration has been improved quite substantially. Basically, if the package you need are in the java, javax, org or com packages, you can just refer to classes the same way you refer to them in Java (except for classes that doesn't have a capital initial letter, but you don't do that, do you?). Typical JRuby code could look like this:

 require 'java'

TreeMap = java.util.TreeMap
x = TreeMap.new(java.util.Comparator.impl { |m,o1,o2|
            o1 <=> o2
          })

Now, there are two new features in here, and one JRuby idiom that should be used in JRuby code. The first feature is to refer to classes by name, simply. The idiom is to import classes into the current namespace by assigning them to constants. The second interesting feature is the 'impl' method, that is available on all interfaces. This method will create an anonymous implementation of the interface, which will call the block when any of the methods in the interface is called. The method name goes in the parameter 'm' in the example. This allows a very clean syntax for implementing one-method-interfaces, like in the example above. Before these we're added, you had to use import_class for each class, then create a class for the interface, and then use this class. The code would easily have doubled for this example.

A few days ago Charles and Tom made me committer in the JRuby project, which obviously feels very good, though slightly nervous. But I've been able to fix a number of very small bugs since then. The more interesting of these, in no order:

• Correct implementation of default-inspect. (This isn't that interesting for implementations, but it makes debugging much easier)
• Hash#each didn't yield properly in some edge-cases. This fix was submitted by Miguel, and applied by me.
• I also added a fix to improve the Java method matching when the methods had a primitive in their arguments. This means that from now on, if you have two Java-methods, foo(float a) and foo(int a), and call it from Ruby with "foo(32)", the float-one won't get called anymore. Pretty nice.
• A String#crypt implementation, which is based on some stuff I did years ago, which means there are no copyright issues.
• The flash-issue with Rails (a message placed in the flash doesn't get removed). This was a Marshalling issue, which was quite easy to fix. Big win.
• Stack overflow when calling non-implemented methods when subclassing an interface. It calls method_missing instead, now.

There are a few things being discussed on the list right now. We will soon commit a readline.rb that checks if the JNI-based GNU readline-bridge is present, and if so uses it. This means that if you want, you can have basic readline in JIRB.

We are talking about the best way to implement OpenSSL support too. This is quite a big thing, though, and as Tom put it "It will be pretty difficult but the person who does it will be worshipped". I'm not sure about the right way to go with implementation either. It seems the available OpenSSL JNI-implementations aren't good enough, so the best route seems to be JSSE. If you have any opinions or suggestions, please get in touch.

In other news, it seems I'm going to be talking about JRuby at JavaForum in Malmö. The date is not final but it seems likely to be the 23 October. If you're in Malmö, please stop by. I'll post more information as soon as everything is clear.

Announcing ActiveRecord-JDBC 0.2.2

Version 0.2.2 of ActiveRecord-JDBC have now been released. It contains numerous smaller bug fixes, but more importantly the support for MimerSQL. The internals have been slightly refactored to allow easier change of database specific instructions further down the road.

The release can be found at http://rubyforge.org/frs/?group_id=2014 or installed through RubyGems.

The JRuby Tutorial #3: Playing with Mongrel

This part of the tutorial will be based on some slightly not-released software, but since it is so cool, I bet you will try it anyway. Basically, what I'm going to show you is how to get Mongrel 0.4 working with JRuby, and then how you can serve your JRuby on Rails-application with said version of Mongrel.

What you'll need
First of all, check out the latest trunk version of JRuby. There are some smoking new fixes in there that is needed for this hack. Next, you will also need to check out the 0.4-branch of Mongrel. This can be done with the following command:

svn co svn://rubyforge.org/var/svn/mongrel/branches/mongrel-0.4

You need to manually copy two parts of mongrel into your JRuby home. If $MONGREL_SRC is the name of the directory where you checked out mongrel, these commands will suffice:

cp -r $MONGREL_SRC/lib/mongrel* $JRUBY_HOME/lib/ruby/site_ruby/1.8
cp $MONGREL_SRC/projects/gem_plugin/lib/gem_plugin.rb $JRUBY_HOME/lib/ruby/site_ruby/1.8
echo '#\!/usr/bin/env jruby' > $JRUBY_HOME/bin/mongrel_rails
cat $MONGREL_SRC/bin/mongrel_rails >> $JRUBY_HOME/bin/mongrel_rails
chmod +x $JRUBY_HOME/bin/mongrel_rails

You will need to download the JRuby-specific http11-extension library. This can be downloaded here, and should also be put in the $JRUBY_HOME/lib/ruby/site_ruby/1.8-directory.

You're now set to go.

Simple web hosting
I will now show how to set up at small web server, that can serve both files and servlets. There really isn't much to it. First of all, we need to include some libraries:

require 'mongrel'
require 'zlib'
require 'java'
include_class 'java.lang.System'

Next step is to create a simple HttpHandler (which is like a Servlet, for you Java-buffs):

class SimpleHandler < Mongrel::HttpHandler
def process(request, response)
  response.start do |head,out|
    head["Content-Type"] = "text/html"
    results = <<-"EDN";
<html>
<body>
  Your request:<br/>
  <pre>#{request.params.inspect}</pre>
  <a href=\"/files\">View the files.</a><br/>
  At: #{System.currentTimeMillis}
</body>
</html>
EDN
    if request.params["HTTP_ACCEPT_ENCODING"] == "gzip,deflate"
      head["Content-Encoding"] = "deflate"
      # send it back deflated
        out << Zlib::Deflate.deflate(results)
    else
      # no gzip supported, send it back normal
        out << results
    end
  end
end
end

Now, this handler basically just generates a bunch of HTML and sends it back. The HTML contains the request parameters. Just to show how easy it is to combine Java-output with Ruby-output, I have added a call to System.currentTimeMillis. This could of course by anything. The last part is to actually make this handler active also. To finalize, we also start the server:

@simple = SimpleHandler.new
@http_server = Mongrel::HttpServer.new('0.0.0.0',3333)
@http_server.register("/", @simple)
if ARGV[0]
@files = Mongrel::DirHandler.new(ARGV[0])
@http_server.register("/files", @files)
end

puts "running at 0.0.0.0:3333"

@http_server.run

If you start this script with:

jruby testMongrel.rb htdocs

you can visit localhost:3333 and expect to see some nice output.

Making it work with Rails
A prerequisite for this part is that you have a functional JRuby on Rails-application using ActiveRecord-JDBC. If that is the case, you just need to go your application directory and execute this command:

$JRUBY_HOME/bin/mongrel_rails --prefix "" start

and everything should just work.

So, that's it. JRuby on Rails, with Mongrel. Enjoy.

In Joy and Sorrow with Continuations

Continuations is one of those topics that tend to crop up now and again. This is not strange, of course, since they happen to be one of the more powerful features of certain languages, but also is one of the most confusing one. I would like to stick my head out and say that continuations are probably up there besides real macros in power. The reason for this is that you can implement so many language features in terms of them.

Since there still seem to be some confusion about them, I'll write my piece on the. Not just for you readers of course, but more importantly for myself. I intend to get a good grip on continuations in Ruby by writing this (and this is incidentally one of the best ways to learn about something confusing; try to write about).

First of all, exactly what is a continuation? Basically, at every point in the evaluation of an expression, there will be one or more continuations lurking. For example, if we take the very simple expression foo = 13 * (10 - 7). In this place there is 4 interesting continuations waiting. (There are actually 8 of them all in all, but only 4 interesting.) We start by looking at the expression 10 - 7. If we look at the rest of the expression like this: foo = 13 * [] where the square brackets is the place where the result of the expression 10 - 7 will go. What's actually happening is that those square brackets is the continuation of the complete expression. The result of evaluating 10 - 7 will be injected into the rest of the expression, and that is what the continuation is.

Until now, I have spoken about continuations as a concept. Those of you who know the Ruby interpreter knows that it isn't coded in continuation-passing style. But it could be, and it doesn't really matter, since we still have a way to get at the current continuation. So, how should a continuation be represented, though? The way most languages choose to do it, a continuation is nothing but an anonymous closure, which takes one parameter, which is the result to return to the evaluation. In the example above, if we inject the callcc-primitive into the mix, we will have code that looks like this:

foo = 13 * callcc {|c| c.call(10-7) }

His doesn't really look that spectacular, of course. The above code will have exactly the same effect as the first example, namely binding the variable 'foo' with the value 39.

If you want to, you can look at every computation like this. It sometimes helps to imagine that you just wring the evaluation inside out.

So. What can you do with them? Mostly anything, actually. Many parts of Scheme is implemented in CPS (continuation passing style). But for a few concrete things that can be implemented easily: exceptions, throw/catches, breaks, returns, coroutines, generators and much more. As an example, we can implement a return like this:

 def val
  callcc do |ret|
    1000.times do |v|
      if v == 13
        ret.call(v+1)
      end
    end 
  end
end
bar = val
puts bar

What happens here is exactly the same result as if we had used the keyword return. Most of the other flow control primitives can be implemented this way too.

What has made continuations trendy lately is something called continuation web servers. The idea is to make the statelessness of the web totally transparent by hiding the client round trips inside methods, and these methods save the current continuation, and then breaks of evaluation. When the result from the server arrives, the continuation will be looked up from some session storage, and then restarted again, where it was. Basically, this allows web applications to work more or less exactly the same as a console application. This is very powerful, but as I hope this small post have shown, continuations have much more to give.

YAML and JRuby - the last bit

An hour ago I sent the patches to make JRuby's YAML support completely Java-based. What I have done more specifically, is to remove RbYAML completely, and instead used the newly developed 0.2-support of JvYAML. There were a few different parts that had to be done to make this possible, especially since most of the interface to YAML was Ruby-based, and used the slow Java proxy-support to interact with JvYAML.

So, what's involved in an operation like this? Well, first I created custom versions of the Representer and the Serializer. (I had a custom JRubyConstructor since May). These weren't that big, mostly just delegating to the objects themselves to decide how they wanted to be serialized. And that leads me to the RubyYAML-class, which is what will get loaded when you write "require 'yaml'" in JRuby from now on. It contains two important parts. First, the module YAML, and the singleton methods on this module, that is the main interface to YAML functionality in Ruby. This was implemented in RbYAML until now.

The next part is several implementations of the methods "taguri" and "to_yaml_node" on various classes. These methods are used to handle the dumping, and it's really there that most of the dumping action happens. For example, the taguri method for Object says that the tag for a typical Ruby object should be "!ruby/object:#{self.class.name}". The "to_yaml_node" for a Set says that it should be represented as a map where the values of the set are keys, and the values for these keys are null.

So, when this support gets into JRuby trunk it will mean a few things, but nothing that is really apparent for the regular JRuby user. The most important benefits of this is part performance, and part correctness. Performance will be increased since we now have Java all the way, and correctness since I have had the chance to add lots of unit tests and also to fix many bugs in the process. Also, this release makes YAML 1.0-support a reality, which means that communication with MRI will work much better from now on.

So, enjoy. If we're lucky, it will get into the next minor release of JRuby, which probably will be here quite soon.

Announcing JvYAML 0.2.1

The last few days have been spent integrating the JvYAML dumper with JRuby, and also to make YAML support in JRuby totally implemented in Java. As a side effect I have been able to root out a few bugs in JvYAML. Enough of them to warrant a minor release, actually. So, what's new? Working binary support, support for better handling of null types, better 1.o-support and a few hooks to make it possible to remove anchors in places where it doesn't make sense. (Like empty sequences.)

The url is http://jvyaml.dev.java.net and I recommend everyone to upgrade.

Two things in Rails

This will be a short in-between post. Don't expect to be annoyed, enlightened or even trivially entertained. I'm just going to describe two small things I do in all my Rails-projects, and I haven't found a way to do them as plugins. This is very annoying, of course, so I hope someone from the Rails team will eventually see this and tell me how to do it DRY.

1. Add a production_test environment
I feel constrained by the three environments that get delivered by Rails out of the box. And I find that for every project where the customer isn't myself and the codebase is bigger than about 50 lines of (hand-written) code, I tend to add a new environment to Rails; 'production_test'. The problem this environment solves is the situation where I want my customers to test out an application, but I don't want them to do it against a real production environment. For example, I did an application called LPW a few months back, that works against a 3rd party web service. This web service has one production environment and one test environment. I want the production_test to be as fast, responsive and generally as much as the production environment as possible, but not go against the production web service. I solve this by adding a production_test env which is exactly like the production environment, except I can just change the address to the web service endpoint to the test one.

I usually do this, so I can give my customers a nice application that they can play with, but without worrying about them damaging production data.

2. Add plugin environment configuration
This is actually a major pain. I have developed a few plugins, and generally I want them to have configurations based on which environment we are in. For example, the CAS authentication plugin shouldn't really redirect to the CAS server when in development environment. But, I can't set this in any good way, since the plugins will be loaded after the environment-specific files have been loaded. So, what I do is simply to add a new directory, called config/plugin and in environment.rb I have this:

 plugin_environment = File.join(RAILS_ROOT,'config', 'plugin', "#{ENV['RAILS_ENV']}.rb")
load plugin_environment if File.exist?(plugin_environment)

This solution sucks, but it works.

The Ruby singleton class

After my post on Meta-programming techniques I got a few comments and questions about the singleton-class. This feature seem to be quite hard to understand so I have decided that I will try to clarify the issue by first describing what it is, and then detail why it is so useful. This entry will be concept-heavy and code-light.

What it is
A child with many names, the singleton class has been called metaclass, shadow class, and other similar names. I will stay with singleton class, since that's the term the Pickaxe uses for it.

Now, in Ruby, all objects have a class that it is an instance of. You can find this class by calling the method class on any object. The methods an object respond to will originally be the ones in that objects class. But as probably know, Ruby allows you to add new methods to any object. There are two syntaxes to do this:

 class << foo
def bar
puts "hello world"
end
end

and

 def foo.bar
puts "hello, world"
end

To the Ruby interpreter, there is no difference in this case. Now, if foo is a String, the method bar will be available to call on the object referenced by foo, but not on any other Strings. The way this works is that the first time a method on a specific object is defined, a new, anonymous class will be inserted between the object and the real class. So, when I try to call a method on foo, the interpreter will first search inside the anonymous class for a definition, and then go on searching the real class hierarchy for an implementation. As you probably understand, that anonymous class is our singleton class.

The other part of the mystery about singleton classes (and which is the real nifty part) is this. Remember, all objects can have a singleton class. And classes are objects in themselves. Actually, a class such as String is actually an instance of the class Class. There is nothing special about these instances, actually. They have capitalized names, but that's because the names are constants. And, since every class in Ruby is an instance of the class Class, that means that what's called class methods, or static methods if you come from Java, is actually just singleton methods defined on the instance of the class in question. So, say you would add a new class method to String:

 def String.hello
puts "hello"
end

String.hello

And now you see that the syntax is actually the same as when we add a new singleton method to any other object. This only difference here is that that object happens to be an instance of Class. There are two other common ways to define class methods, but they work the same way:

 class String
def self.hello
puts "hello"
end
end

class String
class << self
def hello
 puts "hello"
end
end
end

Especially the second version needs explaining, for two reasons. First, this is the preferred idiom in Ruby, and it also makes explicit the singleton class. What happens is that, since the code inside the "class String"-declaration is executed in the scope of the String instance of Class, we can get at the singleton class with the same syntax we used to define foo.bar earlier. So, the definition of hello will happen inside the singleton class for String. This also explain the common idiom for getting the singleton class:

 class << self; self; end

There is no other good way to get it, so we extract the self from inside a singleton class definition.

Why is it so useful for metaprogramming?
Obviously, you can define class methods with it, but that's not the main benefit. You can do many metaprogramming tricks with it, that are impossible without. The first one is to create a super class that can define new class methods on sub classes of itself. That is the use I show cased in my earlier blog entry. The problem is that you can't just use self by itself, since that only gives the class instance. This code with results show the difference:

 class String
p self
end # => String

class String
p (class << self; self; end)
end # => #<Class:String>

And, if you want to use define_method, module_eval and all the other tricks, you need to invoke them on the singleton-class, not the regular self. Basically, if you need to dynamically define class methods, you need the singleton-class. This example will show the difference between defining a dynamic method with self or the singleton class:

 class String
self.module_eval do
  define_method :foo do
    puts "inside foo"
  end
end

(class << self; self; end).module_eval do
  define_method :bar do
    puts "inside bar"
  end
end
end

"string".foo # => "inside foo"
String.bar # => "inside bar"

As you can see, the singleton class will define the method on the class instead. Of course, if you know the class name it will always be easier to avoid having an explicit singleton class, but when the method needs to defined dynamically you need it. It's as simple as that.

Announcing JvYAML 0.2

I'm very pleased to announce that JvYAML 0.2 was released a few minutes ago. The new release contains all the things I've talked about earlier and a few extra things I felt would fit good. The important parts of this release are:

• The Dumper - JvYAML is now a complete YAML processor, not just a loader.
• Loading and dumping JavaBeans - This feature is necessary for most serious usage of YAML. It allows people to read configuration files right into their bean objects.
• Loading and dumping specific implementations of mappings and sequences. Very nice if you happen to need your mapping to be a TreeMap instead of a HashMap.
• Configuration options to allow 1.0-compatibility with regard to the ! versus !! tag prefixes.
• The simplified interface have been substantially improved, adding several utility methods.
• Lots and lots of bug fixes.

So, as you can see, this release is really something. I am planning on spending a few nights this week integrating it with JRuby too. And soon after that we will be able to have YAML completely in Java-land. That is great news for performance. It also makes it easier to just have one YAML implementation to fix bugs in, instead of two.

A howto? Oh, you want a guide to the new features? Hmm. Well, OK, but it really isn't much to show. How to dump and object and get the YAML string back:

 YAML.dump(obj);

or dump directly to a file:

 YAML.dump(obj,new FileWriter("/path/to/file.yaml"));

or dump with version 1.0 instead of 1.1:

 YAML.dump(obj, YAML.options().version("1.0"));

dumping a JavaBean:

 String beanString = YAML.dump(bean);

and loading it back again:

 YAML.load(beanString);

That's more or less it. Nothing fancy. Of course, all the different parts underneath is still there, and you can provide your own implementation of YAMLFactory to add your own specific hacks. If you want to dump your object in a special way, you can implement the YAMLNodeCreator interface, and your own object will be in charge of creating the information that should be used to represent your object.

Three ways to add Ruby Macros

As most of my readers probably have realized at this point, I have a few obsessions. Lisp and Ruby happens to be two of the more prominent ones. And regarding Lisp, macros is what especially interest me. I have been doing much thinking lately on how you could go about adding some kind of macro facility to Ruby and these three options are the result.

I should begin by saying that none of these options are entirely practical right now. All of them have some serious problems which I frankly haven't been able to come up with an answer for yet. But that doesn't stop me from blogging about my ideas, of course. Another thing to notice is that this is not about hygienic macros. This is the full-blown, power, blow-the-moon away version of macros.

MacRuby - Direct defmacro in Ruby
The first approach rests on modifying the language itself. You can add a defmacro keyword which takes a name and a code block to execute. Each time the compiler/interpreter finds a macro-definition, it will remember the name. When that name is found in the code later on each place will be marked. Then, before execution begins, all places where the call to the macro are will be replaced by the output from sending in the subnodes at that place by the output of calling the macro. An example of a simple macro:

 defmacro log logger, level, *messages
if $DEBUG
  :call, logger, level, *messages
else
  :nop
end
end

log @l, :debug, "value is: #{very_expensive_operation()}"

What's interesting in this case is that the messages will not be evaluated if the $DEBUG flag is not set. This is because the value returned from the macro will be spliced into the AST only if that flag is set. Otherwise a no-op will be inserted instead. Obviously, for this kind of code to work, the interpreter would need to change substantially. There is also a big problem with it, since it's very hard to fit this model into the object-oriented system of Ruby. As I think about it now, it seems macros would be the only non-OOP feature in Ruby, if added in this way. Another big problem with this model is that it is really not that intuitive what the resulting code from the macro will be. As soon as something more advanced needs to be returned, it will be very hard getting it straight in your head. One solution to this would be to do it the standard CL way. First write the output from the macro in several different instances. Then transform this to the AST code through a tool that parses the code. Then transform this into the macro. This process would be helped by tools, of course.

Back-and-Lisp-Ruby - Write macros in Lisp, translate Ruby back and forth
Another way to achieve this power in Ruby would be to separate the macro language from the main language. In effect, the macros would be a classic pre-processor. To offer the same power level as Lisp and others, the best way would be to write the macros themselves in a Lisp dialect, then transform Ruby in a well-defined way to Lisp and back again. (See the next version for more about this idea.) In this situation the same macro as before could look like this:

 (defmacro log (logger level &rest messages)
 (if $DEBUG
     `(,level ,logger ,@messages)
     '()))

The main difference in this code is that the macro and the output from the macro is Lisp. We have gotten rid of the ugly :call and :nop return values, and to me this seems quite readable. Of course, I'm not sure everyone else feels the same way. And we still have the same problem with Object Orientedness. It's missing.

RoCL - Ruby over Common Lisp
The final idea is to build a Ruby runtime within Common Lisp and transform Ruby into Common Lisp before running it. The macros could either be added as Ruby code or Lisp code. Everything will be transformed into the equivalent code in Lisp, maybe using CLOS as the Object-system, or building something based on Ruby's. Of course, the semantics of many things would change, and many libraries would need to rewritten. But in the end, there would be incredible power available. Especially if we can make it go both ways, so that Common Lisp can use Ruby libraries.

An example transformation could look like this. From this Ruby:

 class String
  def revert(a, *args)
    if block_given?
      yield a
    else
      args + [a]
    end
  end
end

"abc".revert "one" do |x|
  puts x
end

This is nonsense code, if you hadn't noticed. =)

 (with-class "String" nil
            (def revert (a block &rest args)
              (if block
                  (apply block a)
                  (+ args [a]))))
(revert "abc" "one" #'(lambda (x)
                        (puts self x)))

Conclusions
It is very hard to actually retrofit macros into Ruby after the fact. I'm still not sure it can be done and keep enough of Ruby's semantics to make it meaningful. It seems that we need a new language. But if I had to choose among these approach, the RoCL one seems the most interesting and also the most fun to implement. If I have a motto it would have to be something in the line of "best of all worlds". I want the best from Ruby, Java, Lisp, Erlang and everything I can find.

The Dark Ages of programming languages

We seem to be living in the dark ages of programming languages. I'm not saying this to bash everything; I'm actually being totally objective right now. Obviously, our situation right now is much better than it was 10 years ago. Or even 5 years ago. I would actually say that it's really much better now, than 1 year ago. But programming is still way too painful in almost all cases. We are doing so much stuff by hand that obviously should be done be computer.

I spend quite much time learning new languages now and then, to try to find something that's really good for me. So far, the best contestants are Ruby, Erlang, OCaml and Lisp, but all of those have their share of problems too. They just suck less than the alternatives.

• Ruby... I really like Ruby. Ruby is such an improvement that I really want to do almost everything in it nowadays. I think in Ruby half the time and in Lisp the other half. But it's not enough. It is still clunky. I want tail calls. I want real macros. I want blazing speed and complete integration with good libraries for everything and more. I'm just a sucker for power, and I want more of it in Ruby.
• Erlang and OCaml. These languages are really great. For specific applications. Specifically, Erlang is totally superior for concurrent programming. And OCaml is incredibly fast, very typesafe and has great GUI libraries. So, if I was asked to do something massively concurrent I would probably choose Erlang, and OCaml if it was GUI programming. But otherwise... Well, Erlang does have some neat functional properties, but not any nice macro support. It doesn't have a central code repository and many other things you expect from a general purpose language. OCaml suffers from the same things.
• Lisp is the love of my life. But as so many people before me has noted, all the implementations are bad in some way or another. Scheme is lovely; for research. Common Lisp is so powerful, but it needs users. Lots of them, creating libraries for every little data format there can be, creating competing implementations of particularly important API's; like databases.

Conclusion. Nothing is good enough, right now. I see two two paths ahead. Two ways that could actually end in the "100-year language".

The first path is one new language. This language will be based on all the best features of all current languages, plus a good amount of research output. I have a small list what this language would need to be successful as the next big one:

• It needs to be multiparadigm. I'm not saying it can't choose one paradigm as the base, but it should be possible to program in it functionally, OOP, AOP, imperative. It should be possible to build a declarative library so you can do logic programming without leaving the language.
• It should have static type inference where possible. It should also allow optional type hints. This is so important for creating great implementations. It can also increase readability in some cases.
• It needs all the trappings of functional languages; closures, first-order functions and lambdas. This is essential, to avoid locking the language into an evolutionary corner.
• It needs garbage collection. Possibly several competing implementations of GC's, running evolutionary algorithms to find out which one is best suited for long running processes of the program in question.
• A JIT VM. It seems almost a given right now that Virtual Machines are a big win. They can also be made incredibly fast.
• Another JIT VM.
• A non-VM implementation. Several competing implementations for different purposes is important to allow competition and experimentation with new features of implementation.
• Great integration with legacy languages (Java, Ruby (note, I'm counting on all Rubyists moving to this new language when it gets out, making Ruby legacy), Cobol). This is obvious. There are to many things lying around, bitrotting, that we will never get rid of.
• The language and at least one production quality implementation needs to be totally open-source. No lock-in of the language should be possible.
• Likewise, good company support is essential. A language needs money to be developed.
• A centralized code/library repository. This is one of Java's biggest failings. Installing a new library in Java is painful. We need something like CPAN, ASDF, RubyGems.
• The language needs great, small and very orthogonal libraries. The libraries included with the language needs to be great, since they have to be small but still pack all the most needed punch.
• Concurrency must be a breeze. There should be facilities in the language itself for making this obvious. (Like Erlang or Gambit Scheme).
  
• It should be natural to do meta-programming in it (in the manner of Ruby).
• It should be natural to solve problems bottom-up, by implementing DSL's inside or outside the language.
• The languages needs a powerful macro facility that isn't to hard to use.
  
• Importantly, for the macro facility, the language needs to have a well-defined syntax tree of the simplest possible kind, but it also needs to have optional syntax.

So, that's what I deem necessary (but maybe not sufficient) for a really useful, good, long term programming language. When I read this list, it doesn't seem that probables that this language will show up any time soon, though. Actually, it seems kinda unrealistic.

So maybe the other way ahead is the right one? The other way I envision is that languages become easier and easier to create, and languages have their strength in different places. Along this path I envision the descendants of Ruby and Erlang exploiting what they're good at and eschewing everything else. But for this strategy to work, the first thing implemented in each language needs to be a seamless way to integrate to other languages. Maybe there will come an extremely good glue-language (not like Perl or Ruby, but a language that only will serve as glue between programming languages), and all languages will implement good support for that language. For example you could code a base Erlang concurrent framework, which uses G (the glue language) to implement some enterprise functionality in Java sandboxes, and some places where Ruby through G will implement a DSL, which have subparts where Ruby uses G to run Prolog knowledge engines.

If you had to choose among the two futures, I am frankly more inclined towards the one-language one. But the multi-language way seems much more probable. And since I'm trying to choose way now, I'm placing my bets on the second option. We are not ready to implement G yet, but I do think that as many p-language techs as possible should do their best to learn how languages can cooperate in different ways, to prepare this project.

The difference between Kernel#` and Kernel#system

Today I had a fun learning experience. It cost me several hours of work, so I will post a small notice about it here so Google can make other developers lives easier. Or maybe I'm the only one who did this mistake.

Anyway. What I was trying to do was to start an external Ruby script (from another Ruby script). This other Ruby script went daemon, but since I didn't want to install the daemonize package (another bad decision, probably), I just wrote the script in question to fork and detach. Now, I have condensed the question a little, to this Ruby script:

 `ruby -e'if pid=fork; Process.detach(pid); else; sleep(5); end'`

Everyone please raise their hands if it is obvious that this script will sleep for 5 seconds before giving back my prompt. It wasn't obvious for me, since it was a long time I did UNIX System programming.

For those who still want to know, the problem is that backtick binds to the started process' STDIN, STDOUT and STDERR. As long as STDOUT is live, backtick will wait. And since the forking and detaching doesn't redirect all the STD* streams, this will wait until both processes has finished.

There are two ways to fix this. One right way, and one fast way. The right way is to detach the rebind the streams after forking. This can easily be done with this code:

 STDIN.reopen('/dev/null')
STDOUT.reopen('/dev/null')
STDERR.reopen('/dev/null')

The faster way is to replace backtick with a system call. Since system isn't interested in the output from the process, it will not bind those streams. So just running this instead, will work:

 system "ruby -e'if pid=fork; Process.detach(pid); else; sleep(5); end'"

I have learned the lesson. I have bought a copy of the Stevens book. (UNIX Network Programming, which detail the interaction between fork and ports, which was what my original problem was about.)