Faster YAML with byte processing

As noted in my last post, I have started work on converting JvYAML into JvYAMLb. Right now I have finished the work on the Scanner and the Parser, and it's looking quite good. The numbers I reported in the last post for regular JvYAML performance was wrong though. We're looking at about 7.8s to 10.0s for scanning that 3.5MB gemspec file. (And that's only the scanning, not file IO). But with the Scanner converted to use bytes and ByteList, the same processing takes 2.8s. That's a substantial difference. But it doesn't end with that.

As I said I also converted the Parser. It doesn't do any String processing at all, so I didn't expect either a speedup or slowdown except for that from the Scanner. But... Before, parsing the gemspec took 18.515s, but after, it runs in 4s. That's a dramatic speedup, and I don't really know where it comes from. Unless the earlier implementation generated so much more garbage, and used more memory, that it was noticeable in speed. Anyway, this looks good for JRuby YAML processing, since I expect big reductions in complexity in the callpath and generation of objects after the YAML processor is byted all the way through.

But tomorrow it's time to work on the Resolver, and that's going to be hard. Optimally, it would be nice to have a byte-based Regexp engine. And maybe that would be something for JRuby too, know? Our Regular Expressions must be dead slow now that they have to convert to strings all the time.

Announcing JvYAMLb, a fork

The conversion to using byte-arrays as the basis of our String work in JRuby has led me to realize that JvYAML just doesn't cut it anymore. The performance wasn't good to begin with, and it's even worse having to convert EVERY SINGLE STRING read into bytes. That's no good. As an example why something needs to be done I'm going to describe the transformations that happen to data in JRuby if executing this code:

YAML.load_file "gems.yml"

First, the file is opened, and wrapped inside a RandomAccessFile. Then data is read from it by YAML. Reading will proceed like this:
1. Bytes are read through the RAF, hopefully in chunks.
2. Those bytes are wrapped in a RubyString so they can be returned from the IO#read method.
3. An IOReader wraps that RubyIO object, gets the RubyString and converts it from bytes into a String, and this String gets converted into a char array.
4. That char array is returned to the YAML Scanner.
5. The chars from the char array is collected in a StringBuffer, and saved in various Strings as token values.
6. The parser, resolver and constructor work on these Strings in various ways.
7. The JRubyConstructor takes these Strings and creates RubyString objects from them and in the process converting the String back to a byte array.

Is there any doubt that this process is slow? Well, it hasn't been that big of a problem until now, since we are doing so well on performance in other parts of the system.

So, the radical decision is to rewrite JvYAML, making it more SYCK-compliant, working with InputStreams and byte-arrays, and in the process get away from several of the steps above. So that's what I'm going to do. I hereby create JvYAMLb. It will only be a part of the JRuby codebase, but it will be reasonably separate, so it can be extracted for other purposes. I will not stop work on regular JvYAML, but will maintain both projects.

Since the objective of this new project is blazing speed, I will post some numbers on this now and again. But first I will show you the speed of the regular system. JvYAML's Scanner can scan an old gem source index (about 3.5MB) of 435654 tokens in about 1654ms. This is the baseline I'm going to use to test performance, and I'll post more on this as soon as the byte-based Scanner is ready to try out.

Bytes bites. Or maybe not.

Well, the byte arrays are in, for good and evil. We had to wrap them in a counterpart to StringBuffer, but backed by byte[] instead, since all that explicit allocation and deallocation was way unperformant.

Of course, we aren't seeing any performance benefits from this right now. The problem is that there is still many places that use IRubyObject#toString to get at the contents. That operation is very expensive right now, so gem installs are slower, for example. But we have good hopes on improving the situation, and many parts of the codebase have become much clearer without the need to do String-to-byte[] and byte[]-to-String all over the place.

Fractured blogging

My blogging in the future will be a little bit fractured (or more fractured, some might say), since I have been invited to write at Inside Java for APress. The address is http://java.apress.com and I do recommend that you subscribe to the feed if you're interested in Java or associated technologies. My first posting was about the two closure proposals for Java 7, and I will try to focus my posting there to be more Java specific, mostly in article format. But if I write anything I deem to be exceptionally good, I promise to link from here to there.

Go subscribe now!

Serial JRuby

Things are really moving along faster than ever in JRuby land. It's so fun! As my last entry told you, Hpricot is now available for JRuby (and Java) people. I need to share a few lines from the logs of yesterday evenings conversation at #jruby:

<headius> seeya ola!
* shellac does some xsl-ing, plays on the wii, 
  then finds ola got HPRICOT working in that time
<shellac> I'm wasting my life

Some would say that what I do with JRuby is a waste of life... Well, we'll see about that.

Anyway, what's happened in JRuby world since last week? First, and most important, Charles has changed our RubyString implementation. It used to be backed by either a Java String or a StringBuffer. The problem with both of these is that Ruby has a tendency to use Strings as byte buckets. And our code was riddled with encoding and decoding into and out of byte arrays. So Charles took the big step, converted RubyString to use a byte-array instead, and fixed all the bugs that he found by doing that. The result is a happier codebase, less encoding and possibly faster Zlib and IO operations. That's big.

Tom is working on removing visibility and refactoring scopes. That could have huge impact too.

This Sunday I merged and fixed some code that allow Ruby code to inherit from Java classes and override methods there, and this overriding will be seen if an instance is sent back to Java. I'm planning on using this for some interesting tricks with Java ContentHandler's, and this functionality is really, really, really important. But it's also complex, since it requires generating bytecode at runtime. Fun, but hard. But now it's in trunk, and it's time to find the bugs in it and fix them.

I also need you to go read what Jonas Bonér has done with JRuby and OpenTerracotta. I could describe it here, but Jonas does a good job of it himself. So go there: http://jonasboner.com/2007/02/05/clustering-jruby-with-open-terracotta/. Very cool stuff, indeed!

So, the future is coming faster each day. JRuby will still conquer the world!

Hpricot goodness

This is just so cool, I cannot contain it. For those of you who haven't heard about Hpricot, it is one of why the lucky stiff's incredibly cool tools (which he probably will use to take over the world any day now...). It's HTML parsing goodness, very flexible, with the goal of being able to parse (and fix) everything that Firefox handles.

"So what?" you're probably asking... Well, Hpricot uses Ragel and some C code to achieve blinding speed. This means JRuby can't run it. Or I should say couldn't run it:

orpheus:~/workspace/jruby> jruby bin/gem install hpricot --source http://code.whytheluckystiff.net
Bulk updating Gem source index for: http://code.whytheluckystiff.net
Select which gem to install for your platform (java)
1. hpricot 0.5.110 (jruby)
2. hpricot 0.5.110 (mswin32)
3. hpricot 0.5.110 (ruby)
4. hpricot 0.5 (ruby)
5. hpricot 0.5 (mswin32)
6. hpricot 0.5.0 (ruby)
7. hpricot 0.5.0 (mswin32)
8. hpricot 0.4.99 (ruby)
9. hpricot 0.4.99 (mswin32)
10. hpricot 0.4.92 (ruby)
11. hpricot 0.4.92 (mswin32)
12. Skip this gem
13. Cancel installation
> 1
Successfully installed hpricot-0.5.110-jruby
Installing ri documentation for hpricot-0.5.110-jruby...
Installing RDoc documentation for hpricot-0.5.110-jruby...

That's right, Hpricot is now more promiscuous than any other gem with native parts.
What can you do with it? Well, I'm just going to point you to _why's own description of it. All he says at http://code.whytheluckystiff.net/hpricot/ will work fine in JRuby!

How did this come to be? Well, me and _why did some joint hacking, which was helped along by the fact that Adrian Thurston (the genius behind Ragel) recently added Java support to it. So, basically, most of the Ragel definition is exactly the same for both the C and the Java versions. The native code has been factored out, and both versions are buildable with rake from _why's code repository.

This is important. Don't think anything else. This strategy will, and can, be used for other gems with native parts. It's just a question of time.

JFokus post-mortem

Yesterday marked the first one day conference in Sweden focused on Java. All in all, it was a resounding success. I am personally very happy about how well everything ran. The only part that wasn't really up to par was the panel discussion.

So, a quick rehash of the day. First Simon Phipps did his thing in the keynote, speaking much about where Sun's open source initiative comes from. I would have liked some more specific info on the future though, but I guess that's hard for him to say...

After that, Thorbiörn Fritzon from Sun in Sweden started of the Language track with talking about some new features in Mustang, specifically some Swing topics and the JSR 223 support (scripting). After that he showcased a really cool stack-based calculator he'd hacked during the weekend, where you could attach the functioning of the buttons to different scripts, and the scripts could be implemented in any language on the classpath that supported JSR 223. The final showdown of calculator buttons was first a script in Haskell (using Jaskell) that pushed a lambda on the stack. This lambda curried addition, with the first object on the stack. Then another script written in Python popped the lambda from the stack, popped the next value and called the lambda with the value from the stack, in effect doing an addition the longer way...

After that it was time for my presentation. It felt really good and I believe most in the audience was satisfied.

After that I sat in on Dan Bergh Johnsson from Omegapoint talking about domain driven design, and how to refactor to find your domain. I found it quite elementary, and nothing really new.

Last, Jonas Bonér talked about OpenTerracotta, which is an incredibly cool tool for clustering your JVM applications, and by doing that achieving many different cool things. Look it up! I wish there had been more time for Jonas to get in depth about the implementation of Terracotta, though. Most interesting thing he told us was about a situation where they wanted about 50-80 clients pounding in a cluster at the same time, and they used Terracotta for this too, using a CyclicBarrier that was synchronized over all the JVM's. Very nice.

The panel debate wasn't that interesting. Most of the questions asked by the moderator didn't really achieve any discussion. Open sourcing Java doesn't seem like such a controversial topic anymore.

After the debate, the conference was over, but then BEA provided a pub evening with entertainment, so we stayed and networked a little, talked with interesting people and generally had fun. After that, six of us continued to a pub in Stockholm, called Monk's Cafe, where we drank good Belgium beer and talked until our throats were sore. A very interesting day, no doubts.

Current JRuby status - AKA The what's-cool-and-happening

This is just a small update on what's going on in JRuby development at the moment. The first and most important item is that Tom has merged his block-work to trunk. He has done a monumental achievement with a patch on over 17k lines. The most amazing thing is that my main test case for block problems (Camping) runs perfectly. Tom also reports some performance improvements, but that isn't the important thing. What this block patch is all about is to remove the block stacks from our ThreadContext, and instead just pass the current block along through the Java stack (in other words, as a parameter to the method call in question). This simplifies many things and also make it possible for our compiler to finally compile closures and blocks. Charles will soon continue the work on making this happening, but being able to do that means the JIT will work for vastly more of the code than it does at the moment.

The second work we're hard at right now is Rails compatibility. Tom is looking at ActiveRecord, Charles have improved Marshalling (with the side effect that the gem local source cache finally works) and also did some work on Multibyte::Chars. This work is going fast forward. My part in this is ActionPack, where we right now are down to 9 failures and 0 errors, from 23 failures and 8 errors. Those numbers are for 1150 tests and 5202 assertions, so there isn't much left to get this working.

My work on the YARV compiler continues, and I will soon post an update about this too.

Focus on Java, this Tuesday

This Tuesday is JFokus in Stockholm, where I will talk about Java, JRuby and RSpec and how to combine these in exciting ways. Let me tell you, I look forward to speaking about this. I believe there is something incredibly useful in combining technologies like this.

Of course, it would also have been great fun to take about where JRuby is at right now, since there is so much cool stuff going on at the moment. When the 0.9.3 release comes along, I bet many people will be surprised and as excited as I am about this. I will get back with more information what has happened in the codebase lately quite soon.

If you're going to JFokus, please say hi if you see me!

Advice

I usually don't like to use this blog for strictly personal reasons, but I will do an exception for once, to get advice from y'all.

Hypothetically, I would like to know what kind of work opportunities I should look for, if I was in a situation where my current work situation was alright, but not satisfactory (since I've been at the same place for 6 years.) Further, I'm interested in getting away from Sweden. The places I would most likely want to be situated in would be SF or London.

The picture is complicated by my lack of academic degrees. The kind of experience I have is strictly professional and recreational. (In which category should I put my work on JRuby?). Apart from that, my interests are in the region of work where I can use different languages, and will not be constrained by Java. I haven't done consulting, and maybe I should do that. On the other hand, I'm not sure that is the right thing for me either.

So, please. I need advice. What should I do with my professional life at this moment?

YARV tail call optimization in JRuby

After some further developments on the YARVMachine, I can report that JRuby trunk supports most of the optimized instructions that YARV makes use of. The big thing missing is the local cache optimization. But that is just an aside from the main course.

Because right now there is support in JRuby's YARVMachine to handle a limited form of tail call optimization. Specifically, if the last call in a method is to the method we're in and the receiver of that call is the same as the current self, tail call optimization will happen. That allows JRuby to run this script:

def fact(n, acc)
if n == 0
acc
else
fact n-1, n*acc
end
end

p fact(25000, 1)

That will explode in both YARV and MRI, long before getting anywhere, since the stack goes out. But this will work correctly in JRuby with -y, provided that the tail call optimization is actually turned on. To turn it on, use $JAVA_OPTS and set the variable jruby.tailcall.enabled to true, like this:

JAVA_OPTS="-Djruby.tailcall.enabled=true"
jruby -y testRecFact.rbc

Of course, you need latest trunk for this to work, and you need to compile the YARV file as described in my previous post. But provided you have done that, the above will actually work perfectly, without deepening the stack at all. The reason this is not turned on by default in the YARVMachine is that it isn't completely safe yet. There is one small kink, which is that if you redefine the current method within the current method, and then call self, this will not work as expected. It should be easy to add checks for this behavior, though, so that will come soon.

Of course, the big problem with something like this is the stack trace. How to handle it? The implementation behavior would make it look like those 25000 calls to fact was just a single one. In the same manner, trace methods won't get called.

But trunk will actually report when these tail calls have been done. Say you have a script like this:

def a(n)
if n > 20000
  raise "hell"
end
a(n+1)
end

a(0)

(it is designed to explode at a certain depth, of course.) In current JRuby trunk, executed with -y, this will generate an output that looks like this:

testRaise.rb:-1:in `a' TAIL CALL(20001): hell (RuntimeError)
       from testRaise.rb:-1

So it is quite obvious that the exception was raised in the 20001:th invocation of 'a'.

This work goes forward very fast, and it's funny to see progress happen this quickly. Yesterday SASADA Koichi visited the #jruby and #rubinius IRC channels, and we had some very interesting discussions on further collaboration. The future looks bright indeed, and this weekend I'll probably have time enough to take a first crack at a YARV compiler for JRuby.

Executing YARV (in JRuby)

I guess the title says it all. Oh, it doesn't? You want what? A tutorial on how to play with this? Well, alright then.

What you will need to get started is first and foremost JRuby trunk. Download and build that. You will also need either a very recent version of YARV or Ruby trunk. Download either and build that too. I have the source in $YARV_SRC and $RUBY_TRUNK_SRC. I've also installed them with suffixes, so I have ruby-yarv and ruby-trunk globally available. That's a quite good setup.

Before compiling some Ruby, we need to change the change the compile-script slightly. So open up the file in $YARV_SRC/tool/compile.rb and disable the peephole_optimization and inline_const_cache, since JRuby doesn't really support these features yet. If you use Ruby trunk, you also need to do a global search-and-replace in this file, from YARVCore to VM.

Now you're finished to compile something. The goal for this execution have been to get an iterative fibonacci sequence running, so save this code in a file called fib.rb:

def fib_iter_ruby(n)
  i = 0
  j = 1
  cur = 1
  while cur <= n
    k = i
    i = j
    j = k + j
    cur = cur + 1
  end
  i
end

puts fib_iter_ruby(5000)

Next, compile this file:

ruby-yarv $YARV_SRC/tool/compile.rb -o fib.rbc fib.rb

Modify as needed if you use Ruby trunk instead. Now, how to run this? With the y-flag to JRuby:

jruby -y fib.rbc

As you can see, simple files like this just work. Of course, there are much missing yet, but basic interpretation works really nice. Some more additions to the YARV machine will come quite soon, probably. But the next step will be to implement a compiler for YARV too. From that point it should be possible to execute Ruby-files in JRuby in several different ways; by regular interpretation, YARV compilation, YARV interpretation, Java bytecode AOT compiling, Java bytecode JIT compiling and also possibly Rubinius interpretation. The future is, as always, interesting.

Rubinius and Lisp

Yesterday on the #rubinius-channel, me, Evan and MenTaLguY had the beginnings of a discussion about a Lisp dialect for the Rubinius VM. Since this sound exactly like the kind of thing I've been thinking about for some time, I got excited. But anyway, Pat Eyler covered the issue much better and did a small interview with us that can be read here. Ruby is wonderful. Rubinius is sweet. With Lisp added to the mix, it will be better.

I'll probably follow this up with some more information and ideas, as soon it's fleshed out.

That Rails Thing

I need to join the fray. Pat Eyler has announced a competition together with APress. The first installment is about how Rails has made me a better programmer. Details to be found here.

So, let's get at it. Rails has inspired and affected me in numerous ways. You got to realize that I have been programming web applications in Java for way too many years. Before that I've done both ASP and PHP. Nothing ever felt natural. I've tried many of the LISP frameworks (and Uncommon Web is really cool). I've embraced continuation-based frameworks. But none felt so powerful, yet nonintrusive. Rails embodies the best parts of Ruby.

For me, Rails is very interesting for several reasons. First and foremost is the fact that it is an incredibly good framework. It's really amazing how good it is. And that in itself acts as an inspiration. To know that software can be this good makes me want to strive to perfect my API's, make my libraries more usable, and finding new and novel ways to improve my DSL's.

But that's only part of it. The flip side of this is that Rails is not perfect. In fact, there are myriad ways it can be improved. And that gives me hope, because it also means that no matter how good my code gets, it can always get better. And that means I'll never be totally bored!

In a very literal sense, Rails has made me a better programmer in another way. Since Ruby on Rails was the first Ruby-application to make it into Karolinska Institutet, it means I can thank Rails for being able to write more Ruby at work. And writing Ruby instead of Java must make you a better programmer, neh?

Of course, Rails is an excellent testing tool for seeing how far we have left with JRuby. It goes without saying that you become a better programmer by implementing a language...

I have several applications and libraries that I keep in mind when writing my code. Those libraries act as a gauge against which I measure my code. Code quality, testing ability, interface, sheer trickyness; in all these areas, Rails is one of the top frameworks I know.

Five Things.

Oh well. It has finally hit me, as I knew it would sooner or later. Charles O Nutter managed to nail me.

If you don't know what this is, it's called the "5 Things Meme" and it means I'll tell you 5 things about me that you probably don't know, and then pass the baton on to 5 other people. We'll see about that.

• I am an "academic failure" in that I haven't even finished the Swedish equivalent to High School (in Swedish it's Gymnasium). There is no thing about computers I have learned in school. And even though I have no education, I actually work centrally at Karolinska Institutet, which is one of the more esteemed universities in Sweden.
• I am an amateur musician. I play guitar, some piano, sing quite a lot and play a few other instruments (but not very good). I create some music, but nothing that has been finished enough to make public. Until recently I was a singer in a Swedish barbershop chorus called the Entertainmen. We came 14:th on the World Barbershop Competition in Salt Lake City 2 years ago. I used to sing Lead, but switched to Bari.
• I train martial arts. I have been doing so for most of my life, but not with enough regularity to attain a black belt yet. What I've been training for the last few years is called Bujinkan Budo Taijutsu, and it seems I will stay with it.
• I am a vegetarian, and has been for the past 10 years. I don't eat meat, fowl or fish, but I do eat egg and dairy products.
  
• I live in a former mental hospital, in a beautiful park 20 minutes from central Stockholm. The hospital was built 1908 and looks very much like a castle.

So. Now I've done my share. What's left is to give this burden to someone else. Since I don't like the concept of chain *anything* I will introduce a small twist by tagging 5 people who probably won't do it, for several reasons. But if they do it, it will be all the more interesting. So, without further ado:

• Steve Yegge
• Jonathan Schwartz
• why the lucky stiff
• Matz
• Martin Fowler

I will present at JFokus

A few weeks ago I noted that Sun in Sweden announced on the JavaForum meetup that they were going to have a full day Java conference in January. This conference is called JFokus and will have 4 different tracks with 4 speakers on each track. I will feature in one of these tracks, talking about testing Java code with JRuby and RSpec, and also how to build a DSL for doing database integration testing using Ruby. I believe it will be very interesting, and the rest of the day looks incredible nice too.

So, if you're in Sweden, Jan 30:th, look into it!

More information can be found at the official site: http://www.jfokus.se/.

Profile support in JRuby trunk

As noted a few posts ago, Sandbox has been ported and run in JRuby. You did need to use a special version, only available from a branch in the JRuby source tree. But now this support has been added to trunk. This means you can check out the latest version of JRuby trunk, gem install javasand, and everything will work.

For more info, see my previous post on this: here.

A Personal blog

Due to lots and lots and lots of stuff happening in my life right now, I haven't had the time to write anything substantial here in a while. Hopefully, this will change.

I have also decided to start a new blog, for more... personal musings. Books, music, movies. Stuff like that. Nothing that would interest the regular readers of this blog, probably. But anyway, here's the address: http://olabini.blogspot.com.

Ah, right: Have a merry christmas and a happy new year, everyone!

Another neat trick

Another sweet thing, that I've actually wondered idly about from time to time, is how you could get "provided?"-style parameters in Ruby. If you have used a language with this functionality, like Common Lisp for example, you know that this can be highly useful.

And just the other day, while reading Hal Fultons excellent (but poorly proof read) The Ruby Way (2nd ed) I found the way. Ergo, it looks like this:

def foo(bar, baz = (baz_provided=true; 42))
 p [bar,baz,baz_provided]
end

foo(13,47)
foo(13)
foo(13,42)

This will provide the output:

[13, 47, nil]
[13, 42, true]
[13, 42, nil]

which illustrates the usage pretty well. The magic, of course, lies in the fact that default parameters are evaluated, just the same as anything else. You could do some very crazy things inside that default value specification if you wanted. Anyway, just a nugget.

A very small Ruby method

I haven't had much time or inclination for blogging lately. Not much happening at the moment, actually. But I came up with one small thing I wanted to document. Just a practical thing for certain situations. Basically it's a with-method, that works fairly well. It's nothing magical and the trick is basic. It's more or less an alias, actually:

module Kernel
 def with(obj = nil, &block)
   (obj || self).instance_eval &block
 end
end

with("abc") do
 puts reverse
end

"abc".with do
 puts reverse
end

As you can see, insstance_eval can be used like JavaScript or VB's with. This is nice for the simple reason of documentation. I find this usage much easier to read and understand than most usages of instance_eval that I've seen.

So, that's it for today. I'll probably be back soon with some recent JRuby developments too.

The Curse of Camping: The story continued

It's time for another installment in the Camping saga. The last time I wrote about 1.5, and some things that didn't work properly in JRuby. This time, the announcement is once again this: Camping 1.5 works in JRuby. It works very well, actually. To show this, I will make it very easy for you to follow along in how to achieve it.

First, have JRuby installed, either from trunk or version 0.9.2 if that version has been released when you read this.

Secondly, gem install ActiveRecord, Camping and all their requirements.

Thirdly, gem install ActiveRecord-JDBC.

Fourth, put you MySQL JDBC driver on your classpath (or any other JDBC driver you care to use).

Download blog.rb from the Camping examples here. Modify it by adding the line

require 'jdbc_adapter'

after the other require's at the head of the file.

Finally, you need to create a database file, since JRuby doesn't support the standard choice of camping (which is SQLite3). To do this, create a file that's named (on Linux) $HOME/.campingrc, or on Win32 %APPDATA%/Campingrc. (%APPDATA% is on most Win32-boxes something like c:\Documents and Settings\olagus\Application Data). This file should contain your database definition in standard ActiveRecord fare. My example looks like this:

database:
 adapter: jdbc
 driver: com.mysql.jdbc.Driver
 url: jdbc:mysql://localhost:3306/blog
 username: blog
 password: blog

Of course, you should use your own individual settings and driver information.

Camping does only require that the database exists. It will add migrate the blog application automatically, and as soon as you've started the application by doing:

jruby $JRUBY_HOME/bin/camping blog.rb

it will listen on 3301 and be ready to serve an example blog application.

Have fun and enjoy!

Freaky freaky Sandbox has come to JRuby

I would like to announce the first release of JavaSand, which brings _why's spooky Sandbox to JRuby.

For those of you who hasn't played with Sandbox, it allows you to create a new area for executing Ruby where you have control on which classes are loaded and available. It's good for lots of things. It can be made very secure, for example. Or you can load several applications in different Sandboxes and they won't interfere with each other in any way.

Now, in MRI, this is done through some fairly involved swapping of symbol tables. In JRuby things are quite a lot easier. Actually, it's only a matter of loading the right classes and remove all methods not allowed.

So, how do you get to use sandbox from JRuby? First of all, you need a special JRuby, that can be checked out and compiled from svn://svn.codehaus.org/jruby/branches/sandbox. This functionality will come to regular JRuby to, but not until after 0.9.2 has been released. So, when you have this special version downloaded, you just install the RubyGem called javasand, and you're set to go. If you would like a fun example to try out, you can download sandbox_server directly from _why's code repository http://code.whytheluckystiff.net/svn/sandbox/trunk/bin/sandbox_server. To run it:

jruby -rubygems sandbox_server

and you can connect to your machine at port 5000, press enter, and you have an IRB session. Wow!

So, the project JavaSand is part of JRuby-extras and the source is in that repository, at RubyForge! Go ahead and test it now!

The FINAL OpenSSL post?

Possibly.

I've checked in all functionality I will add to OpenSSL support in JRuby at this point. Of course, there will be more, but not concentrated in a spurt like this. Tomorrow I will modify the build process and then merge everything I've done into trunk.

Let's back up a little. What have I accomplished? This: All OpenSSL tests from MRI run (except PKCS#7). That includes tests of SSL server and SSL client. Simple https-request also works. This is sweet. Everything else there is tests for in Ruby works. But... this is also the problem. Roughly half of Ruby's OpenSSL library is not tested at all. And since the current OpenSSL initiative from my part is based on tests, I haven't done anything that isn't tested for.

So, some things won't work. There is no support for Diffie-Hellman keys right now, for example. Will be easy to add when the time comes, but there isn't any testing so I haven't felt the need.

The only thing not there, as I said, is PKCS#7. That was just too involved. I'll take care of that some other time, when someone says they want it... Or someone else can do it? =)

So, what this boils down too is that JRuby trunk will have OpenSSL support sometime tomorrow. Hopefully it will be useful and I can get on to other JRuby things. I have a few hundred bugs I would like to fix, for example...

Oh yeah, that's true. Tomorrow will also be YAML day. I'll probably fix some bugs and cut a new release of JvYAML. It's that time, the bug count is bigger than it was, and JRuby needs some fixes. So that's the order of day for tomorrow. First OpenSSL and then YAML. Any comments on this, please mail or comment directly here.

G'night.

JavaForum: last night

And now it's over. Last night was the last presentation for me in a while. Over 200 people where there and the place was packed. I must say I'm very pleased with how my part went. I felt really good, much flow and the demonstrations went perfect.

Rob Harrop and Adryan Colyer talked about Spring 2.0 and AOP. It was fairly interesting, but since I've done some work with AOP before I felt that the level was a bit to introductory. No real meat.

JavaForum announced a day focused on Java in January, with four tracks and some great speakers. Looks very nice.

JRuby in Stockholm, JavaForum on Tuesday

On Tuesday it's time for the big JavaForum in Stockholm. I'm looking very much forward to it and have spent some hours today sharpening the presentation. Of course it will be very interesting to see Rob Harrop too.

In other news, something quite exciting is probably just around the corner...

JRuby presence at JavaOne 2007

The call for papers went out two days ago, and we already have several good proposals in line that will be submitted to JavaOne. Hopefully we will get many interesting talks this year. As far as I know, there are at least 5 JRuby-specific proposals submitted, and there will hopefully be more.

I am sending in two different proposals this year, one about JRubyEE (it's called "Agile Enterprise Development with JRuby"). That will be interesting to do and I really believe there is much for Java enterprise development to use in JRuby.

The second proposal is for a BOF about several JVM languages called "Lambda on the JVM: JRuby, Jython and Lisp". I'll talk about how to utilize different JVM alternative languages to do interesting and useful things. Since this is my venue, I will specialize on languages that are derived from Lisp (implicitly or explicitly). (I include both Ruby and Python in this category).

Hopefully at least one of the proposals will get through.

The obligatory meta-blog post.

So, this is post number 100, since I started blogging. And that's almost exactly one year ago. Isn't that interesting? I've been writing something here each 3.65'th day. Often it's been uninteresting, very often indifferent, but sometimes I have manage to write something that people liked. So, I'm happy about this small anniversary.

So, I plan to do what almost every blogger do at least once: turn the writing process inwards and investigate why people blog, why the person in question blogs, and perhaps, if we are really META, why the current blog post is actually written. Hopefully I'll know why I've written this by the end of this post. Otherwise, it'll probably be something indifferent or boring.

Let's begin. "Why blog?", the general question. When I started out, I had no real goal. I knew that I wanted to get better at technical writing and that was largely it. I hoped I would get some new contacts and interesting discussions but the base reason was for the writing ability. As such, I wouldn't have to write public; I could hone my writing skill in my basement and never publish anything to the (sometimes scary) scrutinizing eyes of the public. But that's the problem. If I didn't publish it, I wouldn't have as much incentive to become better. Now I absolutely have to get better at writing, otherwise I will feel ashamed about the filth that will for all time be remembered by Google.

That's reason number one. The second reason, that I've come to realize this last year but didn't know when I started out, is that I learn by writing about a subject. If it's a technical topic I basically have to research the topic quote thoroughly, so I won't look like a fool. And that is really good, that is incredibly good. I have learned very much by writing about things I find interesting. The first learning when I research and think about what I should write, and the second learning when people read it and comment and correct me.

Reason number three can be thought of as hubris. I believe that some of the things I know can be useful for other to know, and I believe that some of the problems and bugs I uncover in open software should be documented somewhere so others won't have to go through the same bug finding escapades as I did on that subject.

And here's where we come to the point. The point is that if you're working with software (or with anything where information sharing is important, really), you should write about. Because the writing benefits you and it benefits me. If you find a problem, write about it. Doesn't matter if someone actually reads it or not, for the most important aspects of writing is about yourself.

Steve Yegge wrote about this in his "Blog Or Get Of The Pot", and also in one of this older blogs ("Why you should blog" I believe it's called).

Coda: Why did I write this post? To make it clear in my head why I blog. To make it obvious and explicit why I do it. So that's what this post is about, and it's a recursive post since it talks about itself. I'm satisfied.

Some notes from the Stockholm Rails meet

Last night (Wednesday) about 35 developers with an unhealthy interest in Rails met up in Stockholm, at Valtech's offices, to share some experiences and talk shop.

It was very interesting and fun to meet people I can relate to. We ended up talking programming languages for a few hours after the main event ended.

Peter Marklund did a presentation on a CRM system in Rails, and Christian and Albert from Adocca talked about caching, and handling a Rails application that needs to scale into the millions. Nice stuff.

Last, I did an improvised presentation on LPW, a small Rails application that is fed its main data through Web Services instead of ActiveRecord. I believe it went fairly well, even though I had no slides and almost no preparation... =)

Speakings this and next week

Just a quick notice. I will be talking this wednesday at a Rails group in Stockholm. The subject will (for once) not be about JRuby, at least not primarily. Instead I will talk about an application called LPW, which is interesting since it doesn't use a database as the main data feed, but a bunch of Web Services that are implemented by a third party and deployed with EJB's in Axis. I had all kinds of challenges, but in the end the result became very good. I will talk a little about what you can expect when trying to interface with Java in this way, and good tricks on how to handle an alternative to ActiveRecord as model.

I've written a bit about LPW sometime this summer, if anyone is interested.

Next week is the JavaForum meet up in Stockholm, and I will speak after Rob Harrop. The subject is JRuby and the day is Tuesday. As far as I know, the meet up is full.

The presentation will be pretty standard JRuby fare for Java developers.

Further OpenSSL progress

Sometimes it feels like the progress I've done with the OpenSSL library is almost swallowed up by all the new functions I've found that needs to be implemented. It's an uphill battle.

implementation doesn't handle the But, I'm happy to say that the latest digression is finished. I have successfully implemented the X509_STORE-family of functions plus all dependencies. In the end I had to create a wrapper for X509Certificate and my own PEM-reader and PEM-writer, since BouncyCastle'sOpenSSL specific aux trust information that can be appended to these certificates.

But anyway, that means there is a Java implementation of X509_STORE, X509_STORE_CTX, X509_VERIFY_PARAM, X509_LOOKUP, X509_LOOKUP_METHOD, X509_TRUST, X509_PURPOSE and a whole slew of others too. About the only thing I didn't implement was X509_POLICY_TREE. That was just too much. I'll tackle that hurdle if it seems Ruby needs it.

So, what is the status then? Coming in at almost 10 000 lines of code there is only three real parts left to do. Or, three parts left that have MRI test cases, I should say. Since there are quite a few files not tested in the MRI implementation. Like DH keys. Wow. But I ignore that for now. The current goal is the OpenSSL::X509::Store-family, the OpenSSL::PKCS7-family, and the SSL-stuff, the rest of the way. So, wish me luck.

An Emacs diversion: Font sizes

After a few weeks of very intense JRuby-OpenSSL hacking, I felt the need to do something different, so I've spent a few hours with my slightly rusty Emacs Lisp skills, trying to fix something that I really need. Namely, control over font-size and fonts in Emacs, on Linux. I want it inside Emacs and customizable by EL. To my surprise I couldn't find anything like that anywhere.

For me personally, it's necessary when presenting, since I usually code with a small font in Emacs, the code will be totally unreadable when presenting. And since I don't have a fancy MacBook Pro, I need to be able to zoom in and out inside Emacs.

Presto, it wasn't easy, but I've managed it. For some reason, font handling seems quite backward in Emacs. I had to extract the current font, and then split it and join the new array together again. Not neat and my way of doing it is not the best. But, for your pleasure, here is the code to do it, and also some code that establishes a font ring of the standard fonts in different sizes that can be walked through:

(defun inc-font-size ()
 (interactive)
 (let* ((current-font (cdr (assoc 'font (frame-parameters))))
        (splitted (split-string current-font "-"))
        (new-size (+ (string-to-number (nth 7 splitted)) 1))
        (new-font (concat (nth 0 splitted) "-"
                          (nth 1 splitted) "-"
                          (nth 2 splitted) "-"
                          (nth 3 splitted) "-"
                          (nth 4 splitted) "-"
                          (nth 5 splitted) "-"
                          (nth 6 splitted) "-"
                          (number-to-string new-size) "-*-"
                          (nth 9 splitted) "-"
                          (nth 10 splitted) "-"
                          (nth 11 splitted) "-*-"
                          (nth 13 splitted))))
   (if (> (length splitted) 14)
       (dotimes (n (- (length splitted) 14))
         (setq new-font (concat new-font "-" (nth (+ n 14) splitted)))))
   (set-default-font new-font t)
   (set-frame-font new-font t)))

(defun dec-font-size ()
 (interactive)
 (let* ((current-font (cdr (assoc 'font (frame-parameters))))
        (splitted (split-string current-font "-"))
        (new-size (- (string-to-number (nth 7 splitted)) 1))
        (new-font (concat (nth 0 splitted) "-"
                          (nth 1 splitted) "-"
                          (nth 2 splitted) "-"
                          (nth 3 splitted) "-"
                          (nth 4 splitted) "-"
                          (nth 5 splitted) "-"
                          (nth 6 splitted) "-"
                          (number-to-string new-size) "-*-"
                          (nth 9 splitted) "-"
                          (nth 10 splitted) "-"
                          (nth 11 splitted) "-*-"
                          (nth 13 splitted))))
   (if (> (length splitted) 14)
       (dotimes (n (- (length splitted) 14))
         (setq new-font (concat new-font "-" (nth (+ n 14) splitted)))))
   (set-default-font new-font t)
   (set-frame-font new-font t)))

(defvar *current-font-index* 0)

(defconst *font-ring* '(
                       "-urw-nimbus mono l-regular-r-normal--15-*-88-88-p-*-iso8859-1"
                       "-urw-nimbus mono l-regular-r-normal--17-*-88-88-p-*-iso8859-1"
                       "-Adobe-Courier-Medium-R-Normal--14-*-100-100-M-*-ISO8859-1"
                       "-Adobe-Courier-Medium-R-Normal--16-*-100-100-M-*-ISO8859-1"
                       "-Adobe-Courier-Medium-R-Normal--18-*-100-100-M-*-ISO8859-1"
                       "-Adobe-Courier-Medium-R-Normal--20-*-100-100-M-*-ISO8859-1"
                       "-Adobe-Courier-Medium-R-Normal--22-*-100-100-M-*-ISO8859-1"
                       "-Adobe-Courier-Medium-R-Normal--24-*-100-100-M-*-ISO8859-1"
                       "-Adobe-Courier-Medium-R-Normal--26-*-100-100-M-*-ISO8859-1"
                       "-Adobe-Courier-Medium-R-Normal--28-*-100-100-M-*-ISO8859-1"
                       "-Adobe-Courier-Medium-R-Normal--30-*-100-100-M-*-ISO8859-1"
                       "-Adobe-Courier-Medium-R-Normal--32-*-100-100-M-*-ISO8859-1"
                       "-Adobe-Courier-Medium-R-Normal--34-*-100-100-M-*-ISO8859-1"
                       "-Adobe-Courier-Bold-R-Normal--14-*-100-100-M-*-ISO8859-1"
                       "-Adobe-Courier-Bold-R-Normal--16-*-100-100-M-*-ISO8859-1"
                       "-Adobe-Courier-Bold-R-Normal--18-*-100-100-M-*-ISO8859-1"
                       "-Adobe-Courier-Bold-R-Normal--20-*-100-100-M-*-ISO8859-1"
                       "-Adobe-Courier-Bold-R-Normal--22-*-100-100-M-*-ISO8859-1"
                       "-Adobe-Courier-Bold-R-Normal--24-*-100-100-M-*-ISO8859-1"
                       "-Adobe-Courier-Bold-R-Normal--26-*-100-100-M-*-ISO8859-1"
                       "-Adobe-Courier-Bold-R-Normal--28-*-100-100-M-*-ISO8859-1"
                       "-Adobe-Courier-Bold-R-Normal--30-*-100-100-M-*-ISO8859-1"
                       "-Adobe-Courier-Bold-R-Normal--32-*-100-100-M-*-ISO8859-1"
                       "-Adobe-Courier-Bold-R-Normal--34-*-100-100-M-*-ISO8859-1"
                       "-Misc-Fixed-Medium-R-SemiCondensed--10-*-75-75-C-*-ISO8859-1"
                       "-Misc-Fixed-Medium-R-SemiCondensed--12-*-75-75-C-*-ISO8859-1"
                       "-Misc-Fixed-Medium-R-SemiCondensed--13-*-75-75-C-*-ISO8859-1"
                       "-Misc-Fixed-Medium-R-Normal--13-*-75-75-C-*-ISO8859-1"
                       "-Misc-Fixed-Medium-R-Normal--14-*-75-75-C-*-ISO8859-1"
                       "-Misc-Fixed-Medium-R-Normal--15-*-75-75-C-*-ISO8859-1"
                       "-Misc-Fixed-Medium-R-Normal--16-*-75-75-C-*-ISO8859-1"
                       "-Misc-Fixed-Medium-R-Normal--17-*-75-75-C-*-ISO8859-1"
                       "-Misc-Fixed-Medium-R-Normal--18-*-75-75-C-*-ISO8859-1"
                       "-Misc-Fixed-Medium-R-Normal--19-*-75-75-C-*-ISO8859-1"
                       "-Misc-Fixed-Medium-R-Normal--20-*-75-75-C-*-ISO8859-1"
                       "-Schumacher-Clean-Medium-R-Normal--12-*-75-75-C-*-ISO8859-1"
                       "-Schumacher-Clean-Medium-R-Normal--13-*-75-75-C-*-ISO8859-1"
                       "-Schumacher-Clean-Medium-R-Normal--14-*-75-75-C-*-ISO8859-1"
                       "-Schumacher-Clean-Medium-R-Normal--15-*-75-75-C-*-ISO8859-1"
                       "-Schumacher-Clean-Medium-R-Normal--16-*-75-75-C-*-ISO8859-1"
                       "-Schumacher-Clean-Medium-R-Normal--17-*-75-75-C-*-ISO8859-1"
                       "-Schumacher-Clean-Medium-R-Normal--18-*-75-75-C-*-ISO8859-1"
                       "-Sony-Fixed-Medium-R-Normal--14-*-100-100-C-*-ISO8859-1"
                       "-Sony-Fixed-Medium-R-Normal--16-*-100-100-C-*-ISO8859-1"
                       "-Sony-Fixed-Medium-R-Normal--18-*-100-100-C-*-ISO8859-1"
                       "-Sony-Fixed-Medium-R-Normal--20-*-100-100-C-*-ISO8859-1"
                       "-B&H-LucidaTypewriter-Medium-R-Normal-Sans-14-*-100-100-M-*-ISO8859-1"
                       "-B&H-LucidaTypewriter-Medium-R-Normal-Sans-16-*-100-100-M-*-ISO8859-1"
                       "-B&H-LucidaTypewriter-Medium-R-Normal-Sans-18-*-100-100-M-*-ISO8859-1"
                       "-B&H-LucidaTypewriter-Bold-R-Normal-Sans-20-*-100-100-M-*-ISO8859-1"
                       "-B&H-LucidaTypewriter-Bold-R-Normal-Sans-24-*-100-100-M-*-ISO8859-1"
                       "-B&H-LucidaTypewriter-Bold-R-Normal-Sans-30-*-100-100-M-*-ISO8859-1"
                       "-B&H-LucidaTypewriter-Bold-R-Normal-Sans-34-*-100-100-M-*-ISO8859-1"
                           ))

(defun font-next ()
 (interactive)
   (let ((len (length *font-ring*))
         (next-index (+ *current-font-index* 1)))
     (if (= next-index len)
         (setq next-index 0))
     (setq *current-font-index* next-index)
     (message (concat "setting " (nth *current-font-index* *font-ring*)))
     (set-default-font (nth *current-font-index* *font-ring*) t)
     (set-frame-font (nth *current-font-index* *font-ring*) t)))

(defun font-prev ()
 (interactive)
   (let ((len (length *font-ring*))
         (next-index (- *current-font-index* 1)))
     (if (= next-index 0)
         (setq next-index (- len 1)))
     (setq *current-font-index* next-index)
     (set-default-font (nth *current-font-index* *font-ring*) t)
     (set-frame-font (nth *current-font-index* *font-ring*) t)))

(defun font-current ()
 (interactive)
 (cdr (assoc 'font (frame-parameters))))

(defun font-set (ix)
 (setq *current-font-index* ix)
 (set-default-font (nth *current-font-index* *font-ring*) t)
 (set-frame-font (nth *current-font-index* *font-ring*) t))

(provide 'fontize)

I also bound these methods to keys, like this:

(global-set-key [?\C-+] 'inc-font-size)
(global-set-key [?\C--] 'dec-font-size)
(global-set-key [?\M-+] 'font-next)
(global-set-key [?\M--] 'font-prev)

Hope this helps someone in the same situation.

Nooks and Crannies of Ruby

There are many small parts of Ruby, tips, tricks and strange things. I thought that I would write about some of the more interesting of these, since some of them are common idioms in the Ruby community. The basis for the information is as always from the Pick-axe, but how these things are used in real life comes from various places.

The splat operator

The asterisk is sometimes called the splat operator when not used for multiplication. It is used in two different places for opposite cases. When on the right hand side of an expression, it is used to convert an array into more than one right hand value. This makes splicing of lists very easy and nice to do.

a,b,c = *[1,3,2]

Second, it's used at the left hand side to collect more than one right hand value into an arra

*a = 1,3,2

This makes no difference if you're calling a method or assigning variables. What matters is as usual with programming languages; that there is a left hand side and a right hand side (lhs and rhs from now on):

def foo(a,*b)
p b
end

foo 1,2,3,*[4,5,6]

This is all old news, and not very exciting. It's useful and the basis for some niceties, but nothing overwhelming. The thing that is really nice about the rhs version of the splat operator is what it does if the value it's applied to isn't an array. Basically, the interpreter first checks if there is a to_ary-method available. If not, it goes for the to_a method. Now, Kernel has a default to_a-method so all objects will respond to to_a. This method is deprecated to call directly, though, but if called through splat or Kernel#Array it doesn't generate a warning. So:

a = *1

will result in the same thing as

a = 1

except for jumping through some unnecessary hoops underneath the covers. But say that you have an object that implements Enumerable and you want to do something with. Maybe transform a Hash into an array of 2-element arrays, you can do it like this:

*a = *{:a=>1,:b=>2}

Now, this still isn't that useful. Oh, it's slightly useful but there is a method in Hash that does this too. But say that we have a file object:

*a = *open('/etc/passwd')

Since File includes Enumerable, it also has a to_a method which creates the array by using each to iterate and collect all elements. In this case all the lines in the file.

def foo(*args)
bar(*args)
end

Camping uses the splat operator at many places, mostly with the common idiom to take any arguments offered and passing them all on as separate arguments again:

Symbols and to_proc

I hesitate to use the word neat, but I can't really find anything that better describes the sweet, sweet combination of symbols and to_proc. I'm going to show you a small example of how it's used before I explain this very common practice:

[1e3,/(foo)/,"abc",:hoho].collect &:to_s

Now, this code will not run without a small addition to your code base. But first of all, let's just walk through the code. First we define a literal array that contains four elements of different type. One Float, one Regexp, a String and a Symbol. Then we call collect to make a new array out of this. But where we usually provide collect with a block, we instead see the ampersand that symbolizes that we want to turn a Proc-object into a block argument for a method. But what comes next is not a variable, but a symbol. So, what happens? Well, the ampersand checks if the value provided to it is a Proc, and if not it calls to_proc on the value in question, if such a method is defined. And how should this method look? Like this:

class Symbol
 def to_proc
lambda { |o| o.send(self) }
 end
end

Now, this method is nothing much. But it employs some fun trickery. It first creates a Proc by calling Kernel#lambda with a literal block. This block takes one argument, and the block calls the method send on the argument with itself as argument. As self in this case would be a symbol, and specifically the symbol :to_s in the above example, the end result is that the Proc returned will call to_proc on each object yielded to the block. So, with this explanation it's easier to understand what the first example does. In effect it is exactly the same as

[1e3,/(foo)/,"abc",:hoho].collect {|v| v.to_s}

but without that nasty duplication of the v-argument. It's not a big saving, but many small savings...

I recommend installing facets, which include numerous small, nice solutions like this. They can also be required separately, so if you have facets installed, just require 'facet/symbol/to_proc' to get this specific functionality included.

Using operators as method names

Ruby allows much more operators to be redefined than most languages. This makes some interesting tricks possible, but most importantly it can make your code radically more readable. An excellent example of this can be found in the net/ldap-library (available as ruby-net-ldap from RubyGems). Now, LDAP uses something called filters for searching, and the syntax for filters are basically prefix notation with ampersand, pipe and exclamation mark for and, or and not, respectively. Now, with the net/ldap-library you can define a combined filter like this:

include Net
f = (LDAP::Filter.eq(:cn,'*Ola*') & LDAP::Filter.eq(:mail,'*ologix*')) |
    LDAP::Filter.eq(:uid,'olagus')

This defines a filter that basically says: find all entries where cn is '*Ola*' and mail is '*ologix*' or uid is 'olagus'. This is very readable thanks to the infix operators, that for everyone who knows LDAP will be easy to understand.

The next example comes from Hpricot, where _why puts the slash to good use:

doc = Hpricot(open("http://redhanded.hobix.com/index.html"))
(doc/"span.entryPermalink").set("class", "newLinks")

Note how neatly doc/"span..." fits in, and it looks like XQuery, or any other path query syntax. But it's just regular Ruby code and the slash is just method call. I'm really sad that /. isn't allowed as a method in this way... =)

Now, ackording to the Pickaxe, all of these infix operators will be translated from arg1 op arg2 into arg1.op(arg2). But Ruby still needs to be able to parse everything. This means that most operators need to have one required argument. Trying this with a home defined *-operator will not work:

x = a *

But, an experimental syntax for importing packages in JRuby actually used this effect:

import java.util.*

This is just a simple exploatation of the fact that * is a regular method name and used like this will be parsed by Ruby like that too, which means it doesn't need an argument. So, which operators are available for your leisure? Ackording to the Pickaxe, these are [], []=, **, !, ~, + (unary), - (unary), *, /, %, +, -, >>, <<, &, ^, |, <=, <, >, >=, <=>, ==, ===, !=, =~, !~.
Note that the method names when implementing the unary + and - is +@ and -@:

class String
 def -@
swapcase
 end
end

The most important thing to remember when reusing operators like this is to not overdo it. Use it where it makes sense and is natural but not elsewhere. Remember that Ruby code should follow the principle of least surprise. The above example of using unary minus to return a swapcased version of the string is probably not obvious enough to warrant its use, for example.

Using lifecycle methods to simplify daily life

Inversion of control is all the rage in the Java world right now, but using callbacks of call kinds have always been a great way to make readable and compact. The Observer pattern is used in many places, and I suspect it's implemented without any knowledge of the pattern in most places.

Ruby contains a few callback methods and lifecycle hooks that make life that much easier for the Ruby library writer. Probably the most useful of these are Module#included. Basically, this is a method you define like this:

module Enumerable
def self.included(mod)
puts "and now Enumerable has been used by #{mod.inspect}..."
end
end

It will be called every time a module is included somewhere else.

There are other callbacks that can be useful. Module#method_added, Module#method_removed, Module#method_undefined and counterparts for Kernel with singleton prefixed. Class#inherited is interesting. Through this you can actually keep track of all direct subclasses of your class and with some metaprogramming trickery (basically writing a new inherited for each subclass that does the same thing) you can get hold of the complete tree of subclasses. If you want that for some reason. I would for example use this approach for Test::Unit, rather than iterating over ObjectSpace. But I guess that's a matter of taste.

Class variables versus Class instance variables

This is one thing that always trips people up. Including me. Class variables are special variables that are associated with a class. They are referenced with two at-signs and a name, like @@name. So far, it's simple. But classes are also instances of Class, which means that these instances can have regular one-at-sign instance variables. These are not the same thing. Not at all. Something like this:

class Foo
@@borg = []
@me = nil

def initialize
@me = self
Foo::add_borg
end

def self.add_borg
@@borg << @me
end
end

will result in a @@borg-list filled with nils. This is because the first @me refers to an instance variable in the Foo instance of Class; not the @me instance variable associated with an instance of the Foo-class.

Condensed lesson: Class have instance variables of themselves, these are rarely useful; they usually contribute to hard-to-find-errors. And don't confuse them with class variables which is a totally different kind of beast.

Shortcuts: __FILE__ and ARGF

Ruby contains a myriad of shortcuts, many influenced from Perl and other invented to make it easier to write condensed programs. The regexp result globals are always good to have, but there are other that can be very useful too. Two that I like most are __FILE__ and ARGF. __FILE__ is also part of a very, very common idiom that the Pickaxe details. Combined with the global $0 it makes it easy to differ execution when a file is required, and when it's executed. Basically, $0 contains the name of the file that has been executed. In C this would be argv[0]. __FILE__ is the full filename of the file the code can be found in. If these are the same, the current file is the one asked to execute. This is useful in many places. I use it often in gemspecs:

if $0 == __FILE__
Gem::manage_gems
Gem::Builder.new(spec).build
end

If I run the file above with gem build, this part will not execute, but if I execute the file directly, it will run.

Matz sometimes likes to show how to implement the UNIX utility cat in Ruby:

puts *ARGF

This combines tip number uno in this blog entry with the constant ARGF. ARGF is a nice special object that when you reference it will open all the files named in ARGV. If you have any options in your ARGV you'd better remove them before referencing ARGF, though. Basically what you get when referencing ARGF is a file handle to the files named on the command line. And since a File has Enumerable and thus to_a, splat will read all the lines in all the files and combine them into an array and then splay the array into the call to puts which will print each line. Here you are, cat!

There are other globals and constants available, but most aren't as useful as the previously named. For example you can use __END__ on an empty line, and the code interpolation will stop there and the rest of the file will be available as the constant DATA. I haven't seen anyone use this. It's a remnant from when Ruby was a tool to replace Perl, and the other scripting tools in UNIX.

Everything is runtime

Basically, the whole difference in Ruby compared to compiled languages is that everything happens at runtime. Actually, this difference can be seen when looking at Lisp too. In Common Lisp there are three different times when code can be evaluated: at compile-time, load-time and eval-time. In Java class-structure is fixed. You can't change class structure based on compile parameters (oh boy, sometimes I miss C-style macros). But in Ruby, everything is runtime. Everything happens at that time (except for constants... this is a different story). This means that class definitions can be customized based on environment. A typical example is this:

class Foo
include Tracing if $DEBUG
end

This class will include some methods when the -d flag is provided, and others when it's not. Basically there isn't much syntax in Ruby that couldn't be implemented in the language itself. A class declaration can be be duplicated with

Class.new(:name) do
#class declarations go here
end

And almost all parts of a method-definition with def can be provided with define_method. The glaring mismatch (blocks) will be corrected with 1.9. Except for that, it's just sugar. If statements could be implemented with duck typing/polymorphism:

class TrueClass
def if(t,f)
t.call
end
end

class FalseClass
def if(t,f)
f.call if f
end
end

x = true

x.if lambda{ puts "true" }, lambda{ puts "false"}

And that's the real Lisp inheritage of Ruby. There really isn't any essential syntax. Everything can be implemented with the basics of receiver, message, arguments, and blocks. Just remember that. It's the basis for all useful metaprogramming. There is no compile-time. Everything can change. "There is no spoon".

Announcing ActiveRecord-Mimer 0.0.1

The initial version of ActiveRecord-Mimer have been released.

The project aims to provide complete ActiveRecord support for the Mimer SQL database engine. This initial release provides the basis for that. Most operations work, including migrations. The only exceptions are rename_column and rename_table which isn't supported by the underlying database engine.

The project resides at RubyForge: http://rubyforge.org/projects/ar-mimer

and can be installed with RubyGems by
gem install activerecord-mimer

The code is released under an MIT license.

Another OpenSSL woe.

My interesting OpenSSL implementation exercise continues. I am now very close. Very, very close. I'm actually so close that SSLSocket and SSLServer actually works, provided that you use Anonymous Diffie-Hellman (which is suicidal, but that's another story). All of this have been submitted to my openssl-branch. What's missing is the X509-store and PKCS#7. And the X509-store doesn't really look good. Not good at all. It's needed for full SSL support. But the bad thing is this: there isn't any Java libraries that duplicate the functionality. Nada. At least not that I can find. The functionality needed is to read and write X509_store-formatted files and directories, to be able to add certificates and CRL's and to verify against these a certificate, based on various interesting OpenSSL rules.

I wouldn't say that I mislike OpenSSL. I wouldn't say that I hate it either. It's very impressive in many ways. But boy. It seems I have to port a substantial part of it to Java, and I'm not looking forward to it. I need to to do both a port, and add support for KeyStore and CertStore so the Java SSLEngine also can use the information. Will this be an interesting exercise? Oh yes.

So, without further ado, this is the plea of this blog post: If you know of any easier way to do this, please tell me. Now! (where "this" is the X509_STORE-family of functions.)

JRuby versus Camping: Round 2

As undoubtedly some of you have discovered while trying, JRuby doesn't run Camping anymore. The culprit is a small feature in Ruby that we don't support yet. For some reason this feature is the preferred idiom for option parsing and Camping 1.5 introduced it, which means that Camping 1.5 will fail miserably in JRuby right now. There are more or less two ways around it, though. The first one is easy; just use a Camping version that is

opts.on("-p", "--port NUM", "Port for web server (defaults to #{conf.port})")
     { |conf.port| }

To fix this particular instance, just change it into this:

opts.on("-p", "--port NUM", "Port for web server (defaults to #{conf.port})")
     { |v| conf.port=v }

Not much of a difference, really. On all places where assignment to a non-obvious node is done in the manner above, just replace it with a regular assignment, and Camping will run.

Now, we are planning on fixing this syntax, but there is much development going down right now, and this change requires some changes in the parser, which is always interesting. But it will be there.

Introducing TIJuAVA - Java with Type Inference

Every time I've written Java code lately, I've been painfully aware of how much unnecessary code I write every time. And most of this is Java's fault. This blog post is a very small thought experiment. TIJuAVA does not exist as software. Yet. If I someday have the time I would love to implement it, but there are more pressing needs right now.

So, what are the rules? Basically, all valid Java programs are valid TIJuAVA programs. Some valid TIJuAVA programs are not valid Java programs. Simply put, the main difference is that you don't need to declare a type for any local variables or member variables. Type declarations are only necessary in method declarations. You can declare local variables and member variables if you want to, and in certain very unlikely circumstances you will need too.

Let's take a very simple example. This code is taken from the JRuby source code, but I have added one or two things to make it easier to showcase:

package org.jruby.util.collections;

import java.util.ArrayList;
import java.util.Collection;
import java.util.Iterator;

public class IdentitySet {
   private items = new ArrayList();

   public void add(Object item) {
       items.add(item);
   }

   public void remove(Object item) {
       iter = items.iterator();
       while (iter.hasNext()) {
           storedItem = iter.next();
           if (item == storedItem) {
               iter.remove();
           }
       }
   }

   public boolean contains(Object item) {
       iter = items.iterator();
       while (iter.hasNext()) {
           storedItem = iter.next();
           if (item == storedItem) {
               return true;
           }
       }
       return false;
   }

   private Collection getItems() {
       return items;
   }

   private void something(java.util.AbstractSet inp) {
       val1 = inp;
       for(iter = val1.iterator();iter.hasNext();) {
           System.err.println(iter.next());
       }
   }
}

This code doesn't really show all that can be done with this approach, and if I were to show a real example, this blog would be unbearably filled with code. So, this is just a tidbit.

The TIJuAVA system would need to be implemented as a Java two-pass compiler. Basically, the first pass finds all variable names that need to have a type inferred, and then walks through the information it's got, basic on method signatures and methods called on the variable. In almost all cases it will be possible to come to one conclusion on which type to use. The compiler would then generate regular Java byte code, basically the same bytecode that would have been generated had you written the types by hand.

Of course, most people use IDE's to write code nowadays. Wizards and code generators and what not. So why something like this? Well, even though your IDE writes your code for you, it is still there, and you still have to understand it at some level. If not when writing, you would still need to read it. And boy does type declarations clutter things. Especially generics. And here is one interesting tidbit. Generic types would also be possible to infer in most cases.

Another thing that could be easily added is some kind of in-place literal syntax for lists and maps. This would be more like a macro feature, but the list syntax would mostly just be a call to Array.asList, which isn't to bad.

An objection that I anticipate is from people who think that the code will be less readable by removing the type pointers. This should be more of a problem when you have large methods, but everyone these days use refactorings so they won't have methods with a LOC over 20. And if that's the case, the local variables should be easily understood by the operations that are used on them.

So. Someday, when I have time, this may be reality. If anyone is interested, that is.

Why Open Source?

Sometimes, when discussing my interest and work with JRuby and other Open Source projects, people get all upset. It's really hard to justify for a non-programmer the benefits of Open Source, but sometimes it can be as hard to explain to corporate programmers. "What? You give away code for free, for anyone to use?". Not only that, I also do it mostly in my spare time.

I'm not planning for this blog entry to be some kind of manifesto. I'll just detail the (very logical) reasons why I do what I do.

At work
I work at Karolinska Institutet (KI) in Sweden. KI is a University, and like all Universities in Sweden, we are part of the government. That means that most of our funding is tax-based. We use lot of Open Source in our work, providing services to the campus. We run mostly on Linux servers, we use Open Source frameworks when building internal systems. Needless to say, we have saved uncountable millions with this strategy. And the Swedish government did a review a few years ago that recommended that all tax-funded organizations should use Open Source software is possible. From this perspective, it is very rational to try to give something back. We mostly do this by trying to release everything we develop internally that can be repackaged for distribution easily. We also allocate some time for all our developers to work on Open Source projects of their choice. Needless to say, I mostly use this time to work on JRuby.

In my spare time
Rationalizing Open Source from a corporate perspective is quite easy. It's harder to answer why I also do it in my spare time. I spend on average about two-three hours a day on Open Source, with about 80%-85% of that on JRuby. So why?

First of all, I recognize that Open Source is important in itself. I firmly believe that the current market model for software will soon disappear. It is obvious that the classical, shrink-wrapped model doesn't really work. I want to make that happen faster. Contributing to Open Source projects make that happen.

Secondly, JRuby is important. Ruby is great, but it has it's short comings. I believe that JRuby will be the necessary bridge between the two camps that Java and Ruby seems to move towards. JRuby will bridge the gap and give crucial capabilities to both platforms, and that can't happen soon enough.

Third, I believe it's important to hone my skills. Doing software development means always have to become better and better in your areas, and new areas. This is hard to do in the confines of regular corporate culture. Open Source is practice for me. It makes me better and I learn crucial new tools and techniques. Reading other peoples code is an excellent way to learn, and that's easy with Open Source.

Fourth. And most importantly; I'm a coder. This is my passion. Code is art and coding is what I really like doing. I have code in my head always, 24 hours. I dream about code. I design software more or less irrespectively of what my front lobes are engaging in. I'm getting more and more that my subconscious uses programming languages to get around Sapir-Whorf, helping me think from other angles.

Charles wrote about this a few months ago, here. He describes mostly what I feel about coding.

Of course I have other interests. I'm very musical, creating and listening to music constantly. I'm very fond of Go. I read incredibly much (too much if you ask some). But coding is what I like doing best, and when you get down to it, that's why I do Open Source.

Speaking engagements

This Monday (30:th October) I'll go to Malmö to present on JRuby at JavaForum, as I posted a few weeks ago. After everything is over I'll have an hour or two to spend before my train leaves, so if anyone is up for beer I'm game.

I will also present at JavaForum in Stockholm 21:th November. The presentation will be mostly the same, except for possible interesting developments during that time. At the same night Rob Harrop of Interface21 will present on Spring, AOP and ActiveJ. That promises to be very interesting, so I recommend everyone in the region to get there. =)

More information at http://javaforum.se.

OpenSSL status report

I just checked in a few updates to my openssl branch for JRuby. Boy is it tricky getting everything right. It seems like every DER format Java crypto emits differs from the OpenSSL DER output. And it's really incompatible. As an example I have been forced to reimplement the DER dumping for X509 certificates myself, and that's not the only place.

But the work is actually going forward; as fast as I can make it when I'm only doing this in my spare time and my regular work takes lots of time right now. I can't say for sure when it will be finished or usable, but I know for a fact that most of the MRI tests run now. What's missing is PKCS#7, X509 CRL's and X509 cert-stores, plus the regular SSL socket support. Not much, compared to what actually works.

But that leads to me to two issues. We have recently agreed that OpenSSL support will require BouncyCastle and Java 5. There is really no other way to get this working. 1.4.2 is fine for basic Digest support and some of the RSA/DSA support, but Java is sorely lacking in the ASN.1 and X509 department. Nothing whatsoever. Which is why we need BouncyCastle, which is fairly complete. I have only been forced to reimplement one or two central classes. Quite good. But SSL support is another story. As you may know, 1.4.2 has SSLSocket and SSLServerSocket. The problem is this: they aren't any good. As a first, they are blocking, and there isn't any support in 1.4.2 for NIO SSL sockets. Whoopsie. Which explains the requirement on Java 5. Tiger adds the SSLEngine class which can be used to implement NIO SSL, with the caveat that it heightens complexity. I have only taken a cursory look at this yet. Right now I want the other stuff working first, since there are so many dependencies on them.

But it's really going forward. Now, if I only had this as my day job, this would be finished in a few days... Alas, that's not the way it is. Expect further updates in a week or two.

Meta-meta-information?

It seems that the current trend in Internet services is all the rage about meta information; ways to find the information you want. These services come in various guises; some collecting pictures or movies, collecting and classifying links or finding out what other people are looking at. The quintessential application right now must be the search engine and Google is in the forefront with this, as well as with many other current meta information processing applications. But what is the next level? The next evolutionary step in information processing?

Is meta-meta-information enough?
The obvious answer would be that we need meta information about the meta services; a way to search, handle and collect this information. Which meta services are the people I admire using? But is that really enough? It places a big burden on the individual trying to keep up with the information flow. Tagging and such helps the situation a bit, but it's still way to much for one person. I already feel slightly overwhelmed right now, trying to keep up with everything that happens in my current domains of interest at the moment. It seems to me that meta services for finding meta information wouldn't be enough, since it doesn't solve the underlying problem of actually helping to manage the information.

Are there any alternatives?
The development in services from this point on seems to revolve around two solutions; the semantic web, RDF and all those shenanigans, and intelligent agents.

The semantic web is a collection of technologies - some that exist and some that don't, yet - that enables more collaboration and interaction through Internet. Typical examples include ontologies for classifying information, two-ended links, user annotations and other ways to include self-description in the data itself.

Intelligent agents are probably the easiest AI application to rationalize, since it doesn't require strong AI, and it's obvious how helpful they would be. Intelligent agents are typically cast in the same box as semantic web, but I feel that intelligent agents (IA from now on) could probably exist and be effective without semantic web metadata. What makes IA more or less necessary very soon, is that automating handling of information is the solution to the meta-meta-problem. Most of my current meta information handling is done with the help of programs that I have customized in various ways to make information easier for me to handle, but manually customizing handling of meta meta information suffers from the same problem as manually tagging spam; it is just too much information. IA could help, by first getting to know the habits and interests of it's user, and then extrapolating what information would be useful and what could be thrown away.

Finalities
Is semantic web or IA enough in it's own right? Could the one solve the problem without the other? I think not, or at least not solve it permanently. I believe semantic web tech will enter Internet before IA, but I don't believe the information flow will be solved without them. At most, semantic web will buy us one or two years to catch our breath. IA is interesting technology, and we really need it now, especially if are a knowledge worker.

JRuby 0.9.1 released

We in the JRuby team are proud to announce the release of JRuby 0.9.1.

Download: http://dist.codehaus.org/jruby/.

This release contains numerous new features, fixes and improvements:

• Overall performance is 50-60% faster than JRuby 0.9.0
• Improved Rails support
• New syntax for including Java classes into Ruby
• New interpreter design
• Refactoring of method dispatch, code evaluation, and block dispatch code
• Parser performance enhancement
• Rewriting of Enumerable, StringScanner and StringIO in Java
• New experimental syntax for implementing interfaces
• 86 Jira issues resolved since 0.9.0

This past months have been great for JRuby, and I know that it will get even better from now on. My personal goal for 0.9.2 is to have complete Java YAML support in, and a working OpenSSL library. Obviously, all bugs should be fixed too... =)