torsdag, augusti 17, 2006

The JRuby Tutorial #2: Going Camping

So, for the 2nd, (actually third installment, but the JIRB one doesn't really count), I was thinking that we should take a look at a few different subjects. The red thread will be a simple web based Blog application built with Camping. Camping is a microframework for web development, by whytheluckystuff, is insanely small and incredibly powerful. It uses another library called Markaby, which generates markup based on pure Ruby code. I will show the application in smaller parts, with explanations, and at the end include a link to the complete source file.

First of all we have to have a working JRuby built from the latest version of trunk. After JRuby is working we need to install a few gems. Follow these commands and you'll be good:
jruby %JRUBY_HOME%\bin\gem install camping --no-ri --no-rdoc --include-dependencies
jruby %JRUBY_HOME%\bin\gem install ActiveRecord-JDBC --no-ri --no-rdoc
This installs Camping, ActiveRecord, ActiveSupport, Markaby, Builder, Metaid and ActiveRecord-JDBC. We don't generate RDoc or RI for these gems, since that's one part of JRuby that still is pretty slow.

The Blog application
The first thing the blog application needs is a database. I will use MySQL, but other databases may work through the JDBC-adapter, but there is still some work to be done in this area. I will have my MySQL server on localhost, so change the configuration if you do not have this setup. You'll need a database for the blog application. I've been conservative and named the database "blog" and the user "blog" with the password "blog". Easy to remember, but not so good for production.

Update: As azzoti mentioned, you have to set your classpath to include the MySQL JDBC-driver, which can be downloaded from http://www.mysql.org/.

Now, open up a new Ruby file and call it blog.rb. The name of the file is important; it has to have the same name as the Camping application. Now, first of all we include the dependencies:
require 'rubygems'
require 'camping'
require 'camping/session'
require_gem 'ActiveRecord-JDBC'
require 'active_record/connection_adapters/jdbc_adapter'
require 'java'

include_class 'java.lang.System'
These statements include Rubygems, Camping, Camping session support, ActiveRecord-JDBC and Java support. It also includes the Java class named java.lang.System for later use. The next step is to include some configuration for our application.
Camping.goes :Blog
Blog::Models::Base.establish_connection :adapter => 'jdbc', :driver => 'com.mysql.jdbc.Driver',
:url => 'jdbc:mysql://localhost:3306/blog', :username => 'blog', :password => 'blog'

module Blog
include Camping::Session
end
These statements first names our application with the Camping.goes-statement. This includes some fairly heavy magic, including reopening the file and rewriting the source to include more references to the Camping framework. But this line is all that is needed. The next line establishes our connection to the database, and it follows standard JDBC naming of the parameters. Of course, these should be in a YAML file somewhere, but for now we make it easy. The last part makes sure we have Session support in our Blog application.

Now we need to define our model, and this is easily done since Camping uses ActiveRecord:
module Blog::Models
def self.schema(&block)
@@schema = block if block_given?
@@schema
end

class Post < Base; belongs_to :user; end
class Comment < Base; belongs_to :user; end
class User < Base; end
end
The first part of this code defines a helper method that either sets the schema to the block given, or returns an earlier defined schema. The second part defines our model, which includes Post, Comment and User, and their relationships.

The schema is also part of the application, and we'll later see that Camping can automatically create it if it doesn't exist (that's why we didn't need to create any tables ourselves, just the database).
 Blog::Models.schema do
create_table :blog_posts, :force => true do |t|
t.column :id, :integer, :null => false
t.column :user_id, :integer, :null => false
t.column :title, :string, :limit => 255
t.column :body, :text
end
create_table :blog_users, :force => true do |t|
t.column :id, :integer, :null => false
t.column :username, :string
t.column :password, :string
end
create_table :blog_comments, :force => true do |t|
t.column :id, :integer, :null => false
t.column :post_id, :integer, :null => false
t.column :username, :string
t.column :body, :text
end
end
This defines the three tables needed by our blog system. Note that the names of the tables include the name of the application as a prefix. This is because Camping expects more than one application to be deployed in the same container, using the same database.

When we have defined the schema, it's time to define our controller actions. In Camping, each action is a class, and each action class define a method for get, one for post, etc. These classes will be defined inside the module Blog::Controllers. The first action we create will be the Index action. It looks like this:
     class Index < R '/'
def get
@posts = Post.find :all
render :index
end
end
This defines a class that inherits from an anonymous class defined by the R method. What it really does, is bind the Index action to the /-path. It uses ActiveRecord to get all posts and then renders the view with the name index.

The Add-action adds a new Post, but only if there is a user in the @state-variable, which acts as a session. If something is posted to it, it creates a new Post from the information and redirects to the View-action:
     class Add
def get
unless @state.user_id.blank?
@user = User.find @state.user_id
@post = Post.new
end
render :add
end
def post
post = Post.create :title => input.post_title, :body => input.post_body,
:user_id => @state.user_id
redirect View, post
end
end
As you can see, there's not much to it. Instance variables in the controller will be available to the view later on. Note that this action doesn't inherit from any class at all. This means it will only be available by an URL with it's name in it.

We need a few more controllers. View and Edit are for handling Posts. Comment adds new comments to an existing Post. Login and Logout are pretty self explanatory. And Style returns a stylesheet for all pages. Note that Style doesn't render anything, it just sets @body to a string with the contents to return.
     class View < R '/view/(\d+)'
def get post_id
@post = Post.find post_id
@comments = Models::Comment.find :all, :conditions => ['post_id = ?', post_id]
render :view
end
end

class Edit < R '/edit/(\d+)', '/edit'
def get post_id
unless @state.user_id.blank?
@user = User.find @state.user_id
end
@post = Post.find post_id
render :edit
end

def post
@post = Post.find input.post_id
@post.update_attributes :title => input.post_title, :body => input.post_body
redirect View, @post
end
end

class Comment
def post
Models::Comment.create(:username => input.post_username,
:body => input.post_body, :post_id => input.post_id)
redirect View, input.post_id
end
end

class Login
def post
@user = User.find :first, :conditions => ['username = ? AND password = ?', input.username, input.password]
if @user
@login = 'login success !'
@state.user_id = @user.id
else
@login = 'wrong user name or password'
end
render :login
end
end

class Logout
def get
@state.user_id = nil
render :logout
end
end

class Style < R '/styles.css'
def get
@headers["Content-Type"] = "text/css; charset=utf-8"
@body = %{
a {
text-decoration: none;
}
a:hover {
text-decoration: underline;
}
body {
font-family: Utopia, Georga, serif;
}
h1.header {
background-color: #fef;
margin: 0; padding: 10px;
}
div.content {
padding: 10px;
}
div.post {
background-color: #ffa;
border: 1px solid black;
padding: 20px;
margin: 10px;
}
}
end
end
Also note how easy it is to define routing rules with the help of regular expressions to the R method.

Next up, we have to create our views. Since Camping uses Markaby, we do it in Ruby, in the same file. Views are methods in the module Blog::Views with the same name as referenced inside the controllers call to render. There is a special view called layout which get called for all views, if you don't specify otherwise in the call to render. It looks like this:
     def layout
html do
head do
title 'Blog'
link :rel => 'stylesheet', :type => 'text/css',
:href => '/styles.css', :media => 'screen'
end
body do
h1.header { a 'Blog', :href => R(Index) }
div.content do
self << yield
end
end
end
end
As you can see, standard HTML tags are defined by calling a method by that name. The contents of the tag is created inside the block sent to that method, and if it makes sense to give it content as an argument, this works too. Title, for example. If you give a block to it, it will evaluate this and add the result as the title, but in this case it's easier to just provide a string. Note how a link is created, by the method R (another method R this time, since this is in the Views module). Finally, the contents of the layout gets added by appending the result of a yield to self.

The index view is the first we'll see when visiting the application, and it looks like this:
     def index
if @posts.empty?
p 'No posts found.'
else
for post in @posts
_post(post)
end
end
p { a 'Add', :href => R(Add) }
p "Current time in millis is #{System.currentTimeMillis}."
end
I've also added a call that writes out the current time in milliseconds, from Java's System class, to show that we're actually in Java land now, and potentially could base much of our application on data from Java. We check if there are any posts, and if so iterate over them and write them out with a partial called _post. We also create a link to add more posts. The rest of the views look like this:
     def login
p { b @login }
p { a 'Continue', :href => R(Add) }
end

def logout
p "You have been logged out."
p { a 'Continue', :href => R(Index) }
end

def add
if @user
_form(post, :action => R(Add))
else
_login
end
end

def edit
if @user
_form(post, :action => R(Edit))
else
_login
end
end

def view
_post(post)

p "Comment for this post:"
for c in @comments
h1 c.username
p c.body
end

form :action => R(Comment), :method => 'post' do
label 'Name', :for => 'post_username'; br
input :name => 'post_username', :type => 'text'; br
label 'Comment', :for => 'post_body'; br
textarea :name => 'post_body' do; end; br
input :type => 'hidden', :name => 'post_id', :value => post.id
input :type => 'submit'
end
end
And the three partials:
     def _login
form :action => R(Login), :method => 'post' do
label 'Username', :for => 'username'; br
input :name => 'username', :type => 'text'; br

label 'Password', :for => 'password'; br
input :name => 'password', :type => 'text'; br

input :type => 'submit', :name => 'login', :value => 'Login'
end
end

def _post(post)
div.post do
h1 post.title
p post.body
p do
a "Edit", :href => R(Edit, post)
a "View", :href => R(View, post)
end
end
end

def _form(post, opts)
p do
text "You are logged in as #{@user.username} | "
a 'Logout', :href => R(Logout)
end
form({:method => 'post'}.merge(opts)) do
label 'Title', :for => 'post_title'; br
input :name => 'post_title', :type => 'text',
:value => post.title; br

label 'Body', :for => 'post_body'; br
textarea post.body, :name => 'post_body'; br

input :type => 'hidden', :name => 'post_id', :value => post.id
input :type => 'submit'
end
end
In my opinion, this code is actually much easier to read than HTML and most of it is fairly straight forward. One interesting part is the add and edit methods, which checks if a user is logged in, otherwise uses the _login-partial instead of showing the real content.

Finally, we will define a create-method for Camping, which is responsible for creating the tables for our model:
 def Blog.create
Camping::Models::Session.create_schema
unless Blog::Models::Post.table_exists?
ActiveRecord::Schema.define(&Blog::Models.schema)
Blog::Models::User.create(:username => 'admin', :password => 'camping')
end
end

This first creates a table for the session information, and then checks if the Post-table exists; if not all the tables in the schema defined before is created.

Now you have seen the complete application. If you have no interest in writing this by hand, the complete code can be found here.

Running the application
To run a Camping application, you need to run the camping executable that has been installed into your %JRUBY_HOME%\bin on your application file. In my case I run it like this:
jruby %JRUBY_HOME%\bin\camping blog.rb
in the directory where my blog.rb exists and I very soon have a nice application at http://localhost:3301/blog which works wonderfully. Startup is a little bit slow, but as soon as WEBrick has started listening the application is very snappy. You can try changing your blog.rb-file too; Camping will automatically update your application without having to restart the server. As I said above, I included a call to System.currentTimeMillis, to show that we are actually using Java in this blog-application. If that isn't apparent from the call to System, remember that we are actually using JDBC to talk to our database, and very soon you will be able to use the ActiveRecord-JDBC adapter to connect to any databases Java can talk too. That's a brigth future.

The JRuby Tutorial #1.5: Using JIRB to check Java behaviour

I was planning on this issue to be on Camping, but I realized that it would be good to show how to use IRB with JRuby to make life as a Java developer easier. Since Java is a compiled language, it's exceedingly hard to just test things out. My hard drive is completely littered with 5-line programs testing one aspect of the Java libraries or whatnot. If I could have an easy way to interface with the libraries without having to go through a compile cycle, this would ease my life very much. And actually, the last months I have been using JIRB in this way. Of course there is BeanShell and Groovy, but I feel they get in the way. And I'm very fond of IRB, of course.

IRB stands for Interactive Ruby, and JIRB is the name of the program for starting IRB with JRuby. Starting it should be as easy as:
%JRUBY_HOME%\bin\jirb
and after two or three seconds you will get the Ruby prompt:
irb(main):001:0>
At this point, the first thing to do is to require the java libraries, so we can have something to work with:
irb(main):001:0> require 'java'
=> true
I would first like to show how I would go about finding the behaviour of the caret in a Java regexp. (I know I could find it in the API, but this is just to show how you could do it.)

First of all, I will include the relevant class and see what I can do:
irb(main):002:0> include_class 'java.util.regex.Pattern'
=> ["java.util.regex.Pattern"]
irb(main):003:0> Pattern.new
NameError: wrong # of arguments for constructor
from D:\Project\jruby-trunk\src\builtin\javasupport.rb:231:in `initialize'
from (irb):1:in `new'
from (irb):1:in `binding'
from D:/project/jruby-trunk/lib/ruby/1.8/irb.rb:150:in `eval_input'
from D:/project/jruby-trunk/lib/ruby/1.8/irb.rb:70:in `signal_status'
from D:/project/jruby-trunk/lib/ruby/1.8/irb.rb:189:in `eval_input'
from D:/project/jruby-trunk/lib/ruby/1.8/irb.rb:70:in `each_top_level_statement'
from D:/project/jruby-trunk/lib/ruby/1.8/irb.rb:190:in `loop'
from D:/project/jruby-trunk/lib/ruby/1.8/irb.rb:190:in `catch'
from D:/project/jruby-trunk/lib/ruby/1.8/irb.rb:190:in `eval_input'
from D:/project/jruby-trunk/lib/ruby/1.8/irb.rb:70:in `start'
from D:\project\jruby-trunk\bin\jirb:13:in `catch'
from D:/project/jruby-trunk/lib/ruby/1.8/irb.rb:71:in `start'
from D:\project\jruby-trunk\bin\jirb:13
irb(main):004:0> Pattern.methods
=> ["matches", "quote", "compile", "setup_constants", "setup", "singleton_class", "setup_attributes", "new_instance_for", "[]", "const_missing" ...]
irb(main):005:0> Pattern.methods.grep /compile/
=> ["compile"]
Ok, first I include the class, then I try it out. Whoopsie, there was no empty constructor for Pattern. I'm don't really remember what methods are in there, though, so I check out which methods Pattern have. There is a good many of them (and I haven't written them all out either), so I try again by grepping for a method name I believe is in there. Once I've found the method I'm looking for, I call it with the pattern I want to compile, and inspect the result in various ways. I vaguely remember a matcher method, and it seems to work as I remember:
irb(main):008:0> pt = Pattern.compile("^foo")
=> ^foo
irb(main):009:0> pt.inspect
=> "^foo"
irb(main):010:0> pt.java_class
=> java.util.regex.Pattern
irb(main):011:0> pt.class
=> Pattern
irb(main):012:0> m = pt.matcher("foo")
=> java.util.regex.Matcher[pattern=^foo region=0,3 lastmatch=]
Ok, now I have something to work with. Good, and the standard output from JRuby gives me some insight into the internals of the matcher too; interesting stuff. There is a find method, and a matches method on the Matcher:
irb(main):013:0> m.find
=> true
irb(main):014:0> m.find
=> false
irb(main):015:0> m.matches
=> true
irb(main):016:0> m.search
NoMethodError: undefined method `search' for java.util.regex.Matcher[pattern=^foo region=0,3 lastmatch=foo]:#
from (irb):1:in `method_missing'
from (irb):1:in `binding'
from D:/project/jruby-trunk/lib/ruby/1.8/irb.rb:150:in `eval_input'
from D:/project/jruby-trunk/lib/ruby/1.8/irb.rb:70:in `signal_status'
from D:/project/jruby-trunk/lib/ruby/1.8/irb.rb:189:in `eval_input'
from D:/project/jruby-trunk/lib/ruby/1.8/irb.rb:70:in `each_top_level_statement'
from D:/project/jruby-trunk/lib/ruby/1.8/irb.rb:190:in `loop'
from D:/project/jruby-trunk/lib/ruby/1.8/irb.rb:190:in `catch'
from D:/project/jruby-trunk/lib/ruby/1.8/irb.rb:190:in `eval_input'
from D:/project/jruby-trunk/lib/ruby/1.8/irb.rb:70:in `start'
from D:\project\jruby-trunk\bin\jirb:13:in `catch'
from D:/project/jruby-trunk/lib/ruby/1.8/irb.rb:71:in `start'
from D:\project\jruby-trunk\bin\jirb:13
I also belived there was a search method, and also a way to get back to the initial state of the Matcher. Hmm. Time to check out the methods of this object too:
irb(main):017:0> m.methods
=> ["hashCode", "start", "matches", "useAnchoringBounds", "region", "reset"...] irb(main):019:0> m.reset
=> java.util.regex.Matcher[pattern=^foo region=0,3 lastmatch=]
irb(main):020:0> m.find
=> true
irb(main):021:0> m.group 0
=> "foo"
irb(main):022:0>
I found the method I was looking for; reset, calls it and tries find again, and then gets the group with index 0, which is the complete match.

Some Swinging
JIRB works great for trying out Swing interactively. I'm just going to show a tidbit of this, and since this is one of the more common examples of JRuby, you will be able to find many different demonstrations of this on the net.

We first require java-support, and including the required classes for this example:
irb(main):001:0> require 'java'
=> true
irb(main):002:0> include_class(['javax.swing.JFrame','javax.swing.JLabel','javax.swing.JButton','javax.swing.JPanel'])
=> ["javax.swing.JFrame", "javax.swing.JLabel", "javax.swing.JButton", "javax.swing.JPanel"]

Then we create a frame and some panels:
irb(main):003:0> frame = JFrame.new("JRuby panel")
=> javax.swing.JFrame[frame0,0,0,0x0,invalid,hidden,layout=java.awt.BorderLayout,title=JRuby panel,resizable,normal,defaultCloseOperation=HI
DE_ON_CLOSE,rootPane=javax.swing.JRootPane[,0,0,0x0,invalid,layout=javax.swing.JRootPane$RootLayout,alignmentX=0.0,alignmentY=0.0,border=,fl
ags=16777673,maximumSize=,minimumSize=,preferredSize=],rootPaneCheckingEnabled=true]
irb(main):004:0> basePanel = JPanel.new
=> javax.swing.JPanel[,0,0,0x0,invalid,layout=java.awt.FlowLayout,alignmentX=0.0,alignmentY=0.0,border=,flags=9,maximumSize=,minimumSize=,pr
eferredSize=]
irb(main):005:0> labelPanel = JPanel.new
=> javax.swing.JPanel[,0,0,0x0,invalid,layout=java.awt.FlowLayout,alignmentX=0.0,alignmentY=0.0,border=,flags=9,maximumSize=,minimumSize=,pr
eferredSize=]
irb(main):006:0> buttonPanel = JPanel.new
=> javax.swing.JPanel[,0,0,0x0,invalid,layout=java.awt.FlowLayout,alignmentX=0.0,alignmentY=0.0,border=,flags=9,maximumSize=,minimumSize=,pr
eferredSize=]
After this, we add some labels and some buttons to the panes, and then add the panes to the base pane, and the base pane to the frame.
irb(main):007:0> labelPanel.add JLabel.new("Isn't JRuby cool?")
=> javax.swing.JLabel[,0,0,0x0,invalid,alignmentX=0.0,alignmentY=0.0,border=,flags=8388608,maximumSize=,minimumSize=,preferredSize=,defaultI
con=,disabledIcon=,horizontalAlignment=LEADING,horizontalTextPosition=TRAILING,iconTextGap=4,labelFor=,text=Isn't JRuby cool?,verticalAlignm
ent=CENTER,verticalTextPosition=CENTER]
irb(main):008:0> labelPanel.add JLabel.new("JIrb is very useful")
=> javax.swing.JLabel[,0,0,0x0,invalid,alignmentX=0.0,alignmentY=0.0,border=,flags=8388608,maximumSize=,minimumSize=,preferredSize=,defaultI
con=,disabledIcon=,horizontalAlignment=LEADING,horizontalTextPosition=TRAILING,iconTextGap=4,labelFor=,text=JIrb is very useful,verticalAlig
nment=CENTER,verticalTextPosition=CENTER]
irb(main):009:0> buttonPanel.add JButton.new("Pushbutton without action")
=> javax.swing.JButton[,0,0,0x0,invalid,alignmentX=0.0,alignmentY=0.5,border=javax.swing.plaf.BorderUIResource$CompoundBorderUIResource@1354
355,flags=296,maximumSize=,minimumSize=,preferredSize=,defaultIcon=,disabledIcon=,disabledSelectedIcon=,margin=javax.swing.plaf.InsetsUIReso
urce[top=2,left=14,bottom=2,right=14],paintBorder=true,paintFocus=true,pressedIcon=,rolloverEnabled=true,rolloverIcon=,rolloverSelectedIcon=
,selectedIcon=,text=Pushbutton without action,defaultCapable=true]
irb(main):010:0> basePanel.add labelPanel
=> javax.swing.JPanel[,0,0,0x0,invalid,layout=java.awt.FlowLayout,alignmentX=0.0,alignmentY=0.0,border=,flags=9,maximumSize=,minimumSize=,pr
eferredSize=]
irb(main):011:0> basePanel.add buttonPanel
=> javax.swing.JPanel[,0,0,0x0,invalid,layout=java.awt.FlowLayout,alignmentX=0.0,alignmentY=0.0,border=,flags=9,maximumSize=,minimumSize=,pr
eferredSize=]
irb(main):012:0> frame.add basePanel
=> javax.swing.JPanel[,0,0,0x0,invalid,layout=java.awt.FlowLayout,alignmentX=0.0,alignmentY=0.0,border=,flags=9,maximumSize=,minimumSize=,pr
eferredSize=]
Then it's just to pack the contents and show the frame:
irb(main):013:0> frame.pack
=> nil
irb(main):014:0> frame.show
=> nil
To continue playing the Swing, just poke away at the methods. You can always change it if something goes wrong. After you're finished you can dump most of the structure very easy. Just remember to call show again to see some of the updates. Certain changes call redraw directly, but mostly it's always a good choice to call show after updates on the structures.

The JRuby Tutorial #1: Getting started

This is the first installment in a series of tutorials I will write about how to use JRuby for fun and profit. I'm planning on covering a wide range of things you can do with JRuby, from Web frameworks with JDBC to writing your own JRuby extensions in Java. This time I'll take some time covering the basics; what you can do and how you can do it.

I'm writing these tutorials on Windows 2003, but most operations are not OS-centric, so hopefully people with other systems can use these instructions unchanged.

This tutorial will not teach neither Java or Ruby. A basic knowledge about both programming languages are assumed, but most examples will be in Ruby and focus on Ruby development with help from Java (the issue on JRuby extensions is a notable exception.)

What is JRuby
To quote the homepage: "JRuby is a 100% pure-Java implementation of the Ruby programming language." Almost complete support for all Ruby constructs work, and the idea is that you should be able to just use your Ruby scripts with JRuby instead of MRI (Matz' Ruby Implementation, the original Ruby, written in C). But the goal goes beyond that. Java is a powerful platform and there are millions of lines of Java code being written each month, that the world will have to live with for a long time from now. By leveraging Java the platform with the power of the Ruby programming language, programmers get the best from both worlds.

Limitations and caveats
Of course, nothing is simple, so JRuby can't do everything that MRI can. Some of these limitations are due to basic constraints in the Java platform. For example, JRuby will never support a fine-grained timed, until Java does it. Some File-operations just won't work through the Java API. Other limitations are due to differences in implementation, and will probably go away Any Time Now (TM). Some of these include the lack of support for continuations (which Ruby 2.0 won't support anyway), that JRuby's threads are real operating system threads instead of Green Threads (which Ruby 2.0 will have too). Some problems exist with extending Java classes from Ruby and having the result show up from Java. These issues are under development and will be available as soon as enough man hours get put on the problem.

Downloading and installing
The first step in downloading JRuby is to decide whether you should go with a released version (which would be 0.9.0 at this time of writing), or run JRuby from trunk. The last few months, development on JRuby have been quite hectic. Much has been done, and trunk already is quite far away from 0.9.0, both in terms of features, performance and bug fixes. This tutorial will be based on trunk, and if there are any special considerations regarding version, this will be noted in the relevant issue.

To download JRuby from trunk, you need to install a Subversion client. One of these can be found at http://subversion.tigris.org and is very easy to install. After you've downloaded and installed Subversion you're ready to decide where you want JRuby to call home. After you've decided this, you have to set an environment variable called JRUBY_HOME to this directory. For example, I would do this:
set JRUBY_HOME=d:\project\jruby-trunk
in a command window. (In the long run, it would be better to put this somewhere so that it always get set. In Win32 I go to the System configuration on my Control Panel, choose the Advanced tab and press Environment Variables.)

Now we're ready to check out the latest JRuby version:
svn co svn://svn.codehaus.org/jruby/trunk/jruby %JRUBY_HOME%
After you have downloaded JRuby you have to build it, so you have to have a recent version of Ant installed. To build and test JRuby, just enter these commands:
cd %JRUBY_HOME%
ant test
The testing will take a while, but it's good to make sure your installation is fully functional. After you've got the final "BUILD SUCCESSFUL", you can be reasonably sure that your JRuby is functional.

Using Java from Ruby
OK, so now that we've got JRuby up and running, we should try a simple example to see that it actually works correctly and that we can use Java classes to. Write this into a file called testJava.rb:
require 'java'
include_class 'java.util.TreeSet'
set = TreeSet.new
set.add "foo"
set.add "Bar"
set.add "baz"
set.each do |v|
puts "value: #{v}"
end

and run with the command
jruby testJava.rb
You will get this output:
value: Bar
value: baz
value: foo
So, what's happening in this small script is that we first include JRuby's java-support. This provides lots of interesting capabilities, but the most important one is the method Kernel#include_class, which is used to get hold of a Java-class. In most cases you can just include the classes you're interested in and use them as is. In our case, we instantiated the TreeSet as usual in Ruby, by calling new on the class. After this we added some strings to the set, and since JRuby automatically transforms Ruby-Strings into Java Strings, these get sorted by TreeSet in ASCII-order. You may have noticed that we could use a standard each-call to iterate over the entries in the set. This is one of the nice features of JRuby. Even though you're actually working with a Java Collection, it looks and feels (and quacks) like a regular Ruby collection object. All Java classes that implements java.util.Collection automatically gets an each method and includes Enumerable.

The regular include_class function does not work correctly if you try to import a Java class with the same name as an already existing Ruby class. This won't work, for example:
include_class 'java.lang.String'
since there already exists a class called String in Ruby. The first way of handling this problem is to rename the class when including it, by adding a block to the call to include_class:
include_class('java.lang.String') {|package,name| "J#{name}" }
In this case, the class added to the Ruby name space is called JString and doesn't collide with the Ruby String.

The second solution uses the fact that include_class includes the classes mentioned into the current name space, so you can call include_class from inside a module definition, and the classes will be available there:
module Java
include_class 'java.lang.String'
end
The Java String is now available as the class Java::String. If you don't want to include all classes specifically, you can also include a complete package, with include_package. It works exactly like include_class, but takes a package name instead, and makes all classes in that package (but not sub packages) available as classes in the current context.

This is mostly it. JRuby tries as hard as possible to follow the Ruby way, and make Java integration as natural as possible. The ideal is that you should be able to guess how it works, and it will work that way.

Conclusion
The last 6 months, JRuby have become a very real alternative to MRI. As performance gets better and better, the promise of integration with legacy Java systems will get more and more convincing. And using Ruby for tools and tasks where Java only incurs overhead is a very compelling way for Java developers to get the most out of their current tools.

The next issue is slated to be about using Camping from JRuby, together with ActiveRecord and JDBC. Camping is microframework, that together with ActiveRecord lets you write small applications very compact and efficient.

onsdag, augusti 16, 2006

Byte-lexing JRuby.

Since I begun the effort to recreate our lexer with JFlex, me, Charles and Wes Nakamura have discussed how MRI's lexer can be so much faster than JRuby's. One thing discussed was that it could maybe be because JRuby lexes with Readers, instead of handling bytes directly. I didn't believe this would be a big difference, but I set out to test it, anyway. Basically, I replaced all references to chars with bytes, and Readers to InputStreams in our Lexer. There were a few other places that had to be changed. I haven't done anything else to the lexer (there were many places were we could do some good optimization). The results amazed me. My first test case was completely focused on only the parsing step. There is no evaluation in these numbers. First, trunk JRuby:

Did 10_000 parses in 4336
Did 10_000 parses in 3885
Did 10_000 parses in 3866
Did 10_000 parses in 3835
Did 10_000 parses in 3836
Did 10_000 parses in 3815
Did 10_000 parses in 3836
Did 10_000 parses in 3896
Did 10_000 parses in 3845
Did 10_000 parses in 3926
-- Full time for 1_000_000 parses: 39076

and with my byte enhancements:

Did 10_000 parses in 3595
Did 10_000 parses in 3214
Did 10_000 parses in 3215
Did 10_000 parses in 3205
Did 10_000 parses in 3204
Did 10_000 parses in 3184
Did 10_000 parses in 3205
Did 10_000 parses in 3235
Did 10_000 parses in 3204
Did 10_000 parses in 3215
-- Full time for 1_000_000 parses: 32476

These times show about 18% increase in parsing speed, just by working with raw bytes instead of characters.

After I integrated the changes and fixed a few bugs, I finally could test the gain in regular JRuby evaluation speed. My test case this time consists of requiring webrick and IRB. First, trunk JRuby:

real 0m5.191s
user 0m0.000s
sys 0m0.010s

And with the byte-patch:

real 0m4.743s
user 0m0.010s
sys 0m0.010s

So, for this particular test case, we get about 5-8%, depending on circumstances. I would never have imagined that it was such an expense to work with Readers instead of InputStreams. The question now is; is it worth it? Should we sacrifice full unicode (or at least non-astral plane) reading for speed? I'm not sure.

söndag, augusti 13, 2006

Lexing Ruby.

It has become apparent that JRuby's hand coded Lexer is a liability, in the long run. The code is hard to maintain and probably suboptimal with regards to speed. So two weeks ago I decided to check out the possibility of a Lexer generator. I haven't had much time, but the last days I've started reimplementing the JRuby lexer with the help of JFlex.

The first parts have been really easy, actually. I already have a simple version tied in with JRuby, working. There's not much of the syntax there yet, but some of it works really fine. Fine enough to check performance and see the road ahead. So, I have two test cases, the first one looks like this:


class H < Hash
end

puts H.new.class
puts H[:foo => :bar].class

and the second like this:

class H < Hash
def abc a, b, *c
1_0.times {
puts "Hellu"
}
end

def / g, &blk
2.times do
puts "well"
end
end
end

H.new.abc 1, 2
H.new/3

The first one test a corner case in subclassing core JRuby classes. The second one is just nonsense to test different parts of the syntax. Both of these parse and run correctly on JRuby with the JFlex-based lexer. It's not much, but it's something to build on.

Regarding performance, I've created a test program that uses the JRuby parser to parse a specified file 10 x 10_000 times, reporting each time and the total. I've run it on both the first and the second example, and both are about 8 to 10 percent faster with the new lexer. I also expect performance to improve more for bigger files. Right now the lexer keeps track on line number, offset and column number, which is also a performance drain. Removing it gives about 2-3 percent more.

fredag, augusti 11, 2006

Speed in JRuby.

I had really wanted my next post here to be a tutorial about how to get Camping working with ActiveRecord-JDBC and JRuby, but I managed to find two quite serious bugs related to blocks while trying things out. One of the bugs is really strange, and manifests in some of the Markaby CSS-Proxy internals. So, Markaby works, if you refrain from add css class names to tags.

Since Monday I've been busy with various things in JRuby. I've looked around for easy performance fixes, which there are loads of. Many of them didn't give much, but a few things actually had a noticeable difference. One of these places was in the loading process, where it's important to do as few searches as possibly, since most of them are really expensive. Actually, in a few situations JRuby actually parsed and ran a file several times. This happened in WEBrick among others.

I've also implemented a Java version of Enumerable, which was very trick to get working at first, since JRuby don't really support calling methods that yield from Java. So, I had to devise a way of doing this. The generic case seems to be to hard to do right now, but the specific case of calling each and collect the results in a List are finally working well.

I devised a test case where all Enumerable-operations are used 2 or 3 times, and ran this test case 1_000.times. The results for trunk JRuby was ~7s, while the Java implementation of Enumerable took that down to ~5s, which is nice.

Anyway. That's mostly all. I'm off on vacation now.

JvYAML status

I have finally started work on JvYAML again, and I hope to have a new release together within 2 weeks, which will contain JavaBean materialization and the completed emitter. Hopefully I'll manage to integrate this with JRuby not long after.

It's been really hard to get going with this, since I have so many different projects going simultaneously.

måndag, augusti 07, 2006

Some JRuby tidbits

I've spent some time this weekend taking a look at various JRuby issues.

  • Camping:
    The most interesting was trying to get Camping to work. Camping is a microframework for certain kinds of web applications. Very neat, but uses lots of Ruby tricks to function. As such it's a really good test of JRuby capabilities, but it's quite hard to debug. I have got it working really good for basic applications, in the process finding a bug in our Hash#[] implementation that fortunately was easy to fix. But today I've unearthed something really hairy. The test case demands two classes with two methods each, and it's some really strange block tweaking that happens. I hope Charlie is up to the challenge. Markaby (which Camping uses), needs this functionality to work.
  • Mongrel:
    I've started to take a serious look at the Mongrel support. Danny's work on the parser seems really great, so I've added basic JRuby integration to our Subversion, and plan to try getting it to work with Mongrel-0.4 later on today.
  • ActiveRecord-JDBC:
    Last week I added some Oracle functionality to it, but this still needs some work. I'm thinking that we need to factor out some driver-specific functionality soon. Anyway, I released the a version 0.0.1-gem so that people easily can use what functionality we have.
  • JvYAML:
    I've finally begun writing on the Emitter in ernest again. I hope to have most of it finished by Friday, before I go on vacation.
Expect a tutorial on using JRuby and Camping together very soon; hopefully tomorrow. And in a few days after, maybe we can showcase a functional Mongrel in JRuby too?

torsdag, augusti 03, 2006

Announcing Swedish Rails

I would like to announce my plugin Swedish Rails.

Rails provide many goodies and helpers for you, but some of these are dependant on your user interface being in english. Date controls, for example. Pluralization is another. And have you ever tried to capitalize a string with Swedish letters in them? Then you know it doesn't work that well. Until now, that is. You install this plugin, make sure all your Swedish source uses UTF-8, and most of these problems disappear. Downcase, upcase, swapcase and capitalize work as expected. Integer#ordinalize gives Swedish ordinals. And month names and weekday names are in Swedish.

This plugin isn't only for people looking to create Swedish Web interfaces. It can also act as a map for creating a plugin like this for any western language. Just replace all the text-strings in this plugin, change the module name and you're set to go. (Of course you'll have to know the language you're porting too, though).

The plugin can be found at:
http://svn.ki.se/rails/plugins/swe_rails
and some information here:
http://opensource.ki.se/swe_rails.html

The plugin itself is fairly simple. Most of it monkey patches different parts of Ruby proper or aspects of Rails. One of the more interesting parts (which could also be usable by itself) translates all internal month and weekday names in Swedish. This is probably the most interesting when doing Rails GUI's, since you don't have to create your own date_select tag, and all the other variations on this. I've also added som inflector rules, but only for the pluralize helper (so don't start naming database tables in Swedish, please!). The problem is that english pluralization rules are pretty simple. Just tuck an -s on the end and hope for the best. Swedish uses 5 or six declinations for nouns, and most of these are irregular. This makes it slightly hard to describe inflection rules for Swedish, but I gave it a try anyway.

So, enjoy! I just wish I had done this a year ago. Then I wouldn't have had to write sublty different versions of this code all over the place.

onsdag, augusti 02, 2006

Rails, SOAP4R and Java.

I've spent the last three weeks working part time on a project called LPW at work. LPW is a set of web services that talk with the Swedish student databases. The libraries are implemented in Java and deployed with Axist. My project was to create a web system that students can use to access various LPW services. We had a an old implementation of this, written in Java with uPortal as a framework, but for the new implementation we decided that using Ruby on Rails would be interesting and probably worthwhile.

I've spent about 40 hours on the implementation. I did everything except the user interface. I got finished HTML-pages and integrated these with the system. Initially we expected the complete development to take about 120-150 hours with Ruby and about 250 if we did it in Java. In retrospect, I'm pretty sure the Java version would have taken more than that; probably between 350 and 400 hours. Since I wrote the first version in Java I can say this pretty accurately.

So, did I have any interesting experiences while writing this system? Oh yes; otherwise I wouldn't be writing. First of all, I managed to release two plugins in the process of this project. More information about them can be found in older blog entries. I also tried Mongrel for the first time, and I just have to say that I will never go back to WEBrick.

But the most interesting part with using Ruby was that I would have to get SOAP4R and Axis to work together. In the process I found some interesting things. Let me describe the relevant layout of my application.

The project goal was to implement 4-6 different services, which are all available as separate WSDL-files. I generated the drivers for these with wsdl2ruby. After generating the clients for each, I renamed the driver.rb into for example addr_driver.rb, and created a file called lpw_valueobjects.rb where I put all valueobjects found in defaultDriver.rb. I then repeated this process for each service. I then put all these files into the directory RAILS_ROOT/vendor/lpw.

Now, hitting one or two web services each time a student goes to a page isn't really realistic, so I decided early to implement some way of caching the results of the service calls. Since the data in question is pretty static this works well.

So, to integrate the services into Rails, I created a base class called LPWObject in the models-directory. In this class I defined self.wsdl_class and a few other helpers that let me transparently handle the caching of service-classes without actually having them inside the LPWObject itself. This becomes important when I want to save the results in the session. Ruby can't marshal classes which have singleton methods, and WSDL2Ruby depends heavily on singleton methods for the service client.

The first problem with getting Ruby and Java to work over Soap was with swedish characters. When I added something like "Gävlegatan 74" to an attribute and then tried to send this over soap, Soap4R transformed the attribute type into Base64 instead of xsd:String. Suffice to say, Axis didn't really like this. The solution was to add $KCODE="UTF8" to environment.rb. This let's Soap4R believe that åäöÅÄÖ is part of regular strings.

The next problem came when I tried to save some of the value objects into the session. After looking for a long while, I found that if the value object had an attribute called "not", WSDL2Ruby didn't generate an
attr_accessor :not
for this until at runtime, which creates singleton methods on the value object. The solution was to add these accessors by myself. I'm not sure why wsdl2ruby does it like this, but probably there is some weird interaction with the not keyword in Ruby.

The final problem - which I'm not sure if I should blame Axis or Soap4R for - came when one of the value objects contained a byte array with a PDF-file. For some reason Axis sends the regular response XML looking fine, but before and after there are some garbled data. It looks like Axis actually sends the byte data as an attachment too, not just inside the byte array. Soap4R didn't handle this at all, and I got an "illegal token" from the XML parser. The solution to this problem is the worst one, and I should really send a bug report about this, but I haven't had the time yet. Anyway, to fix it, I monkeypatched SOAP::HTTPStreamHandler, and aliased the method send. Inside my new send method I first called the old one, then use a regexp to extract only the xml-parts of the response and reset the receive_string to this. It fixes my problem and works fairly well, but it isn't pretty.

So, in conclusion, Soap4R and Axis seem to work good together, except for a few corner cases. I'm really happy about a project that could've taken 10 times longer to complete, though.

lördag, juli 22, 2006

InPlaceEditor with Autocompletion.

Since the AJAX controls in Rails are so neat, and the InPlaceEditor is probably the neatest since it's so easy and useful, I immediately decided to start fixing one small feature I needed/wanted very much. That is, autocompletion on InPlaceEditor-fields. So, I've created a plugin that does this. It was actually much easier than I thought, since the prototype JavaScript library makes JavaScript almost pain-less to work with. Not quite nice, but not bad either.

Anyway, it's dead simple to use, do it like you've done it with InPlaceEditor and you'll be find. More information can be found here (I've finally convinced my employer to host a place where we can release open source, so it can be seen that KI actually supports open source). The Subversion path is: http://svn.ki.se/rails/plugins/in_place_completer.

Much joy!

söndag, juli 16, 2006

CAS Rails filter

Yesterday I released a Rails filter for doing authentication with CAS. This is really neat, but it's the Rails plugins that make it so neat, since you just have to install the plugin, add three configuration parameters and everything will just work out of the box with all your controllers protected by authentication.

During development it's often practical to not do real authentication, but rather just get a username back as if you'd actually been authenticated already. This can be easily accomplished in three ways with the filter, by a configuration options:

CAS::Filter.fake = "testuid"

which returns the string provided as a username

CAS::Filter.fake = :param

which takes the value of params[:username] and returns this as the authenticated user, and lastly

CAS::Filter.fake = lambda { |controller| ['testuser1','testuser2','testuser3'][rand(3)] }

which invokes the proc every time the filter is called, and uses the string returned as username.

Anyway, authentication is just one of those things that you don't want to think about. It should just be there. And now it is:

script/plugin install http://svn.ki.se/rails/plugins/cas_auth

Enjoy.

Rails and plugin irritation.

I've just started work on a medium sized Rails-application (100-200 hours), and about the first thing I wanted to do was to add my CAS filter (see next posting) to the system. Now, what I'd really like is to be able to set the filter to fake authentication while doing development on my box but going to real authentication when in production. I first tried using $RAILS_ROOT/config/environments/development.rb to set this option, but this doesn't work. Plugins are loaded really late in the startup sequence so my best choice is to add

load "plugin/#{ENV['RAILS_ENV'].rb" if File.exist?("plugin/#{ENV['RAILS_ENV'].rb")

at the end of environment.rb and add my specific configuration to $RAiLS_ROOT/plugin/development.rb

This works, of course, but it is something that I'd thought would already be part of the startup process. Oh well, you can't have everything.

söndag, juni 18, 2006

The perils of hashCode

A few days ago, me and two colleagues tried to track down a very tricky bug. After some hours looking, we finally found it, and it was actually due to a misconception that I had about the workings of HashSet and HashMap. I'm not sure if I'm the only one that didn't know this, but it's very logical once you've found it out. You see, if you save an object in a HashSet, and then change the object in such a way that the hashCode changes, then you won't find that object in the Set anymore. It will still be there, you will still iterate over it, but if you ask for example set.contains(obj), then it will return false. If you iterate over the set, and call Iterator#remove, this will silently fail to remove anything, since the HashSet can't find the object you want to remove. So, if you save things in a HashSet or use them as keys in a HashMap, make sure that the object is immutable, otherwise you'll get extremely hard-to-find bugs.

Incidentally, one of the best newsletters about Java programming wrote about this issue ages ago. Regardless if you work with Java professionally or just for fun I implore you to subscribe to JavaSpecialists by Dr Heinz Max Kabutz. It can be found here.

lördag, juni 17, 2006

JRuby developments

I've spent another saturday at work, mostly working with KIMKAT, but taking some time to look at different JRuby-issues before the 0.9 release. The first - and only major - one, was that Zlib didn't seem to work for Charlie and Tom. I couldn't reproduce this error locally, which was very strange. But finally I found the error in my code, and also the reason I couldn't reproduce it in my own JRuby.

I'd managed to not remove a zlib.rb from one of the directories, and my test cases worked (albeit slowly), because it was using the former Zlib-code, and not my newritten one. After I found this problem, it went fairly fast to isolate the real error, in my GzipWriter. The problem was that I didn't call finish on the GZIPOutputStream which means the gzip-stream doesn't end with a gz-trailer. Once this was added, everything seems to work.

Problem number two seemed harder, initially, and the reason for this was that I needed to muck with the innards of JRuby threads to get it working. The problem concerns Signal handling. In Ruby you can trap operating system signals and execute blocks of code when they happen. This is obviously very powerful and useful for lots of things. My first solution for this, was to just save the block, and then call it when the signal occured. This didn't play well with the rest of JRuby, and Tom got some other strange errors from this. So, after some thinking I came to the realization that I had to create a new RubyThread, and start this when the signal occured, and then JRuby's internal thread scheduling would acertain that nothing untoward happens. This works really well, but I just realized one drawback with it, so I'm going back in to fix it now. The current approach can only execute the block once. Ouch.

So, now I've fixed a new approach. This is much more involved though. I had to get really into the internals of the thread architecture so I could actually create a new JRuby-thread with an existing RubyProc, Frame and Block object. So now I create a new RubyThread each time the signal is called, and sets the block-information manually. This is needed so we can run it multiple times.

Anyway, the real point of this blog is the last problem. The speed of RDoc. RDoc does some fairly heavy lifting, and it itsn't especially fast on C Ruby either:

$ time ruby d:/programming/ruby/bin/gem install rake
Attempting local installation of 'rake'
Successfully installed rake, version 0.7.1
Installing RDoc documentation for rake-0.7.1...

real 0m13.915s
user 0m0.000s
sys 0m0.000s

But now, JRuby isn't near this speed in any way at all:

$ time bin/jruby.bat bin/gem install rake
Attempting local installation of 'rake'
Successfully installed rake, version 0.7.1
Installing RDoc documentation for rake-0.7.1...

real 1m42.985s
user 0m0.000s
sys 0m0.010s
Yes, it's about one magnitude slower. Yesterday I found a big reason for this. JRuby does a lot of hashing and spends very much time in HashMap#get. So I was curious what hashCode-algorithm was used internally. What I found was this:
    public final int hashCode() {
return RubyNumeric.fix2int(callMethod("hash"));
}

Which actually calls the Ruby hash-method every time someone wants a plain hashCode. This explained some of the bad performance. I added some better hashCodes to different common Ruby-objects, like RubyString, RubyHash, RubyFixnum and others. After this small change I got this:

$ time bin/jruby.bat bin/gem install rake
Attempting local installation of 'rake'
Successfully installed rake, version 0.7.1
Installing RDoc documentation for rake-0.7.1...

real 0m59.946s
user 0m0.000s
sys 0m0.010s

... That is almost double the speed for general JRuby performance. By just adding a few simple hashCode-methods at different places. So this is the lesson from all this: if you have performance problems; look at your hashCode(), it may be easy to fix!

torsdag, juni 15, 2006

Working towards 0.9

These last days I've been focusing on fixing issues with RubyGems and Rails, so that JRuby 0.9 will be a really great release. It already is, of course, but all we can do to make it better feels nice. These are the things I've done since sunday.

StringIO
I finally took the time to rewrite all of StringIO to Java, and also really test so that it works correctly. I managed to make the more common usage patterns between 8 and 10 times faster, which I feel is sufficient for now.

Signal and Kernel#trap
Ruby has a close ties to C, which means you can trap low level POSIX signals quite easily. This is hard to support in Java, though. One common use case that most Ruby programs do, is to trap INT so they can break the program gracefully. Rails and WEBrick does this. The current Kernel#trap and Signal was just stubbed out. I found a way to support signal handling on Sun JVM's at least, through the undocumented class sun.misc.Signal. JRuby feels if this is available, if so it uses it, otherwise trapping signals doesn't work. The implementation is really easy, I just grab the block provided and saves this in a Runnable that will be executed when the trap happens.

Zlib, IOInputStream and IOOutputStream
When working with Ruby IO-like objects from Java, it is often very convenient to wrap these in a Input/Output-stream. This isn't totally obvious how to do, though. My first implementation worked, but was intolerably slow. My last post about the plaincharset tells the tale how these things can go wrong. Suffice to say, I managed to fix the streams pretty good, and then started the real work: Reimplementing Zlib::GzipReader and Zlib::GzipWriter in Java. This wasn't as hard as I thought, once I got the IO-streams working as it should.

There was one bug which was really hard to find though, and it was caused by a minor difference between Java's read(n) and Ruby's read(n) methods. In Java, if we ask to read n bytes, we don't necessarily get all the bytes we asked for, even if there are that many bytes available. This is why those read-methods return how many bytes actually were read. But Ruby's read(n) doesn't act this way. It either reads n bytes, or to end of stream, depending on which come first. That one took a while to find.

The reimplementation of Zlib isn't complete yet, but the important classes are done. The deflater and inflater classes already use a backing Java class for performance, and the checksum classes don't need that kind of speed. The performance improvement from reimplementing GzipReader into Java was great, though. It seems to be between 15 and 20 times faster, most often. RubyGems is really useful now.

What's left?
Now, these things are quite minor. They improve different parts of Ruby that are used quite often. RubyGems seem to work more or less perfectly. Rails generation works too. The server-script for Rails almost works, but there is something strange going on with WEBrick yet. I'm wondering if this has something to do with our Socket-code, which I'm not totally into yet. But these are the issues I'll be looking at before 0.9.

In the long (longer at least), I have two main points that interest me. First I want to complete JvYAML, and integrate it with JRuby. And secondly I'm thinking about ways to byte compile (to some Ruby bytecode) parts of the AST tree, much in the same way Charles have been toying with compiling parts of it to Java bytecode. This is mostly just ideas in my head yet, but that's probably something I'll write more about quite soon here.

Java Charsets

Today I've spent some time pounding java.lang.String to give me a byte array I can use as a data format. This seems harder than I thought it should be. So, why can't I use getBytes() or getBytes("ascii") or getBytes("iso-8859-1") or getBytes("utf-8")? Those are fine for certain tasks, but I'm looking for a very specific translation from chars to bytes. The application I was working on is Zlib in Java, for JRuby. Since Ruby have the somewhat funny custom of using Strings as byte buffers this means the output I get from a Ruby IO-operation is a RubyString.

The reason I started trying different paths for this was that Zlib didn't work as it should. Not at all. I knew it worked when I did it one char at a time, because then I casted the char to an int instead (since InputStream#read() returns a int). So, I created this small program:


final byte[] chrs = new byte[256];
for(int i=0,j=chrs.length;i<j;i++) {
chrs[i] = (byte)i;
}
final String str = new String(chrs);
final byte[] bts = str.getBytes();
for(int i=0,j=chrs.length;i<j;i++) {
System.out.println("[" + i + "]= " + (int)chrs[i] + ", " + bts[i] + " ... should be: " + (byte)chrs[i]);
}

to see what happened here. Now, I won't bore you with the complete printout from this. But there are a few specific portions that I'd like to share:

[127]= 127, 127 ... should be: 127
[128]= -128, -128 ... should be: -128
[129]= -127, 63 ... should be: -127
[130]= -126, -126 ... should be: -126

[140]= -116, -116 ... should be: -116
[141]= -115, 63 ... should be: -115
[142]= -114, -114 ... should be: -114
[143]= -113, 63 ... should be: -113
[144]= -112, 63 ... should be: -112
[145]= -111, -111 ... should be: -111

and

[156]= -100, -100 ... should be: -100
[157]= -99, 63 ... should be: -99
[158]= -98, -98 ... should be: -98

Those 63-values keep showing up and destroying everything. If I try another encoding in the getBytes-method it actually gets worse. I couldn't find any way to get this to write the expected output. So, I embarked on a quest. A quest to solve this small trouble, forever and always. The result is plaincharset, a small project consisting of 4 classes. Nothing spectacular, but if you add the jar-file to your classpath you can now use the charset name "PLAIN" to get every byte correctly from getBytes and new String. If you have characters that are not within 0..255 I cannot guarantee anything at all. I hereby release the project in the public domain. The source can be found here, and if you just want the jar-file, download it here.

So, what is the secret behind this marvel? In one word: NIO. The jar-file contains a subclass of CharsetProvider, a subclass of Charset, one CharsetDecoder and one CharsetEncoder. The only classes with anything in them is the decoder and encoder, which gets an input NIO-buffer and an output NIO-buffer. I just read from the input and write to the output, casting where necessary. There is also one service-provider file in the META-INF directory in the jar, which says to use the com.ologix.charset.PlainCharsetProvider as a provider for charsets.

And did this work for my Zlib-implementation? I'm happy to say that it did. It works very well and is both smaller in code length, and much, much faster. I'm happy.

The arcitecture of a Meta Directory system

One of the major projects at Karolinska Institutet the last 3 years is called KIMKAT. It has gotten the moniker of a meta directory system, but in many ways that's not entirely correct. Since I've been one of the lead architects and developers in this project, I wanted to write about the technical architecture for the system, what choices panned out well, and things I would change if I could.

The problem
Karolinska Institutet is a fair sized university (quite big for Sweden). We have about 20k students and something like 5-7000 employees. Universities have a tendency to become decentralized, with local solutions for all problems. KI is no exception. Information about people at KI exists in more than 10 disparate systems only at the administration. This information costs much money to keep fresh, and it's also very inefficient. The vision with KIMKAT is to have one central source for all datums about persons, organizations and resources, where external systems can find up-to-date and current data. It should also be possible for these systems to contribute domain specific data into KIMKAT. (For example, our phone directory system should probably be the source for phone numbers.)

Different parts of the solution
We have worked with this problem in a project which has oscillated between 10 and 20 project members during 2½ years. Of course, there are many ways to solve this kind of problem, and the main differentiation between them is how much of a hack you want it to be. KI specifically wanted to avoid yet another hack solution (since other uni's in Sweden has gone down this route, and it seems both costly and becomes unmaintainable in only one or two years), so the focus of the the project group was to find and implement a solution that would hold for many years to come.

The first problem was that we had no definite source for organization information. Neither did we have anything for our affiliates. All that information was stored ad-hoc in the mail system, and on paper. So, we instigated a regime where each datum of information has one primary source which is responsible for maintaining and updating it. For employees this is our HR-system (called Primula). For students, it's our student database called LADOK. For organizations we created a new primary source called KOrg, and for affiliates KAff. We also created a database for all KIMKAT information that we couldn't write back to the other primary sources. This effectively became it's own primary source.

Our meta directory solution is based around a meta engine; a scripted system that reads and collects data from our primary sources and writes this in a unified data structure to the central KIMKAT database OmniaS. For this task we evaluated several different products, and also considered writing our own, but in the end we chose IBM Tivoli Directory Integrator, which is Java-based and very easy to use for easier situations. It uses BSF to allow scripting, which is almost always needed for more intricate solutions. Suffice to say, our final system contains lots of ITDI scripting.

The OmniaS database are surrounded with an EJB tier with Hibernate and accessors for reading the data in a well defined onthology implemented with JavaBeans. There are no Entity EJB's in KIMKAT. (Nor anywhere else at KI, as far as I know - and hope.)

Right now the primary user of the OmniaS information is our web interface, KKAWeb, which is used to update information in our primary sources. It's also used to establish new affiliations and organizations. There are some information the we deemed was necessary for KKAWeb and other applications that would eventually display KIMKAT-data in some way, but wasn't really business data. Because of this we created an external data source called KDis for this data. KKAWeb and another application called KIKAT uses this data extensively, but mostly for things like I18N and sorting.

Since our primary sources are of a very diverse nature, KKAWeb doesn't write to them directly. Instead the updated value objects are sent to an update topic with JMS, and every primary source has one or more message driven beans listening for just those messages that pertains to it. Then that bean updates the database.

The final part of the puzzle is another topic which ITDI sends all updates to, as soon as something is changed in one of the primary sources. This allows external systems to get hold of change events in whatever way they need. The change information is sent with JMS using S-expressions to represent data, since not all consumers will be Java-based.

Lessons learned
The KIMKAT project is late, and the deadline have been put off several times. There are several reasons for this, but the main one is probably due to our inability to give good time estimates. There are a few architectural and design decisions that I would probably have done another way if we would start everything over. The biggest one is there data source called KDis. That was a big mistake for several reasons. First of all, having joins between different databases is a pain. And having database constraints between databases are also really cumbersome.

If we did this again, I would probably drop one of the tiers between the sesssion EJB's and the OmniaS database. Right now, much time goes to serialize and deserialize between different kinds of value objects.

And third, our updating service is a really good idea, actually, but I think that the implementation could have been done in a better way. There is something nagging me with it, but I can't but the finger on it right now.

But all in all, our architecture have been really successful, and it feels like something that will be able to stand the test of time. Now we just have to integrate all other systems with this, so everyone gets all the real benefits from this solution.

måndag, juni 12, 2006

A YAML dumper in Java

Today I've begun work on porting the RbYAML dumper to Java. The work will be done in three separate layers: the emitter, the serializer and the representer. I've just decided the structure for the emitter, but the basic premise is really simple since emitting is just about low level IO stuff.

The emitter will be a stack based finite state machine, and it actually closeley resembles the ParserImpl part of the loader. There are 17 states and they are represented as anonymous implementations of a simple interface. If this was a Java 5 project, I would've implemented it with an Enums, like this:

public enum EmitterState {
StreamStart {
public void expect(final EmitterEnvironment env) {
env.expectStreamStart();
}
}
}

with anonymous Enum subclasses for each state. A very fine solution, since I can just fetch the next state and execute it directly, instead of doing something like this:

STATES[STREAM_START].expect();

Alas, that's not the way it will be.

One of the more interesting parts of this implemention will be finding a nice way to represent the options hash from Syck and RbYAML 0.2. With standard Ruby keyword arguments, the options hash is very practical to use, instead of having to supply 17 different arguments - or even worse fact(17) different constructors to allow for default arguments. The correct way in Ruby is to have a defaults Hash, merge this with the arguments provided, and lookup the configurations options here. Obviously, I could do this with maps, but it's very cumbersome and requires more than one line to provide an option; something I would like to avoid.

The solution to this dilemma came from a part of Joshua Blochs Effective Java technical session at JavaOne. (I've written a little about it in my entry on JavaOne, day 2). Anyway, the solution is to provide an YAMLOptions class with a build factory method. I'm not totally finished with the syntax yet, but I'm thinking that a toYaml call could look something like this:

YAML.toYaml(theObj, YAMLOptions.build().useVersion(true).useDouble(true).indent(4));


There are a few benefits with this syntax. The first is obvious: it takes a whole lot of options to span this over more than one line, and if you want that many, maybe you should save this options-object as a constant somewhere? Another great benefit is type safety. This isn't really an issue in Ruby, since there are no casts, but in Java I like the guarentee that the indent-option is an integer. I can save the defaults inside YAMLOptions, and if I define an interface for the YAMLOptions someone can implement it, while providing more options if needed. For example, right now I'm sure that JRubyYAMLOptions will be implemented; probably as a subclass to YAMLOptions, with some new options that only the JRuby-specific representers will use. Oh how I would love to be able to specify that static method build()with an interface...

JvYAML, RbYAML and JRuby

There has been much activity on several points these last weeks, but the guiding light has been on getting JRuby as good as possible. JRuby 0.9 will arrive sometime this or next week, and to that point I managed to rework the emitter in RbYAML and release version 0.2 a few days ago. This release will be a part of JRuby 0.9. About 10 days ago I finished the parser implementation for JvYAML and released version 0.1 of it.

Today I've spent some time working on porting the emitter from RbYAML to Java too, but I haven't really gotten that far yet. I hope I can get it finished in time for JRuby 0.9, but it's no big catastrophe if not. The essential part with YAML emitting is that it's correct, not fast. But it would be really nice to get it included in this major release too. We'll have to see.

I realize I haven't really blogged about neither RbYAML 0.2 or JvYAML, just announcing them. But don't worry, it will arrive shortly.

As for now, the more interesting thing is JRuby. The 0.9 release will contain some truly spectacular things, and I'm prepared to say that JRuby is now usable in many commercial environments. We have RubyGems working correctly (albeit slowly, right now), we have the initial parts of Rails, we have the ActiveRecord JDBC connector, we have massive speed improvements. We have also received official sanctioning from Matz to include C Ruby's core libraries in the JRuby distribution (this means we no longer have to hack fileutils.rb each time checking out a new copy of JRuby). The IO work from Evan Buswell means Webrick will very soon work correctly from JRuby. All in all, it's a really major improvement, and I'm proud of being part of it. Stay tuned for more info.

A few hours with Emacs

Today I've spent some hours upgrading Emacs to version 22, and getting various stuff to work. It was an interesting experience, and I've found lots of neat tricks I didn't know about before. But first, Emacs 22! This is in no way a released version, it is still in beta and reportedly has some bugs left in the code base. That said, I've been programming and working in it during the day and not experienced any trouble at all.

The new version brings some major updates, especially in the department of internationalization. The support looks great, actually. Another cool feature that Steve Yegge have blogged about (see link to the right), is the new support for embedding elisp inside replacement in "query-replace-regexp". That's really neat.

First installation wasn't totally smooth; my JDEE environment didn't really want to play, but after a while of fiddling with the code for it, I noticed that a new minor version was available. I added this to my local installation and everything justed worked. Even my own JDEE extensions continued working.

I've never really liked the completion in JDEE, but now I've finally written a small script to make is usable when I want it, at least:

(defun indent-or-complete-jde ()
"Complete if point is at end of a line, otherwise indent line."
(interactive)
(if (looking-at "$")
(jde-complete)
(indent-for-tab-command)))
(define-key jde-mode-map [tab] 'indent-or-complete-jde)

This code does code-completion if at end of a line, which is the most common case when you need completion. It's important to have the jde-global-classpath point to all available classes, though. (I've created a global prj.el with common options preset for JDEE, and loads this from my various projects prj-files.) Completion is still slow, especially the first time, but when you really need it, it's there. I toyed with getting dabbrev or pabbrev in there also, but that just doesn't seem right for Java.

Another nice feature I got working was integrating imenu with JDEE, so I can Shift-right-click in any Java buffer and get a nice menu list of all class definitions with method and field hyperlinks. That's probably the only thing I'll use the mouse for in Emacs, ever, and it's more like using it as a key anyway. The only annoying thing with imenu is that I want it to sort the classes alphabetically. I've set this both with imenu-sort-function and jde-imenu-sort, but it doesn't really bite.

What I haven't had time to test yet - and the things that'll probably need some work - is my tramp-paths for interactively working with files on other computers through Emacs and plink (which also enables me to use Slime for LISP editing in-process on other servers). Right now this doesn't seem to work, but I read somewhere that the plink-support has been upgraded in Emacs 22.

But all in all I'm very satisfied. I recommend any Emacs maven to try out the new version, and especially if you need to work in Cygwin or on the Mac. The support for these environments are now there.

söndag, juni 11, 2006

Announcing RbYAML version 0.2

Another major release, with most changes in the dumper:
http://rbyaml.rubyforge.org
or to download directly:
http://rubyforge.org/frs/?group_id=1658

Changes:
  • Performance has been greatly improved
  • Rewritten the representer to use a distributed representation model
  • Much improvement of test cases
  • And many bug fixes

tisdag, juni 06, 2006

Announcing JvYAML.

I am pleased to announce JvYAML, version 0.1. JvYAML is a Java YAML 1.1 loader that is both easy to extend and easy to use. JvYAML originated in the JRuby project (http://jruby.sourceforge.net), from the base of RbYAML (http://rbyaml.rubyforge.org). For a long time Java have lacked a good YAML loader and dumper with all the features that the SYCK using scripting communities have gotten used to. JvYAML aims to rectify this.

Of major importance is that JvYAML works the same way as SYCK, so that JRuby can rely on YAML parsing and emitting that mirrors C Ruby.
JvYAML is a clean port of RbYAML, which was a port from Python code written by Kirill Simonov for PyYAML3000.

Simple usage:
import org.jvyaml.YAML;

Map configuration = (Map)YAML.load(new FileReader("c:/projects/ourSpecificConfig.yml"));
List values = (List)YAML.load("--- \n- A\n- b\n- c\n");
There is also support for more advanced loading of JavaBeans with automatic setting of properties with the use of domain tags in the YAML document.

More information:
At java.net: http://jvyaml.dev.java.net

Download: https://jvyaml.dev.java.net/servlets/ProjectDocumentList

License:
JvYAML is distributed with the MIT license.

lördag, juni 03, 2006

Transforming RbYAML

RbYAML went through some big changes from release 0.0.2 to 0.1. My intentions are to detail some of these changes, what implementation choices I did, and why.

First, conversion from Mixins to Classes. The original Python implementation used multiple inheritance, created several base classes (Reader, Scanner, Parser, etc) and then created one several versions of a Loader class which inherited from the different base classes. My first implementation mirrored this approach, but used Modules instead of base classes and mixed in different versions of these in the different Loader classes. This approach was quite limiting since mixing in code into other Modules doesn't really work as you expect, and this is no substitute for subclassing. For example, I had a BaseResolver module, a SafeResolver module which mixed in BaseResolver and added code of it's own, but this were quite cumbersome.
The solution to this was simply to convert all Modules to class, and make all calls to the other tiers explicit. For example, instead of having the Parser module just assume that you've mixed in a Scanner and call check_token on itself, I have the Parser class take a Scanner instance at initialization and call check_token on this instance instead.
This works very well, and probably makes the code easier to understand. Another positive of this is that the interface between the layers are more apparent. For inclusion in JRuby, this will make it easier to replace certain parts with Java implementations.

The next piece on the agenda was a rewrite of the Parser. The original Python implementation used Python generators (which are almost like coroutines, but not quite). My first port of this code just parsed the whole stream, saved all events and then passed these on after parsing. This was good enough for smaller YAML documents, but when trying to parse the RubyGems gemspec, the memory and time requirements became to prohibitive. In the course of making the generator algorithm explicit I totally rewrote the Parser from the beginning, making it hybrid table driven instead of recursive-descent as the original was. I actually believe the new Parser is both easier to understand and faster. Just as an example, this is the code for block_sequence:

def block_sequence
@parse_stack += [:block_sequence_end, :block_sequence_entry, :block_sequence_start]
nil
end

where @parse_stack contains the next productions to call after block_sequence has finished. The main generator method just keeps calling the next production until it arrives to a terminal, and then returns the value of this:

def parse_stream_next
if !@parse_stack.empty?
while true
meth = @parse_stack.pop
val = send(meth)
if !val.nil?
return val
end
end
else
return nil
end
end

Another benefit of this is that this code is dead simple to port to other languages, once again probably easier than the Python version.

The third improvement was performance. I have no trustworthy numbers of the improvement, but it's in the order of 5-8 times faster than from the beginning. I achieved by some easy fixes, and some harder ones. I removed the Reader class and inlined those methods into the Scanner. I tested each case where I tested if a character was part of a String and checked were a Regexp was faster. And added some hard coded, unrolled loops in the most intense parts of the code, which was peek(), forward(), prefix() and update(). Every microsecond improvement in these methods counted since they are called so many times. I didn't do all this work blind, though. The Ruby profiler is really good. Just take a script, run it with ruby -rprofile script.rb and you get output that's incredibly good. I tested most of my changes this way, and the end result is about as fast as the JRuby RACC-based YAML parser, which was my goal.

Since version 0.1 I've spent some time getting JRuby to work flawlessly with RubyGems, and this work have uncovered some small bugs in RbYAML (and in SYCK, for that matter), so a new minor release will probably come soon. Until then the CVS is up to date.

Getting RubyGems to work with JRuby

I'm sorry if the title gives it away, but here are some recent output in my terminal window:

#bin/jruby bin/gem install rails --include-dependencies
Attempting local installation of 'rails'
Local gem file not found: rails*.gem
Attempting remote installation of 'rails'
Updating Gem source index for: http://gems.rubyforge.org
Successfully installed rails-1.1.2
Successfully installed activesupport-1.3.1
Successfully installed activerecord-1.14.2
Successfully installed actionpack-1.12.1
Successfully installed actionmailer-1.2.1
Successfully installed actionwebservice-1.1.2
Installing RDoc documentation for activesupport-1.3.1...
Installing RDoc documentation for activerecord-1.14.2...
Installing RDoc documentation for actionpack-1.12.1...
Installing RDoc documentation for actionmailer-1.2.1...
Installing RDoc documentation for actionwebservice-1.1.2...
So, we have RubyGems mostly working. Right now there are two caveats. First, during the YAML parsing, we get some InterruptedExceptions for some reason. This doesn't seem to impair functionality, though. The second problem is that it takes serious time. Between 30 minutes and an hour for this. The two parts that are time hogs are the YAML parsing of the Gemspec, and the RDoc stuff, for some reason.

So, what do you need to do, to get this working?
  • Start from a newly checked out JRuby.
  • Add patch for RubyTime and TimeMetaClass. (Adds gmt_offset and utc_offset. This patch can be found in the jruby-devel archives.)
  • Checkout the latest version of RbYAML from RubyForge, and put this in $JRUBY_HOME/lib/ruby/site_ruby/1.8.
  • Add the contents from the C Ruby libraries.
  • Change fileutils.rb, so that RUBY_PLATFORM works.
  • Replace the file $JRUBY_HOME/src/builtin/yaml.rb with the yaml.rb for RbYAML, that can be found here.
  • Change the jruby and jirb scripts by adding -Xmx512M. (I'm not sure 512 is really needed, actually. Maybe 256 or 128 suffices.)
And this should be everything that's needed to get the same results as me when trying to install Rails (provided you've got the patience).