Dynamic Ruby power and static balance

Update: This post has been updated to explain, clarify and remove certain things that sounded like an attack on people that didn't agree with me, especially Austin. This was certainly not my intent when writing it. Added explanations will be highlighted with italic text.

Sir Bedevere: And what do you burn, apart from witches?
Peasant 1: More witches.
Peasant 2: Wood.
Sir Bedevere: Good. Now, why do witches burn?
Peasant 3: ...because they're made of... wood?
Sir Bedevere: Good. So how do you tell whether she is made of wood?
Peasant 1: Build a bridge out of her.
Sir Bedevere: But can you not also build bridges out of stone?
Peasant 1: Oh yeah.
Sir Bedevere: Does wood sink in water?
Peasant 1: No, no, it floats!... It floats! Throw her into the pond!
Sir Bedevere: No, no. What else floats in water?
Peasant 1: Bread.
Peasant 2: Apples.
Peasant 3: Very small rocks.
Peasant 1: Cider.
Peasant 2: Gravy.
Peasant 3: Cherries.
Peasant 1: Mud.
Peasant 2: Churches.
Peasant 3: Lead! Lead!
King Arthur: A Duck.
Sir Bedevere: ...Exactly. So, logically...
Peasant 1: If she weighed the same as a duck... she's made of wood.
Sir Bedevere: And therefore...
Peasant 2: ...A witch!
(quotes from Monty Python and the Holy Grail, courtesy of IMDB)

My post announcing Ducktator seems to have stirred up a few emotions on Ruby-talk. Of course, most of this is my fault, by naming the library in such a frivolous way and not explaining the domains for its usage correctly. But on the other hand, there seems to be a general confusion about the concept of Duck typing, dynamic versus static typing, validation and other issues. Actually, I get a whiff of religion when my mention of Duck typing engendered such a diverse set of responses.

Of course, my reaction about duck typing was as religious. I see this is a general trap when discussing programming languages. The Ruby community is altogether very good at avoiding religion, which caused me to be quite startled when I found hints of it. Duck typing as a concept seem to be very loaded right now. I'm merely pointing this out as something that we should take care to be on the watchout for. Just as I will do from now on, I suggest people in the Ruby community should try to be as objective as possible, when discussing this.

And everyone and their aunt seem to have different opinions on what duck typing really is. It's all quite fun, actually, except for the fact that it misses the point. I should have avoid mentioning ducks. I should have avoiding saying anyting at all about typing, since that isn't the point. And I bloody well shouldn't have used the class-validator in my example. Well, done is done. And this post won't be about that. Just the next paragraph.

The Ducktator disclaimer

I won't mention the words duck typing from here on. I would change the name of the project if it wasn't so damn hard in RubyForge. But what I want to explain is this. Ducktator is about validating things. But not everywhere. You shouldn't use Ducktator at those places where you have one or two checks for something in an object. You should really only use it at the borders of your code. The borders where you you will receive complex objects. Really complex objects where a method_missing won't tell anyone anything useful at all. The use case I had in mind when writing the library was for RubyGems, when the YAML spec for a Gem has been loaded, to check that the important parts actually have what it takes to get into the source index. Since I managed to break RubyGems this way, I feel that this kind of validation can be really important. Once again, this is validation of live Ruby objects. Nothing else. You can check practically anything you want, but the easiest examples have been about each, class and respond_to. Hope this clarifies things a bit.

I removed the entire paragraph about typing. But my recommendation still stands; if you find formal types in programming languages interesting and/or confusing, read Programming Language Pragmatics, and you will be enlightened.

The main point

The reaction to my possibly improper use of the term Duck typing engendered a very strange response, which I hadn't expected. Of course, I realize that this is a very obvious community effect. Since Duck typing is one of the trademarks of the Ruby community, it also means everyone has opinions on it, and more importantly feel the need to defend it as soon as some threat is perceived. Steve Yegge has written lots and lots about what language religion is really about, and I feel that this is an extension of that issue, so I won't write more about it here either. You can find more in many of this excellent Drunken Blog Rants.

Finally. Balance is what I'm after. One person (Austin) said that the d**k t****g philosophy (I had written the word 'issue' here. That seems to have been misinterpreted. I blame that on my poor grasp of English, since my mother tounge is Swedish. =) is about TRUST. That you should trust the caller of your library to read your documentation (which - obviously - is perfect), and supply the correct objects. This isn't too much to task if your docs are up to notch. And if the caller is the same one that will suffer if he mishandles your library. But trust isn't enough when you're at the borders. When talking to other languages through shaky serialization systems. When talking with clients that possibly could be hostile. (Yes, in this case setting $SAFE helps, but it doesn't go all the way). (Sandbox is - or will be - a good alternative here, but I still see places where object validation is a better solution.)

Further, Austin responded in his blog post that he thinks I have 'set up a false dichotomy here: people who are for duck typing as trusting your caller are against validation'. This wasn't my intention. Actually more the other way around. I am for duck typing, in most places. What I'm saying is that no solution is perfect at all points in your code and duck typing is good fit in many, but not all. Further, the next paragraph clarifies my wish for balance.

What I'm saying is, most of the time you won't need it, but in some cases, some kind of interface validation really helps a lot. I know the so called dynamic community doesn't like to hear this. But what is so dynamic about failing without control? (The arguments I heard about letting code fail when the method isn't there sounded very much to me like failing without control. That was my interpretation of the argument that you don't need to use respond_to? for duck typing.) I know that I, as a developer isn't infallible. I make mistakes. Most of the times I am in control of all my objects, but there are times when I'm not. For example, there are situations where I develop smaller applications for other (non-programmers) people. I like to create configurations and rules in YAML for these projects and leave the client in charge of configuring the application. But, what if he/she/it makes a mistake? Using the 'other' way, I would fail when trying to call protocol on something that should have been an URI, but wasn't because the person made a typo and put an illegal character inside the URL. Will that message help the person doing the configuration? Should you wrap your calls in rescue's all over the place and give the same explanation? Should you trust that the (non-programming) client should be able to read your RDoc and figure out that a method (which I bet you didn't name get_uri_from_yaml_configuration) failed because of something they did in the configuration? I believe not.

What I'm really ranting about is balance. There needs to be a balance between checking and laissez-faire. In most places, just calling the method is fine. In other places it's appropriate to check with respond_to?, in some cases you need to check the class. We're programmers. We are supposed be good at judging which technique to use where. Yes, Ruby is dynamic language. Yes, Ruby is very easy to learn. Yes, Ruby makes most stuff very easy on you. That doesn't mean you should stop thinking. It doesn't mean you should be lazy. We are programmers, and we should be able to adapt.

One more time. Balance. Balance. Everywhere. And I do love the Ruby community. It is the best. Even though people get mad at each other, we can solve our differences. I'm proud of being a part of it.

MySQL, some concrete suggestions!

After my post Rails, Databases, ActiveRecord and the path towards damnation, I got an e-mail from Mårten Mickos, the CEO of MySQL. He asked me to provide concrete suggestions on how to improve MySQL (since the other post just contained some unspecified not-like vibes), so that's the rationale for this post. I'm going to point at a few things I see as a problem for using MySQL as a production database right now. Standard disclaimer stands: these are my opinions, my own only, and my employer doesn't necessarily agree or disagree with them on any level.

Let us jump into the fray:

• Sequences. I would like real, nice and sweet sequences. I really don't like to have no control of my primary key generation, and I especially don't like that I can't have sequences for anything else. The recommended solution according to the manual is to create a table with one auto-increment column in it, and use this as a sequence. That's not acceptable, especially since I cannot tie this so-called sequence to the generation of id's on other tables with subselects and other fun things.
• OK, I really don't like the auto-increment feature. Why not provide an IDENTITY keyword like the non-core feature ID T174+T175 specifies?
• Real, honest-to-god, boolean types. Real ones. Not tinyint(1)s. Not enums. Not tinyint's hidden behind the word boolean (like JDBC). Real boolean types.
• I would like table1 and Table1 to be different (as per the spec). Oh yes, we seem to live in an insensitive world (case and otherwise) with Windows all over the place. But in my database I want that kind of control.
• Limiting the return values of result sets. Now, I have no problem with LIMIT and friends, but since there is a spec, and that spec has a feature for this functionality too (T611), why can't that be in MySQL?
• Time-types should be able to store fractional seconds and time zones.
• And what's the matter with the TIMESTAMP type? That doesn't really do what the standard says it should do. Please give it a name not in the standard.
• And for Pete's sake, double bars is for concatenation in SQL. || is for 'or' in programming, but SQL is a DSL. This screams leaky abstractions and is very annoying.
• Stability of 5.0 features. I know triggers, foreign keys and stored procedures are all there now. But frankly, I don't trust my referential integrity with them yet. Not from a database vendor that a few years ago wrote in their manual that the only reason for foreign keys was to be able to let GUI's diagram relationships between database objects. Not from a vendor that said that you don't need transactions to ensure data integrity. All in all, I want these features to be around a few hours, get the bugs hashed out, let them be pounded on for a while. But that's not going to happen if people move to Rails, since Rails doesn't believe in data integrity or foreign keys.

Well, that's that. Only my opinions, remember? Anyway, for small and fast development, MySQL is really useful. I'm just arguing that a big production system should choose something else.

Ruby Metaprogramming techniques

Updated: Scott Labounty wondered how the trace example could work and since a typical metaprogramming technique is writing before- and after-methods, I have added a small version of this.
Updated: Fixed two typos, found by Stephen Viles

I have been thinking much about Metaprogramming lately. I have come to the conclusion that I would like to see more examples and explanations of these techniques. For good or bad, metaprogramming has entered the Ruby community as the standard way of accomplishing various tasks, and to compress code. Since I couldn't find any good resources of this kind, I will start the ball running by writing about some common Ruby techniques. These tips are probably most useful for programmers that come to Ruby from another language or haven't experienced the joy of Ruby Metaprogramming yet.

1. Use the singleton-class

Many ways of manipulating single objects are based on manipulations on the singleton class and having this available will make metaprogramming easier. The classic way to get at the singleton class is to execute something like this:

 sclass = (class << self; self; end)

RCR231 proposes the method Kernel#singleton_class with this definition:

 module Kernel
def singleton_class
class << self; self; end
end
end

I will use this method in some of the next tips.

2. Write DSL's using class-methods that rewrite subclasses

When you want to create a DSL for defining information about classes, the most common trouble is how to represent the information so that other parts of the framework can use them. Take this example where I define an ActiveRecord model object:

 class Product < ActiveRecord::Base
set_table_name 'produce'
 end

In this case, the interesting call is set_table_name. How does that work? Well, there is a small amount of magic involved. One way to do it would be like this:

 module ActiveRecord
class Base
def self.set_table_name name
define_attr_method :table_name, name
end

def self.define_attr_method(name, value)
singleton_class.send :alias_method, "original_#{name}", name
singleton_class.class_eval do
 define_method(name) do
   value
 end
end
end
end
 end

What's interesting here is the define_attr_method. In this case we need to get at the singleton-class for the Product class, but we do not want to modify ActiveRecord::Base. By using singleton_class we can achieve this. We have to use send to alias the original method since alias_method is private. Then we just define a new accessor which returns the value. If ActiveRecord wants the table name for a specific class, it can just call the accessor on the class. This way of dynamically creating methods and accessors on the singleton-class is very common, and especially so in Rails.

3. Create classes and modules dynamically

Ruby allows you to create and modify classes and modules dynamically. You can do almost anything you would like on any class or module that isn't frozen. This is very useful in certain places. The Struct class is probably the best example, where

 PersonVO = Struct.new(:name, :phone, :email)
p1 = PersonVO.new(:name => "Ola Bini")

will create a new class, assign this to the name PersonVO and then go ahead and create an instance of this class. Creating a new class from scratch and defining a new method on it is as simple as this:

 c = Class.new
c.class_eval do
define_method :foo do
puts "Hello World"
end
end

c.new.foo    # => "Hello World"

Apart from Struct, examples of creating classes on the fly can be found in SOAP4R and Camping. Camping is especially interesting, since it has methods that creates these classes, and you are supposed to inherit your controllers and views from these classes. Much of the interesting functionality in Camping is actually achieved in this way. From the unabridged version:

 def R(*urls); Class.new(R) { meta_def(:urls) { urls } }; end

This makes it possible for you to create controllers like this:

 class View < R '/view/(\d+)'
def get post_id
end
end

You can also create modules in this way, and include them in classes dynamically.

4. Use method_missing to do interesting things

Apart from blocks, method_missing is probably the most powerful feature of Ruby. It's also one that is easy to abuse. Much code can be extremely simplified by good use of method_missing. Some things can be done that aren't even possible without. A good example (also from Camping), is an extension to Hash:

 class Hash
def method_missing(m,*a)
if m.to_s =~ /=$/
  self[$`] = a[0]
elsif a.empty?
  self[m]
else
  raise NoMethodError, "#{m}"
end
end
end

This code makes it possible to use a hash like this:

 x = {'abc' => 123}
x.abc # => 123
x.foo = :baz
x # => {'abc' => 123, 'foo' => :baz}

As you see, if someone calls a method that doesn't exist on hash, it will be searched for in the internal collection. If the method name ends with an =, a value will be set with the key of the method name excluding the equal sign.

Another nice method_missing technique can be found in Markaby. The code I'm referring to makes it possible to emit any XHTML tags possible, with CSS classes added into it. This code:

 body do
h1.header 'Blog'
div.content do
'Hellu'
end
end

will emit this XML:

  <body>
<h1 class="header">Blog</h1>
<div class="content">
Hellu
</div>
</body>

Most of this functionality, especially the CSS class names is created by having a method_missing that sets attributes on self, then returning self again.

5. Dispatch on method-patterns

This is an easy way to achieve extensibility in ways you can't anticipate. For example, I recently created a small framework for validation. The central Validator class will find all methods in self that begin with check_ and call this method, making it very easy to add new checks: just add a new method to the class, or to one instance.

 methods.grep /^check_/ do |m|
self.send m
end

This is really easy, and incredibly powerful. Just look at Test::Unit which uses this method all over the place.

6. Replacing methods

Sometimes a method implementation just doesn't do what you want. Or maybe it only does half of it. The standard Object Oriented Way (tm) is to subclass and override, and then call super. This only works if you have control over the object instantiation for the class in question. This is often not the case, and then subclassing is worthless. To achieve the same functionality, alias the old method and add a new method-definition that calls the old method. Make sure that the previous methods pre- and postconditions are preserved.

 class String
alias_method :original_reverse, :reverse

def reverse
 puts "reversing, please wait..."
 original_reverse
end
end

Also, a twist on this technique is to temporarily alias a method, then returning it to before. For example, you could do something like this:

 def trace(*mths)
add_tracing(*mths) # aliases the methods named, adding tracing
  yield
remove_tracing(*mths) # removes the tracing aliases
end

This example shows a typical way one could code the add_tracing and remove_tracing methods. It depends on singleton_class being available, as per tip #1:

 class Object
  def add_tracing(*mths)
    mths.each do |m|
      singleton_class.send :alias_method, "traced_#{m}", m
      singleton_class.send :define_method, m do |*args|
        $stderr.puts "before #{m}(#{args.inspect})"
        ret = self.send("traced_#{m}", *args)
        $stderr.puts "after #{m} - #{ret.inspect}"
        ret
      end
    end
  end
 
  def remove_tracing(*mths)
    mths.each do |m|
      singleton_class.send :alias_method, m, "traced_#{m}"
    end
  end
end

"abc".add_tracing :reverse

If these methods were added to Module (with a slightly different implementation; see if you can get it working!), you could also add and remove tracing on classes instead of instances.

7. Use NilClass to implement the Introduce Null Object refactoring

In Fowlers Refactorings, the refactoring called Introduce Null Object is for situations where an object could either contain an object, or null, and if it's null it will have a predefined value. A typical exampel would be this:

 name = x.nil? ? "default name" : x.name

Now, the refactoring is based on Java, which is why it recommends to create a subclass of the object in question, that gets set when it should have been null. For example, a NullPerson object will inherit Person, and override name to always return the "default name" string. But, in Ruby we have open classes, which means you can do this:

 def nil.name; "default name"; end
x # => nil
name = x.name # => "default name"

8. Learn the different versions of eval

There are several versions of evaluation primitives in Ruby, and it's important to know the difference between them, and when to use which. The available contestants are eval, instance_eval, module_eval and class_eval. First, class_eval is an alias for module_eval. Second, there's some differences between eval and the others. Most important, eval only takes a string to evaluate, while the other can evaluate a block instead. That means that eval should be your absolutely last way to do anything. It has it's uses but mostly you can get away with just evaluating blocks with instance_eval and module_eval.

Eval will evaluate the string in the current environment, or, if a binding is provided in that environment. (See tip #11).

Instance_eval will evaluate the string or the block in the context of the reveiver. Specifically, this means that self will be set to the receiver while evaluating.

Module_eval will evaluate the string or the block in the context of the module it is called on. This sees much use for defining new methods on modules or singleton classes. The main difference between instance_eval and module_eval lies in where the methods defined will be put. If you use String.instance_eval and do a def foo inside, this will be available as String.foo, but if you do the same thing with module_eval you'll get String.new.foo instead.

Module_eval is almost always what you want. Avoid eval like the plague. Follow these simple rules and you'll be OK.

9. Introspect on instance variables

A trick that Rails uses to make instance variables from the controller available in the view is to introspect on an objects instance variables. This is a grave violation of encapsulation, of course, but can be really handy sometimes. It's easy to do with instance_variables, instance_variable_get and instance_variable_set. To copy all instance_variables from one object to another, you could do it like this:

 from.instance_variables.each do |v|
to.instance_variable_set v, from.instance_variable_get(v)
end

10. Create Procs from blocks and send them around

Materializing a Proc and saving this in variables and sending it around makes many API's very easy to use. This is one of the ways Markaby uses to manage those CSS class definitions. As the pick-axe details, it's easy to turn a block into a Proc:

 def create_proc(&p); p; end
create_proc do
puts "hello"
end       # => #<Proc ...>

Calling it is as easy:

 p.call(*args)

If you want to use the proc for defining methods, you should use lambda to create it, so return and break will behave the way you expect:

 p = lambda { puts "hoho"; return 1 }
define_method(:a, &p)

Remember that method_missing will provide a block if one is given:

 def method_missing(name, *args, &block)
block.call(*args) if block_given?
end

thismethoddoesntexist("abc","cde") do |*args|
p args
end  # => ["abc","cde"]

11. Use binding to control your evaluations

If you do feel the need to really use eval, you should know that you can control what variables are available when doing this. Use the Kernel-method binding to get the Binding-object at the current point. An example:

 def get_b; binding; end
foo = 13
eval("puts foo",get_b) # => NameError: undefined local variable or method `foo' for main:Object

This technique is used in ERb and Rails, among others, to set which instance variables are available. As an example:

 class Holder
def get_b; binding; end
end

h = Holder.new
h.instance_variable_set "@foo", 25
eval("@foo",h.get_b)

Hopefully, some of these tips and techniques have clarified metaprogramming for you. I don't claim to be an expert on either Ruby or Metaprogramming. These are just my humble thoughts on the matter.

Announcing Ducktator - A Duck Type Validator

As I hinted in my last post, I feel with all my heart that there should be some way to actively validate my static expectations on certain kinds of objects. Now, respond_to? and friends are fine, but they do not scale. Not at all. So, I have built Ducktator - a duck type validator. It uses a very recursive, extensible rule syntax. Rules can be specified in either YAML or simple Ruby, with hashes and arrays and all that stuff.

First though, where would that be useful? Not everywhere of course, but these are the places that just drops into my head when writing this: Validating objects that have been serialized or marshallad. Validating what you get when loading YAML files, so that the object graph matches what your code does. Write test cases that expect a complicated object back. The possibilities are many.

Ducktator is very easy to extend. Basically, you just create a method on the validator whose name begins with "check_" and this will be automatically called for all objects. The base library is divided into modules that are mixed-in to the central Validator. I won't detail exact usage here, but just show an example. First, the rule file, which resides in rules.yml:

---
root:
class: Hash
 each_key: {class: String}
 each_value:
  class: Array
   value:
   - - 0
     - class: Symbol
  - - 1
     - class: Integer
     - max: 256

Then, our code to create a Validator from this:

require 'ducktator'
v = Ducktator::from_file('rules.yml')

And lastly, to use it to validate the objects foo and bar:

foo = {'baz' => 13}
bar = {'b1' => [:try1, 130],
      'q16' => [:foobaz, 255]}
v.valid?(foo) # => false
v.valid?(foo,bar) # => false
v.valid?(bar) # => true

Whereto
Now, you'll certainly be wondering where to get this interesting code. As always, it will be found on RubyForge here, and the first release is available through gems, so just gem install ducktator and you should be set to go.

It is licensed with a nice MIT license and I am the project creator, maintainer et al.

YAML needs schema

It has been said before and it needs to be said again. YAML really needs schema. Now, before all your enterprisey warning bells start ringing I want to add that I'm only proposing this for specific applications. Most uses of YAML can continue gladly without any need for schema. But for some cases the security and validation capabilities of a good YAML schema would be invaluable. One example could be for RubyGems. It shouldn't be possible to crash RubyGems with bad YAML. Also, in all cases where Ruby emits objects as YAML it should be possible to automatically generate a schema specification from the object structure. This means that in many cases you may not need to create your schema by hand. You could just serialize your domain objects to YAML, take the schema generated and modify it as needed.

What would the advantages of YAML schema be? Numerous:

• Validation: Validate that a YAML file conforms to your expectations before loading it
• Default values: The possibility to provide default values for missing parts of the YAML, making convention over configuration even more powerful. With reasonable defaults most YAML documents could shrink dramatically in size.
• Tool help: GUI builders and other tools would be able to help you construct your YAML-file from scratch. I like being able to auto-complete XML with nXML in Emacs. Very neat. I just wish I had that capability with yaml-mode too.
• Loading hints and instructions: A schema could specify that the key named 'foo' always has a value with the tag !ruby/object:Gem::Specification or that all integer values should be decimal, regardless of leading zeroes. Many instructions that you at this point need to customize your YAML system to achieve would be automatic.
• Remove clutter from YAML-file: If the schema defines the tags for values, it means that this information doesn't need to appear in the YAML file itself, reducing clutter and noise. This would make it even easier to edit YAML files by hand.

A YAML schema format should be specified in YAML, and it should be self hosting (meaning it's format language should be definable in itself). For most parts it seems we can use ideas from XML Schema. The only part I'm not really sure about for YAML schema is how to bind a document to a schema. Maybe the best way would just be to add a new directive that specifies the schema for that document. I don't believe that YAML needs different schema for different parts of documents right now, though. I don't think we need the proliferation of schema metadata inside the YAML document that XML experiences. (Anyone tried to manually work with a WSDL-file which includes all requires namespaces and such? Nightmare!)

There are a few different parts needed for this to work. I believe it could be done with the current YAML spec (and retrofitted on YAML 1.0 too), since the only real change to the document would be a new directive in the stream header. The next step is that someone starts defining a format for schema. Then, a tool would be needed that could validate against schema. This wouldn't reap us all benefits of schema, but it's a start. The final step would be to integrate schema support in existing YAML libraries, to allow validation and using schema for metadata information.

Actually, this solves exactly half the problem, the part of the problem I call the external validation. The other part is not YAML specific, and it's something I've been thinking about for Ruby. This regards validation of object hierarchies in the current language. Expect some more info on this in one or few days. I want to have something usable to release. But I believe the Ducktator will be really useful for certain use cases.

MetaProgramming Refactoring

Reflexive metaprogramming have been part of programmer consciousness for a long time. It's been possible in many languages in one way or another. Some have embraced it more than other, and among these the most prominent are Lisp, SmallTalk, Python and Ruby. But it's not until Ruby entered the common programmers mind that Metaprogramming actually starts to become common place. The discussion on DSL's is also relevant for metaprogramming issues, since implementing a DSL (in the same language, of course) is very hard without reflexive metaprogramming.

I recently reread my copy of Refactoring, and as usual I was amazed by how on-topic it was, and how easy and useful the tips in it where. But, I also started thinking that there is something missing. Refactoring is specifically about Object Oriented Programming, but I'm heading more and more towards Language Oriented Programming, with DSL's, reflexive metaprogramming, introspection and Meta-class extensions, These approaches make the base OOP system much more powerful. This is also very prominent when prototyping smaller
systems. I find that I start by writing methods in such a way that I have to do very much by hand, and in the next stage I fold my code, as much as possible, both for readability and laziness.

What is this blog about, then? Well, I propose that the time is right and nigh for a catalog of Metaprogramming Refactorings. I'm not saying I should do them, not at all, but it would be something very nice to have. Maybe as a wiki somewhere, where some of our metaprogramming luminaries could write about their experiences. DHH? _why? Weirich? Dave Thomas?

Anyway, just to make it very apparent what kind of refactorings I'm talking about, I will provide a somewhat contrived example. This is more or less what a mock implementation of something could look like. Some log calls, and quite much repetition.

 def startup
   @log.info { "-startup()" }
   self.startup_foo
   self.startup_bar
 end
 
 def init
   @log.info { "-init()" }
   self.init_vars
   self.init_constants
   self.init_other
 end
 
 def main
   @log.info { "-main()" }
   self.run_main
   puts "hello from main"
 end
 
 def close
   @log.info { "-close()" }
   self.close_bar
   self.close_foo
 end
 
 def shutdown
   @log.info { "-shutdown()" }
   self.all_shutdown
   self.run_shutdown
 end

I present the Extract Code Template metarefactoring. The first step is to take all the method names that should be handled and put these in a list, like this:

 [:startup, :init, :main, :close, :shutdown]

Then we walk through these definitions, and provide empty bodies for each, like this:

 [:startup, :init, :main, :close, :shutdown].each do |name|
   define_method(name) do
   end
 end

We then have to change the list to a hash, making each method-name point to the methods to call in the method, like this:

 { :startup => [:startup_foo, :startup_bar], 
   :init => [:init_vars, :init_constants, :init_other], 
   :main => [:run_main], 
   :close => [:close_bar, :close_foo], 
   :shutdown => [:all_shutdown, :run_shutdown] }

When this has been done, we have to add the part of the main method which can't be extracted in the same way, which we do with a proc:

 { :startup => [:startup_foo, :startup_bar], 
   :init => [:init_vars, :init_constants, :init_other], 
   :main => [:run_main, lambda { puts "hello from main" }], 
   :close => [:close_bar, :close_foo], 
   :shutdown => [:all_shutdown, :run_shutdown] }

The next step is to walk through the method names and values, and define the method contents. We then remove the original methods. Finally, our code may look like this:

 { :startup => [:startup_foo, :startup_bar], 
   :init => [:init_vars, :init_constants, :init_other], 
   :main => [:run_main, lambda { puts "hello from main" }], 
   :close => [:close_bar, :close_foo], 
   :shutdown => [:all_shutdown, :run_shutdown] }.each do |name, methods|
   define_method(name) do
     @log.info { "-#{name}()" }
     methods.each do |m|
       if m.is_a? Proc
         m.call
       else
         self.send m
       end
     end
   end
 end

Now, in this case I'm not sure I would do this refactoring at all. This serves more as an example of the kinds of refactorings I would like to see in a catalog like this. Refactorings like Extract DSL, Create Class Dynamically, Extend From Anonymous Class and others. This is something I really feel would be useful in today's programming environment.

Comments and tips are very welcome.

JvYAML and RbYAML - what's to come?

I know I've posted about the next release of JvYAML a few times, but going has gotten tough, since there are so many interesting projects to work at. Anyway, I was thinking that I would blog a bit again, since a significant new feature has reared it's head. It's about YAML 1.0 compatibility. This needs to be added to both JvYAML and RbYAML in fact. The reason is a recent incident where YAML communication between JRuby and Ruby broke in a slightly embarrassing way. Most of you probably know which incident I'm talking about.

The problem is very simple. YAML 1.1 is _almost_ backwards compatible to 1.0, with the exception for a few points. The point that broke is the shorthand tags from the YAML type repository. In YAML 1.0 you could prefix a value with !str and this means it shouldn't be interpolated as another type of value. A typical example (and actually the example that triggered the incident) is this:

version: !str 0.2

Now, in YAML 1.1, it doesn't look the same way. It was decided that a single exclamation point is actually shorthand for the user namespace, while a double exclamation point means the yaml.org:2002-namespace. So, the above example in YAML 1.1 is

version: !!str 0.2

This is a tiny change, but it breaks, since YAML 1.0 handles !!str as a private type, and YAML 1.1 handles !str as a private type. Not a really nice situation.

The solution in my case will be to add a flag for RbYAML and JvYAML that specifies that you want 1.0-compatibility. When that flag is turned on, some of these issues will be handled correctly by the parser, and emitted in a way an 1.0 parser could read. This will be the only change in RbYAML. But JvYAML will contain (as detailed before) an emitter, JavaBean materialization and many bug fixes.

Rails, Databases, ActiveRecord and the path towards damnation

The last two days I've spent some thought on databases and Rails. I haven't gotten far, but I do know that Rails have serious trouble with regard to databases. Some of this thinking comes from David Blacks talk, other parts from DHH's rant after the panel discussion, and some from discussions with other Rubyists and Railites.

So. What are these problems? The first one is MySQL. Now, I don't want to bash MySQL. Not really. But it is not a good database. Until recently it's been very bad on SQL compliance. It's slow. It's cumbersome. The foreign keys are annoyingly incomplete. And some MySQL-extensions have a tendency to be preached as gospel by people who doesn't understand databases. (But I guess this isn't really the fault of MySQL). Actually, the worst part with MySQL is that ActiveRecord have been designed based on it. Now, I really do understand 37signals point of view on this. Of course, if MySQL is good for them, I understand that they have built that support in deep. But this puts the rest of the world using Rails in a tight spot. It is incredibly hard to get other databases working with Rails, and even if you do get them working, it will be slow. Really slow. Take Oracle. Oracle lives and dies by prepared statements. But there is no sane way to do this in Rails. Instead, SQL is generated dynamically and Ruby code is used to quote variables instead of doing this as part of the prepared statement. This is obviously very much painful when doing the JDBC adapter, but it is really important for all serious databases. Having prepared statements would also cut down on much of the database specific code in Active Record, since quoting would be up to the database driver, as it should be.

OK, problem number two. Real world database design. As Black has noticed, this isn't talked about in the Rails community. At all. Of course, as one person in the audience noted, this is partly since using ActiveRecord and designing your objects with care results in 3rd normal form without effort, but this is not the whole story. There are much more important issues in database engineering than normalization, and lets face it, most Rails developers produce pretty crappy databases. This needs to be investigated, talked about, discussed. It needs to come out into the open. This discussion is needed for many reasons. I want to be able to use Rails for all applications where it makes sense. But most of those places won't be possible until the database support doesn't kill Oracle when trying to use it. Or requires a database with no good management tools.

The limits of power: What Lisp can do but Ruby can't

I've for a long time been thinking about where Ruby's limits are, compared to Lisp. As I see Lisp as the ultimate power, this is mostly about trying to gauge the power of Ruby, in a very unscientific, highly opinionated and very subjective way. I talked some with Jim Weirich, _why, Charles and few others about this in London which started me thinking again. Actually, I've only find one macrotype that really isn't possible in Ruby. And it's only impossible if you require that the syntax doesn't change.

First, required reading for this post is Why Ruby is an acceptable Lisp by Erich Kidd. I happen to agree with this, in almost all cases, but there are a few corner cases where Lisp is just more convenient. As a first example, let's take a typical AOP task. I want to define a method to execute before the method foo. The definition of foo is like this:

def foo(arg1, arg2, arg3)
do.something {}
end

And my before-advice, which I would really like to be able to write like this:

defbefore foo(arg1, *args)
puts "before foo with first arg #{arg1}"
end

This isn't possible. The closest I really can come up with is this:

defbefore :foo do |arg1, *args|
puts "before foo with first arg #{arg1}"
end

This isn't so bad, of course. But it puts a disconnect between the language and your extensions. The most powerful macro facility is invisible to the programmer. There should be no division between how keywords work and how macros could function. To take another example of this, take the classical pattern for logging:

logger.debug{"baz: #{expensive_formatting_operation(baz)}"}

where the block is used to avoid calling expensive_formatting_operation if debug-logging isn't turned on. This is neat. But it isn't neat enough. I would like to be able to write

logger.debug "baz: #{expensive_formatting_operation(baz)}"

and avoid having expensive_formatting_operation run if debug-logging is off. In the general case this isn't possible in Ruby. There is parts of the execution process that there are no hooks into.

Of course, as I said in the beginning, this doesn't really matter in most cases. In almost all cases the convenience of Ruby's powerful syntax, emerging libraries and great frameworks is what you get for that small power-tradeoff. But even so, I would like to be able to go that extra distance in power. Could something like this ever be possible in a language that has syntax? I'm not sure. Maybe if there was a well defined way that Ruby translates into something that resembles S-expressions. In that case you could have macros that work on these internal concepts instead of on pure Ruby-syntax. Of course, this means a division into two languages, but it would give that extra power.

RailsConf EU, part 3

Drat. I missed _why's keynote speech. Oh well, I hear they recorded everything in the congress hall, so now I'm just waiting for a podcast of it. What I didn't miss was Jim Weirichs keynote which was really great. It was about some of Ruby's more dangerous features and how to avoid people getting really mad at you when using them in a library. Like, for example changing the semantics of Kernel#require. Or making Logger singleton. And other fun stuff. The tips where quite basic, common sense things that Rubyists really need to think more about. For example; don't pollute the global namespace with classes named Node. Or name your namespace HTML. The other part of the speech focused on Design by Contract in the Bertrand Meyer sense. This was the watered-down version, looking at preconditions on method arguments and postconditions on the return value. I would have liked seeing something more about preconditions, postconditions and invariants on objects too, since this is a really important part of DoC in my opinion. But it was still great.

I had to run before _why's speech and only came back in the middle of the MySQL optimization talk. (The reason for leaving will probably be obvious in one or two weeks time.)

Now, MySQL optimization... Oh wow, I'm twisted. I used to like MySQL. But now I have a really big problem with it, and especially in the Rails crowd. Actually, after David Blacks session my problems with MySQL and databases are so big I really have to spend an entire blog entry for it. Coming up.

Anyway, I can't say much about the session on MySQL exception: "What are you smoking, man, and can I have some of that?" to quote DHH in an unrelated talk. This guy from MySQL actually said some really nasty things about PostgreSQL and DB2 since (in his opinion) they were less SQL compliant than MySQL. Oh wow. Oh wow. I'm really upset about this, so I'll quickly continue writing about the next session.

Oh yeah, a small interlude to all Rubyists out there. Charles pointed something out to me the other day in London and I can't stop thinking about it. What he asked me was this: "Have you ever seen a unit test attached to patches submitted for Ruby-core?". No, I haven't, and it scares me. Really much. Anyone wants to inject an opinion on this matter?

Talking about Charles Nutter; his JRuby talk was next up and boy was it great? I know what it would contain but it still is really nice to see. JMX control over Rails. EJB session beans feeding data into Rails. JDBC database backend (was MySQL, could've been Apache Derby, Mimer SQL or something entirely other). All of this, with such small amounts of code. I really got the impression that the Ruby crowd actually grok why the JVM can give them even more power now. I felt a big change in the climate. No more Java bashing. Things are moving forward.

James Duncan Davidsson ended RailsConf for me, since I had to move towards Heathrow at the moment he finished. He talked about how to scale our applications in a different way, using techniques we can only guess at that Google and Amazon has deployed internally. Big, clustered memory databases and distributed file systems on a scale not really possible for most of us right now. It was interesting and it got me very excited about the close future.

I didn't get to see Dave Thomas either, but I'm counting on the podcasts once again. All in all, it's been two great days. Very intense, very fun and really interesting. It's been great meeting all of these people, especially seeing Charles again, talking to Jim Weirich and why the lucky stiff. It's been great. When's next time?

RailsConf EU, part 2

After watching the Rails speaks C-session, I went to lunch. Got back to see the session on Capistrano. Sadly, it wast most about the new features and not much at all about the basics of Capistrano. If I understood the audience correctly, most were expected something like that.

Next session for the day was Camping: Going of the Rails with Ruby, by Eleanor McHugh. She had a companion with her, but I don't remember his name. Actually, the presentation was really crazy. It was good fun, really crazy, but I'm not sure I actually learned so much. It was fun that she'd included that JRuby runs Camping in her presentation.

Next session was David Black's Database Engineering and Rails, which more or less stated loads of questions and undecided issues regarding databases and Rails. There are many things people aren't talking about, and DB design is one of them. Davids talk was mostly questions and a general idea that we need to do better in this area.

Last session for the day was Ugo Cei's talk about Ruby for Java programmers. As Ugo noted in the beginning, there were no abstracts posted on the schedule, and the title for his talk was slightly confusing because of that. The presentation was about getting Ruby and Java to inter operate, not to compare languages to each other, or learn Java programmers how to code Ruby. As such, I had known this from before but my colleagues didn't, and were pleasantly surprised. The presentation was really good. Ugo did a great job of detailing all different approaches, the pros and cons of each and also showed some demos, including my JRuby Camping. Of course, the demo demon was there, and the RubyGems bug surfaced alot.

Oh, and as a side note, that issue will hopefully soon be fixed. Charles and me talked with Jim Weirich about it yesterday, and we may be close to a solution.

After the days session I went back to my hotel to freshen up a little bit. I then went back to the conference center in time to see the Rails Core Team panel discussion. This was quite interesting, but the question I wanted to have answered didn't make it into the session, since I submitted it to late. A few interesting tidbits of information was given. First, SimplyHelpful made some people afraid there would be div_for-helpers all over the place, giving rise to much overhead in terms of unnecessary CSS classes and ids. This is not the case, since div_for should only be used for special cases where the repetition is apparent. Another interesting issue was the proliferation of globalization and internationalization plugins for Rails. The question was whether the core would some day contain these things. The answer is a clear no, but some things need to be better in the core, for example making strings available to change.

The question I wanted to have answered was if ActiveRecord someday will be refactored or rewritten. There are two reasons I see for doing this; to remove the MySQL-specific assumptions from the AR core, and to make it possible to have Prepared Statements. Alas, I didn't get answers to this.

After that, DHH ended the session by spending 20 minutes ranting about the recent security issue and more interesting; the communities reaction to this issue. His core point is simple: he doesn't owe anyone anything just for downloading and using Rails. If you have contributed to the community or Rails that's another thing entirely; that means you're vested in the framework and entitled to some information. It was really interesting. I hadn't realized how irritated DHH had become by those discussions.

Oh well. Now it's time for Jim Weirich and Why The Lucky Stiff to hold a general session. It's bound to be interesting. More reports later.

RailsConf EU, part 1

About ten minutes ago, Kathy Sierra finished her part of the plenary session for RailsConf. She was preceded by DHH and they talked for about an hour each. Of course, DHH's talk was quite technical and mostly about Rails 1.2 and 2.0. Let me tell you, there are some really awesome stuff there. I really liked SimplyHelpful which is smart helpers for your view, mostly like ActiveRecord for models and SimplyRestful for controllers. It makes repetitions disappear in a very pleasing way. There were of course lots of other small things talked about. ActiveResource looks really nice. I just wish I had an interface like that for the SOAP servers I work against. Oh well.

Kathy Sierra. Wow. Really interesting talk. Incredible actually. She talked about creating passion in users, how you can work with selling your product to people who are already customers in such a way that they become passionate and really make your product advertise itself. She talked much about how our brains work regarding these topics and how you can "cheat" to grab attention. A very telling point in her speech was when she showed the Canon advertisement for one of their cameras, and then their manual for those who had actually bought the camera. Of course, the manual was sleep inducing and probably bad for your health just to look at, but the advertisement was cool, "sexy" and all those ad buzzwords.

The current session I'm in is about Rails and C, and how you can apply C extensions to make your applications blazing fast. This is a really appropriate topic if you look at the current war between Joel Spolsky (Language Wars, Wasabi, Ruby Performance) and DHH (FUD, Wasabi and Performance). Also, the last few months have seen many discussions on ruby-talk about performance (or lack of) in Ruby. David Goodlad starts the session by speaking about different ways of interfacing with C. The ways detailed are regular C extensions, Ruby/DL, RubyInline and SWIG.
An interesting take on the Rails issue, and how to get great performance from one server running mostly C code and the then have your 5 Rails boxes talking to this with BackgrounDRb. Really neat.

London arrival

My London arrival yesterday was really abysmal. Our flight was two hours late and when we finally arrived at our hotel (Russell Hotel: never live there), they told us one of our five reserved room was damp. So, we told her to fix another hotel close to the conference center. At that point she gave our rooms to somebody else. As if that was not enough, they were incredibly rude and tried to blame "management" for what had happen. A management that conveniently wasn't around and would talk to us on phone.

Finally, after two hours waiting, we got rooms at Hilton Waldorf. When I finally got to bed the clock was 3am local time and I was to stressed out to get any restful sleep.

The good part of the sleeplessness is that I had time to do some hacking. I've implemented support for Mimer SQL in ActiveRecord-JDBC and also fixed a small error that Peter Chan found while trying it out.

For those who don't know about it, I really recommend Mimer. It's a great database, really fast and extremely focused on complete SQL compliance in all ways.

I'm writing this a few minutes after the first plenary session for RailsConf was supposed to start. This will be interesting. I have already said hi to Charles, and it looks to be two interesting days. Envy me! =)

My London Schedule

Tomorrow evening (Wednesday), me and my colleagues will land in London for RailsConfEU. It will be a hectic time but I'm counting on many interesting presentations. I will post my current schedule here, so people interested in saying hi can know where I will be.

• Arrival: We land at Heathrow airport 20:15 GMT. This means I will not be at Pizza on Rails (sadly enough).
• Thursday: RailsConf starts. I'm planning on attending sessions all day from 8:30 to 20:30. I'm not sure exactly which ones yet, but there are many interesting ones (and some conflicting). I plan on looking at David Black's talk on Database Engineering and Eleanor McHugh's session on Camping (since I have a special interest in Camping). Ugo Cei's talk on Ruby for Java will be most interesting too.
• Thursday evening: A friend will show me the night life in London.
• Friday morning: I'm planning on going to the plenary session but I won't go to the first sessions after that.
• Friday lunch time: Lunch and an hour of sightseeing with Meg.
• Friday afternoon: I'm not sure which sessions to go to. I'm a little bit mad about the planning that conflicts the four most interesting sessions at the same time. I'll go see Charlies JRuby-presentation, though.
• Friday evening: We fly from Heathrow at 20:00 GMT, so, once again, we won't be able to go to the JRuby meet up.

First the Sun, then the world

As has been announced at various places today, Sun has made it official that they hire Charles O Nutter and Thomas Enebo to work full time on JRuby and Ruby development tools. This is obviously great news for JRuby fans, Ruby evangelists and Java developers. Because this news is something we all will benefit from.

So, what does this mean to JRuby, Ruby and Java? Well, JRuby will gain even more traction. It's fair to say that the next few months probably will see an explosion of fixes to JRuby correctness and performance. I'm willing to bet money that JRuby will run Rails as fast as MRI by New Years. Of course, this is something that the Ruby language will benefit greatly from. JRuby won't be a competitive Ruby implementation; it will be a complementary one where the resources of Java will be available to systems that need the agile development process of Ruby. Other clear benefits is the possibility of creating a real Ruby language specification and test suite.
For Java the issue isn't as clear cut. Java the language will probably lose developers to Ruby. But they won't move from the Java platform; and this is the important point. It's the platform that counts; and Ruby with the power of the Java platform will be something incredible to behold.

I guess the next question for me and other JRuby developers will be what next? Should we lie down on the couch, take a rest and let the big guys take care of JRuby from now on? Of course not. Actually, the situation is the complete inverse. It is now we should make an extra effort. It is now that we are needed. Two great developers working full time will not be enough. It will never be enough. This is really a call to arms. Now is the time for all those developers that I know are out there, thinking about pitching in, but haven't decided to do it yet. Now is the time to help. There are test cases to be written, documentation is needed. There are core changes that need to be implemented. There are extensions to write, and applications to test. We have one gem released on RubyForge that demands JRuby. In a years time there should be one hundred!

ActiveRecord-JDBC 0.2.0

Earlier today I finally released version 0.2.0 of ActiveRecord-JDBC. Since it contains fairly big changes I decided to bump the version number a bit, from 0.0.1 to 0.2.0. This blog entry will first describe what's new and cool about this release, and then show how to get started with this adapter inside Rails.

What's new
Of course, to warrant a new release, there's got to be some good stuff in it, and I can really say I'm proud of the funcionality included in this package. I feel that the first release was much proof-of-concept, and to make it easy for people to follow along in the Camping tutorial. But with this release, ActiveRecord-JDBC is definitely an option for a real project.

ActiveRecord-JDBC (AR-JDBC from now on) ships with support for 8 databases, including migrations. Adding support for more is often very trivial and we hope to very soon have support for most major databases. What works right now is:

• MySQL
• PostgreSQL
• Oracle
• HSQLDB
  
• Microsoft SQL Server (except for change_column_default)
• DB2 (except change_column, change_column_default, rename_column, remove_column,
  add_index, remove_index and rename_table)
• Derby (except change_column, change_column_default, remove_column, rename_column)
• FireBird (except change_column_default and rename_column)

The exceptions in this list will probably not be fixed, since there is no support in the database for this functionality. Of course, some of it could be faked by expensive copy-table operations, but this seem to be detrimental since it often doesn't work for sequences and identity columns. We felt it was better to just don't support renaming tables in those drivers.

Putting JDBC on Rails
Now, I will detail how to create a basic Rails application using JRuby and AR-JDBC. I assume you have a working JRuby installation, and have installed Rails. Since I have rails and gem in my global path from regular Ruby, I have added symlinks in $JRUBY_HOME/bin to make it easier using these commands. Specifically, I will use jem for gem, jails for rails and jake for rake. This makes the command lines very much shorter indeed.

The first step is to install the AR-JDBC gem: (we don't generate rdoc and ri since these are still expensive operations in JRuby)

jem install -y ActiveRecord-JDBC --no-ri --no-rdoc

After this operation has completed, the next step is to create our rails-application:

jails test_arj

This will take a while and generate the usual files. Now, to actually make Rails use the JDBC-connector, we need to change environment.rb to load our jdbc_adapter. There's a thread on the rails-core-list about making this step unnecessary, but right now it's needed. Anyway, modify the head of the file to look like this:

require File.join(File.dirname(__FILE__), 'boot')
require 'jdbc_adapter'
Rails::Initializer.run do |config|

that is, add a require for the jdbc adapter. The next step is to configure your database. I will the Derby embedded driver for this, but the concept is the same; just change the adapter name, add driver, url and possibly username and password. This is the configuration any JDBC driver takes so it shouldn't come as a surprise for Java developers. Make sure the driver is on your CLASSPATH before running anything, though. My development-database specification looks like this:

adapter: jdbc
driver: org.apache.derby.jdbc.EmbeddedDriver
url: jdbc:derby:test_arjdbc;create=true

and you see here that there isn't anything strange going on, we just configure the JDBC driver with the same parameters as usual.

From now on, the operations are the same as when creating a regular Rails application. First, create the model:

jruby script/generate model Product

and then edit db/migrate/001_create_products.rb by adding this:

t.column :title, :string
t.column :description, :text
t.column :image_url, :string
t.column :price, :float

Nothing strange. When this is done, we can create our data:

jake db:migrate

which, since we've chosen to use Derby, will create some database files in the current directory. This is what I want for now. After this step, some scaffolding is in place:

jruby script/generate scaffold product

You will notice we have to use jruby explicitly for some scripts. This is because the shebang doesn't get set correctly for these files.

After this step I copied my development configuration to the production configuration in the database and started webrick in production mode (since it's much faster right now):

jruby script/server -e production

Now you should be able to visit http://localhost:3000/products and going wild crudding your products.

... And the rest? There is nothing more. From here on, it's a regular Rails application, just with the added benefit of all the Java resources in the world available to you.