måndag, september 18, 2006

MetaProgramming Refactoring

Reflexive metaprogramming have been part of programmer consciousness for a long time. It's been possible in many languages in one way or another. Some have embraced it more than other, and among these the most prominent are Lisp, SmallTalk, Python and Ruby. But it's not until Ruby entered the common programmers mind that Metaprogramming actually starts to become common place. The discussion on DSL's is also relevant for metaprogramming issues, since implementing a DSL (in the same language, of course) is very hard without reflexive metaprogramming.

I recently reread my copy of Refactoring, and as usual I was amazed by how on-topic it was, and how easy and useful the tips in it where. But, I also started thinking that there is something missing. Refactoring is specifically about Object Oriented Programming, but I'm heading more and more towards Language Oriented Programming, with DSL's, reflexive metaprogramming, introspection and Meta-class extensions, These approaches make the base OOP system much more powerful. This is also very prominent when prototyping smaller
systems. I find that I start by writing methods in such a way that I have to do very much by hand, and in the next stage I fold my code, as much as possible, both for readability and laziness.

What is this blog about, then? Well, I propose that the time is right and nigh for a catalog of Metaprogramming Refactorings. I'm not saying I should do them, not at all, but it would be something very nice to have. Maybe as a wiki somewhere, where some of our metaprogramming luminaries could write about their experiences. DHH? _why? Weirich? Dave Thomas?

Anyway, just to make it very apparent what kind of refactorings I'm talking about, I will provide a somewhat contrived example. This is more or less what a mock implementation of something could look like. Some log calls, and quite much repetition.

def startup
@log.info { "-startup()" }
self.startup_foo
self.startup_bar
end

def init
@log.info { "-init()" }
self.init_vars
self.init_constants
self.init_other
end

def main
@log.info { "-main()" }
self.run_main
puts "hello from main"
end

def close
@log.info { "-close()" }
self.close_bar
self.close_foo
end

def shutdown
@log.info { "-shutdown()" }
self.all_shutdown
self.run_shutdown
end

I present the Extract Code Template metarefactoring. The first step is to take all the method names that should be handled and put these in a list, like this:

[:startup, :init, :main, :close, :shutdown]

Then we walk through these definitions, and provide empty bodies for each, like this:

[:startup, :init, :main, :close, :shutdown].each do |name|
define_method(name) do
end
end

We then have to change the list to a hash, making each method-name point to the methods to call in the method, like this:

{ :startup => [:startup_foo, :startup_bar],
:init => [:init_vars, :init_constants, :init_other],
:main => [:run_main],
:close => [:close_bar, :close_foo],
:shutdown => [:all_shutdown, :run_shutdown] }

When this has been done, we have to add the part of the main method which can't be extracted in the same way, which we do with a proc:

{ :startup => [:startup_foo, :startup_bar],
:init => [:init_vars, :init_constants, :init_other],
:main => [:run_main, lambda { puts "hello from main" }],
:close => [:close_bar, :close_foo],
:shutdown => [:all_shutdown, :run_shutdown] }

The next step is to walk through the method names and values, and define the method contents. We then remove the original methods. Finally, our code may look like this:

{ :startup => [:startup_foo, :startup_bar],
:init => [:init_vars, :init_constants, :init_other],
:main => [:run_main, lambda { puts "hello from main" }],
:close => [:close_bar, :close_foo],
:shutdown => [:all_shutdown, :run_shutdown] }.each do |name, methods|
define_method(name) do
@log.info { "-#{name}()" }
methods.each do |m|
if m.is_a? Proc
m.call
else
self.send m
end
end
end
end

Now, in this case I'm not sure I would do this refactoring at all. This serves more as an example of the kinds of refactorings I would like to see in a catalog like this. Refactorings like Extract DSL, Create Class Dynamically, Extend From Anonymous Class and others. This is something I really feel would be useful in today's programming environment.

Comments and tips are very welcome.

17 kommentarer:

S sa...

Hi Ola, I am new to Ruby and metaprogramming. your post is very interesting. thanks for posting it
-SPuppala

toby sa...

Ola,

I have had the same thought a while ago as well. One of the reasons rails is so good is the use of metaprogramming to make the "user code" as minimal as possible. I think this is where Ruby excels, in making metaprogramming accesible. I made a short list of such patterns a while ago, and am trying to remember them...
1. macros - I think this is the most commonly used pattern that I see in the ruby community right now. Basically, you are wrapping up some metaprogramming code inside a class method or instance method. Examples are the rails methods: scafford, has_many, belongs_to etc. A lot of times these methods would take in a symbol or a string, and would generate a set of methods based on the symbol or string.
2. DSL (domain specific languages) - this is a really cool thing you can do in ruby. Basically, you make your ruby code look like another language, and this is possible because of the flexible syntax of Ruby. Technically, this is called internal DSL, since you are really writting ruby code at the end of the day, you don't need to write another parser and compiler. As an example, you could write SQL code like this:
q = sql {
select name, age, height
from patient, doctors
where (age > min_age).and(
name.like(first_name + '%')).or(
name.in([1, 2, 3])).and(
age == height)
order_by age
}
This is valid ruby, but it looks a lot like SQL. It also has the advantage over normal DSL's in that you can use the full power of ruby inside your DSL.
3. inter-language integration - when you integrate ruby with another language, it helps if the other language has reflection-like capabilities, because then you can automatically generate the ruby integration layer. Two examples are SQL, Webservices. ActiveRecord uses this pattern with respect to SQL. It basically scans the database tables, and uses the naming conventions to generate all the ruby attributes and finder and save methods for you, so that there is no configuration required. For webservices there's a similar library. Basically you write one line of code:
stub = SOAP::WSDLDriverFactory.new(wsdlLocation).create_rpc_driver
in order to get the stub given only the url of the webservice, and then you can go to work. No code generation needed.

That's all I can think of right now. I am a big fan of metaprogramming and I think it's getting more interest as ruby is getting into the mainstream.

futuremint sa...

Would a simpler way of re-factoring those methods be to just make one parameterized method that can optionally yield a block inside of it? If these methods are already used and you don't want to change the interface then this meta re-factoring might be helpful, and I've used a similar style to this on occasion, but to me it seems a little bit like there is some simpler way to do that.

Maybe it is just a bad problem used to illustrate the concept?

Ola Bini sa...

Futuremint!

Yes, as I say at the end, this is probably not the best example to showcase this kind of metaprogramming. Which is why I will not write the book. =) Someone who's better at coming up with good examples should do it. Martin Fowler, anyone?

Ryan Bates sa...

I love Refactoring, and I find the concept of mixing Refactoring and DSL very intriguing. I too would be interested in a catalog like this with examples.

One thing that I am coming to realize more and more is the importance of readability. IMO, the end result of your refactoring example is much more difficult to read than the original.

You have already mentioned the example is not the greatest, but I think this is a problem that every DSL faces. Yes it may still be Ruby, but it is interpreted in a different way to the point that it's almost like learning a completely new language. This comes with a learning curve. If the user has not learned this "new language", the readability is greatly hindered.

I believe this is why creating a DSL is so difficult and why you don't see many examples or refactorings of the process.

For a DSL to be successful, the benefits it brings should far outweigh the cost of the learning curve. It seems more often than not the DSL's I see do not meet up to this.

abc sa...

I'm tired of Ruby folks acting like there language is somehow special. You can do all of this in Perl.

Anonym sa...

This isn't meta-programming. Had you done this in a functional language or even python no one who knew what they were talking about would ever call it meta-programming.

Ola Bini sa...

I'm tired of Ruby folks acting like there language is somehow special. You can do all of this in Perl.

This can be done in Lisp too. Has been possible for many years before both Perl and Ruby. The point is that Ruby is getting so much traction for (so called) real programming which means that metaprogramming is now accessible to the masses, as I argue it wasn't when confined to Perl and Lisp

Ola Bini sa...

This isn't meta-programming. Had you done this in a functional language or even python no one who knew what they were talking about would ever call it meta-programming.

We are defining methods on the fly. How much more meta do you need? I would say that those techniqes in functional languages (and python) also merit the name meta-programming.

Anonym sa...

(different anon from above)

wtf is "real programming"? Google is not enlightening me, except providing me with a million pages of people saying "X isn't a -real- programming language, because it doesn't have what -Y- does." You're not saying that, you are saying X has what Y has, but it made it real.

chromatic sa...

I'm not sure what you mean by "the masses". Are you suggesting that there are a few million Ruby programmers somewhere, happily doing metaprogramming? Or are you suggesting that Ruby makes these complex concepts so easy that anyone who found metaprogramming difficult with another language will understand it much better?

(A third option, I suppose, is that you didn't understand it until you learned Ruby, but projecting that to "the masses" may be an unfair argument.)

autocrat sa...

More ruby hype.

weeeee.... let's hop on the band-wagon!

What matters is that we look cool!

Anonym sa...

Right. Let's flame ruby because it's the hip thing to do right now. Nobody can be cool and use a popular language at the same time, right?

Get over the fact that the content of the article wasn't "Ruby rocks, [your language of choice] sucks". The point of the article was general enough, it just happened to use ruby for the example.

Sheez.

LanguageWarsAreEvil sa...

Yes this article did choose ruby as its , but it (wrongly) states that this type of functionality was not availble until ruby came along. I am a long time perl programmer, and recently, .NET. I love ruby, and I think that it is awesome. That said, don't get so upset when people (correctly) point out that lisp, ml, scheme, perl, and many other languages have supported this functionality for some time. They are right.
Look, no one is saying that ruby sucks. It is a great language, for many tasks. But ruby didn't invent anything, it merely took the cool features of many languages, learned from some mistakes other languages have made, and combined these features to make one slick language. That said, ruby is not well suited for all tasks. Every language has a purpose (even if it is just to prove that something can be done!). Ruby excels at a bunch of tasks, but falls short at others. Ruby (like all other interpreted, vm, etc languages) will be slower than compiled c or c++ in general for most tasks, and there will be a huge difference in others. The key is that ruby makes some tasks quite simple, and has a nice syntax (my opinion). However, with all languages, ruby's great features also cause it to be unsuitable for other tasks. For example, 'convention over configuration' may be great for getting something out quick, but there is a learning curve associated with it. In addition, once you step out of what ruby provides out of the box (say, you need to do things differently than its default implementation), then you are in the same boat as with a language without built-in support for those tasks; You have to write code! No language is perfect. If one were, than there wouldn't be any others. There is a reason to use a given language for a given project or task. All programmers don't face the same tasks. So don't get upset when someone prefers a different language. Unless ruby is the only language you know, you'll understand this, and realize that the languages you previously were using still have their place.
There is a tendency to look at the new shiny object and think that everything else is junk, especially when there is so much hype surrounding a new technology. Take XML, for example: to be sure, XML is a great thing, and is very useful. It did not, however, obsolete every other data representation, nor did xml/xsl take the place of html, or databases, as was widely predicted. A similar hype surrounded the release of Java. Yet people still use C. Java is great, for certain needs, but it didn't replace C/C++, nor did it replace DHTML or flash. As a matter of fact, due to the explosion of languages and technologies, a developer now must usually be proficient in many tools and languages (especially true of web developers, and SOA devs). The reason is that existing technologies continue to be viable, and in many cases, are better for tasks than a newer tech. Just look at the 'web 2.0' hype. It boils down to javascript, and out of band (xmlhttp) requests, which have been around for years, only now, it's called AJAX, and is all the rage.
If you are a ruby dev, and you believe that because of ruby's data support you won't have to learn any javascript, or SQL, you are in for a surprise when you try to use it for a real application, rather than a canned demo. You may have to do all of your data code explicitly, because your company requires stored procedures to access the database. You probably can't just create your tables however you want, you'll have a DB team that does that. In these cases, you'll find ruby's 'automagic' O-R mapping leaves something to be desired. On top of that, since ruby is relatively new (in the US, and english speaking countries in general), there will be many growing pains as new problems (both bugs and performance issues) surface, and on top of that, we all know that ruby isn't the best documented language in the world (unless you happen to speak japanese). The features that make ruby so appealing to beginners also cause problems with its adoption in the workplace because of lack of documentation, and proof that it can handle large scale/high performance application development. Users who are afraid to learn javascript and sql, will be lured in by ruby's language level support for these features (especially with rails). However, users who are afraid to learn new (or old) complex technologies generally aren't the type who can deal with poor documentation. Experienced perl (for which there is so much documentation and examples, as well as paid support) developers have no problems looking at the source code (of perl) to learn something they can't find in documentation, I doubt that will be the case for ruby devs (at least until it has been put through the paces and is embraced by these same developers).
Ruby is cool, and I believe that it has its place, but it is naive to believe that this one language came up with all of this stuff on its own. Like all great things, it was built on the technology that came before it. It is also an infant as languages go, so show some respect to devs of mature languages, and don't diss all other languages because you have a shiny new toy.

Ola Bini sa...

LanguageWarsAreEvil:
I'm pretty sure that I never said anything about metaprogramming being new for Ruby. Actually, let me quote the first two sentences: "Reflexive metaprogramming have been part of programmer consciousness for a long time. It's been possible in many languages in one way or another. Some have embraced it more than other, and among these the most prominent are Lisp, SmallTalk, Python and Ruby"

Does this seem like I say that metaprogramming is new in Ruby?

chromatic sa...

Ola, it was the very next sentence (the fourth in this post) that struck me badly. The (so far, unsupported) assertion that Ruby has entered the common programmer's mind is only half of the problem.

matte sa...

Thanks for your interesting article. I've just translated the article in italian on my blog, Rails on the Road. I'm also part of the Rubylation Network, a community envolves on translating ruby article in many languages!