JavaOne 2008, the other half

So, the Thursday got a late start. For some strange reason I didn't feel motivated to go see the Intel General Session, so I showed up for Nick's session about JRuby on Rails deployment instead. Nick did a good job of outlining both the problem and the solution, and I have to say that this presentation was a good end cap for the JRuby week at JavaOne.

I had planned to go to some sessions after that, but I ended up hacking on JRuby instead. An interesting parser bug reared its head.

So the next session I went to was the Filthy Rich Clients one. Quite entertaining, although my interest in Swing is not what it used to be.

The final session of the day was the BOF about writing great technology books. This proved highly enjoyable since joining Josh Bloch and Brian Goetz as panelists were Burt Bates and Kathy Sierra. They did a wonderful job talking about how to write books that captures the readers attention and how to correctly use the brains weaknesses against it. I am tempted to say that this was the best session of the whole JavaOne. Brilliant.

On the Friday I was up early and sat in on Goslings Toy Story. Always funny, and some cool things there. For a geek like me, the CERN stuff and jMars was especially cool.

I managed to see quite a lot of sessions during the rest of the day. More Effective Java was useful as always, the Maxine Virtual Machine looks really cool, Neal Ford did an excellent job of comparing JRuby and Groovy, highlighting both the differences and similarities between the two. Finally the Jython session talked about some of the implementation challenges we in the JRuby team have wrestled with too, implementing a highly dynamic language on top of the JVM.

All in all, this JavaOne definitely stood out as the first non-Java-language JavaOne for me. And I didn't even attend a single one of the gazillions of JavaFX presentations. It was a good year in general, and specifically for dynamic languages.

Oh, and my book is the 5th bestseller in the conference bookstore. Yay!

JavaOne halfway point

To say that I am seriously tired of JavaFX at this point, would be a gross understatement. So let's not even go there.

CommunityOne was a nice event. I like the feeling of it more and more, and the new open spaces approach seemed to be really successful. The alternative languages presentations were well attended and good. I recommend anyone in the Bay Area, or anyone attending JavaOne, to make the effort to go to CommunityOne next year. It's definitely worth it.

Tuesday was, for undisclosed reasons, a day where I didn't attend so many presentations. In fact, I missed both the keynote and Tom&Charlies JRuby talk. Bad on me. I did manage to go to both the technical session and the BOF about upcoming language features in Java 7. Let me say immediately that I don't want pluggable type systems as a part of Java. Yes, they are useful in certain settings, but they don't fit Java. Not at all. I'm all for having it possible to have annotations in more places, but not for type systems.

Wednesday was "my" day of the conference. Started out early with the Script Bowl, where Groovy, JRuby, Jython and Scala faced off in three different challenges. I was one of the three judges. It was a actually a great fun. Some people have gotten annoyed at it, but I feel that they are taking it too seriously. This format was a good way of introducing the audience to the capabilities of four languages that are sometimes very much alike and sometimes very different.

It's also interesting to note that Charles was the only one of the panel who solved his challenges by asking the community for help with it. In fact, Charles didn't do any of them himself, while the other three did their code in isolation. My opinion is that this shows a difference in attitude between the communities, and it's very interesting.

After that I took it easy for some time, and then it was time for my JRuby on Rails presentation. It went fairly well and was also well attended. I did a serious mistake in my database configuration, but Tom helped me out and the rest of the demonstrations was good.

I did a book signing session after that and then headed back to the office for a while before coming back and holding my JRuby at ThoughtWorks BOF which was also nice.

Today I'm going to take it easy and relax. I'll go to Nick's session in an hour and then I'll go to Josh and Brian's BOF tonight. After that we host a party which I'm going to attend for a while. It's going to be a nice day.

Faulty routes on MacOS X

This has been driving me insane, but I haven't had time to actually try to fix the problem until now, so this is a small writeup about the symptom and what you can do about it. Sadly, it's kinda a hard problem to search for, unless you already knows whats wrong.

This was my problem: At work, accessing LiveJournal worked fine, but at home it didn't. Basically I got a Server Not Found from Firefox when trying to access from home. After a while I realized that the distinction wasn't between home/work, but wireless or wirebound network. See, at work I usually use a wire to connect, but I only use wireless at home.

My first shot at a solution was something that solved a similar problem a while back - namely flushing the DNS cache. Of course, doing an nslookup and verifying it told me that the cache wasn't the problem, but I tried anyway, because MacOSX sometimes plays funny tricks with this cache. So, to do this, just do "sudo lookupd -flushcache" in a console window, and then restart the network interface.

I tried to figure out what was wrong by pinging, and doing traceroute. Both of these gave clues that something was wrong. Traceroute gave me "Can't assign requested address" from the bind system call. That finally made me realize that the route tables for the network interface was screwed. Lo and behold, doing a "sudo route flush" and restarting the interface solved my problems. Apparently there were about 10 faulty routes stuck in the cache, and for some reason MacOS X didn't flush this cache automatically when connecting to a different wireless network. Highly annoying, but it's finally solved.

Just add scaling!

(To protect the innocent, all names of programming languages have been changed. All likeness with existing names is purely for dramatic effect.)

I have heard many times now that LRM doesn't scale. People have been telling it to me in all these different circumstances. They tell me that languages like Deep Throttle and Moulder have scaling but LRM doesn't. Even the Scally language has scaling - you could understand that from the name I guess. So why doesn't LRM has this feature?

Since I'm rather fond of LRM, and really don't want to go back to either Deep Throttle or Moulder, I decided to add scaling to LRM. I mean, how hard could it be if all these other languages have it? I have implemented langauges before, so this should just be a case of finding the correct implementation of the feature, and grafting it onto LRM's code base, and then provide a patch to the mailing lists.

Since I knew LRM didn't have scaling, I decided to investigate other languages that have it. I began with the original language, the one people write operating systems in. Obviously a language like Deep Throttle has scaling. So I started reading the books, looking through the syntax. But I couldn't find anything there. What was I missing? Was there some obscure syntactic element that wasn't documented, that provided scaling? Or was it hidden somewhere? Maybe the pointers were in reality another word for scaling? But no, it didn't seem that way. I decided to break open GDPC, the Gnu Deep Throttle Compiler, and see if this scaling thing was actually something the compiler did autoamtically for you - a bit like register allocation. But I didn't find any scaling allocation. There was lots of other neat stuff there, but all of it was quite mundane translations from the Deep Throttle code into machine code.

Feeling utterly dejected I turned my attention to Moulder. I assume that you all are familiar with the language - it's used in many places around the world and provides lots of really advanced features such as garbage collection, a virtual machine, and yes - scaling! Of course, this is where it would be. Maybe Deep Throttle had hidden the feature somewhere obscure, but I was sure Moulder would make it easier to find. I started out with the syntax again, looking through the things you could do with the language. Interestingly, Moulder is a really small language, when you get down to it. It's small but you need to say lots of things to get anything done. I couldn't find the syntax for scaling, so I started entertaining other ideas. Maybe all those extra words "protected final static Foo" was actually necessary. Maybe the scaling happened in the spaces between the words, or in some strange interaction between them? The only way to find out was to crack open the compiler and look what was happening. In this case it was actually easier, because the compiler did even less than Deep Throttle's compiler. Frankly, the Moulder compiler just output something bytecodes. There were no hidden scaling inlining in the Moulder compiler.

You see where this is heading? Right, I had to investigate the bytecodes that Moulder use. Where is that precious scaling bytecode? I assume it would be called something like invoke_scaling or maybe ldscaled. But nothing. There was no scaling bytecode. All of the existing bytecodes just did the same mundane things that I know LRM already can do. I was now feeling ready to give up, but I came up with one final idea. Scally is famous for being a language with scaling. It's creators even named it after that feature. So if both Moulder and Scally has scaling, maybe it's somewhere in Moulders gigantic standard library?

But guess what? I didn't find it. I still haven't found anyone who knows how you implement Scaling in a language, so I guess that LRM will never have it... Anyone who care to enlighten me, please send me a detailed email with an implementation of Scaling. I really feel the need to know how this thing works.

The Twitter Conspiracy

First, let me warn you. This is a conspiracy theory. It's got all the usual logical fallacies and problems of a conspiracy theory. Add to that the fact that it's almost guaranteed to not be true, you might ask why I'm writing about it. Well, the thing is, if I was in charge of Twitter, I would do something like this. Of course, I'm not in charge, and the people in charge are probably more sane and well adjusted persons than me. But still, who knows?

What is the idea then? Well, it's actually really simple. Think about the Twitter network, the kind of people who connect there, and the way things spread. What is the difference between Twitter and mobile texting for example? First, everything is by default multicast. It's not reciprocal - you don't need to know how many hundreds or thousands read what you write. And you don't care how many others read the persons you read. You are restricted in length. And, the whole thing is open enough that you can follow all the tweets going on in the system.

The characteristics I've described means that Twitter is more or less the ideal memetic engine. What I mean by this is that it's a wonderful way to spread your ideas, if you can express them in a concise and readable way. This means that certain memes doesn't work well in this setting, but most do. And you can convince more people to join, because if your tweets are interesting enough, someone will notice them in the all-tweet. Also, you can see who the people you are interested in follows, which means that you can spread your network selectively, but really quickly.

These are not really part of the theory. They are just the axioms. So what's the theory then? Well, what are Twitter doing with all this data? If I would have been them, I would have used it to do research on memetic spread and viral marketing. I would use it to try out ideas based on how good uptake they have. Finding this information is not really hard when you have control over all the messages happening. In fact, you could actually do it even outside of Twitter, by using the published tools correctly.

What got me thinking along these lines? Well, the whole TechCrunch debacle was the thing that triggered the idea. How would it work in practice? Well, first, the Twitter gang couldn't necessarily know what kind of people would take up Twitter the most, so the cultural fit of Twitter is actually mostly self organizing. The people and groups taking part of twitter select themself for this experiment. Now, of course there are lots of overlapping groups, and that gives even more interesting possibilities for the sociodynamics of meme transfer between non-overlapping social circles.

Take the Ruby people, who have a quite significant presence on Twitter. They are one of the test groups in my theory, and the TechCrunch article was a very directed way of inserting a meme and see how fast and to how many it propagated. It was very easy to insert this into the blog-sphere, since Twitter could have had any amount of people "leak" the information in the TechCrunch article. Once it was out, they just needed to set up some suitable filters and follow the spread. They also inserted a couonter meme, through one of their employees, to see if it work out as an "antidote" to the first meme, or which one of them was stronger. All in all, I think they got enough material for several research articles out of this stunt.

OK, so really, you don't need to grasp for conspiracy theories to explain the TC debacle. It's not necessary, so Occam's razor demands that we choose the simplest available conclusion that explains all the facts. This theory does not fall into this category. But it's still an entertaining notion.

And I predict that sooner or later, someone will use Twitter, or another network like it, to do this kind of research. It's a question of time. This kind of information is way to valuable for marketing purposes and also for the understanding of the human mind, that it will happen. The question is: will you know about it, when you're participating in their research?

Let's not forget the lovely recursive interpretation that this blog post is a way of doing the same kind of research I've just described.

RbYAML in Google Summer of Code

Great news for all Ruby implementations around. A project to bring RbYAML up-to-date and perform better has been accepted for Google Summer of Code. Long Sun is the name of the student, and me and Xue Yong Zhi will jointly mentor this effort.

In fact, I'm very excited about this news. RbYAML was an incredibly important piece of the puzzle to get JRuby to finally work with RubyGems, and that kickstarted our possibilities to start testing numerous other applications. I soon ported RbYAML to Java, and created the JvYAML and JvYAMLb projects, to get better efficiency. Sadly, this left RbYAML without any TLC. That changed a while back when Rubinius picked up the project to get their YAML support going, and now that Long Sun will work on it, hopefully we will finally get an extremely compliant and bug free YAML implementation for Ruby.

This will obviously benefit Rubinius, but it will also be very good for both JRuby and IronRuby. The work will be test-driven which means a more complete test suite will be built around YAML in Ruby.

If you're interested in following the project, it's now hosted at Google Code (due to problems with RubyForge from China) at http://code.google.com/p/rbyaml/. Long Sun will also blog about his progress here: http://rbyaml.blogspot.com/.

Exciting news indeed.

Ruby Design meeting

Yesterday marked the first of the Ruby design meetings, where most of the Ruby implementers got together on IRC and started hashing out the solutions to several current concerns, and worked on getting more cooperation in the design of Ruby features.

This is slated to become a weekly meeting, and it's a huge deal. This will make the lives of all Ruby implementations much easier, and the meeting yesterday actually accomplished some very nice things.

There were representatives from JRuby, Rubinius and macruby present, and of course also Matz, Koichi, Nobu and Tanaka from the Ruby core team.

Some of the highlights was a decision to start working on a common API for MultiVM, initial acceptance to add the RubySpecs (coming from Rubinius originally) into the 1.8 and 1.9 build process, meaning that regression testing will be much better from now on, and also having most implementations using the same specs for compliance testing. In reality, this takes us one step closer to a real executable specification that everyone agrees on and has official blessing from Matz.

In conjunction with this, a decision to set up continuous integration for 1.8 and 1.9 was made. The exact practicalities is still to be decided, but the decision to get it done is also very important.

All in all, these are excellent news, and I'm feeling extremely hopeful about more cooperation between the Ruby implementors.

If you're interested in exactly what happened, you can find the agenda, action items and log here: http://ruby-design.pbwiki.com/Design20080421.

JtestR 0.2 released!

And so, JtestR 0.2 has finally been released. The highlights include support for Expectations and TestNG, RSpec stories and lots of other goodies.

Here is the release announcement:

JtestR allows you to test your Java code with Ruby frameworks.

Homepage: http://jtestr.codehaus.org
Download: http://dist.codehaus.org/jtestr

JtestR 0.2 is the current release of the JtestR testing tool. JtestR integrates JRuby with several Ruby frameworks to allow painless testing of Java code, using RSpec, Test/Unit, Expectations, dust and Mocha.

Features:
- Integrates with Ant and Maven
- Includes JRuby 1.1, Test/Unit, RSpec, Expectations, dust, Mocha and ActiveSupport
- Customizes Mocha so that mocking of any Java class is possible
- Background testing server for quick startup of tests
- Automatically runs your JUnit and TestNG codebase as part of the build

Getting started: http://jtestr.codehaus.org/Getting+Started

New and fixed in this release:
JTESTR-10 It should be possible to run TestNG tests
JTESTR-12 Buildr support
JTESTR-13 CC.rb should be able to run JtestR tests
JTESTR-17 Tests should be groupable and runnable per groups
JTESTR-21 Support RSpec stories
JTESTR-28 JtestR should include expectations
JTESTR-30 code coverage support
JTESTR-31 Autoloading of Java constants
JTESTR-32 Can't load IA 32-bit .so on a IA 32-bit platform
JTESTR-33 JtestR should use latest version of JRuby
JTESTR-34 Errors when project is in path with spaces on Windows XP
JTESTR-37 Can't expect a specific Java exception correctly
JTESTR-38 Problem with mocking Java classes
JTESTR-39 RSpec story runner seems to require rubygems
JTESTR-40 Package missing or.jruby.exceptions.RaiseException
JTESTR-43 It should be possible to get the generated mock class without instantiation
JTESTR-44 New output files start with a whitespace
JTESTR-45 RSpec raise_error and Test/Unit assert_raise and assert_nothing_raised handled JRuby NativeException stuff correctly.

Team:
Ola Bini - ola.bini@gmail.com
Anda Abramovici - anda.abramovici@gmail.com

Connecting languages (or polyglot programming example 1)

Today I spent some time connecting two languages that are finding themselves popular for solving wildly different kinds of problems. I decided I wanted to see how easy it was and if it was a workable solution if you would want to take advantage of the strengths of both languages. The result is really up to you. My 15 minutes experiment is what I'll discuss here.

If you'd like, you can see this as a practical example of the sticky part where two languages meet, in language-oriented programming.

The languages under consideration is Ruby and Erlang. The prerequisite reading is this eminent article by my colleague Dennis Byrne: Integrating Java and Erlang.

The only important part is in fact the mathserver.erl code, which you can see here:

-module(mathserver).
-export([start/0, add/2]).

start() ->
Pid = spawn(fun() -> loop() end),
register(mathserver, Pid).

loop() ->
receive
  {From, {add, First, Second}} ->
      From ! {mathserver, First + Second},
      loop()
end.

add(First, Second) ->
mathserver ! {self(), {add, First, Second}},
receive
  {mathserver, Reply} -> Reply
end.

Follow Dennis' instructions to compile this code and start the server in an Erlang console, and then leave it there.

Now, to use this service is really easy from Erlang. You can really just use the mathserver:add/2 operation directly or remotely. But doing it from another language, in this case Ruby is a little bit more complicated. I will make use of JRuby to solve the problem.

So, the client file for using this code will look like this:

require 'erlang'

Erlang::client("clientnode", "cookie") do |client_node|
server_node = Erlang::OtpPeer.new("servernode@127.0.0.1")
connection = client_node.connect(server_node)

connection.sendRPC("mathserver", "add", Erlang::list(Erlang::num(42), Erlang::num(1)))

sum = connection.receiveRPC

p sum.int_value
end

OK, I confess. There is no erlang.rb yet, so I made one. It includes some very small things that make the interfacing with erlang a bit easier. But it's actually still quite straight forward what's going on. We're creating a named node with a specific cookie, connecting to the server node, and then using sendRPC and receiveRPC to do the actual operation. The missing code for the erlang.rb file should look something like this (I did the minimal amount here):

require 'java'
require '/opt/local/lib/erlang/lib/jinterface/priv/OtpErlang.jar'

module Erlang
import com.ericsson.otp.erlang.OtpSelf
import com.ericsson.otp.erlang.OtpPeer
import com.ericsson.otp.erlang.OtpErlangLong
import com.ericsson.otp.erlang.OtpErlangObject
import com.ericsson.otp.erlang.OtpErlangList
import com.ericsson.otp.erlang.OtpErlangTuple

class << self
def tuple(*args)
  OtpErlangTuple.new(args.to_java(OtpErlangObject))
end

def list(*args)
  OtpErlangList.new(args.to_java(OtpErlangObject))
end

def client(name, cookie)
  yield OtpSelf.new(name, cookie)
end

def num(value)
  OtpErlangLong.new(value)
end

def server(name, cookie)
  server = OtpSelf.new(name, cookie)
  server.publish_port

  while true
    yield server, server.accept
  end
end
end
end

As you can see, this is regular simple code to interface with a Java library. Note that you need to find where JInterface is located in your Erlang installation and point to that (and if you're on MacOS X, the JInterface that comes with ports doesn't work. Download and build a new one instead).

There are many things I could have done to make the api MUCH easier to use. For example, I might add some methods to OtpErlangPid, so you could do something like:

pid << [:call, :mathserver, :add, [1, 2]]

where the left arrows sends a message after transforming the arguments.

In fact, it would be exceedingly simple to make the JInterface API downright nice to use, getting the goodies of Erlang while retaining the Ruby language. And oh yeah, this could work on MRI too. There is an equivalent C library for interacting with Erlang, and there could either be a native extension for doing this, or you could just wire it up with DL.

If you read the erlang.rb code carefully, you might have noticed that there are several methods not in use currently. Say, why are they there?

Well, it just so happens that we don't actually have to use any Erlang code in this example at all. We could just use the Erlang runtime system as a large messaging bus (with fault tolerance and error handling and all that jazz of course). Which means we can create a server too:

require 'erlang'

Erlang::server("servernode", "cookie") do |server, connection|
 terms = connection.receive
 arguments = terms.element_at(1).element_at(3)
 first = arguments.element_at(0)
 second = arguments.element_at(1)

 sum = first.long_value + second.long_value
 connection.send(connection.peer.node, Erlang::tuple(server.pid, Erlang::num(sum)))
end

The way I created the server method, it will accept connections and invoke the block for every time it accepts a connection. This connection is yielded to the block together with the actual node object representing the server. The reason the terms are a little bit convoluted is because the sendRPC call actually adds some things that we can just ignore in this case. But if we wanted, we could check the first atoms and do different operations based on these.

You can run the above code in server, and use the exact same math code if you want. For ease of testing, switch the name to servernode2 in both server and client, and then run them. You have just sent Erlang messages from Ruby to Ruby, passing along Java on the way.

Getting different languages working together doesn't need to be hard at all. In fact, it can be downright easy to switch to another language for a few operations that doesn't suit the current language that well. Try it out. You might be surprised.

Pragmatic Static Typing

I have been involved in several discussions about programming languages lately and people have assumed that since I spend lots of time in the Ruby world and with dynamic languages in general I don't like static typing. Many people in the dynamic language communities definitely expresses opinions that sound like they dislike static typing quite a lot.

I'm trying to not sound defensive here, but I feel the need to clarify my position on the whole discussion. Partly because I think that people are being extremely dogmatic and short-sighted by having an attitude like that.

Most of my time I spend coding in Java and in Ruby. My personal preference are to languages such as Common Lisp and Io, but there is no real chance to use them in my day-to-day work. Ruby neatly fits the purpose of a dynamic language that is close to Lisp for my taste. And I'm involved in JRuby because I believe that there is great worth in the Java platform, but also that many Java programmers would benefit from staying less in the Java language.

I have done my time with Haskell, ML, Scala and several other quite statically typed languages. In general, the fact that I don't speak that much about those languages is that I have less exposure to them in my day-to-day life.

But this is the thing. I don't dislike static typing. Absolutely not. It's extremely useful. In many circumstances it gives me things that really can't have in a dynamic language.

Interesting thought: Smalltalk is generally called a dynamic language with very late binding. There are no static type tags and no type inference happening. The only type checking that happens will happen at runtime. In this regard, Smalltalk is exactly like Ruby. The main difference is that when you're working with Smalltalk, it is _always_ runtime. Because of the image based system, the type checking actually happens when you do the programming. There is no real difference between coding time and runtime. Interestingly, this means that Smalltalk tools and environments have most of the same features as a static programming language, while still being dynamic. So a runtime based image system in a dynamic, late bound programming language will actually give you many of the benefits of static typing at compile time.

So the main take away is that I really believe that static typing is extremely important. It's very, very useful, but not in all circumstances. The fact that we reach for a statically typed programming language by default is something we really need to work with though, because it's not always the right choice. I'm going to say this in even stronger words. In most cases a statically typed language is a premature optimization that gets extremely much in the way of productivity. That doesn't mean you shouldn't choose a statically typed language when it's the best solution for the problem. But this should be a conscious choice and not by fiat, just because Java is one of the dominant languages right now. And if you need a statically typed language, make sure that you choose one that doesn't revel in unnecessary type tags. (Java, C#, C++, I'm looking at you.) My current choice for a static language is Scala - it strikes a good balance in most cases.

A statically typed language with type inference will give you some of the same benefits as a good dynamic language, but definitely not all of them. In particular, you get different benefits and a larger degree of flexibility from a dynamic language that can't be achieved in a static language. Neal Ford and others have been talking about the distinction between dynamic and static typing as being incorrect. The real question is between essence and ceremony. Java is a ceremonious language because it needs you to do several dances to the rain gods to declare even the simplest form of method. In an essential language you will say what you need to say, but nothing else. This is one of the reasons dynamic languages and type inferenced static languages sometimes look quite alike - it's the absence of ceremony that people react to. That doesn't mean any essential language can be replaced by another. And with regads to ceremony - don't use a ceremonious language at all. Please. There is no reason and there are many alternatives that are better.

My three level architecture of layers should probably be updated to say that the stable layer should be an essential, statically typed language. The soft/dynamic layer should almost always be a strongly typed dynamic essential language, and the DSL layers stays the same.

Basically, I believe that you should be extremely pragmatic with regards to both static and dynamic typing. They are tools that solve different problems. But out industry today have a tendency to be very dogmatic about these issues, and that's the real danger I think. I'm happy to see language-oriented programming and polyglot programming get more traction, because they improve a programmers pragmatic sensibilities.

JvYAMLb finally released as separate project

So I've finally made the time to extract JvYAMLb from JRuby. That means that JvYAML is mildly deprecated. Of course, since JvYAMLb only uses ByteLists it might not be the best solution for everyone.

If you're interested in downloading the 0.1 release, you can do it at the Google Code download site http://code.google.com/p/jvyamlb/downloads/list.

Presentations this and next week

Wow. JRuby 1.1 is out! Go get! (And incidentally, wouldn't JRuby be the perfect way for Google to support Ruby in Google Apps Engine?)

I'll be doing a talk tonight at .NETAkademien about Ruby and Rails. It's mostly an introduction.
You can find more info here.

On Thursday I'll speak about JRuby at Developer Summit in Stockholm. I'll be in the Dynamic Languages track and you can read the abstract and more information here.

And finally, next Monday I'll talk about JRuby at the Stockholm Ruby User Group. You can find out more information about that event here.

Overall, it feels like the conference season is starting up again. I'll be presenting at JavaOne in a month too.

I've been extremely quiet here lately. It's been a bit slow in JRuby land, actually, and my focus has been elsewhere. Hopefully I'll be able to start posting some really cool stuff soon - I have something up my sleeve that should be possible to talk about soon.

Meta-level thinking

I have been trying to figure out what characterizes some of the best programmers I know, or know about. It's a frustrating endeavor of course, since most of these people are very different from each other, and have different experiences and ways to think and learn about stuff. Not to mention the fact that programmers tend to be highly individualistic.

But I think that I'm finally zeroing in on something that is general enough but also specific enough to categorize most of these people, and the general mind needed to be a really good programmer. In this blog post I'll call that "meta-level thinking", and I'll explain more about what I mean as we go along.

When people try to become better programmers there are generally a few different kinds of advices you can get. The ones that seem prevalent right now is (among others):

• Learn - and understand - Lisp
• More generally, learn a new language that is sufficiently different from your current knowledge
• Work with domain specific languages
• Understand metaprogramming
• Read up on the available data structures (Knuth anyone)?
• Implement a compiler and runtime system for a language
  

Of course, these are only a few examples, and the categories are a bit fuzzy. These items are based on most of my experiences so they tend to be a bit language heavy. But I believe that what you practice by choosing any of these routes is an ability to abstract your thinking. In short, I believe that being able to abstract and understand what goes on in a programming language is one way to become more proficient in that language, but not only that - by changing your thinking to see this part of your environment you generally end up programming differently in all languages.

This is why Lisp has a tendency to change the way people write Java code. Lisp teaches you metaprogramming and DSL's but they don't really do it in a way that need words. DSLs and metaprogramming are just part of Lisp. It's so much a part of the structure that you don't see it. But when you turn to Java you'll need to start working with these abstractions on a different level. Ruby programmers embrace metaprogramming and this changes the way these Ruby programmers think about things.

I'm really happy to see metaprogramming and DSLs getting more and more focus, because I really believe that understanding them is a really good way to make programmers better. Of course, you can get the same effect by writing a copmiler and runtime system - as Steve Yegge proposes. But I disagree that you really need that experience. There are other ways you can get the same way of looking at the world.

I call this meta-level thinking. I think it's mostly a learned ability, but also that there is an aptitude component. Some of the best programmers I've met just have a mind that fits very well. It's interesting to note that this kind of meta-level thinking is not an ability that is only applicable to programming. In fact, that's probably just a matter of genetic luck that the same ability works very well for programming as for many other things. I think that there is a connection between certain abilities and a capacity for meta-level thinking too. Like music - it's interesting to see how many programmers that have artistic leanings, and specifically towards music. It would be interesting to see some statistics about this.

In conclusion, this is my somewhat fuzzy view about what is one of the most important abilities that contributes to create brilliant programmers. If you have any interesting ideas about other ways you can reliably practice this ability, please improve on my list.

The contract of IO#read

It's interesting. After Charlie made an immense effort and rewrote our IO system, basically from scratch, I have started to find bugs. But these are generally not bugs in the IO code, but bugs in Ruby libraries that depend on the way MRI usually works. One of the more annoying ones are IO#read(n), where n is the length you want to read.

This method is not guaranteed to return a string of length n, even if we haven't hit EOF yet. You can NEVER be sure that what you get back is the length you requested. Ever. If you have code that doesn't check the length of the returned string from read, you are almost guaranteed to have a bug just waiting to happen.

Of course, it might work perfectly on your machine and every other machine you test it on. The reason for this is that read(n) will usually return n bytes, but that depends on the socket implementation or file reading implementation of the operating system, it depends on the size of the cache in the network interface, it depends on network latency, and many other things. Please, just make sure to check the return values length before going ahead and using it.

Case in point: net/ldap has this exact problem. Look in net/ber.rb and you will see. There are also two - possibly three - bugs (couple of years old too) that reports different failures because of this.

One thing that makes this problem hard to find is the fact that if you insert debug statement, you will affect the timing in such a way that the code might actually work with debug statement but not without them.

Oh, did I mention that StringIO#read works the same way? It has exactly the same guarantees as IO#read.

After QCon London 2008

This week has mostly been taken up with QCon London. I spent most of Monday, Wednesday, Thursday and Friday here, and I thought that I'd take the opportunity to write up some of my impressions and thoughts about the sessions I attended.

First, in general the conference definitely didn't disappoint me. It held at least as high standard as I had expected from earlier QCon and JAOO conferences. Solid speakers, a wide range of exciting topics and lots of interesting people made for a grade A conference.

I started out on the Monday with listening to my colleagues Neal Ford, Rebecca J Parsons and Martin Fowler give a tutorial on domain specific languages. I've seen bits and parts of this tutorial before, but seeing as the three speakers are working on evolving it to a full and coherent "pedagogical framework" for teaching DSLs, the current presentation had changed quite a bit since the last time. I really liked it and I recommend it to anyone interested in getting a firm grasp about what DSLs are. Having Rebecca talk about external DSLs in the context of parsers and grammars makes total sense, and the only real thing I would say was a problem was the time allotted to it. Her part of the subject was large enough that 75 minutes felt a bit rushed. Of course, I don't see how Martins or Neals parts could be compressed much more either, so maybe the subject actually is too large for a one day tutorial? Anyway, great stuff.

For several reasons I decided to spend the Tuesday working from the office instead of attending tutorials again.

During the Wednesday I mostly spent my time in the exhibition hall, talking to people and doing general networking. For some reason the tracks I was least interested in had all been scheduled on the same day, so I was lazy and worked on other stuff in the ThoughtWorks booth.

The evening keynote on Wednesday by Martin Fowler and Jim Webber was hilarious, and also managed to get a quite important message across. I had a good time.

Thursday started the session attending for me, beginning with Markus Völters presentation of XText in the DSL track. Highly informative and something that I'll keep in mind if I see something that would be benefit from it. The approach is definitely not for all problem domains, of course.

After that. Venkat Subramaniam gave a talk about how to blend dynamic languages with Java. This talk was useful in explaining why you'd want to do something like this, and why it's such a powerful technique. It also served to set up my talk - which was next in that track - about evolving the Java platform. My talk went well, but I had the timing for it really messed up, so I ran out of material 10 minutes earlier than I expected. Neal Gafter was in the audience and helped out with some corrections during the talk. =)

Finally I headed back to the DSL track and saw Avi Bryant talk about DSLs in Smalltalk and then Magnus Christerson and Henk Kolk talk about the Intentional Workbench. Lots of neat stuff in both of these presentations.

Then there was the speakers dinner... Lots of interesting discussions with lovely people. =)

And then, more quickly than I had expected, the final day of QCon arrived. Me as a Ruby person and programming language nerd had quite a good selection of tracks. I ended up seeing Ted's presentation on F#, which made me feel: wow! Microsoft took ML and did exactly what they've done to all languages on the CLR - added support for .NET objects in the mix. The talk ended with a quite strange discussion about whether F# actually helps with concurrent programming or not, and why a functional programming language has primitives that allow you to have mutable state.

After that I did my talk in the Ruby track, talking about more advanced things regard JRuby. It ended up being great fun, and I spent lots of time in the talk answering questions and showing how seamlessly things work with JRuby. I ended up eating up 10 minutes of everyone's lunch time, but I had a great time and I thing most in the audience had too.

Feeling happy and finished with my contributions, I ended up in the Erlang talk by Joe Armstrong. It gave a quite good overview of why Erlang was created and how it solves some of the problems in that particular problem domain. There is no doubt that Armstrong is an entertaining talker, but his buffoon image gets a little tiring and repetitive after a while. Some of the things that interested me in the talk was missing too. He started out saying that Erlang solves a particular problem, but then expanded that into something that sounded like "Erlang should be used for everything, everywhere". I tried to ask a question related to that, but the answer didn't really go in the direction I was interested in.

I stayed in the languages track and saw the introduction to Scala, which is always fun, except that I'd already learned most of the things showcased. The most interesting about the presentation was the audience interest and questions.

Finally I realized that my contributions were not over at all, since I'd agreed to be part of the closing Ruby panel. This ended up revolving quite a lot around the question whether Ruby and Windows ever will be a good match, if this is important, and if we really want to push Ruby into all kinds of environments.

The closing panel were OK, but nothing special. It ended the day on a good note, but at that time I was tired enough to fall asleep in my chair. For some reason this always happens that last day of conferencing.

Anyway. I had a great time and I look forward to being back the next time. I can definitely recommend QCon as one of the best conferences around in this industry.

I'm at QCon

I'm attending QCon in London this whole week. I'll give a talk on Thursday called "Evolving the Java platform", talking about different things that might be a part of the next generation Java platform. On Friday I give the talk "JRuby: Power on the JVM", which is a more advanced JRuby talk than the regular introduction talks I usually give.

Hope to see you around!

JRuby Google Tech talk online

The talk I gave at Google last week is now online: http://youtube.com/watch?v=PfnP-8XbJao

Language design philosophy: more than one way?

When talking about how dynamic scripting languages are designed, people have a tendency to divide them into "There is more than one way to do it" and "There is one way to do it". Perl is the quintessential example of "more than one way", and Python is the opposite, going so far as to make it impossible to have your own indentation.

These different ways of doing things really divide programmers. Some hate the way Python gives you a bondage strait jacket and no scissors, while some programmers love that they don't need to make any choices and that everyone's code will be equally readable.

From a pure perspective, the Python philosophy seems to be the right one, but it just doesn't work for me. In the same way, I agree with Perl's way of doing things, but the problems that cause in most Perl code is just amazing. Sometimes it feels like the Python way is actually directly a result of someone reacting extremely badly to a Perl code base and decided to never allow that to happen in Python.

So what's the point? Well, the point is Ruby. In fact, Ruby has almost all of the flexibility of Perl to do things in different ways. But at the end of the day, none of the Perl problems tend to show up. Why is this? And why do I feel so comfortable in Ruby's "There is more than one way to do it" philosophy, while the Perl one scares me?

I think it comes down to language design. The Python approach is impossible for the simple reason that what the language designer chooses is going to be the "one way", by fiat. Some people will agree, and some will not. But what I'm seeing in Ruby is that the many ways have been transformed into idioms and guidelines. There are no hard rules, but the community have evolutionary evolved idioms that work and found out many of the ways that doesn't work. This seems to be the right way - if you do the choice as a language designer, you have actually chosen the people that will use your language: that's going to be the persons who doesn't dislike the language designers choices. But if you leave it open enough for evolutionary community design to happen you can actually get the best of both world: both a best way to do things, and something that works for a much larger percentage of the programmer world.

I have come to believe that this is one of the major reasons that Ruby feels so good to me, and is such a good language. And it's a lesson for language designers. When you choose philosophy, make sure to take the possibility of communities and idiom evolution into account.

SF presence

So, next week and the week after that I'll be in San Francisco. I'll participate in the ThoughtWorks code jam on Tuesday and Wednesday, and do a tech talk at Google sometime by the end of the week.

If someone is interested in meeting up, talking about JRuby, Ruby or programming languages in general, just get in touch.

Is Groovy like Java?

I usually don't mention the G word in my blog - the results have a tendency to be less than palatable. This time I'm going to make an exception though. The reason is that this meme has been floating around as an argument for Groovy, and I really don't agree with it. In fact, the reason I'm writing about it is because I think the meme in itself generally can be bad for Groovy.

The thing I'm talking about is the saying that Groovy is just like Java. That you can start using Groovy directly without learning anything about Groovy and that basically all Java syntax is valid Groovy.

Now, the last part has never been really true, since there are a few valid Java constructs that are not supported in Groovy (anonymous classes, I'm looking at you). This might be fixed in the 1.5 version. I don't really know, but that's not the point. Since Groovy isn't using the Java parser there will always be a few small corner cases that are not valid in Groovy but Java accepts. This is not a problem, by the way. The only way it can be a problem is when you're using Groovy the wrong way, or you absolutely need to do something that Groovy doesn't support. Since one of the Groovy catch calls is that you shouldn't implement everything in Groovy, this can't really be a concern either. If you're using Groovy correctly, you should just drop down to the Java layer for that particular feature.

Now that that part is out of the way, let's take a look at the other parts of this idea. Namely that you should choose Groovy because it's just like Java, and you can start writing Java code directly in Java. These statements are definitely true, since you can absolutely pick up a language that way. In my mind, though, this misses the whole point of using another language on top of the JVM. If you're building an application and chooses to use Groovy for this instead of Java, isn't the reason that you're doing that the fact that you expect Groovy to give you something Java does not? And if that's the case, doesn't it sound like a disservice to people picking up Groovy to say that they can begin by just programming Java. There is no gain here. Really. In fact, the equivalent program in Groovy and Java will perform much worse on Groovy since there are several layers of dispatching in Groovy that you don't have in Java. The result? An application with exactly the same source code, but running 10 times slower.

Since this message is mostly used in marketing and introduction facilities, the people it's geared at are by definition newbies to the Groovy world. Many of them are totally new to dynamic languages. And the first thing they do is write Java in Groovy, find it slow and obviously, from that point don't understand why you should use Groovy. This is bad for Groovy. Groovy has some extremely powerful features and libraries that can really make your life on the JVM blissful (compared to using Java, for example). Groovy's integration with Java is top notch, which means that you can always fall back to Java when you need Java specific features. If you're using Groovy, I think you should use the full power of the language so you can actually get the full benefit of Groovy.