Meta-level thinking

I have been trying to figure out what characterizes some of the best programmers I know, or know about. It's a frustrating endeavor of course, since most of these people are very different from each other, and have different experiences and ways to think and learn about stuff. Not to mention the fact that programmers tend to be highly individualistic.

But I think that I'm finally zeroing in on something that is general enough but also specific enough to categorize most of these people, and the general mind needed to be a really good programmer. In this blog post I'll call that "meta-level thinking", and I'll explain more about what I mean as we go along.

When people try to become better programmers there are generally a few different kinds of advices you can get. The ones that seem prevalent right now is (among others):

• Learn - and understand - Lisp
• More generally, learn a new language that is sufficiently different from your current knowledge
• Work with domain specific languages
• Understand metaprogramming
• Read up on the available data structures (Knuth anyone)?
• Implement a compiler and runtime system for a language
  

Of course, these are only a few examples, and the categories are a bit fuzzy. These items are based on most of my experiences so they tend to be a bit language heavy. But I believe that what you practice by choosing any of these routes is an ability to abstract your thinking. In short, I believe that being able to abstract and understand what goes on in a programming language is one way to become more proficient in that language, but not only that - by changing your thinking to see this part of your environment you generally end up programming differently in all languages.

This is why Lisp has a tendency to change the way people write Java code. Lisp teaches you metaprogramming and DSL's but they don't really do it in a way that need words. DSLs and metaprogramming are just part of Lisp. It's so much a part of the structure that you don't see it. But when you turn to Java you'll need to start working with these abstractions on a different level. Ruby programmers embrace metaprogramming and this changes the way these Ruby programmers think about things.

I'm really happy to see metaprogramming and DSLs getting more and more focus, because I really believe that understanding them is a really good way to make programmers better. Of course, you can get the same effect by writing a copmiler and runtime system - as Steve Yegge proposes. But I disagree that you really need that experience. There are other ways you can get the same way of looking at the world.

I call this meta-level thinking. I think it's mostly a learned ability, but also that there is an aptitude component. Some of the best programmers I've met just have a mind that fits very well. It's interesting to note that this kind of meta-level thinking is not an ability that is only applicable to programming. In fact, that's probably just a matter of genetic luck that the same ability works very well for programming as for many other things. I think that there is a connection between certain abilities and a capacity for meta-level thinking too. Like music - it's interesting to see how many programmers that have artistic leanings, and specifically towards music. It would be interesting to see some statistics about this.

In conclusion, this is my somewhat fuzzy view about what is one of the most important abilities that contributes to create brilliant programmers. If you have any interesting ideas about other ways you can reliably practice this ability, please improve on my list.

The contract of IO#read

It's interesting. After Charlie made an immense effort and rewrote our IO system, basically from scratch, I have started to find bugs. But these are generally not bugs in the IO code, but bugs in Ruby libraries that depend on the way MRI usually works. One of the more annoying ones are IO#read(n), where n is the length you want to read.

This method is not guaranteed to return a string of length n, even if we haven't hit EOF yet. You can NEVER be sure that what you get back is the length you requested. Ever. If you have code that doesn't check the length of the returned string from read, you are almost guaranteed to have a bug just waiting to happen.

Of course, it might work perfectly on your machine and every other machine you test it on. The reason for this is that read(n) will usually return n bytes, but that depends on the socket implementation or file reading implementation of the operating system, it depends on the size of the cache in the network interface, it depends on network latency, and many other things. Please, just make sure to check the return values length before going ahead and using it.

Case in point: net/ldap has this exact problem. Look in net/ber.rb and you will see. There are also two - possibly three - bugs (couple of years old too) that reports different failures because of this.

One thing that makes this problem hard to find is the fact that if you insert debug statement, you will affect the timing in such a way that the code might actually work with debug statement but not without them.

Oh, did I mention that StringIO#read works the same way? It has exactly the same guarantees as IO#read.

After QCon London 2008

This week has mostly been taken up with QCon London. I spent most of Monday, Wednesday, Thursday and Friday here, and I thought that I'd take the opportunity to write up some of my impressions and thoughts about the sessions I attended.

First, in general the conference definitely didn't disappoint me. It held at least as high standard as I had expected from earlier QCon and JAOO conferences. Solid speakers, a wide range of exciting topics and lots of interesting people made for a grade A conference.

I started out on the Monday with listening to my colleagues Neal Ford, Rebecca J Parsons and Martin Fowler give a tutorial on domain specific languages. I've seen bits and parts of this tutorial before, but seeing as the three speakers are working on evolving it to a full and coherent "pedagogical framework" for teaching DSLs, the current presentation had changed quite a bit since the last time. I really liked it and I recommend it to anyone interested in getting a firm grasp about what DSLs are. Having Rebecca talk about external DSLs in the context of parsers and grammars makes total sense, and the only real thing I would say was a problem was the time allotted to it. Her part of the subject was large enough that 75 minutes felt a bit rushed. Of course, I don't see how Martins or Neals parts could be compressed much more either, so maybe the subject actually is too large for a one day tutorial? Anyway, great stuff.

For several reasons I decided to spend the Tuesday working from the office instead of attending tutorials again.

During the Wednesday I mostly spent my time in the exhibition hall, talking to people and doing general networking. For some reason the tracks I was least interested in had all been scheduled on the same day, so I was lazy and worked on other stuff in the ThoughtWorks booth.

The evening keynote on Wednesday by Martin Fowler and Jim Webber was hilarious, and also managed to get a quite important message across. I had a good time.

Thursday started the session attending for me, beginning with Markus Völters presentation of XText in the DSL track. Highly informative and something that I'll keep in mind if I see something that would be benefit from it. The approach is definitely not for all problem domains, of course.

After that. Venkat Subramaniam gave a talk about how to blend dynamic languages with Java. This talk was useful in explaining why you'd want to do something like this, and why it's such a powerful technique. It also served to set up my talk - which was next in that track - about evolving the Java platform. My talk went well, but I had the timing for it really messed up, so I ran out of material 10 minutes earlier than I expected. Neal Gafter was in the audience and helped out with some corrections during the talk. =)

Finally I headed back to the DSL track and saw Avi Bryant talk about DSLs in Smalltalk and then Magnus Christerson and Henk Kolk talk about the Intentional Workbench. Lots of neat stuff in both of these presentations.

Then there was the speakers dinner... Lots of interesting discussions with lovely people. =)

And then, more quickly than I had expected, the final day of QCon arrived. Me as a Ruby person and programming language nerd had quite a good selection of tracks. I ended up seeing Ted's presentation on F#, which made me feel: wow! Microsoft took ML and did exactly what they've done to all languages on the CLR - added support for .NET objects in the mix. The talk ended with a quite strange discussion about whether F# actually helps with concurrent programming or not, and why a functional programming language has primitives that allow you to have mutable state.

After that I did my talk in the Ruby track, talking about more advanced things regard JRuby. It ended up being great fun, and I spent lots of time in the talk answering questions and showing how seamlessly things work with JRuby. I ended up eating up 10 minutes of everyone's lunch time, but I had a great time and I thing most in the audience had too.

Feeling happy and finished with my contributions, I ended up in the Erlang talk by Joe Armstrong. It gave a quite good overview of why Erlang was created and how it solves some of the problems in that particular problem domain. There is no doubt that Armstrong is an entertaining talker, but his buffoon image gets a little tiring and repetitive after a while. Some of the things that interested me in the talk was missing too. He started out saying that Erlang solves a particular problem, but then expanded that into something that sounded like "Erlang should be used for everything, everywhere". I tried to ask a question related to that, but the answer didn't really go in the direction I was interested in.

I stayed in the languages track and saw the introduction to Scala, which is always fun, except that I'd already learned most of the things showcased. The most interesting about the presentation was the audience interest and questions.

Finally I realized that my contributions were not over at all, since I'd agreed to be part of the closing Ruby panel. This ended up revolving quite a lot around the question whether Ruby and Windows ever will be a good match, if this is important, and if we really want to push Ruby into all kinds of environments.

The closing panel were OK, but nothing special. It ended the day on a good note, but at that time I was tired enough to fall asleep in my chair. For some reason this always happens that last day of conferencing.

Anyway. I had a great time and I look forward to being back the next time. I can definitely recommend QCon as one of the best conferences around in this industry.

I'm at QCon

I'm attending QCon in London this whole week. I'll give a talk on Thursday called "Evolving the Java platform", talking about different things that might be a part of the next generation Java platform. On Friday I give the talk "JRuby: Power on the JVM", which is a more advanced JRuby talk than the regular introduction talks I usually give.

Hope to see you around!

JRuby Google Tech talk online

The talk I gave at Google last week is now online: http://youtube.com/watch?v=PfnP-8XbJao