Last evening was very nice. We usually have a good time at the Rails meetups, hosted by Valtech. Yesterday it started out with Dr Nic William, who spoke about some metaprogramming techniques. Very neat, and the best part was his Magic Wiggly Lines. I'm not sure if it's genius or mental. But it's cool, at least; it's basically a const_missing hook, which makes it possible to use class names even if you spell them wrong. Hehe.
After that I spoke about new JRuby thing. I have a cold, so I took it pretty easy, and it was very free form with questions. There was many interesting questions, actually, and I think Nic taped it all...
After that, we drank beer and talked technology, which is always fun!
onsdag, mars 21, 2007
söndag, mars 18, 2007
Rails meet in Stockholm
And it's time again for a Rails meetup in Stockholm, at Valtech. This Tuesday (20th) 6pm at Valtechs offices. It is usually very interesting and neat to meet other people involved in Rails and Ruby here in Stockholm, and it's usually very nice, all around.
I will have a short presentation on the new features in JRuby 0.9.8. I will provide a fast intro to JRuby, but this is not the introductionary presentation; I will also try a somewhat different presentation technique, so it is sure to be interesting. At least for me... =)
Show up! It's worth it. You need to reserve a place, though, which can be done at http://www.rails.se/rails/show/Railstr%C3%A4ff+20+Mars+2007
I will have a short presentation on the new features in JRuby 0.9.8. I will provide a fast intro to JRuby, but this is not the introductionary presentation; I will also try a somewhat different presentation technique, so it is sure to be interesting. At least for me... =)
Show up! It's worth it. You need to reserve a place, though, which can be done at http://www.rails.se/rails/show/Railstr%C3%A4ff+20+Mars+2007
fredag, mars 16, 2007
ThoughtWorks
I have resigned from my position at Karolinska Institutet. Instead, I will work for ThoughtWorks. I am extremely excited about this, obviously. I will start in London in the beginning of June. I'll stay in London for 6 months and then move to the San Francisco office.
As you may know, ThoughtWorks is doing lots of Ruby projects in the enterprise and are really helping spread Ruby to the masses. And that's what I will be working with. Ruby and JRuby for ThoughtWorks.
What's more, I will be doing development on the JRuby core at ThoughtWorks!
As an aside, I would like to mention two very interesting things that TW have announced very recently. First, CruiseControl.rb, which looks very nice, and secondly, Mingle. Both of these products have great potential.
I'm just so excited about this!
As you may know, ThoughtWorks is doing lots of Ruby projects in the enterprise and are really helping spread Ruby to the masses. And that's what I will be working with. Ruby and JRuby for ThoughtWorks.
What's more, I will be doing development on the JRuby core at ThoughtWorks!
As an aside, I would like to mention two very interesting things that TW have announced very recently. First, CruiseControl.rb, which looks very nice, and secondly, Mingle. Both of these products have great potential.
I'm just so excited about this!
måndag, mars 12, 2007
JRuby Regular Expressions
The Regular Expression support in JRuby is about to be revamped. I will here detail my plans for this work, and also some of the reasons for it. This post is as much for people interested in JRuby, as for the JRuby developers.
Stage 0: java.util.regex
JRuby has traditionally used java.util.regex. We stopped doing that March 11:th, 2007. The main reasons are because of the disconnect with MRI. Some of the operators work very differently, there are some problems with UTF-8, we can't support SJIS or EUC, nor posix-classes. And java.util.regex also uses a recursive implementation which means it can't handle certain large inputs. Further, we would like to be able to modify the implementation to work with the same stuff that backs the RubyString, to increase performance.
Stage 1: JRegex
Yesterday (March 11:th), I merged JRegex as the main regular expression engine for JRuby. The main reasons for this is twofold. First, we can change the implementation quite easily, and second, it is an iterative algorithm, which means it doesn't fail on the input that java.util.regex does. Since this caused problems with some Rails tests (and also in multipart handling in all libraries using cgi), I decided to merge this as a stopgap until the next incarnation of regex support.
Stage 2: REJ
In about 2 weeks, I hope to be able to merge REJ with JRuby. At the point of merging, it should be a better replacement than both JRegex and java.util.regex. REJ is a project I've started, which will be a direct port of the MRI 1.8.6 regular expression engine. The important thing about this is that the semantics for JRuby will match MRI very closely. We will be able to match UTF-8, SJIS and EUC regular expressions, and we are able to have the same quirks as MRI, even though people shouldn't depend on such quirks. In the process of writing REJ, I will also create a large suite of test cases for regular expressions, based on Henry Spencer's test files. I'll probably submit something initial to a separate repository very soon. If I get my wish, REJ is what will be the regular expression engine for JRuby 1.0.
Stage 3: Ojiguruma
After 1.0 has been released, I think it's time to make the Regexp engine in JRuby really extensible, and provide an interface from Ruby to change which engine to use. After that is done, I would be very interested in doing a port of Oniguruma to Java, which would give us far better multilanguage support, and also some interesting features. The reason I'm choosing to not do this right now is because Oniguruma is just too large.
Stage 4: (No official name yet)
Another engine that some in the JRuby/Ruby community has started working on is an engine which will be based on Ragel for parsing and a modified version of Thompson NFA and TCL-style backreferences for matching. It's an interesting project but it will take some time before it's usable.
Stage 0: java.util.regex
JRuby has traditionally used java.util.regex. We stopped doing that March 11:th, 2007. The main reasons are because of the disconnect with MRI. Some of the operators work very differently, there are some problems with UTF-8, we can't support SJIS or EUC, nor posix-classes. And java.util.regex also uses a recursive implementation which means it can't handle certain large inputs. Further, we would like to be able to modify the implementation to work with the same stuff that backs the RubyString, to increase performance.
Stage 1: JRegex
Yesterday (March 11:th), I merged JRegex as the main regular expression engine for JRuby. The main reasons for this is twofold. First, we can change the implementation quite easily, and second, it is an iterative algorithm, which means it doesn't fail on the input that java.util.regex does. Since this caused problems with some Rails tests (and also in multipart handling in all libraries using cgi), I decided to merge this as a stopgap until the next incarnation of regex support.
Stage 2: REJ
In about 2 weeks, I hope to be able to merge REJ with JRuby. At the point of merging, it should be a better replacement than both JRegex and java.util.regex. REJ is a project I've started, which will be a direct port of the MRI 1.8.6 regular expression engine. The important thing about this is that the semantics for JRuby will match MRI very closely. We will be able to match UTF-8, SJIS and EUC regular expressions, and we are able to have the same quirks as MRI, even though people shouldn't depend on such quirks. In the process of writing REJ, I will also create a large suite of test cases for regular expressions, based on Henry Spencer's test files. I'll probably submit something initial to a separate repository very soon. If I get my wish, REJ is what will be the regular expression engine for JRuby 1.0.
Stage 3: Ojiguruma
After 1.0 has been released, I think it's time to make the Regexp engine in JRuby really extensible, and provide an interface from Ruby to change which engine to use. After that is done, I would be very interested in doing a port of Oniguruma to Java, which would give us far better multilanguage support, and also some interesting features. The reason I'm choosing to not do this right now is because Oniguruma is just too large.
Stage 4: (No official name yet)
Another engine that some in the JRuby/Ruby community has started working on is an engine which will be based on Ragel for parsing and a modified version of Thompson NFA and TCL-style backreferences for matching. It's an interesting project but it will take some time before it's usable.
Two more days in Kraków
I was very lazy on Saturday. Took it very easy, then went to a dinner with some of the presenters and organizers for SFI. And after that I gave my presentation, which I personally felt went very well. I had to adapt the technical level some, because it seemed most of the audience didn't know much about compilers. After that, Lukas held his presentation about Seaside, which also went well.
Then we celebrated. Then there was headache. And a look at the Cathedral. And a taxi driver who didn't understand one word of English. And he took me to the wrong terminal at the airport. But I finally got on the right flight and got home.
Then we celebrated. Then there was headache. And a look at the Cathedral. And a taxi driver who didn't understand one word of English. And he took me to the wrong terminal at the airport. But I finally got on the right flight and got home.
fredag, mars 09, 2007
Two days in Kraków
So, I've been in Kraków for two days now. I will fly back on Sunday. It's a very interesting town. On one part, it's lovely. The buildings are absolutely amazing, and very beautiful. On the other hand, many of them are very run down, and the wear and tear is obvious all around. To me, the whole town seems sort of depressing, but on the other hand the people here are very up beat, and from some of the presentations I've seen, the technological future for Kraków looks very bright indeed.
I haven't really been able to see most of the presentations. Almost all have been in Polish, sadly enough. I did see two interesting ones yesterday. The first was Michael Foords talk about IronPython. This was very neat and I've talked some with him after that too. I'll get back to that. The other talk was about Google Ads, by Greg Badros. As always, hearing about how Google does things internally is always amazingly interesting. I would have enjoyed hearing more about the machine learning and NLP stuff they're doing, but obviously they can't discuss that too much.
Today I haven't been able to see any presentation, due to Polish. On the other hand me, Michael and Lukas Renggli have had some very interesting discussions both yesterday and today. Now, the company Michael's part of has the largest IronPython code base in the world, as far as I understand it. It's basically 90 000 LOC, where 20 000 is production code and 70 000 are testing code. That sounds about right... =)
Lukas Renggli is one of the core developers of Seaside, a framework which I'm quite fond of. What we three have in common is our interest in dynamic languages, so we had some very common ground to talk about. What's nice is that many of the things Michael talked about in his presentation is stuff we in JRuby also speak much about. In the same manner, it seems the IronPython guys have had basically the same problems we in JRuby has had. It seems there is some common ground to be found here, and possibly also a basis for conversations. I for one would find that very interesting, since I enjoy hearing about dynamic languages getting foot holds on statically typed virtual machines; that is just such a sweet spot.
Tomorrow evening is my talk. I'm going to be at an official dinner in the middle of the day, and then get over to the conference and give my presentation at 17:00. I think it's going to be interesting, and it will be an obvious counterpart to Michael's talk, since the reason for existing is so similar for IronPython and JRuby. It will be interesting to see how the audience react.
After that, I'll see Lukas talk about Seaside, and after that people are talking about having a party, so tomorrow will be quite intense. And finally, on Sunday I'm going back to Sweden. As far as I know now, I won't travel that much for at least a month now, which feels sort of nice. This week has been far to intense, and also have had some really great - but tiring - moments.
I haven't really been able to see most of the presentations. Almost all have been in Polish, sadly enough. I did see two interesting ones yesterday. The first was Michael Foords talk about IronPython. This was very neat and I've talked some with him after that too. I'll get back to that. The other talk was about Google Ads, by Greg Badros. As always, hearing about how Google does things internally is always amazingly interesting. I would have enjoyed hearing more about the machine learning and NLP stuff they're doing, but obviously they can't discuss that too much.
Today I haven't been able to see any presentation, due to Polish. On the other hand me, Michael and Lukas Renggli have had some very interesting discussions both yesterday and today. Now, the company Michael's part of has the largest IronPython code base in the world, as far as I understand it. It's basically 90 000 LOC, where 20 000 is production code and 70 000 are testing code. That sounds about right... =)
Lukas Renggli is one of the core developers of Seaside, a framework which I'm quite fond of. What we three have in common is our interest in dynamic languages, so we had some very common ground to talk about. What's nice is that many of the things Michael talked about in his presentation is stuff we in JRuby also speak much about. In the same manner, it seems the IronPython guys have had basically the same problems we in JRuby has had. It seems there is some common ground to be found here, and possibly also a basis for conversations. I for one would find that very interesting, since I enjoy hearing about dynamic languages getting foot holds on statically typed virtual machines; that is just such a sweet spot.
Tomorrow evening is my talk. I'm going to be at an official dinner in the middle of the day, and then get over to the conference and give my presentation at 17:00. I think it's going to be interesting, and it will be an obvious counterpart to Michael's talk, since the reason for existing is so similar for IronPython and JRuby. It will be interesting to see how the audience react.
After that, I'll see Lukas talk about Seaside, and after that people are talking about having a party, so tomorrow will be quite intense. And finally, on Sunday I'm going back to Sweden. As far as I know now, I won't travel that much for at least a month now, which feels sort of nice. This week has been far to intense, and also have had some really great - but tiring - moments.
tisdag, mars 06, 2007
Tomorrow is today - or from London to Stockholm to Krakow
Part I: London
I got up at 5am, got dressed and packed and headed of to the airport. Finally got on the plane to London (thank god for online check-in) and tried to get some work done. That didn't go very well, but I got to read some instead. I'm rereading Ubik. Classic. Landed at Heathrow about 10 minutes behind schedule. Ran to the Piccadilly line, tried to talk with people in Sweden, but the connection was kinda bad. Arrived at Covent Garden, walked in brisk pace to the place of my meeting. Finally arrived, 5 minutes late, to find out there was no hurry. Sat down and had a pleasant chat with Dan North and Simon Stewart. Very nice. (Hey Dan, if you read this, get in touch when you're in Stockholm for ExpertZone. I know places with good beer.).
Then, Lunch. Haha. What a joke. I found a place that seemed to offer great vegetarian burgers, so I settled down for a quorn burger and some British chips. After 15 minutes waiting I got told the grill wouldn't light up. Lucky me. Instead, I had to make do with a Pret sandwich and a spicy vegetable Cornish pasty (someone told me it was the day of the Cornish patron saint yesterday, so I guess that fits). A few minutes walk on Oxford street was nice. I got photographed by some guys from an Austrian fashion magazine. Then back for the real meeting.
Cyndi Mitchell is a very formidable person. I was impressed. We had a 2 hour talk; very interesting. I will tell you all about it as soon as I can. But suffice to say, I was very happy when I left. Until I realized that I had exactly 2 hours until my flight taxed out from Heathrow. So, back to Covent Garden. Back on the underground. Which through some bad communication on the drivers part, almost ended badly when the train reached the end stop before Heathrow. Thanks to the people who told me to get of the train and switch to the other side. =) I finally got to Terminal 3, realized I already had a boarding card, found my way to the security check, got my boots scanned two times, and found the gate about 20 times before takeoff. Of course, as soon as the aircraft had taxed out, the flight management decided that we should stay 30 minutes on ground before finally taking off towards Copenhagen. It's a wonder I actually got there in time. So, in Copenhagen I walked 2 klicks to the transfer center, got a new boarding card and walked back to almost the exact place I started from, got on my flight to Stockholm and finally landed. Wow.
Airport coach to Stockholm city, and night bus home to my dear old mental hospital. When I finally got to bed I ached all over, and had been awake and active for 22 hours solid. Sleep was nice at that point. Especially since the day had been a success on all accounts.
Part II: Stockholm
I finally woke up, took care of mail and told everyone about London. Thought I'd take it easy, maybe practice on my presentation for Saturday, possibly start writing on the presentations for the CS course I'm going to speak for this April. But alas, after some mailing with Poland, I realized I've made a big mistake. I thought I was going to Kraków this Friday, and staying until Sunday. That's not entirely right. I'm actually going tomorrow. Surprise!
So.
Part III: Kraków
As mentioned above, I'll land in Kraków tomorrow (Wednesday) evening. This is good in several ways. I will have a chance to see the city, and I will have the opportunity to see some more presentations at the conference I'm at. There are many interesting things to be had; I'm sad half of them are in Polish. I guess I can look at the slides and have some fun trying the guess the meaning. I am going to see the talks about IronPython, Google Ads and Seaside. And, of course, if anyone is nearby and would like to take a beer and talk computers, programming languages, AI, or whatnot, please hit me a mail, or a comment on this post. My evenings will be mostly free it seems.
That is, unless someone knows any good clubs with nice music; especially electronic stuff and/or postpunk, new wave, darkwave or gothy things.
Anyway. Time to sleep.
I got up at 5am, got dressed and packed and headed of to the airport. Finally got on the plane to London (thank god for online check-in) and tried to get some work done. That didn't go very well, but I got to read some instead. I'm rereading Ubik. Classic. Landed at Heathrow about 10 minutes behind schedule. Ran to the Piccadilly line, tried to talk with people in Sweden, but the connection was kinda bad. Arrived at Covent Garden, walked in brisk pace to the place of my meeting. Finally arrived, 5 minutes late, to find out there was no hurry. Sat down and had a pleasant chat with Dan North and Simon Stewart. Very nice. (Hey Dan, if you read this, get in touch when you're in Stockholm for ExpertZone. I know places with good beer.).
Then, Lunch. Haha. What a joke. I found a place that seemed to offer great vegetarian burgers, so I settled down for a quorn burger and some British chips. After 15 minutes waiting I got told the grill wouldn't light up. Lucky me. Instead, I had to make do with a Pret sandwich and a spicy vegetable Cornish pasty (someone told me it was the day of the Cornish patron saint yesterday, so I guess that fits). A few minutes walk on Oxford street was nice. I got photographed by some guys from an Austrian fashion magazine. Then back for the real meeting.
Cyndi Mitchell is a very formidable person. I was impressed. We had a 2 hour talk; very interesting. I will tell you all about it as soon as I can. But suffice to say, I was very happy when I left. Until I realized that I had exactly 2 hours until my flight taxed out from Heathrow. So, back to Covent Garden. Back on the underground. Which through some bad communication on the drivers part, almost ended badly when the train reached the end stop before Heathrow. Thanks to the people who told me to get of the train and switch to the other side. =) I finally got to Terminal 3, realized I already had a boarding card, found my way to the security check, got my boots scanned two times, and found the gate about 20 times before takeoff. Of course, as soon as the aircraft had taxed out, the flight management decided that we should stay 30 minutes on ground before finally taking off towards Copenhagen. It's a wonder I actually got there in time. So, in Copenhagen I walked 2 klicks to the transfer center, got a new boarding card and walked back to almost the exact place I started from, got on my flight to Stockholm and finally landed. Wow.
Airport coach to Stockholm city, and night bus home to my dear old mental hospital. When I finally got to bed I ached all over, and had been awake and active for 22 hours solid. Sleep was nice at that point. Especially since the day had been a success on all accounts.
Part II: Stockholm
I finally woke up, took care of mail and told everyone about London. Thought I'd take it easy, maybe practice on my presentation for Saturday, possibly start writing on the presentations for the CS course I'm going to speak for this April. But alas, after some mailing with Poland, I realized I've made a big mistake. I thought I was going to Kraków this Friday, and staying until Sunday. That's not entirely right. I'm actually going tomorrow. Surprise!
So.
Part III: Kraków
As mentioned above, I'll land in Kraków tomorrow (Wednesday) evening. This is good in several ways. I will have a chance to see the city, and I will have the opportunity to see some more presentations at the conference I'm at. There are many interesting things to be had; I'm sad half of them are in Polish. I guess I can look at the slides and have some fun trying the guess the meaning. I am going to see the talks about IronPython, Google Ads and Seaside. And, of course, if anyone is nearby and would like to take a beer and talk computers, programming languages, AI, or whatnot, please hit me a mail, or a comment on this post. My evenings will be mostly free it seems.
That is, unless someone knows any good clubs with nice music; especially electronic stuff and/or postpunk, new wave, darkwave or gothy things.
Anyway. Time to sleep.
JRuby 0.9.8 is here
The JRuby team is pleased to announce the release of JRuby 0.9.8.
Download at: http://dist.codehaus.org/jruby/
This release has some great improvements:
The amount of IRC conversations, mailing list threads, bug reports, patches, and blog entries in the community has been a great help and our community is really making a huge difference in how fast JRuby is improving. The amount of progress is really staggering!
If you have ever thought that JRuby wasn't mature enough, I would like to contradict that now. With this release we are better than ever.
More information can be found at http://www.jruby.org.
Download at: http://dist.codehaus.org/jruby/
This release has some great improvements:
- Ruby on Rails support. We have been working hard on getting Rails own unit tests running and over 98% of them now run successfully. We feel things are running well enough to invite Ruby users to kick the tires and help root out any final issues.
- Ruby classes can extend concrete/abstract Java classes and override methods
- New Java primitive array syntax
- Reimplementation of String, Numeric classes, and Array to be more correct and performant
- Significant bottlenecks have been identified. In some cases IO is 6.5x faster than previous releases. Java included classes are significantly faster than in the past.
- 220 Jira issues resolved since last release
The amount of IRC conversations, mailing list threads, bug reports, patches, and blog entries in the community has been a great help and our community is really making a huge difference in how fast JRuby is improving. The amount of progress is really staggering!
If you have ever thought that JRuby wasn't mature enough, I would like to contradict that now. With this release we are better than ever.
More information can be found at http://www.jruby.org.
lördag, februari 24, 2007
The world is spinning
If you didn't know that, the title tells it all. Of course, that's old news.
I haven't really been able to blog as much as I wish I could have, lately. There are reasons for this, of course. Two very exciting reasons, in fact. If everything pans out, I will be able to write about it in 2 to 3 weeks time.
In other news, we are gearing up for another JRuby release. This one will be a biggie. Many nice things will be in place, and it will set the record for both new features and bug fixes. I think no one will be disappointed by it, actually.
I have a few presentations lined up too. The closest to now will be in exactly two weeks. I will speak at the Academic Computer Science Festival in Craków, Poland. If you're somewhere close, by all means offer to show me the city. =) I will land March 9:th and fly out again March 11:th. My presentation will be at 17:00 March 10:th, CET. If you would like more information, it can be found at the festivals homepage, here. The presentation will be in English (since I don't speak Polish, obviously), and it will be slightly more technical than the usual JRuby presentations. There will probably be some detail about our runtime, interpreter, parser and lexer, and hopefully I'll get some info in about our YARV and Java bytecode compiler efforts. This will be very exciting to talk about, I'm kinda salivating just thinking about it. =)
Another, more long term presentation, has just been decided. I will attend TheServerSide Java Symposium Europe, in Barcelona, from June 27:th to June 29:th, and talk about JRuby from the perspective of a Java developer, and what it can do for you. Hopefully I will have time to see the city too.
I will update with more information when possible.
I haven't really been able to blog as much as I wish I could have, lately. There are reasons for this, of course. Two very exciting reasons, in fact. If everything pans out, I will be able to write about it in 2 to 3 weeks time.
In other news, we are gearing up for another JRuby release. This one will be a biggie. Many nice things will be in place, and it will set the record for both new features and bug fixes. I think no one will be disappointed by it, actually.
I have a few presentations lined up too. The closest to now will be in exactly two weeks. I will speak at the Academic Computer Science Festival in Craków, Poland. If you're somewhere close, by all means offer to show me the city. =) I will land March 9:th and fly out again March 11:th. My presentation will be at 17:00 March 10:th, CET. If you would like more information, it can be found at the festivals homepage, here. The presentation will be in English (since I don't speak Polish, obviously), and it will be slightly more technical than the usual JRuby presentations. There will probably be some detail about our runtime, interpreter, parser and lexer, and hopefully I'll get some info in about our YARV and Java bytecode compiler efforts. This will be very exciting to talk about, I'm kinda salivating just thinking about it. =)
Another, more long term presentation, has just been decided. I will attend TheServerSide Java Symposium Europe, in Barcelona, from June 27:th to June 29:th, and talk about JRuby from the perspective of a Java developer, and what it can do for you. Hopefully I will have time to see the city too.
I will update with more information when possible.
onsdag, februari 14, 2007
The new base
I have for a time argued that the JVM should become more like an Operating System, and Java the language for OS development. Other languages should run on top of the JVM for running applications. It seems I'm not alone in this line of thought. Robert Varttinen wrote some about it here. To me it seems like a compelling future. But it seems the next logical step for enterprise applications would be further virtualization. I would for example like my beans implemented in Ruby. But not only that, I would want all the business logic to be OS agnostic. I would like my Ruby logic to live on top of a JVM J2EE server, but that logic should be able to move, transparently, to a .NET-server and provide the same business logic at that place. What would be even better is if I didn't have to deploy it manually to all places, but the logic would just move to the places where it's needed. Will we see that anytime soon?
tisdag, februari 13, 2007
RailsConf galore
So, it's decided. Me and two colleagues from Karolinska Institutet will attend RailsConf in Portland, OR. I'm looking very much forward to it, especially since we are also going to JavaOne, and we plan to drive from San Francisco to Portland.
Hopefully I'll get to meet many of the people on the Ruby scene that I've only had mail contact with until now. See you there!
Hopefully I'll get to meet many of the people on the Ruby scene that I've only had mail contact with until now. See you there!
söndag, februari 11, 2007
Ragel performance
I did some performance testing on the old and new Resolver implementation. The testing have some stupid tests that exercise bad parts of both implementations (like longest match, where it can't be decided what type something is until we have to backtrack about 20 characters). I placed these 24 strings in an array, and pounded on it with an instance of the ResolverImpl that is used in exactly the same way on all scalar values in an YAML document. The objective is to find out if the value is an implicit type or not. So basically, we give it a String, and get back a tag URI. So it's not like I'm parsing a language or anything. I'm just doing some recognizing here.
The old implementation was based on a Map> where the first letter of the string to resolve was used as an index to find a list of patterns to try sequentially. This worked fine, and made it extensible. But not very fast. This is the baseline. For 24 different strings, iterated 100 000 times for 2 400 000 resolves it takes 7879ms. That's OK, but not great.
Now, the new Ragel implementation is dead simple. It's just a translation of the regexps in the aforementioned Pattern's into a state machine. At EOF out actions (%/ for people in the Ragel knowhow), I execute an action that sets a local variable to a tag, and at the end of the resolve method returns that tag. Dead simple, and not exercising the full strength of Ragel, of course.
So, for the same number of resolves, this ResolverImpl takes 1288ms. That's 611% improvement in speed. Ain't it nice to have a friend such as Ragel? And the best part is, for harder tasks, these improvements would be even larger.
Finite State Machines are your friends. All your base are belongs to us.
The old implementation was based on a Map
Now, the new Ragel implementation is dead simple. It's just a translation of the regexps in the aforementioned Pattern's into a state machine. At EOF out actions (%/ for people in the Ragel knowhow), I execute an action that sets a local variable to a tag, and at the end of the resolve method returns that tag. Dead simple, and not exercising the full strength of Ragel, of course.
So, for the same number of resolves, this ResolverImpl takes 1288ms. That's 611% improvement in speed. Ain't it nice to have a friend such as Ragel? And the best part is, for harder tasks, these improvements would be even larger.
Finite State Machines are your friends. All your base are belongs to us.
Results of jvYAMLb
Well, the YAML-based loading is in JRuby trunk. On the way, some parts of the codebase got seriously simplified. Very nice. The final result, with regard to performance, is about 20-30% on speed. But the important gain is in memory usage. The new implementation takes only about one fourth of the memory the original used. So that's great.
Regarding the Resolver, as I mentioned in the last post, it required a different approach, since regular JvYAML uses regular expressions to recognize implicit tags. Since that approach isn't good with byte arrays, I decided to use Ragel to generate a recognizer. That approach was very successful. As soon as I got that working it was the obvious approach. Ragel is good. Ragel is great. Ragel is wonderful. I will use the same approach for regular JvYAML to get away from all those Java regexps.
So, next step will be to do the same conversion of the emitter. Of course, at that point performance isn't that important. It's more about memory usage and the need to get away from another external dependency in JRuby.
Regarding the Resolver, as I mentioned in the last post, it required a different approach, since regular JvYAML uses regular expressions to recognize implicit tags. Since that approach isn't good with byte arrays, I decided to use Ragel to generate a recognizer. That approach was very successful. As soon as I got that working it was the obvious approach. Ragel is good. Ragel is great. Ragel is wonderful. I will use the same approach for regular JvYAML to get away from all those Java regexps.
So, next step will be to do the same conversion of the emitter. Of course, at that point performance isn't that important. It's more about memory usage and the need to get away from another external dependency in JRuby.
lördag, februari 10, 2007
Faster YAML with byte processing
As noted in my last post, I have started work on converting JvYAML into JvYAMLb. Right now I have finished the work on the Scanner and the Parser, and it's looking quite good. The numbers I reported in the last post for regular JvYAML performance was wrong though. We're looking at about 7.8s to 10.0s for scanning that 3.5MB gemspec file. (And that's only the scanning, not file IO). But with the Scanner converted to use bytes and ByteList, the same processing takes 2.8s. That's a substantial difference. But it doesn't end with that.
As I said I also converted the Parser. It doesn't do any String processing at all, so I didn't expect either a speedup or slowdown except for that from the Scanner. But... Before, parsing the gemspec took 18.515s, but after, it runs in 4s. That's a dramatic speedup, and I don't really know where it comes from. Unless the earlier implementation generated so much more garbage, and used more memory, that it was noticeable in speed. Anyway, this looks good for JRuby YAML processing, since I expect big reductions in complexity in the callpath and generation of objects after the YAML processor is byted all the way through.
But tomorrow it's time to work on the Resolver, and that's going to be hard. Optimally, it would be nice to have a byte-based Regexp engine. And maybe that would be something for JRuby too, know? Our Regular Expressions must be dead slow now that they have to convert to strings all the time.
As I said I also converted the Parser. It doesn't do any String processing at all, so I didn't expect either a speedup or slowdown except for that from the Scanner. But... Before, parsing the gemspec took 18.515s, but after, it runs in 4s. That's a dramatic speedup, and I don't really know where it comes from. Unless the earlier implementation generated so much more garbage, and used more memory, that it was noticeable in speed. Anyway, this looks good for JRuby YAML processing, since I expect big reductions in complexity in the callpath and generation of objects after the YAML processor is byted all the way through.
But tomorrow it's time to work on the Resolver, and that's going to be hard. Optimally, it would be nice to have a byte-based Regexp engine. And maybe that would be something for JRuby too, know? Our Regular Expressions must be dead slow now that they have to convert to strings all the time.
fredag, februari 09, 2007
Announcing JvYAMLb, a fork
The conversion to using byte-arrays as the basis of our String work in JRuby has led me to realize that JvYAML just doesn't cut it anymore. The performance wasn't good to begin with, and it's even worse having to convert EVERY SINGLE STRING read into bytes. That's no good. As an example why something needs to be done I'm going to describe the transformations that happen to data in JRuby if executing this code:
1. Bytes are read through the RAF, hopefully in chunks.
2. Those bytes are wrapped in a RubyString so they can be returned from the IO#read method.
3. An IOReader wraps that RubyIO object, gets the RubyString and converts it from bytes into a String, and this String gets converted into a char array.
4. That char array is returned to the YAML Scanner.
5. The chars from the char array is collected in a StringBuffer, and saved in various Strings as token values.
6. The parser, resolver and constructor work on these Strings in various ways.
7. The JRubyConstructor takes these Strings and creates RubyString objects from them and in the process converting the String back to a byte array.
Is there any doubt that this process is slow? Well, it hasn't been that big of a problem until now, since we are doing so well on performance in other parts of the system.
So, the radical decision is to rewrite JvYAML, making it more SYCK-compliant, working with InputStreams and byte-arrays, and in the process get away from several of the steps above. So that's what I'm going to do. I hereby create JvYAMLb. It will only be a part of the JRuby codebase, but it will be reasonably separate, so it can be extracted for other purposes. I will not stop work on regular JvYAML, but will maintain both projects.
Since the objective of this new project is blazing speed, I will post some numbers on this now and again. But first I will show you the speed of the regular system. JvYAML's Scanner can scan an old gem source index (about 3.5MB) of 435654 tokens in about 1654ms. This is the baseline I'm going to use to test performance, and I'll post more on this as soon as the byte-based Scanner is ready to try out.
YAML.load_file "gems.yml"First, the file is opened, and wrapped inside a RandomAccessFile. Then data is read from it by YAML. Reading will proceed like this:
1. Bytes are read through the RAF, hopefully in chunks.
2. Those bytes are wrapped in a RubyString so they can be returned from the IO#read method.
3. An IOReader wraps that RubyIO object, gets the RubyString and converts it from bytes into a String, and this String gets converted into a char array.
4. That char array is returned to the YAML Scanner.
5. The chars from the char array is collected in a StringBuffer, and saved in various Strings as token values.
6. The parser, resolver and constructor work on these Strings in various ways.
7. The JRubyConstructor takes these Strings and creates RubyString objects from them and in the process converting the String back to a byte array.
Is there any doubt that this process is slow? Well, it hasn't been that big of a problem until now, since we are doing so well on performance in other parts of the system.
So, the radical decision is to rewrite JvYAML, making it more SYCK-compliant, working with InputStreams and byte-arrays, and in the process get away from several of the steps above. So that's what I'm going to do. I hereby create JvYAMLb. It will only be a part of the JRuby codebase, but it will be reasonably separate, so it can be extracted for other purposes. I will not stop work on regular JvYAML, but will maintain both projects.
Since the objective of this new project is blazing speed, I will post some numbers on this now and again. But first I will show you the speed of the regular system. JvYAML's Scanner can scan an old gem source index (about 3.5MB) of 435654 tokens in about 1654ms. This is the baseline I'm going to use to test performance, and I'll post more on this as soon as the byte-based Scanner is ready to try out.
Bytes bites. Or maybe not.
Well, the byte arrays are in, for good and evil. We had to wrap them in a counterpart to StringBuffer, but backed by byte[] instead, since all that explicit allocation and deallocation was way unperformant.
Of course, we aren't seeing any performance benefits from this right now. The problem is that there is still many places that use IRubyObject#toString to get at the contents. That operation is very expensive right now, so gem installs are slower, for example. But we have good hopes on improving the situation, and many parts of the codebase have become much clearer without the need to do String-to-byte[] and byte[]-to-String all over the place.
Of course, we aren't seeing any performance benefits from this right now. The problem is that there is still many places that use IRubyObject#toString to get at the contents. That operation is very expensive right now, so gem installs are slower, for example. But we have good hopes on improving the situation, and many parts of the codebase have become much clearer without the need to do String-to-byte[] and byte[]-to-String all over the place.
tisdag, februari 06, 2007
Fractured blogging
My blogging in the future will be a little bit fractured (or more fractured, some might say), since I have been invited to write at Inside Java for APress. The address is http://java.apress.com and I do recommend that you subscribe to the feed if you're interested in Java or associated technologies. My first posting was about the two closure proposals for Java 7, and I will try to focus my posting there to be more Java specific, mostly in article format. But if I write anything I deem to be exceptionally good, I promise to link from here to there.
Go subscribe now!
Go subscribe now!
Serial JRuby
Things are really moving along faster than ever in JRuby land. It's so fun! As my last entry told you, Hpricot is now available for JRuby (and Java) people. I need to share a few lines from the logs of yesterday evenings conversation at #jruby:
Anyway, what's happened in JRuby world since last week? First, and most important, Charles has changed our RubyString implementation. It used to be backed by either a Java String or a StringBuffer. The problem with both of these is that Ruby has a tendency to use Strings as byte buckets. And our code was riddled with encoding and decoding into and out of byte arrays. So Charles took the big step, converted RubyString to use a byte-array instead, and fixed all the bugs that he found by doing that. The result is a happier codebase, less encoding and possibly faster Zlib and IO operations. That's big.
Tom is working on removing visibility and refactoring scopes. That could have huge impact too.
This Sunday I merged and fixed some code that allow Ruby code to inherit from Java classes and override methods there, and this overriding will be seen if an instance is sent back to Java. I'm planning on using this for some interesting tricks with Java ContentHandler's, and this functionality is really, really, really important. But it's also complex, since it requires generating bytecode at runtime. Fun, but hard. But now it's in trunk, and it's time to find the bugs in it and fix them.
I also need you to go read what Jonas Bonér has done with JRuby and OpenTerracotta. I could describe it here, but Jonas does a good job of it himself. So go there: http://jonasboner.com/2007/02/05/clustering-jruby-with-open-terracotta/. Very cool stuff, indeed!
So, the future is coming faster each day. JRuby will still conquer the world!
Some would say that what I do with JRuby is a waste of life... Well, we'll see about that.
<headius> seeya ola!
* shellac does some xsl-ing, plays on the wii,
then finds ola got HPRICOT working in that time
<shellac> I'm wasting my life
Anyway, what's happened in JRuby world since last week? First, and most important, Charles has changed our RubyString implementation. It used to be backed by either a Java String or a StringBuffer. The problem with both of these is that Ruby has a tendency to use Strings as byte buckets. And our code was riddled with encoding and decoding into and out of byte arrays. So Charles took the big step, converted RubyString to use a byte-array instead, and fixed all the bugs that he found by doing that. The result is a happier codebase, less encoding and possibly faster Zlib and IO operations. That's big.
Tom is working on removing visibility and refactoring scopes. That could have huge impact too.
This Sunday I merged and fixed some code that allow Ruby code to inherit from Java classes and override methods there, and this overriding will be seen if an instance is sent back to Java. I'm planning on using this for some interesting tricks with Java ContentHandler's, and this functionality is really, really, really important. But it's also complex, since it requires generating bytecode at runtime. Fun, but hard. But now it's in trunk, and it's time to find the bugs in it and fix them.
I also need you to go read what Jonas Bonér has done with JRuby and OpenTerracotta. I could describe it here, but Jonas does a good job of it himself. So go there: http://jonasboner.com/2007/02/05/clustering-jruby-with-open-terracotta/. Very cool stuff, indeed!
So, the future is coming faster each day. JRuby will still conquer the world!
Hpricot goodness
This is just so cool, I cannot contain it. For those of you who haven't heard about Hpricot, it is one of why the lucky stiff's incredibly cool tools (which he probably will use to take over the world any day now...). It's HTML parsing goodness, very flexible, with the goal of being able to parse (and fix) everything that Firefox handles.
"So what?" you're probably asking... Well, Hpricot uses Ragel and some C code to achieve blinding speed. This means JRuby can't run it. Or I should say couldn't run it:
What can you do with it? Well, I'm just going to point you to _why's own description of it. All he says at http://code.whytheluckystiff.net/hpricot/ will work fine in JRuby!
How did this come to be? Well, me and _why did some joint hacking, which was helped along by the fact that Adrian Thurston (the genius behind Ragel) recently added Java support to it. So, basically, most of the Ragel definition is exactly the same for both the C and the Java versions. The native code has been factored out, and both versions are buildable with rake from _why's code repository.
This is important. Don't think anything else. This strategy will, and can, be used for other gems with native parts. It's just a question of time.
"So what?" you're probably asking... Well, Hpricot uses Ragel and some C code to achieve blinding speed. This means JRuby can't run it. Or I should say couldn't run it:
That's right, Hpricot is now more promiscuous than any other gem with native parts.
orpheus:~/workspace/jruby> jruby bin/gem install hpricot --source http://code.whytheluckystiff.net
Bulk updating Gem source index for: http://code.whytheluckystiff.net
Select which gem to install for your platform (java)
1. hpricot 0.5.110 (jruby)
2. hpricot 0.5.110 (mswin32)
3. hpricot 0.5.110 (ruby)
4. hpricot 0.5 (ruby)
5. hpricot 0.5 (mswin32)
6. hpricot 0.5.0 (ruby)
7. hpricot 0.5.0 (mswin32)
8. hpricot 0.4.99 (ruby)
9. hpricot 0.4.99 (mswin32)
10. hpricot 0.4.92 (ruby)
11. hpricot 0.4.92 (mswin32)
12. Skip this gem
13. Cancel installation
> 1
Successfully installed hpricot-0.5.110-jruby
Installing ri documentation for hpricot-0.5.110-jruby...
Installing RDoc documentation for hpricot-0.5.110-jruby...
What can you do with it? Well, I'm just going to point you to _why's own description of it. All he says at http://code.whytheluckystiff.net/hpricot/ will work fine in JRuby!
How did this come to be? Well, me and _why did some joint hacking, which was helped along by the fact that Adrian Thurston (the genius behind Ragel) recently added Java support to it. So, basically, most of the Ragel definition is exactly the same for both the C and the Java versions. The native code has been factored out, and both versions are buildable with rake from _why's code repository.
This is important. Don't think anything else. This strategy will, and can, be used for other gems with native parts. It's just a question of time.
onsdag, januari 31, 2007
JFokus post-mortem
Yesterday marked the first one day conference in Sweden focused on Java. All in all, it was a resounding success. I am personally very happy about how well everything ran. The only part that wasn't really up to par was the panel discussion.
So, a quick rehash of the day. First Simon Phipps did his thing in the keynote, speaking much about where Sun's open source initiative comes from. I would have liked some more specific info on the future though, but I guess that's hard for him to say...
After that, Thorbiörn Fritzon from Sun in Sweden started of the Language track with talking about some new features in Mustang, specifically some Swing topics and the JSR 223 support (scripting). After that he showcased a really cool stack-based calculator he'd hacked during the weekend, where you could attach the functioning of the buttons to different scripts, and the scripts could be implemented in any language on the classpath that supported JSR 223. The final showdown of calculator buttons was first a script in Haskell (using Jaskell) that pushed a lambda on the stack. This lambda curried addition, with the first object on the stack. Then another script written in Python popped the lambda from the stack, popped the next value and called the lambda with the value from the stack, in effect doing an addition the longer way...
After that it was time for my presentation. It felt really good and I believe most in the audience was satisfied.
After that I sat in on Dan Bergh Johnsson from Omegapoint talking about domain driven design, and how to refactor to find your domain. I found it quite elementary, and nothing really new.
Last, Jonas Bonér talked about OpenTerracotta, which is an incredibly cool tool for clustering your JVM applications, and by doing that achieving many different cool things. Look it up! I wish there had been more time for Jonas to get in depth about the implementation of Terracotta, though. Most interesting thing he told us was about a situation where they wanted about 50-80 clients pounding in a cluster at the same time, and they used Terracotta for this too, using a CyclicBarrier that was synchronized over all the JVM's. Very nice.
The panel debate wasn't that interesting. Most of the questions asked by the moderator didn't really achieve any discussion. Open sourcing Java doesn't seem like such a controversial topic anymore.
After the debate, the conference was over, but then BEA provided a pub evening with entertainment, so we stayed and networked a little, talked with interesting people and generally had fun. After that, six of us continued to a pub in Stockholm, called Monk's Cafe, where we drank good Belgium beer and talked until our throats were sore. A very interesting day, no doubts.
So, a quick rehash of the day. First Simon Phipps did his thing in the keynote, speaking much about where Sun's open source initiative comes from. I would have liked some more specific info on the future though, but I guess that's hard for him to say...
After that, Thorbiörn Fritzon from Sun in Sweden started of the Language track with talking about some new features in Mustang, specifically some Swing topics and the JSR 223 support (scripting). After that he showcased a really cool stack-based calculator he'd hacked during the weekend, where you could attach the functioning of the buttons to different scripts, and the scripts could be implemented in any language on the classpath that supported JSR 223. The final showdown of calculator buttons was first a script in Haskell (using Jaskell) that pushed a lambda on the stack. This lambda curried addition, with the first object on the stack. Then another script written in Python popped the lambda from the stack, popped the next value and called the lambda with the value from the stack, in effect doing an addition the longer way...
After that it was time for my presentation. It felt really good and I believe most in the audience was satisfied.
After that I sat in on Dan Bergh Johnsson from Omegapoint talking about domain driven design, and how to refactor to find your domain. I found it quite elementary, and nothing really new.
Last, Jonas Bonér talked about OpenTerracotta, which is an incredibly cool tool for clustering your JVM applications, and by doing that achieving many different cool things. Look it up! I wish there had been more time for Jonas to get in depth about the implementation of Terracotta, though. Most interesting thing he told us was about a situation where they wanted about 50-80 clients pounding in a cluster at the same time, and they used Terracotta for this too, using a CyclicBarrier that was synchronized over all the JVM's. Very nice.
The panel debate wasn't that interesting. Most of the questions asked by the moderator didn't really achieve any discussion. Open sourcing Java doesn't seem like such a controversial topic anymore.
After the debate, the conference was over, but then BEA provided a pub evening with entertainment, so we stayed and networked a little, talked with interesting people and generally had fun. After that, six of us continued to a pub in Stockholm, called Monk's Cafe, where we drank good Belgium beer and talked until our throats were sore. A very interesting day, no doubts.
Prenumerera på:
Inlägg (Atom)
