A few days ago, me and two colleagues tried to track down a very tricky bug. After some hours looking, we finally found it, and it was actually due to a misconception that I had about the workings of HashSet and HashMap. I'm not sure if I'm the only one that didn't know this, but it's very logical once you've found it out. You see, if you save an object in a HashSet, and then change the object in such a way that the hashCode changes, then you won't find that object in the Set anymore. It will still be there, you will still iterate over it, but if you ask for example set.contains(obj), then it will return false. If you iterate over the set, and call Iterator#remove, this will silently fail to remove anything, since the HashSet can't find the object you want to remove. So, if you save things in a HashSet or use them as keys in a HashMap, make sure that the object is immutable, otherwise you'll get extremely hard-to-find bugs.
Incidentally, one of the best newsletters about Java programming wrote about this issue ages ago. Regardless if you work with Java professionally or just for fun I implore you to subscribe to JavaSpecialists by Dr Heinz Max Kabutz. It can be found here.
söndag, juni 18, 2006
lördag, juni 17, 2006
JRuby developments
I've spent another saturday at work, mostly working with KIMKAT, but taking some time to look at different JRuby-issues before the 0.9 release. The first - and only major - one, was that Zlib didn't seem to work for Charlie and Tom. I couldn't reproduce this error locally, which was very strange. But finally I found the error in my code, and also the reason I couldn't reproduce it in my own JRuby.
I'd managed to not remove a zlib.rb from one of the directories, and my test cases worked (albeit slowly), because it was using the former Zlib-code, and not my newritten one. After I found this problem, it went fairly fast to isolate the real error, in my GzipWriter. The problem was that I didn't call finish on the GZIPOutputStream which means the gzip-stream doesn't end with a gz-trailer. Once this was added, everything seems to work.
Problem number two seemed harder, initially, and the reason for this was that I needed to muck with the innards of JRuby threads to get it working. The problem concerns Signal handling. In Ruby you can trap operating system signals and execute blocks of code when they happen. This is obviously very powerful and useful for lots of things. My first solution for this, was to just save the block, and then call it when the signal occured. This didn't play well with the rest of JRuby, and Tom got some other strange errors from this. So, after some thinking I came to the realization that I had to create a new RubyThread, and start this when the signal occured, and then JRuby's internal thread scheduling would acertain that nothing untoward happens. This works really well, but I just realized one drawback with it, so I'm going back in to fix it now. The current approach can only execute the block once. Ouch.
So, now I've fixed a new approach. This is much more involved though. I had to get really into the internals of the thread architecture so I could actually create a new JRuby-thread with an existing RubyProc, Frame and Block object. So now I create a new RubyThread each time the signal is called, and sets the block-information manually. This is needed so we can run it multiple times.
Anyway, the real point of this blog is the last problem. The speed of RDoc. RDoc does some fairly heavy lifting, and it itsn't especially fast on C Ruby either:
Which actually calls the Ruby hash-method every time someone wants a plain hashCode. This explained some of the bad performance. I added some better hashCodes to different common Ruby-objects, like RubyString, RubyHash, RubyFixnum and others. After this small change I got this:
... That is almost double the speed for general JRuby performance. By just adding a few simple hashCode-methods at different places. So this is the lesson from all this: if you have performance problems; look at your hashCode(), it may be easy to fix!
I'd managed to not remove a zlib.rb from one of the directories, and my test cases worked (albeit slowly), because it was using the former Zlib-code, and not my newritten one. After I found this problem, it went fairly fast to isolate the real error, in my GzipWriter. The problem was that I didn't call finish on the GZIPOutputStream which means the gzip-stream doesn't end with a gz-trailer. Once this was added, everything seems to work.
Problem number two seemed harder, initially, and the reason for this was that I needed to muck with the innards of JRuby threads to get it working. The problem concerns Signal handling. In Ruby you can trap operating system signals and execute blocks of code when they happen. This is obviously very powerful and useful for lots of things. My first solution for this, was to just save the block, and then call it when the signal occured. This didn't play well with the rest of JRuby, and Tom got some other strange errors from this. So, after some thinking I came to the realization that I had to create a new RubyThread, and start this when the signal occured, and then JRuby's internal thread scheduling would acertain that nothing untoward happens. This works really well, but I just realized one drawback with it, so I'm going back in to fix it now. The current approach can only execute the block once. Ouch.
So, now I've fixed a new approach. This is much more involved though. I had to get really into the internals of the thread architecture so I could actually create a new JRuby-thread with an existing RubyProc, Frame and Block object. So now I create a new RubyThread each time the signal is called, and sets the block-information manually. This is needed so we can run it multiple times.
Anyway, the real point of this blog is the last problem. The speed of RDoc. RDoc does some fairly heavy lifting, and it itsn't especially fast on C Ruby either:
$ time ruby d:/programming/ruby/bin/gem install rakeBut now, JRuby isn't near this speed in any way at all:
Attempting local installation of 'rake'
Successfully installed rake, version 0.7.1
Installing RDoc documentation for rake-0.7.1...
real 0m13.915s
user 0m0.000s
sys 0m0.000s
$ time bin/jruby.bat bin/gem install rakeYes, it's about one magnitude slower. Yesterday I found a big reason for this. JRuby does a lot of hashing and spends very much time in HashMap#get. So I was curious what hashCode-algorithm was used internally. What I found was this:
Attempting local installation of 'rake'
Successfully installed rake, version 0.7.1
Installing RDoc documentation for rake-0.7.1...
real 1m42.985s
user 0m0.000s
sys 0m0.010s
public final int hashCode() {
return RubyNumeric.fix2int(callMethod("hash"));
}
Which actually calls the Ruby hash-method every time someone wants a plain hashCode. This explained some of the bad performance. I added some better hashCodes to different common Ruby-objects, like RubyString, RubyHash, RubyFixnum and others. After this small change I got this:
$ time bin/jruby.bat bin/gem install rake
Attempting local installation of 'rake'
Successfully installed rake, version 0.7.1
Installing RDoc documentation for rake-0.7.1...
real 0m59.946s
user 0m0.000s
sys 0m0.010s
... That is almost double the speed for general JRuby performance. By just adding a few simple hashCode-methods at different places. So this is the lesson from all this: if you have performance problems; look at your hashCode(), it may be easy to fix!
torsdag, juni 15, 2006
Working towards 0.9
These last days I've been focusing on fixing issues with RubyGems and Rails, so that JRuby 0.9 will be a really great release. It already is, of course, but all we can do to make it better feels nice. These are the things I've done since sunday.
StringIO
I finally took the time to rewrite all of StringIO to Java, and also really test so that it works correctly. I managed to make the more common usage patterns between 8 and 10 times faster, which I feel is sufficient for now.
Signal and Kernel#trap
Ruby has a close ties to C, which means you can trap low level POSIX signals quite easily. This is hard to support in Java, though. One common use case that most Ruby programs do, is to trap INT so they can break the program gracefully. Rails and WEBrick does this. The current Kernel#trap and Signal was just stubbed out. I found a way to support signal handling on Sun JVM's at least, through the undocumented class sun.misc.Signal. JRuby feels if this is available, if so it uses it, otherwise trapping signals doesn't work. The implementation is really easy, I just grab the block provided and saves this in a Runnable that will be executed when the trap happens.
Zlib, IOInputStream and IOOutputStream
When working with Ruby IO-like objects from Java, it is often very convenient to wrap these in a Input/Output-stream. This isn't totally obvious how to do, though. My first implementation worked, but was intolerably slow. My last post about the plaincharset tells the tale how these things can go wrong. Suffice to say, I managed to fix the streams pretty good, and then started the real work: Reimplementing Zlib::GzipReader and Zlib::GzipWriter in Java. This wasn't as hard as I thought, once I got the IO-streams working as it should.
There was one bug which was really hard to find though, and it was caused by a minor difference between Java's read(n) and Ruby's read(n) methods. In Java, if we ask to read n bytes, we don't necessarily get all the bytes we asked for, even if there are that many bytes available. This is why those read-methods return how many bytes actually were read. But Ruby's read(n) doesn't act this way. It either reads n bytes, or to end of stream, depending on which come first. That one took a while to find.
The reimplementation of Zlib isn't complete yet, but the important classes are done. The deflater and inflater classes already use a backing Java class for performance, and the checksum classes don't need that kind of speed. The performance improvement from reimplementing GzipReader into Java was great, though. It seems to be between 15 and 20 times faster, most often. RubyGems is really useful now.
What's left?
Now, these things are quite minor. They improve different parts of Ruby that are used quite often. RubyGems seem to work more or less perfectly. Rails generation works too. The server-script for Rails almost works, but there is something strange going on with WEBrick yet. I'm wondering if this has something to do with our Socket-code, which I'm not totally into yet. But these are the issues I'll be looking at before 0.9.
In the long (longer at least), I have two main points that interest me. First I want to complete JvYAML, and integrate it with JRuby. And secondly I'm thinking about ways to byte compile (to some Ruby bytecode) parts of the AST tree, much in the same way Charles have been toying with compiling parts of it to Java bytecode. This is mostly just ideas in my head yet, but that's probably something I'll write more about quite soon here.
StringIO
I finally took the time to rewrite all of StringIO to Java, and also really test so that it works correctly. I managed to make the more common usage patterns between 8 and 10 times faster, which I feel is sufficient for now.
Signal and Kernel#trap
Ruby has a close ties to C, which means you can trap low level POSIX signals quite easily. This is hard to support in Java, though. One common use case that most Ruby programs do, is to trap INT so they can break the program gracefully. Rails and WEBrick does this. The current Kernel#trap and Signal was just stubbed out. I found a way to support signal handling on Sun JVM's at least, through the undocumented class sun.misc.Signal. JRuby feels if this is available, if so it uses it, otherwise trapping signals doesn't work. The implementation is really easy, I just grab the block provided and saves this in a Runnable that will be executed when the trap happens.
Zlib, IOInputStream and IOOutputStream
When working with Ruby IO-like objects from Java, it is often very convenient to wrap these in a Input/Output-stream. This isn't totally obvious how to do, though. My first implementation worked, but was intolerably slow. My last post about the plaincharset tells the tale how these things can go wrong. Suffice to say, I managed to fix the streams pretty good, and then started the real work: Reimplementing Zlib::GzipReader and Zlib::GzipWriter in Java. This wasn't as hard as I thought, once I got the IO-streams working as it should.
There was one bug which was really hard to find though, and it was caused by a minor difference between Java's read(n) and Ruby's read(n) methods. In Java, if we ask to read n bytes, we don't necessarily get all the bytes we asked for, even if there are that many bytes available. This is why those read-methods return how many bytes actually were read. But Ruby's read(n) doesn't act this way. It either reads n bytes, or to end of stream, depending on which come first. That one took a while to find.
The reimplementation of Zlib isn't complete yet, but the important classes are done. The deflater and inflater classes already use a backing Java class for performance, and the checksum classes don't need that kind of speed. The performance improvement from reimplementing GzipReader into Java was great, though. It seems to be between 15 and 20 times faster, most often. RubyGems is really useful now.
What's left?
Now, these things are quite minor. They improve different parts of Ruby that are used quite often. RubyGems seem to work more or less perfectly. Rails generation works too. The server-script for Rails almost works, but there is something strange going on with WEBrick yet. I'm wondering if this has something to do with our Socket-code, which I'm not totally into yet. But these are the issues I'll be looking at before 0.9.
In the long (longer at least), I have two main points that interest me. First I want to complete JvYAML, and integrate it with JRuby. And secondly I'm thinking about ways to byte compile (to some Ruby bytecode) parts of the AST tree, much in the same way Charles have been toying with compiling parts of it to Java bytecode. This is mostly just ideas in my head yet, but that's probably something I'll write more about quite soon here.
Java Charsets
Today I've spent some time pounding java.lang.String to give me a byte array I can use as a data format. This seems harder than I thought it should be. So, why can't I use getBytes() or getBytes("ascii") or getBytes("iso-8859-1") or getBytes("utf-8")? Those are fine for certain tasks, but I'm looking for a very specific translation from chars to bytes. The application I was working on is Zlib in Java, for JRuby. Since Ruby have the somewhat funny custom of using Strings as byte buffers this means the output I get from a Ruby IO-operation is a RubyString.
The reason I started trying different paths for this was that Zlib didn't work as it should. Not at all. I knew it worked when I did it one char at a time, because then I casted the char to an int instead (since InputStream#read() returns a int). So, I created this small program:
to see what happened here. Now, I won't bore you with the complete printout from this. But there are a few specific portions that I'd like to share:
Those 63-values keep showing up and destroying everything. If I try another encoding in the getBytes-method it actually gets worse. I couldn't find any way to get this to write the expected output. So, I embarked on a quest. A quest to solve this small trouble, forever and always. The result is plaincharset, a small project consisting of 4 classes. Nothing spectacular, but if you add the jar-file to your classpath you can now use the charset name "PLAIN" to get every byte correctly from getBytes and new String. If you have characters that are not within 0..255 I cannot guarantee anything at all. I hereby release the project in the public domain. The source can be found here, and if you just want the jar-file, download it here.
So, what is the secret behind this marvel? In one word: NIO. The jar-file contains a subclass of CharsetProvider, a subclass of Charset, one CharsetDecoder and one CharsetEncoder. The only classes with anything in them is the decoder and encoder, which gets an input NIO-buffer and an output NIO-buffer. I just read from the input and write to the output, casting where necessary. There is also one service-provider file in the META-INF directory in the jar, which says to use the com.ologix.charset.PlainCharsetProvider as a provider for charsets.
And did this work for my Zlib-implementation? I'm happy to say that it did. It works very well and is both smaller in code length, and much, much faster. I'm happy.
The reason I started trying different paths for this was that Zlib didn't work as it should. Not at all. I knew it worked when I did it one char at a time, because then I casted the char to an int instead (since InputStream#read() returns a int). So, I created this small program:
final byte[] chrs = new byte[256];
for(int i=0,j=chrs.length;i<j;i++) {
chrs[i] = (byte)i;
}
final String str = new String(chrs);
final byte[] bts = str.getBytes();
for(int i=0,j=chrs.length;i<j;i++) {
System.out.println("[" + i + "]= " + (int)chrs[i] + ", " + bts[i] + " ... should be: " + (byte)chrs[i]);
}
to see what happened here. Now, I won't bore you with the complete printout from this. But there are a few specific portions that I'd like to share:
[127]= 127, 127 ... should be: 127and
[128]= -128, -128 ... should be: -128
[129]= -127, 63 ... should be: -127
[130]= -126, -126 ... should be: -126
[140]= -116, -116 ... should be: -116
[141]= -115, 63 ... should be: -115
[142]= -114, -114 ... should be: -114
[143]= -113, 63 ... should be: -113
[144]= -112, 63 ... should be: -112
[145]= -111, -111 ... should be: -111
[156]= -100, -100 ... should be: -100
[157]= -99, 63 ... should be: -99
[158]= -98, -98 ... should be: -98
Those 63-values keep showing up and destroying everything. If I try another encoding in the getBytes-method it actually gets worse. I couldn't find any way to get this to write the expected output. So, I embarked on a quest. A quest to solve this small trouble, forever and always. The result is plaincharset, a small project consisting of 4 classes. Nothing spectacular, but if you add the jar-file to your classpath you can now use the charset name "PLAIN" to get every byte correctly from getBytes and new String. If you have characters that are not within 0..255 I cannot guarantee anything at all. I hereby release the project in the public domain. The source can be found here, and if you just want the jar-file, download it here.
So, what is the secret behind this marvel? In one word: NIO. The jar-file contains a subclass of CharsetProvider, a subclass of Charset, one CharsetDecoder and one CharsetEncoder. The only classes with anything in them is the decoder and encoder, which gets an input NIO-buffer and an output NIO-buffer. I just read from the input and write to the output, casting where necessary. There is also one service-provider file in the META-INF directory in the jar, which says to use the com.ologix.charset.PlainCharsetProvider as a provider for charsets.
And did this work for my Zlib-implementation? I'm happy to say that it did. It works very well and is both smaller in code length, and much, much faster. I'm happy.
The arcitecture of a Meta Directory system
One of the major projects at Karolinska Institutet the last 3 years is called KIMKAT. It has gotten the moniker of a meta directory system, but in many ways that's not entirely correct. Since I've been one of the lead architects and developers in this project, I wanted to write about the technical architecture for the system, what choices panned out well, and things I would change if I could.
The problem
Karolinska Institutet is a fair sized university (quite big for Sweden). We have about 20k students and something like 5-7000 employees. Universities have a tendency to become decentralized, with local solutions for all problems. KI is no exception. Information about people at KI exists in more than 10 disparate systems only at the administration. This information costs much money to keep fresh, and it's also very inefficient. The vision with KIMKAT is to have one central source for all datums about persons, organizations and resources, where external systems can find up-to-date and current data. It should also be possible for these systems to contribute domain specific data into KIMKAT. (For example, our phone directory system should probably be the source for phone numbers.)
Different parts of the solution
We have worked with this problem in a project which has oscillated between 10 and 20 project members during 2½ years. Of course, there are many ways to solve this kind of problem, and the main differentiation between them is how much of a hack you want it to be. KI specifically wanted to avoid yet another hack solution (since other uni's in Sweden has gone down this route, and it seems both costly and becomes unmaintainable in only one or two years), so the focus of the the project group was to find and implement a solution that would hold for many years to come.
The first problem was that we had no definite source for organization information. Neither did we have anything for our affiliates. All that information was stored ad-hoc in the mail system, and on paper. So, we instigated a regime where each datum of information has one primary source which is responsible for maintaining and updating it. For employees this is our HR-system (called Primula). For students, it's our student database called LADOK. For organizations we created a new primary source called KOrg, and for affiliates KAff. We also created a database for all KIMKAT information that we couldn't write back to the other primary sources. This effectively became it's own primary source.
Our meta directory solution is based around a meta engine; a scripted system that reads and collects data from our primary sources and writes this in a unified data structure to the central KIMKAT database OmniaS. For this task we evaluated several different products, and also considered writing our own, but in the end we chose IBM Tivoli Directory Integrator, which is Java-based and very easy to use for easier situations. It uses BSF to allow scripting, which is almost always needed for more intricate solutions. Suffice to say, our final system contains lots of ITDI scripting.
The OmniaS database are surrounded with an EJB tier with Hibernate and accessors for reading the data in a well defined onthology implemented with JavaBeans. There are no Entity EJB's in KIMKAT. (Nor anywhere else at KI, as far as I know - and hope.)
Right now the primary user of the OmniaS information is our web interface, KKAWeb, which is used to update information in our primary sources. It's also used to establish new affiliations and organizations. There are some information the we deemed was necessary for KKAWeb and other applications that would eventually display KIMKAT-data in some way, but wasn't really business data. Because of this we created an external data source called KDis for this data. KKAWeb and another application called KIKAT uses this data extensively, but mostly for things like I18N and sorting.
Since our primary sources are of a very diverse nature, KKAWeb doesn't write to them directly. Instead the updated value objects are sent to an update topic with JMS, and every primary source has one or more message driven beans listening for just those messages that pertains to it. Then that bean updates the database.
The final part of the puzzle is another topic which ITDI sends all updates to, as soon as something is changed in one of the primary sources. This allows external systems to get hold of change events in whatever way they need. The change information is sent with JMS using S-expressions to represent data, since not all consumers will be Java-based.
Lessons learned
The KIMKAT project is late, and the deadline have been put off several times. There are several reasons for this, but the main one is probably due to our inability to give good time estimates. There are a few architectural and design decisions that I would probably have done another way if we would start everything over. The biggest one is there data source called KDis. That was a big mistake for several reasons. First of all, having joins between different databases is a pain. And having database constraints between databases are also really cumbersome.
If we did this again, I would probably drop one of the tiers between the sesssion EJB's and the OmniaS database. Right now, much time goes to serialize and deserialize between different kinds of value objects.
And third, our updating service is a really good idea, actually, but I think that the implementation could have been done in a better way. There is something nagging me with it, but I can't but the finger on it right now.
But all in all, our architecture have been really successful, and it feels like something that will be able to stand the test of time. Now we just have to integrate all other systems with this, so everyone gets all the real benefits from this solution.
The problem
Karolinska Institutet is a fair sized university (quite big for Sweden). We have about 20k students and something like 5-7000 employees. Universities have a tendency to become decentralized, with local solutions for all problems. KI is no exception. Information about people at KI exists in more than 10 disparate systems only at the administration. This information costs much money to keep fresh, and it's also very inefficient. The vision with KIMKAT is to have one central source for all datums about persons, organizations and resources, where external systems can find up-to-date and current data. It should also be possible for these systems to contribute domain specific data into KIMKAT. (For example, our phone directory system should probably be the source for phone numbers.)
Different parts of the solution
We have worked with this problem in a project which has oscillated between 10 and 20 project members during 2½ years. Of course, there are many ways to solve this kind of problem, and the main differentiation between them is how much of a hack you want it to be. KI specifically wanted to avoid yet another hack solution (since other uni's in Sweden has gone down this route, and it seems both costly and becomes unmaintainable in only one or two years), so the focus of the the project group was to find and implement a solution that would hold for many years to come.
The first problem was that we had no definite source for organization information. Neither did we have anything for our affiliates. All that information was stored ad-hoc in the mail system, and on paper. So, we instigated a regime where each datum of information has one primary source which is responsible for maintaining and updating it. For employees this is our HR-system (called Primula). For students, it's our student database called LADOK. For organizations we created a new primary source called KOrg, and for affiliates KAff. We also created a database for all KIMKAT information that we couldn't write back to the other primary sources. This effectively became it's own primary source.
Our meta directory solution is based around a meta engine; a scripted system that reads and collects data from our primary sources and writes this in a unified data structure to the central KIMKAT database OmniaS. For this task we evaluated several different products, and also considered writing our own, but in the end we chose IBM Tivoli Directory Integrator, which is Java-based and very easy to use for easier situations. It uses BSF to allow scripting, which is almost always needed for more intricate solutions. Suffice to say, our final system contains lots of ITDI scripting.
The OmniaS database are surrounded with an EJB tier with Hibernate and accessors for reading the data in a well defined onthology implemented with JavaBeans. There are no Entity EJB's in KIMKAT. (Nor anywhere else at KI, as far as I know - and hope.)
Right now the primary user of the OmniaS information is our web interface, KKAWeb, which is used to update information in our primary sources. It's also used to establish new affiliations and organizations. There are some information the we deemed was necessary for KKAWeb and other applications that would eventually display KIMKAT-data in some way, but wasn't really business data. Because of this we created an external data source called KDis for this data. KKAWeb and another application called KIKAT uses this data extensively, but mostly for things like I18N and sorting.
Since our primary sources are of a very diverse nature, KKAWeb doesn't write to them directly. Instead the updated value objects are sent to an update topic with JMS, and every primary source has one or more message driven beans listening for just those messages that pertains to it. Then that bean updates the database.
The final part of the puzzle is another topic which ITDI sends all updates to, as soon as something is changed in one of the primary sources. This allows external systems to get hold of change events in whatever way they need. The change information is sent with JMS using S-expressions to represent data, since not all consumers will be Java-based.
Lessons learned
The KIMKAT project is late, and the deadline have been put off several times. There are several reasons for this, but the main one is probably due to our inability to give good time estimates. There are a few architectural and design decisions that I would probably have done another way if we would start everything over. The biggest one is there data source called KDis. That was a big mistake for several reasons. First of all, having joins between different databases is a pain. And having database constraints between databases are also really cumbersome.
If we did this again, I would probably drop one of the tiers between the sesssion EJB's and the OmniaS database. Right now, much time goes to serialize and deserialize between different kinds of value objects.
And third, our updating service is a really good idea, actually, but I think that the implementation could have been done in a better way. There is something nagging me with it, but I can't but the finger on it right now.
But all in all, our architecture have been really successful, and it feels like something that will be able to stand the test of time. Now we just have to integrate all other systems with this, so everyone gets all the real benefits from this solution.
måndag, juni 12, 2006
A YAML dumper in Java
Today I've begun work on porting the RbYAML dumper to Java. The work will be done in three separate layers: the emitter, the serializer and the representer. I've just decided the structure for the emitter, but the basic premise is really simple since emitting is just about low level IO stuff.
The emitter will be a stack based finite state machine, and it actually closeley resembles the ParserImpl part of the loader. There are 17 states and they are represented as anonymous implementations of a simple interface. If this was a Java 5 project, I would've implemented it with an Enums, like this:
with anonymous Enum subclasses for each state. A very fine solution, since I can just fetch the next state and execute it directly, instead of doing something like this:
Alas, that's not the way it will be.
One of the more interesting parts of this implemention will be finding a nice way to represent the options hash from Syck and RbYAML 0.2. With standard Ruby keyword arguments, the options hash is very practical to use, instead of having to supply 17 different arguments - or even worse fact(17) different constructors to allow for default arguments. The correct way in Ruby is to have a defaults Hash, merge this with the arguments provided, and lookup the configurations options here. Obviously, I could do this with maps, but it's very cumbersome and requires more than one line to provide an option; something I would like to avoid.
The solution to this dilemma came from a part of Joshua Blochs Effective Java technical session at JavaOne. (I've written a little about it in my entry on JavaOne, day 2). Anyway, the solution is to provide an YAMLOptions class with a build factory method. I'm not totally finished with the syntax yet, but I'm thinking that a toYaml call could look something like this:
There are a few benefits with this syntax. The first is obvious: it takes a whole lot of options to span this over more than one line, and if you want that many, maybe you should save this options-object as a constant somewhere? Another great benefit is type safety. This isn't really an issue in Ruby, since there are no casts, but in Java I like the guarentee that the indent-option is an integer. I can save the defaults inside YAMLOptions, and if I define an interface for the YAMLOptions someone can implement it, while providing more options if needed. For example, right now I'm sure that JRubyYAMLOptions will be implemented; probably as a subclass to YAMLOptions, with some new options that only the JRuby-specific representers will use. Oh how I would love to be able to specify that static method build()with an interface...
The emitter will be a stack based finite state machine, and it actually closeley resembles the ParserImpl part of the loader. There are 17 states and they are represented as anonymous implementations of a simple interface. If this was a Java 5 project, I would've implemented it with an Enums, like this:
public enum EmitterState {
StreamStart {
public void expect(final EmitterEnvironment env) {
env.expectStreamStart();
}
}
}
with anonymous Enum subclasses for each state. A very fine solution, since I can just fetch the next state and execute it directly, instead of doing something like this:
STATES[STREAM_START].expect();
Alas, that's not the way it will be.
One of the more interesting parts of this implemention will be finding a nice way to represent the options hash from Syck and RbYAML 0.2. With standard Ruby keyword arguments, the options hash is very practical to use, instead of having to supply 17 different arguments - or even worse fact(17) different constructors to allow for default arguments. The correct way in Ruby is to have a defaults Hash, merge this with the arguments provided, and lookup the configurations options here. Obviously, I could do this with maps, but it's very cumbersome and requires more than one line to provide an option; something I would like to avoid.
The solution to this dilemma came from a part of Joshua Blochs Effective Java technical session at JavaOne. (I've written a little about it in my entry on JavaOne, day 2). Anyway, the solution is to provide an YAMLOptions class with a build factory method. I'm not totally finished with the syntax yet, but I'm thinking that a toYaml call could look something like this:
YAML.toYaml(theObj, YAMLOptions.build().useVersion(true).useDouble(true).indent(4));
There are a few benefits with this syntax. The first is obvious: it takes a whole lot of options to span this over more than one line, and if you want that many, maybe you should save this options-object as a constant somewhere? Another great benefit is type safety. This isn't really an issue in Ruby, since there are no casts, but in Java I like the guarentee that the indent-option is an integer. I can save the defaults inside YAMLOptions, and if I define an interface for the YAMLOptions someone can implement it, while providing more options if needed. For example, right now I'm sure that JRubyYAMLOptions will be implemented; probably as a subclass to YAMLOptions, with some new options that only the JRuby-specific representers will use. Oh how I would love to be able to specify that static method build()with an interface...
JvYAML, RbYAML and JRuby
There has been much activity on several points these last weeks, but the guiding light has been on getting JRuby as good as possible. JRuby 0.9 will arrive sometime this or next week, and to that point I managed to rework the emitter in RbYAML and release version 0.2 a few days ago. This release will be a part of JRuby 0.9. About 10 days ago I finished the parser implementation for JvYAML and released version 0.1 of it.
Today I've spent some time working on porting the emitter from RbYAML to Java too, but I haven't really gotten that far yet. I hope I can get it finished in time for JRuby 0.9, but it's no big catastrophe if not. The essential part with YAML emitting is that it's correct, not fast. But it would be really nice to get it included in this major release too. We'll have to see.
I realize I haven't really blogged about neither RbYAML 0.2 or JvYAML, just announcing them. But don't worry, it will arrive shortly.
As for now, the more interesting thing is JRuby. The 0.9 release will contain some truly spectacular things, and I'm prepared to say that JRuby is now usable in many commercial environments. We have RubyGems working correctly (albeit slowly, right now), we have the initial parts of Rails, we have the ActiveRecord JDBC connector, we have massive speed improvements. We have also received official sanctioning from Matz to include C Ruby's core libraries in the JRuby distribution (this means we no longer have to hack fileutils.rb each time checking out a new copy of JRuby). The IO work from Evan Buswell means Webrick will very soon work correctly from JRuby. All in all, it's a really major improvement, and I'm proud of being part of it. Stay tuned for more info.
Today I've spent some time working on porting the emitter from RbYAML to Java too, but I haven't really gotten that far yet. I hope I can get it finished in time for JRuby 0.9, but it's no big catastrophe if not. The essential part with YAML emitting is that it's correct, not fast. But it would be really nice to get it included in this major release too. We'll have to see.
I realize I haven't really blogged about neither RbYAML 0.2 or JvYAML, just announcing them. But don't worry, it will arrive shortly.
As for now, the more interesting thing is JRuby. The 0.9 release will contain some truly spectacular things, and I'm prepared to say that JRuby is now usable in many commercial environments. We have RubyGems working correctly (albeit slowly, right now), we have the initial parts of Rails, we have the ActiveRecord JDBC connector, we have massive speed improvements. We have also received official sanctioning from Matz to include C Ruby's core libraries in the JRuby distribution (this means we no longer have to hack fileutils.rb each time checking out a new copy of JRuby). The IO work from Evan Buswell means Webrick will very soon work correctly from JRuby. All in all, it's a really major improvement, and I'm proud of being part of it. Stay tuned for more info.
A few hours with Emacs
Today I've spent some hours upgrading Emacs to version 22, and getting various stuff to work. It was an interesting experience, and I've found lots of neat tricks I didn't know about before. But first, Emacs 22! This is in no way a released version, it is still in beta and reportedly has some bugs left in the code base. That said, I've been programming and working in it during the day and not experienced any trouble at all.
The new version brings some major updates, especially in the department of internationalization. The support looks great, actually. Another cool feature that Steve Yegge have blogged about (see link to the right), is the new support for embedding elisp inside replacement in "query-replace-regexp". That's really neat.
First installation wasn't totally smooth; my JDEE environment didn't really want to play, but after a while of fiddling with the code for it, I noticed that a new minor version was available. I added this to my local installation and everything justed worked. Even my own JDEE extensions continued working.
I've never really liked the completion in JDEE, but now I've finally written a small script to make is usable when I want it, at least:
This code does code-completion if at end of a line, which is the most common case when you need completion. It's important to have the jde-global-classpath point to all available classes, though. (I've created a global prj.el with common options preset for JDEE, and loads this from my various projects prj-files.) Completion is still slow, especially the first time, but when you really need it, it's there. I toyed with getting dabbrev or pabbrev in there also, but that just doesn't seem right for Java.
Another nice feature I got working was integrating imenu with JDEE, so I can Shift-right-click in any Java buffer and get a nice menu list of all class definitions with method and field hyperlinks. That's probably the only thing I'll use the mouse for in Emacs, ever, and it's more like using it as a key anyway. The only annoying thing with imenu is that I want it to sort the classes alphabetically. I've set this both with imenu-sort-function and jde-imenu-sort, but it doesn't really bite.
What I haven't had time to test yet - and the things that'll probably need some work - is my tramp-paths for interactively working with files on other computers through Emacs and plink (which also enables me to use Slime for LISP editing in-process on other servers). Right now this doesn't seem to work, but I read somewhere that the plink-support has been upgraded in Emacs 22.
But all in all I'm very satisfied. I recommend any Emacs maven to try out the new version, and especially if you need to work in Cygwin or on the Mac. The support for these environments are now there.
The new version brings some major updates, especially in the department of internationalization. The support looks great, actually. Another cool feature that Steve Yegge have blogged about (see link to the right), is the new support for embedding elisp inside replacement in "query-replace-regexp". That's really neat.
First installation wasn't totally smooth; my JDEE environment didn't really want to play, but after a while of fiddling with the code for it, I noticed that a new minor version was available. I added this to my local installation and everything justed worked. Even my own JDEE extensions continued working.
I've never really liked the completion in JDEE, but now I've finally written a small script to make is usable when I want it, at least:
(defun indent-or-complete-jde ()
"Complete if point is at end of a line, otherwise indent line."
(interactive)
(if (looking-at "$")
(jde-complete)
(indent-for-tab-command)))
(define-key jde-mode-map [tab] 'indent-or-complete-jde)
This code does code-completion if at end of a line, which is the most common case when you need completion. It's important to have the jde-global-classpath point to all available classes, though. (I've created a global prj.el with common options preset for JDEE, and loads this from my various projects prj-files.) Completion is still slow, especially the first time, but when you really need it, it's there. I toyed with getting dabbrev or pabbrev in there also, but that just doesn't seem right for Java.
Another nice feature I got working was integrating imenu with JDEE, so I can Shift-right-click in any Java buffer and get a nice menu list of all class definitions with method and field hyperlinks. That's probably the only thing I'll use the mouse for in Emacs, ever, and it's more like using it as a key anyway. The only annoying thing with imenu is that I want it to sort the classes alphabetically. I've set this both with imenu-sort-function and jde-imenu-sort, but it doesn't really bite.
What I haven't had time to test yet - and the things that'll probably need some work - is my tramp-paths for interactively working with files on other computers through Emacs and plink (which also enables me to use Slime for LISP editing in-process on other servers). Right now this doesn't seem to work, but I read somewhere that the plink-support has been upgraded in Emacs 22.
But all in all I'm very satisfied. I recommend any Emacs maven to try out the new version, and especially if you need to work in Cygwin or on the Mac. The support for these environments are now there.
söndag, juni 11, 2006
Announcing RbYAML version 0.2
Another major release, with most changes in the dumper:
http://rbyaml.rubyforge.org
or to download directly:
http://rubyforge.org/frs/?group_id=1658
Changes:
http://rbyaml.rubyforge.org
or to download directly:
http://rubyforge.org/frs/?group_id=1658
Changes:
- Performance has been greatly improved
- Rewritten the representer to use a distributed representation model
- Much improvement of test cases
- And many bug fixes
tisdag, juni 06, 2006
Announcing JvYAML.
I am pleased to announce JvYAML, version 0.1. JvYAML is a Java YAML 1.1 loader that is both easy to extend and easy to use. JvYAML originated in the JRuby project (http://jruby.sourceforge.net), from the base of RbYAML (http://rbyaml.rubyforge.org). For a long time Java have lacked a good YAML loader and dumper with all the features that the SYCK using scripting communities have gotten used to. JvYAML aims to rectify this.
Of major importance is that JvYAML works the same way as SYCK, so that JRuby can rely on YAML parsing and emitting that mirrors C Ruby.
JvYAML is a clean port of RbYAML, which was a port from Python code written by Kirill Simonov for PyYAML3000.
Simple usage:
More information:
At java.net: http://jvyaml.dev.java.net
Download: https://jvyaml.dev.java.net/servlets/ProjectDocumentList
License:
JvYAML is distributed with the MIT license.
Of major importance is that JvYAML works the same way as SYCK, so that JRuby can rely on YAML parsing and emitting that mirrors C Ruby.
JvYAML is a clean port of RbYAML, which was a port from Python code written by Kirill Simonov
Simple usage:
import org.jvyaml.YAML;There is also support for more advanced loading of JavaBeans with automatic setting of properties with the use of domain tags in the YAML document.
Map configuration = (Map)YAML.load(new FileReader("c:/projects/ourSpecificConfig.yml"));
List values = (List)YAML.load("--- \n- A\n- b\n- c\n");
More information:
At java.net: http://jvyaml.dev.java.net
Download: https://jvyaml.dev.java.net/servlets/ProjectDocumentList
License:
JvYAML is distributed with the MIT license.
lördag, juni 03, 2006
Transforming RbYAML
RbYAML went through some big changes from release 0.0.2 to 0.1. My intentions are to detail some of these changes, what implementation choices I did, and why.
First, conversion from Mixins to Classes. The original Python implementation used multiple inheritance, created several base classes (Reader, Scanner, Parser, etc) and then created one several versions of a Loader class which inherited from the different base classes. My first implementation mirrored this approach, but used Modules instead of base classes and mixed in different versions of these in the different Loader classes. This approach was quite limiting since mixing in code into other Modules doesn't really work as you expect, and this is no substitute for subclassing. For example, I had a BaseResolver module, a SafeResolver module which mixed in BaseResolver and added code of it's own, but this were quite cumbersome.
The solution to this was simply to convert all Modules to class, and make all calls to the other tiers explicit. For example, instead of having the Parser module just assume that you've mixed in a Scanner and call check_token on itself, I have the Parser class take a Scanner instance at initialization and call check_token on this instance instead.
This works very well, and probably makes the code easier to understand. Another positive of this is that the interface between the layers are more apparent. For inclusion in JRuby, this will make it easier to replace certain parts with Java implementations.
The next piece on the agenda was a rewrite of the Parser. The original Python implementation used Python generators (which are almost like coroutines, but not quite). My first port of this code just parsed the whole stream, saved all events and then passed these on after parsing. This was good enough for smaller YAML documents, but when trying to parse the RubyGems gemspec, the memory and time requirements became to prohibitive. In the course of making the generator algorithm explicit I totally rewrote the Parser from the beginning, making it hybrid table driven instead of recursive-descent as the original was. I actually believe the new Parser is both easier to understand and faster. Just as an example, this is the code for block_sequence:
where @parse_stack contains the next productions to call after block_sequence has finished. The main generator method just keeps calling the next production until it arrives to a terminal, and then returns the value of this:
Another benefit of this is that this code is dead simple to port to other languages, once again probably easier than the Python version.
The third improvement was performance. I have no trustworthy numbers of the improvement, but it's in the order of 5-8 times faster than from the beginning. I achieved by some easy fixes, and some harder ones. I removed the Reader class and inlined those methods into the Scanner. I tested each case where I tested if a character was part of a String and checked were a Regexp was faster. And added some hard coded, unrolled loops in the most intense parts of the code, which was peek(), forward(), prefix() and update(). Every microsecond improvement in these methods counted since they are called so many times. I didn't do all this work blind, though. The Ruby profiler is really good. Just take a script, run it with ruby -rprofile script.rb and you get output that's incredibly good. I tested most of my changes this way, and the end result is about as fast as the JRuby RACC-based YAML parser, which was my goal.
Since version 0.1 I've spent some time getting JRuby to work flawlessly with RubyGems, and this work have uncovered some small bugs in RbYAML (and in SYCK, for that matter), so a new minor release will probably come soon. Until then the CVS is up to date.
First, conversion from Mixins to Classes. The original Python implementation used multiple inheritance, created several base classes (Reader, Scanner, Parser, etc) and then created one several versions of a Loader class which inherited from the different base classes. My first implementation mirrored this approach, but used Modules instead of base classes and mixed in different versions of these in the different Loader classes. This approach was quite limiting since mixing in code into other Modules doesn't really work as you expect, and this is no substitute for subclassing. For example, I had a BaseResolver module, a SafeResolver module which mixed in BaseResolver and added code of it's own, but this were quite cumbersome.
The solution to this was simply to convert all Modules to class, and make all calls to the other tiers explicit. For example, instead of having the Parser module just assume that you've mixed in a Scanner and call check_token on itself, I have the Parser class take a Scanner instance at initialization and call check_token on this instance instead.
This works very well, and probably makes the code easier to understand. Another positive of this is that the interface between the layers are more apparent. For inclusion in JRuby, this will make it easier to replace certain parts with Java implementations.
The next piece on the agenda was a rewrite of the Parser. The original Python implementation used Python generators (which are almost like coroutines, but not quite). My first port of this code just parsed the whole stream, saved all events and then passed these on after parsing. This was good enough for smaller YAML documents, but when trying to parse the RubyGems gemspec, the memory and time requirements became to prohibitive. In the course of making the generator algorithm explicit I totally rewrote the Parser from the beginning, making it hybrid table driven instead of recursive-descent as the original was. I actually believe the new Parser is both easier to understand and faster. Just as an example, this is the code for block_sequence:
def block_sequence
@parse_stack += [:block_sequence_end, :block_sequence_entry, :block_sequence_start]
nil
end
where @parse_stack contains the next productions to call after block_sequence has finished. The main generator method just keeps calling the next production until it arrives to a terminal, and then returns the value of this:
def parse_stream_next
if !@parse_stack.empty?
while true
meth = @parse_stack.pop
val = send(meth)
if !val.nil?
return val
end
end
else
return nil
end
end
Another benefit of this is that this code is dead simple to port to other languages, once again probably easier than the Python version.
The third improvement was performance. I have no trustworthy numbers of the improvement, but it's in the order of 5-8 times faster than from the beginning. I achieved by some easy fixes, and some harder ones. I removed the Reader class and inlined those methods into the Scanner. I tested each case where I tested if a character was part of a String and checked were a Regexp was faster. And added some hard coded, unrolled loops in the most intense parts of the code, which was peek(), forward(), prefix() and update(). Every microsecond improvement in these methods counted since they are called so many times. I didn't do all this work blind, though. The Ruby profiler is really good. Just take a script, run it with ruby -rprofile script.rb and you get output that's incredibly good. I tested most of my changes this way, and the end result is about as fast as the JRuby RACC-based YAML parser, which was my goal.
Since version 0.1 I've spent some time getting JRuby to work flawlessly with RubyGems, and this work have uncovered some small bugs in RbYAML (and in SYCK, for that matter), so a new minor release will probably come soon. Until then the CVS is up to date.
Getting RubyGems to work with JRuby
I'm sorry if the title gives it away, but here are some recent output in my terminal window:
So, what do you need to do, to get this working?
#bin/jruby bin/gem install rails --include-dependenciesSo, we have RubyGems mostly working. Right now there are two caveats. First, during the YAML parsing, we get some InterruptedExceptions for some reason. This doesn't seem to impair functionality, though. The second problem is that it takes serious time. Between 30 minutes and an hour for this. The two parts that are time hogs are the YAML parsing of the Gemspec, and the RDoc stuff, for some reason.
Attempting local installation of 'rails'
Local gem file not found: rails*.gem
Attempting remote installation of 'rails'
Updating Gem source index for: http://gems.rubyforge.org
Successfully installed rails-1.1.2
Successfully installed activesupport-1.3.1
Successfully installed activerecord-1.14.2
Successfully installed actionpack-1.12.1
Successfully installed actionmailer-1.2.1
Successfully installed actionwebservice-1.1.2
Installing RDoc documentation for activesupport-1.3.1...
Installing RDoc documentation for activerecord-1.14.2...
Installing RDoc documentation for actionpack-1.12.1...
Installing RDoc documentation for actionmailer-1.2.1...
Installing RDoc documentation for actionwebservice-1.1.2...
So, what do you need to do, to get this working?
- Start from a newly checked out JRuby.
- Add patch for RubyTime and TimeMetaClass. (Adds gmt_offset and utc_offset. This patch can be found in the jruby-devel archives.)
- Checkout the latest version of RbYAML from RubyForge, and put this in $JRUBY_HOME/lib/ruby/site_ruby/1.8.
- Add the contents from the C Ruby libraries.
- Change fileutils.rb, so that RUBY_PLATFORM works.
- Replace the file $JRUBY_HOME/src/builtin/yaml.rb with the yaml.rb for RbYAML, that can be found here.
- Change the jruby and jirb scripts by adding -Xmx512M. (I'm not sure 512 is really needed, actually. Maybe 256 or 128 suffices.)
lördag, maj 27, 2006
RbYAML version 0.1.0 released
Version 0.1.0 of RbYAML has now been released. Most of the interesting work on this was done on the flight from San Francisco and JavaOne to Stockholm. I guess I got tired of all the Java code. Anyhow, this is a major release, which improves almost all areas, with better testing, more functionality, Ruby-fied code, a new parser, and huge performance improvements.
I will take some time later this week to write more about the things I have done, implementation-wise.
I will take some time later this week to write more about the things I have done, implementation-wise.
lördag, maj 20, 2006
RbYAML version 0.0.2 released.
I have released version 0.0.2 of RbYAML. This is mostly fixes and convergence to the current PyYAML codebase, so nothing revolutionary. There are some things working now, that didn't before. I've also added some more automated tests.
The code can be downloaded here.
The code can be downloaded here.
JavaOne, last day.
So. The last day of JavaOne is always a strange experience. Most people are often to tired to stand straight after 3 really intense days of information gathering and people interactions. Personally, I was to tired to go to all sessions, but I managed the general session with Gosling and McNealy, the Mustang scripting session and the one about writing good API's.
All three were worthwhile. Gosling showcased some really amazing toys, as usual. The Mustang scripting session was interesting, mostly so because it seems they've ripped some parts of the Rhino JavaScript engine out, for some reason.
The best session today was the one on writing good API's, though. It had som really interesting advice and tips about API design. Basically you should apply the same rules as when you're doing UI design.
After this, I went to the JRuby meetup, where we sat around talking for a few hours, until I felt the need to go home and pack. JRuby is really on the go now, we have momentum and some really cool stuff almost finished. Stay tuned.
All three were worthwhile. Gosling showcased some really amazing toys, as usual. The Mustang scripting session was interesting, mostly so because it seems they've ripped some parts of the Rhino JavaScript engine out, for some reason.
The best session today was the one on writing good API's, though. It had som really interesting advice and tips about API design. Basically you should apply the same rules as when you're doing UI design.
After this, I went to the JRuby meetup, where we sat around talking for a few hours, until I felt the need to go home and pack. JRuby is really on the go now, we have momentum and some really cool stuff almost finished. Stay tuned.
fredag, maj 19, 2006
JavaOne, day 3.
So, the third day of JavaOne has also featured some interesting presentations. My blog today will not be a blow-for-blow description of these, but more a few interesting tidbits I noticed during the day.
I managed to talk to Gilad Bracha about how I thought his proposal for super packages looked very inspired by Common Lisp packages, and his response was that it was an interesting observation. He hadn't thought that way consciously until I pointed it out, so it was not designed that way, but he said that it was a good sign for the proposal that it looked like Common Lisp packages.
Actually, the first session was probably the most interesting from my perspective. This was Gilad s talk about supporting dynamically typed languages on the JVM. The first part talked about invokedynamic, which is fairly straightforward. The only new information I got about this area was that they're thinking about adding handlers for cases where the JVM can't discern a correct overloaded method to call for a dynamic invocation. In reality, this would more or less be a method_missing, available directly on the JVM, with all the performance characteristics you can get from the JIT. Nice stuff. Probably the handler architecture could also be used to implement some variations of multiple inheritance and mixins, which also is a problem to do efficiently on the JVM.
The second part of his talk was about hotswapping, which I didn't even know they're trying to get into the JVM. Basically hotswapping is what enables eval and replacing, adding and removing methods and types at runtime. This seems to be a very hard problem, but Gilad had some ideas, so it looks promising. It seems that JRuby may actually be able to run completely in JVM bytecode sometime in the future. Very cool.
After this I want to a session about simplifying enterprise development with scripting. This turned out to not match the title; it was basically another presentation on Groovy, and nothing much more.
The session on Compiler Optimizations where really interesting, and full of the kind of vocabulary that makes your head spin (but for different reasons if you're a compiler head or just a regular geek).
The Harmony session where really cool, they actually have a working (but slow) Swing implementation. The demonstration showed JEdit running inside Harmony, which is nice.
The security traps session was mostly basic material. Nothing new at all if you've been reading the books.
The last session for me today was about good ways to both an enterprise application. This presentation was really great, one of the top 3 this JavaOne, and I'm definitely planning on going home to study the slides. (It was TS-5397 if anyone wasn't there). Great stuff, really.
So, the rest of the evening will be After Dark Bash, and then out to make San Francisco unsafe.
I managed to talk to Gilad Bracha about how I thought his proposal for super packages looked very inspired by Common Lisp packages, and his response was that it was an interesting observation. He hadn't thought that way consciously until I pointed it out, so it was not designed that way, but he said that it was a good sign for the proposal that it looked like Common Lisp packages.
Actually, the first session was probably the most interesting from my perspective. This was Gilad s talk about supporting dynamically typed languages on the JVM. The first part talked about invokedynamic, which is fairly straightforward. The only new information I got about this area was that they're thinking about adding handlers for cases where the JVM can't discern a correct overloaded method to call for a dynamic invocation. In reality, this would more or less be a method_missing, available directly on the JVM, with all the performance characteristics you can get from the JIT. Nice stuff. Probably the handler architecture could also be used to implement some variations of multiple inheritance and mixins, which also is a problem to do efficiently on the JVM.
The second part of his talk was about hotswapping, which I didn't even know they're trying to get into the JVM. Basically hotswapping is what enables eval and replacing, adding and removing methods and types at runtime. This seems to be a very hard problem, but Gilad had some ideas, so it looks promising. It seems that JRuby may actually be able to run completely in JVM bytecode sometime in the future. Very cool.
After this I want to a session about simplifying enterprise development with scripting. This turned out to not match the title; it was basically another presentation on Groovy, and nothing much more.
The session on Compiler Optimizations where really interesting, and full of the kind of vocabulary that makes your head spin (but for different reasons if you're a compiler head or just a regular geek).
The Harmony session where really cool, they actually have a working (but slow) Swing implementation. The demonstration showed JEdit running inside Harmony, which is nice.
The security traps session was mostly basic material. Nothing new at all if you've been reading the books.
The last session for me today was about good ways to both an enterprise application. This presentation was really great, one of the top 3 this JavaOne, and I'm definitely planning on going home to study the slides. (It was TS-5397 if anyone wasn't there). Great stuff, really.
So, the rest of the evening will be After Dark Bash, and then out to make San Francisco unsafe.
torsdag, maj 18, 2006
JavaOne, day 2, second part.
So, the second part of day two was composed of a few different BOF's. I won't bother to talk about them all separately, since there really wasn't that much information in them.
First of all I went to the Collections Connection, which is always fun. Josh had most responsibility still, even though he's officially at Google now. They talked about the new collections in Mustang, of which the Deque interface is the most important addition. Also, navigable collections have been added. This is more or less SortedSet and SortedMap done right, with navigability from all ways.
My second BOF talked about identity management and federation. I really didn't get much out of this presentation. The presenter showcased a few standards that should be used, and some fairly complicated graphics showing how to interconnect these data transport protocols. Most of the stuff focused on SAML 2.0, XACML and ID-FF.
After that there was the BOF on Java Language and Compiler Issues, where they talked a little about the new compiler API in Mustang. The new packages javax.tools, javax.lang.model and com.sun.source seems really interesting and usable to do neat stuff. Another cool thing they showed was something called the JavacViewer, which more or less gives access to most information that the different compiler types uses internally. Parse trees, annotation processing, internal labeling; it's all there. Very cool.
Last, but not least at all, the late night BOF called "A script for more powerful Java technology-based applications" which talked about how you can leverage different scripting technologies to add a different interface to your application in a few different ways, by providing plugin possibilities, as a way of adding new features quickly, and also to make macros for getting your power users happy. The presenter used different kinds of scripting to demonstrate these techniques. Some parts integrated BeanShell, and a big part of the demonstration talked about how to write your own domain specific language, and a parser and definition for this. As the session was late at night, and there were fairly few people attending, it tended to drift to different subjects depending on questions from the audience, but this didn't detract at all. It was mostly very interesting and one of the better sessions this JavaOne.
One of the best reasons and rationales for adopting scripting languages as an approach is for your own developer needs. It makes sense to add scripting support so you can explore a huge code base, test out corner cases easily. (I know I constantly do this, start up JRuby or BeanShell inside Emacs, and test something there before using it in a real Java application).
After this session, me, Pop, Bob Evans, Charles Nutter, Thomas Enebo (the JRuby guys) and a few other went to a pub, drank some beer and continued talking scripting, JRuby, Lisp and other cool stuff for some parts of the night. I've learn some very neat stuff, and we've talked some more about the future for implementing RubyGems in JRuby. It will be very soon.
First of all I went to the Collections Connection, which is always fun. Josh had most responsibility still, even though he's officially at Google now. They talked about the new collections in Mustang, of which the Deque interface is the most important addition. Also, navigable collections have been added. This is more or less SortedSet and SortedMap done right, with navigability from all ways.
My second BOF talked about identity management and federation. I really didn't get much out of this presentation. The presenter showcased a few standards that should be used, and some fairly complicated graphics showing how to interconnect these data transport protocols. Most of the stuff focused on SAML 2.0, XACML and ID-FF.
After that there was the BOF on Java Language and Compiler Issues, where they talked a little about the new compiler API in Mustang. The new packages javax.tools, javax.lang.model and com.sun.source seems really interesting and usable to do neat stuff. Another cool thing they showed was something called the JavacViewer, which more or less gives access to most information that the different compiler types uses internally. Parse trees, annotation processing, internal labeling; it's all there. Very cool.
Last, but not least at all, the late night BOF called "A script for more powerful Java technology-based applications" which talked about how you can leverage different scripting technologies to add a different interface to your application in a few different ways, by providing plugin possibilities, as a way of adding new features quickly, and also to make macros for getting your power users happy. The presenter used different kinds of scripting to demonstrate these techniques. Some parts integrated BeanShell, and a big part of the demonstration talked about how to write your own domain specific language, and a parser and definition for this. As the session was late at night, and there were fairly few people attending, it tended to drift to different subjects depending on questions from the audience, but this didn't detract at all. It was mostly very interesting and one of the better sessions this JavaOne.
One of the best reasons and rationales for adopting scripting languages as an approach is for your own developer needs. It makes sense to add scripting support so you can explore a huge code base, test out corner cases easily. (I know I constantly do this, start up JRuby or BeanShell inside Emacs, and test something there before using it in a real Java application).
After this session, me, Pop, Bob Evans, Charles Nutter, Thomas Enebo (the JRuby guys) and a few other went to a pub, drank some beer and continued talking scripting, JRuby, Lisp and other cool stuff for some parts of the night. I've learn some very neat stuff, and we've talked some more about the future for implementing RubyGems in JRuby. It will be very soon.
JavaOne, day 2, first part.
So, this day I've been trying to keep my notes more close to the final result seen in this blog, with the result that I'll actually be able to post information even before the day is over. So, what I'm posting now is information from the beginning of the day, to the JRuby session that ended at 5pm.
Effective Java Reloaded
Effective Java has not been reloaded. Or not yet at least. But there is much material that can be used, and the session went through some great stuff. The presentation were divided into three parts, Object Creation, Generics and Other.
So, the object creation part had some great patterns. The first regarded static factories and how you can use factory methods to improve creation of
generic instances. For example, take this horrible example of creating a HashMap:
There are a few disadvantages that both static factories and constructors share. A big one is optional parameters. There are many ways of solving this, but none good. The pattern to fix this is to use a variation of the builder pattern.
You create a static Builder nested class, this builder constructor takes all required parameters and then provides setters for all optional parameters. It also exposes a build method that returns a created object. An example:
This approach is really powerful. If we're lucky this interface may be added to the JDK in the future:
Then we could stop passing Class objects around, and use the typesafe Builder instead.
The generic part of the session had some interesting information that was new to me, at least.
The first recommendation was to never use raw types anymore. Those are only for legacy code. Raw types are really evil.
You should never ignore compiler warnings. They should be understood and eliminated if possible. If not they should be commented, and suppressed with the SuppressWarnings annotation if it can be proved safe.
Wildcards should be preferred to explicit type parameters. In many cases this makes method signatures clearer, and you don't have
to manage a type variable. The exception to this is conjunctive types (which is really neat too).
Bounded wildcards are almost always better to use in your API, it will make it work for many more cases where people expect it to work.
The usual case when this is a problem is when you're using generics of generic types in your code. The reason this is a problems is that for example Collection<Integer> is NOT a subtype of Collection<Number>.
Bounded wildcards should never be a return type. This forces clients to deal with wildcards explicitly. Only library designers should use wildcards.
Sometimes you actually need to do it, but it's very unlikely.
Generics and arrays don't mix very well, mostly always use generics if you can.
Some people say avoid arrays altogether, but there are cases where arrays are both prettier and faster.
Finally, the presentation ended with a few various recommendations.
Use the @Override annotation. This avoids common problems when you think you're overriding something, but really isn't, for example equals or hashCode.
Final should be used everywhere, except where there really is a reason to not do that. This minimizes mutability and is clearly thread-safe, which means you have one less thing to worry about. The only problem is readObject and clone, so take care with these.
You can use a HashMap makes a fine sparse array, with generics and autoboxing.
The Serialization Proxy pattern is really neat.
Since serialization depends on implementation details you should take care with serialization.
The pattern solves these problem by having you create a new class representing the logical state of your object, and you just use writeReplace and readResolve to use this proxy to serialize your object in an implementation independent way.
Java Puzzlers
There were some really intriguing things showcased here, and everything was Tiger-oriented. I didn't take any notes, since I had way to much fun. But I definitely recommend everyone to have a look at the presentation slides.
Super packages
Gilad Bracha had a small session about the new super packages proposed for Dolphin. As he constantly told us, nothing of this is really ready or finished. The JCP process will hash everything out later.
There is really two processes going on for modularity. One is for super packages, and regards the language changes necessary for this functionality. The other part is a module approach for packaging and distribution. The packaging has nothing to do with the language. The packaging only concerns tools and environment, more or less.
The problem with current packages concern information hiding and encapsulation. There are really hard to do this in a good way in current Java. A few solutions have been proposed for this, that are easier than a real language change.
* Don't document unexposed API
* Using static classes to provide access control to different classes
* Make a small language change that makes packages nested
The conclusion is that these doesn't suffice. They are not good enough, and very hackish solutions.
A real solution will solve the packaging problem, provide encapsulation and also allow separate compilation. All this will use separate module files for changing the semantics of a program, but still having the default way for modularity to look like current Java, for providing backwards compatibility. This is also the reason annotations won't be used for this, since it would change runtime semantics of a program, which annotations should not do.
To my eyes, the syntax and semantics Gilad showed us reminds me very much of Common Lisp packages.
Spring WebFlow
Classical web packages use free navigation, stateless systems. This is not always perfect. Some business scenarios are better represented with a controlled flow of actions. Traditionally this hasn't been the focus of Web tools. Instead, most of the current frameworks focus on providing easy to use solutions for the base case of free simple navigation. There are a few reasons for this, but the simplest reason is that controlled flow is really hard to get right.
In my opinion, WebFlow is a perfect example on how you should not solve this problem. It has the right ideas, but doesn't go far enough.
The idea in Spring WebFlow is basically to describe states and state progressions either declaratively with XML, or programmatically in code. When you've done this, WebFlow takes care of most boring stuff, like state and back buttons. It's really about inverting control to the controller, instead of having the client provide parameters that the web server uses to find out where in the flow they are.
This approach is a really good solution to the problem, but it doesn't go far enough, if you ask me. When I see executable XML I always get scared, and this case is no exception. Spring WebFlow seems to be more or less a (very) poor mans continuation server. Since you can actually have real continuation servers in Java, using an embedded script language like JavaScript or Ruby, this approach isn't good enough for me.
Groovy
Groovy is like Java with some Python, Ruby and Smalltalk. It's object oriented and completely Java compatible. It has iterators, code blocks (closures), and many, many DWIM hacks.
Since the JVM is standardized and more general than Java, it can be used to innovate at the source code level. There are many scripting languages for Java.
Scripting seems to be a good way to glue business code together, since you really don't have that much business code in reality. There also is a drive to test code
with dynamic languages. So scripting is just great glue; it works like programmer duct tape.
The reason for Groovy is to have something Java developers will instantly recognize. Complete binary compatibility with Java, and possibilities to use Java without wrappers and cumbersome API's.
Groovy is basically dynamic, but also supports static typing. There is native support for lists, maps, arrays and beans..
Regexps are also part of the language. There exists some operator overloading, but nothing really lethal. Groovy also adds lots of convenience methods to the JDK, for example lots of new String-methods.
It has BSF support.
There seemed to me to exist some really hairy magic, which means that it's very hard to know exactly what's going on under the covers. A typical example was the actionPerformed parameter to some of the swing builders, which found an ActionListener interface and found the method inside that, and implemented this interface with a closure added, via 3 or 4 levels of indirection.
In conclusion, Groovy looks good on the surface, but beneath, it feels very much like Perl (in the negative sense).
JRuby
JRuby showcased many fun things, the best one was JRuby on Rails actually running. Is that cool or what? A part from that, the talk was mostly aimed at people new to Ruby and JRuby. It was interesting to see that most of the people at the session hadn't heard about either Ruby or Rails one year ago! Major impact or what?
That was my day, to now. I'm off to the Java Certified Professional party! Part two comes later. Maybe much later depending on how much free drinks there are at the party.
Effective Java Reloaded
Effective Java has not been reloaded. Or not yet at least. But there is much material that can be used, and the session went through some great stuff. The presentation were divided into three parts, Object Creation, Generics and Other.
So, the object creation part had some great patterns. The first regarded static factories and how you can use factory methods to improve creation of
generic instances. For example, take this horrible example of creating a HashMap:
Map<String, List<String>> m = new HashMap<String,List<String>>();Instead, HashMap should have a factory method, and then you can do this:
Map<String, List<String>> m = HashMap.newInstance();The recommendation is to always write your generic code like this.
There are a few disadvantages that both static factories and constructors share. A big one is optional parameters. There are many ways of solving this, but none good. The pattern to fix this is to use a variation of the builder pattern.
You create a static Builder nested class, this builder constructor takes all required parameters and then provides setters for all optional parameters. It also exposes a build method that returns a created object. An example:
final NutritionFacts twoLdietCoke = new NutritionFacts.Builder("Diet Coke",240,8).sodium(1).build();or even
final NutritionFacts twoLdietCoke = NutritionFacts.builder("DietCoke",240,8).sodium(1).build();
This approach is really powerful. If we're lucky this interface may be added to the JDK in the future:
public interface Builder<T> {
T build();
}
Then we could stop passing Class objects around, and use the typesafe Builder instead.
The generic part of the session had some interesting information that was new to me, at least.
The first recommendation was to never use raw types anymore. Those are only for legacy code. Raw types are really evil.
You should never ignore compiler warnings. They should be understood and eliminated if possible. If not they should be commented, and suppressed with the SuppressWarnings annotation if it can be proved safe.
Wildcards should be preferred to explicit type parameters. In many cases this makes method signatures clearer, and you don't have
to manage a type variable. The exception to this is conjunctive types (which is really neat too).
Bounded wildcards are almost always better to use in your API, it will make it work for many more cases where people expect it to work.
The usual case when this is a problem is when you're using generics of generic types in your code. The reason this is a problems is that for example Collection<Integer> is NOT a subtype of Collection<Number>.
Bounded wildcards should never be a return type. This forces clients to deal with wildcards explicitly. Only library designers should use wildcards.
Sometimes you actually need to do it, but it's very unlikely.
Generics and arrays don't mix very well, mostly always use generics if you can.
Some people say avoid arrays altogether, but there are cases where arrays are both prettier and faster.
Finally, the presentation ended with a few various recommendations.
Use the @Override annotation. This avoids common problems when you think you're overriding something, but really isn't, for example equals or hashCode.
Final should be used everywhere, except where there really is a reason to not do that. This minimizes mutability and is clearly thread-safe, which means you have one less thing to worry about. The only problem is readObject and clone, so take care with these.
You can use a HashMap makes a fine sparse array, with generics and autoboxing.
The Serialization Proxy pattern is really neat.
Since serialization depends on implementation details you should take care with serialization.
The pattern solves these problem by having you create a new class representing the logical state of your object, and you just use writeReplace and readResolve to use this proxy to serialize your object in an implementation independent way.
Java Puzzlers
There were some really intriguing things showcased here, and everything was Tiger-oriented. I didn't take any notes, since I had way to much fun. But I definitely recommend everyone to have a look at the presentation slides.
Super packages
Gilad Bracha had a small session about the new super packages proposed for Dolphin. As he constantly told us, nothing of this is really ready or finished. The JCP process will hash everything out later.
There is really two processes going on for modularity. One is for super packages, and regards the language changes necessary for this functionality. The other part is a module approach for packaging and distribution. The packaging has nothing to do with the language. The packaging only concerns tools and environment, more or less.
The problem with current packages concern information hiding and encapsulation. There are really hard to do this in a good way in current Java. A few solutions have been proposed for this, that are easier than a real language change.
* Don't document unexposed API
* Using static classes to provide access control to different classes
* Make a small language change that makes packages nested
The conclusion is that these doesn't suffice. They are not good enough, and very hackish solutions.
A real solution will solve the packaging problem, provide encapsulation and also allow separate compilation. All this will use separate module files for changing the semantics of a program, but still having the default way for modularity to look like current Java, for providing backwards compatibility. This is also the reason annotations won't be used for this, since it would change runtime semantics of a program, which annotations should not do.
To my eyes, the syntax and semantics Gilad showed us reminds me very much of Common Lisp packages.
Spring WebFlow
Classical web packages use free navigation, stateless systems. This is not always perfect. Some business scenarios are better represented with a controlled flow of actions. Traditionally this hasn't been the focus of Web tools. Instead, most of the current frameworks focus on providing easy to use solutions for the base case of free simple navigation. There are a few reasons for this, but the simplest reason is that controlled flow is really hard to get right.
In my opinion, WebFlow is a perfect example on how you should not solve this problem. It has the right ideas, but doesn't go far enough.
The idea in Spring WebFlow is basically to describe states and state progressions either declaratively with XML, or programmatically in code. When you've done this, WebFlow takes care of most boring stuff, like state and back buttons. It's really about inverting control to the controller, instead of having the client provide parameters that the web server uses to find out where in the flow they are.
This approach is a really good solution to the problem, but it doesn't go far enough, if you ask me. When I see executable XML I always get scared, and this case is no exception. Spring WebFlow seems to be more or less a (very) poor mans continuation server. Since you can actually have real continuation servers in Java, using an embedded script language like JavaScript or Ruby, this approach isn't good enough for me.
Groovy
Groovy is like Java with some Python, Ruby and Smalltalk. It's object oriented and completely Java compatible. It has iterators, code blocks (closures), and many, many DWIM hacks.
Since the JVM is standardized and more general than Java, it can be used to innovate at the source code level. There are many scripting languages for Java.
Scripting seems to be a good way to glue business code together, since you really don't have that much business code in reality. There also is a drive to test code
with dynamic languages. So scripting is just great glue; it works like programmer duct tape.
The reason for Groovy is to have something Java developers will instantly recognize. Complete binary compatibility with Java, and possibilities to use Java without wrappers and cumbersome API's.
Groovy is basically dynamic, but also supports static typing. There is native support for lists, maps, arrays and beans..
Regexps are also part of the language. There exists some operator overloading, but nothing really lethal. Groovy also adds lots of convenience methods to the JDK, for example lots of new String-methods.
It has BSF support.
There seemed to me to exist some really hairy magic, which means that it's very hard to know exactly what's going on under the covers. A typical example was the actionPerformed parameter to some of the swing builders, which found an ActionListener interface and found the method inside that, and implemented this interface with a closure added, via 3 or 4 levels of indirection.
In conclusion, Groovy looks good on the surface, but beneath, it feels very much like Perl (in the negative sense).
JRuby
JRuby showcased many fun things, the best one was JRuby on Rails actually running. Is that cool or what? A part from that, the talk was mostly aimed at people new to Ruby and JRuby. It was interesting to see that most of the people at the session hadn't heard about either Ruby or Rails one year ago! Major impact or what?
That was my day, to now. I'm off to the Java Certified Professional party! Part two comes later. Maybe much later depending on how much free drinks there are at the party.
onsdag, maj 17, 2006
JavaOne, day 1: some diverse impressions.
So, JavaOne is finally here, and it starts big! This blog will talk some about the different sessions I've been to during the day, but first a few impressions. There are very many people here. John Gage said at the first general session that this is the largest JavaOne ever, and I believe him. I'm sad to say that the WiFi network is very spotty at best.
JavaOne has a new way of getting into sessions; you have to use their schedule builder to reserve places for a session, and then use your RFID chip to register when entering a specific session. I thought this was a mad idea, but it actually works really well, and I'm glad to say that the JavaOne team seems to have solved most of the overflow troubles from last year.
Another small reflection is that the focus on compatibility in the core platform seem to be a major focus this year. I've seen and heard the word spoken more times than I can speak.
General session
The first general session is always one of the more interesting times during JavaOne. Both Sun and other company leaders are just brimming with announcements. One of the more interesting quotes of the day came fairly early, from Rich Green: "It's not a question of whether, it's a question of how". As you may guess, this was an answer to the question of open sourcing Java. In plain language the situation is that Java will be open source, as soon as Sun finds a good way to do it in.
Java Enterprise Edition 5 is now finished, released and of production quality, and these are some highlights from the release:
It's very focused on ease of development.
Contains much Web 2.0 support.
Interoperability with .NET have been greatly enhanced. (Except for compatilibity, interoperatibility is the major illy today).
SOA has been simplified.
They have simplified the programming model, mostly by using annotations.
New EJB version, using plain old Java objects.
A new (annotation-based) persistence API.
During the general session, many, many libraries and applications were open sourced, among these JMS. It's nice to see that Sun really wants open source to work.
Session on EJB 3.0
The idea for EJB 3 was to make it easier for the developer, by making the container harder to implement. This tradeoff seems reasonable in retrospect, but until now most of the EJB specification made it easy for the container.
The new model is based on letting the container provde requested services to the bean, and also using reasonable defaults for most operations.
Once again, the presentation pressed very hard on compatibility. Existing applications had to continue working. Clients using the new API should have no trouble connecting to servers using the old libraries and the other way around.
Most of EJB 3.0 is based on POJOs and declarations with either annotations or XML information.
Environment access is made easier with dependency injection or simple lookup.
Client code also uses dependency injection, which means the new model removes the need for home interfaces, PortableRemoteObject.narrow, RemoteException and other checked exceptions.
The EJB lifecycle doesn't use explicit callbacks anymore. Instead you can annotate the method that should get notifications about a lifecycle change and this will be taken care of by the container. You can also allow a separate interceptor as the notification and callback manager. Very neat.
All in all, EJB seems to be heading the right way, at last. Ease of development really matters, and configuration by reasonable defaults have already been shown to be a viable solution.
Technical general session
The technical session had some interesting information too. Most of the talk was about Mustang (Java SE 6) and Dolphin (Java SE 7), and what we could expect from these releases.
The projections for Mustang looks good, and it is scheduled for release in October. They have been using a new, very open process for Mustang development, which have worked extremely well. The same system will most likely be used even more in Dolphin development.
So, some nuggets of good stuff in Mustang:
Many performance improvements. They showed some pretty convincing performance graphs, and I was duly impressed.
They've fixed the so called gray rect problem in Swing, which results in heightened perceived performance dramatically.
There is many improvements in the monitoring and management areas.
Scripting support will come, with JSR 223. Also, check out http://scripting.dev.java.net.
There are many desktop fixes, and Vista is the desktop focus for Mustang.
The talk went on to the future for Standard Edition:
Probably direct support for XML in the language.
Super packages, and new module system for packaging with versioning.
They're thinking about adding BeanShell to the scripting languages provided by the core language.
And more, more,more desktop stuff.
Hamilton went on to talk about scripting- and dynamic languages, and more or less recommended using them in the Web tier and in situations where a fast cycle of development is required. This layer can then use Java for the business logic. (And this is really what KI is heading for right now; using Ruby on Rails for web, and using SOAP to get at Java business logic exposed with Web Services.)
Anothing thing that's coming is JSR 292: the new bytecode for dynamic invocation.
An interesting demo of a Visual Basic to Java compiler. The system is not a total clone, but will enable people used to Visual Basic to program for the JVM instead. It will not enable translation of existing applications, though.
Mustang and Dolphin session
This session started out with Mark commanding us to upgrade to Tiger now, there is really no reason not to do it.
So, what's new in Mustang?
Some class file changes.
A new Compiler API.
New annotation processors.
JDBC4.
Scripting support.
Streaming API for XML.
Common Annotations.
WS-Metadata
and JAX-WS 2.0.
Mark top ten list of new things in Mustang:
10. Attach-on-demand monitoring.
9. Plugin API for JConsole.
8. jhat OQL (a query language to explore heap dumps).
7. Solaris d-trace.
6. javac will now do annotation processing interleaved with compilation.
5. Classpath wildcards!
4. API for finding free Disk-space.
3. API for password prompting.
2. New Grouplayout for Swing.
1. That JAX-WS can do RESTful web services.
Mustang will also bundle Apache Derby, a small inmemory database.
There are many ideas for what Dolphin will contain. There are some interesting things that can be really cool.
These are mostly core language changes:
Properties, improving getters and setters.
Real method references.
Block closures.
Native XML support.
The new bytecode for dynamic invocation.
Bundling BeanShell.
And beans binding for making Swing more easy to use.
The session also had some very fun information about how the testing of JDK is done. It's really quite amazing. The big trouble with creating new versions is disconcerting fact that running a full test cycle takes 10 weeks.
Session on the new Concurrency features in Java 5
The session began with some talk about the rationale behind the new concurrency features in Java. The easy answer is that real concurrency is hard to do right, and the builtin primitives for threads and locks are, well, primitive.
The new concurrency packages have something for everyone. There is both easy-to-use utilities for mostly anyone, and also some primitives for hard core programmers, that enable some things that just can't be done in Java right now.
Java has always had thread-safe collections, but these are not conccurent. They also had a bad performance structure. Therefore a few new conccurent collections has been added. Most of them allow unlimited reads and up to 16 simultanous writes. Mostly, the semantics are the same, but they differ in a few areas. The most glaring difference is iterators. With the old collections you got a ConcurrentModificationException if someone updated the collection while you were iterating over it. This will not happen anymore. Tiger has a new Map called ConccurentHashMap and Mustang adds a SkipListMap.
A new collection interface has been added to java.util, which is called Queue. This is a subset of the List functionality, that can be used for implementing high performance versions with just this restricted functionality.
The most interesting Queue is the BlockingQueue, which makes explicit a producer-consumer relationship. Mostly all code already uses something like it, but this implementation is industry strength and very easy to use.
They have hadded ThreadPools with a system of Executors and ExecutorServices. They are very easy to use and configure with factory methods.
Another interesting addition is Future and Callable, which is used to put an execution inside another thread and then get the value when it's finished. But it is not
call-by-need, which was what I first thought. The value will always be calculated, even if it's not needed.
Scheduling primitives have been added, to replace Timer and TimerTask.
There are also some new advanced features for locking. There are some things you just can't do with the synchronized keyword in Java. Hand-over-hand locking is one example. The concurrency library adds Locks, Conditions and Semaphores for more advanced use. They are very complicated so they shouldn't be used if you don't really need them. One good reason for this is that you have to release locks manually.
JUnit next generation
JUnit has many warts and problems. How do you run one test in a test case, for example?
It's not really good for anything but unit testing. It has been very few updates and the
protocol is intrusive. It also uses a very static programming model. And it doesn't use the latest Java features.
TestNG is the new JUnit. It uses annotations for most configuration. There are test groups which can be dynamic depending on your needs.
You also have the possibility to have dependent tests, parallell testing, load testing and partial failures. It has also got a very nice plugin API.
All in all, it looks really good. My personal opinion is that there is no reason not to use TestNG instead of JUnit on all
Java 5 projects.
Restructuring a web application with Hibernate and Spring
This was the first BOF for me this JavaOne and I was very disappointed. The presenters use case was a web application that had been badly written from the beginning. They then decided to rewrite it with Hibernate and Spring, but there really didn't seem to be much better code written for this application. I guess it's a testament to how good Spring and Hibernate are, that they got a really good performance improvement anyway.
Testing a persistence layer
Testing a persistence layer is really hard, for a few different reasons. You want it to be fast, and easy to write. There are many different strategies to testing persistence, and the presentation talked about most of them.
There are a few different kinds of persistence layers. They can be SQL-based, Object/Relational-based or using the ActiveRecord pattern (which the presenter viewed as a special case of SQL-based persistence).
As noted above, there are many strategies available.
You can mock the DAO's for testing the business logic. This is easy if you use a dependency injection framework like Spring to initialize the DAO objects. Otherwise this is the main problem. It's very fast but the scaffolding can get hairy to write.
You can also mock the ORM-framework, to test the DAO's. The presenter have written a utility called ORMUnit to faciliate this testing.
Regarding mock frameworks, the presenter prefers JMock, but is planning on migrating to EasyMock instead.
Another strategy is to test the metadata mapping and schema. This checks for stupid errors like forgetting to map a field. It can also checks that all referenced tables and columns actually exist.
Of course, the standard way of testing persistence, by doing CRUD operations is also available, but it's complex, and very slow. You have to write much code to drop and add data for each test.
Another simple way is to check the generated queries directly, to see that they return correct data.
Using an inprocess database can be very fast, but may also lead to trouble with incomplete SQL implementations.
A final strategy for testing slightly things faster is doing operations but never committing transactions. This has the advantage of being fast and also avoids actually changing the DB.
Summary
All in all, it was a busy first day. Much of it was very interesting, and I've learned many new things. I just regret missing the Scripting Languages BOF at the end of the night, but I was just to beat to manage going to it.
JavaOne has a new way of getting into sessions; you have to use their schedule builder to reserve places for a session, and then use your RFID chip to register when entering a specific session. I thought this was a mad idea, but it actually works really well, and I'm glad to say that the JavaOne team seems to have solved most of the overflow troubles from last year.
Another small reflection is that the focus on compatibility in the core platform seem to be a major focus this year. I've seen and heard the word spoken more times than I can speak.
General session
The first general session is always one of the more interesting times during JavaOne. Both Sun and other company leaders are just brimming with announcements. One of the more interesting quotes of the day came fairly early, from Rich Green: "It's not a question of whether, it's a question of how". As you may guess, this was an answer to the question of open sourcing Java. In plain language the situation is that Java will be open source, as soon as Sun finds a good way to do it in.
Java Enterprise Edition 5 is now finished, released and of production quality, and these are some highlights from the release:
It's very focused on ease of development.
Contains much Web 2.0 support.
Interoperability with .NET have been greatly enhanced. (Except for compatilibity, interoperatibility is the major illy today).
SOA has been simplified.
They have simplified the programming model, mostly by using annotations.
New EJB version, using plain old Java objects.
A new (annotation-based) persistence API.
During the general session, many, many libraries and applications were open sourced, among these JMS. It's nice to see that Sun really wants open source to work.
Session on EJB 3.0
The idea for EJB 3 was to make it easier for the developer, by making the container harder to implement. This tradeoff seems reasonable in retrospect, but until now most of the EJB specification made it easy for the container.
The new model is based on letting the container provde requested services to the bean, and also using reasonable defaults for most operations.
Once again, the presentation pressed very hard on compatibility. Existing applications had to continue working. Clients using the new API should have no trouble connecting to servers using the old libraries and the other way around.
Most of EJB 3.0 is based on POJOs and declarations with either annotations or XML information.
Environment access is made easier with dependency injection or simple lookup.
Client code also uses dependency injection, which means the new model removes the need for home interfaces, PortableRemoteObject.narrow, RemoteException and other checked exceptions.
The EJB lifecycle doesn't use explicit callbacks anymore. Instead you can annotate the method that should get notifications about a lifecycle change and this will be taken care of by the container. You can also allow a separate interceptor as the notification and callback manager. Very neat.
All in all, EJB seems to be heading the right way, at last. Ease of development really matters, and configuration by reasonable defaults have already been shown to be a viable solution.
Technical general session
The technical session had some interesting information too. Most of the talk was about Mustang (Java SE 6) and Dolphin (Java SE 7), and what we could expect from these releases.
The projections for Mustang looks good, and it is scheduled for release in October. They have been using a new, very open process for Mustang development, which have worked extremely well. The same system will most likely be used even more in Dolphin development.
So, some nuggets of good stuff in Mustang:
Many performance improvements. They showed some pretty convincing performance graphs, and I was duly impressed.
They've fixed the so called gray rect problem in Swing, which results in heightened perceived performance dramatically.
There is many improvements in the monitoring and management areas.
Scripting support will come, with JSR 223. Also, check out http://scripting.dev.java.net.
There are many desktop fixes, and Vista is the desktop focus for Mustang.
The talk went on to the future for Standard Edition:
Probably direct support for XML in the language.
Super packages, and new module system for packaging with versioning.
They're thinking about adding BeanShell to the scripting languages provided by the core language.
And more, more,more desktop stuff.
Hamilton went on to talk about scripting- and dynamic languages, and more or less recommended using them in the Web tier and in situations where a fast cycle of development is required. This layer can then use Java for the business logic. (And this is really what KI is heading for right now; using Ruby on Rails for web, and using SOAP to get at Java business logic exposed with Web Services.)
Anothing thing that's coming is JSR 292: the new bytecode for dynamic invocation.
An interesting demo of a Visual Basic to Java compiler. The system is not a total clone, but will enable people used to Visual Basic to program for the JVM instead. It will not enable translation of existing applications, though.
Mustang and Dolphin session
This session started out with Mark commanding us to upgrade to Tiger now, there is really no reason not to do it.
So, what's new in Mustang?
Some class file changes.
A new Compiler API.
New annotation processors.
JDBC4.
Scripting support.
Streaming API for XML.
Common Annotations.
WS-Metadata
and JAX-WS 2.0.
Mark top ten list of new things in Mustang:
10. Attach-on-demand monitoring.
9. Plugin API for JConsole.
8. jhat OQL (a query language to explore heap dumps).
7. Solaris d-trace.
6. javac will now do annotation processing interleaved with compilation.
5. Classpath wildcards!
4. API for finding free Disk-space.
3. API for password prompting.
2. New Grouplayout for Swing.
1. That JAX-WS can do RESTful web services.
Mustang will also bundle Apache Derby, a small inmemory database.
There are many ideas for what Dolphin will contain. There are some interesting things that can be really cool.
These are mostly core language changes:
Properties, improving getters and setters.
Real method references.
Block closures.
Native XML support.
The new bytecode for dynamic invocation.
Bundling BeanShell.
And beans binding for making Swing more easy to use.
The session also had some very fun information about how the testing of JDK is done. It's really quite amazing. The big trouble with creating new versions is disconcerting fact that running a full test cycle takes 10 weeks.
Session on the new Concurrency features in Java 5
The session began with some talk about the rationale behind the new concurrency features in Java. The easy answer is that real concurrency is hard to do right, and the builtin primitives for threads and locks are, well, primitive.
The new concurrency packages have something for everyone. There is both easy-to-use utilities for mostly anyone, and also some primitives for hard core programmers, that enable some things that just can't be done in Java right now.
Java has always had thread-safe collections, but these are not conccurent. They also had a bad performance structure. Therefore a few new conccurent collections has been added. Most of them allow unlimited reads and up to 16 simultanous writes. Mostly, the semantics are the same, but they differ in a few areas. The most glaring difference is iterators. With the old collections you got a ConcurrentModificationException if someone updated the collection while you were iterating over it. This will not happen anymore. Tiger has a new Map called ConccurentHashMap and Mustang adds a SkipListMap.
A new collection interface has been added to java.util, which is called Queue. This is a subset of the List functionality, that can be used for implementing high performance versions with just this restricted functionality.
The most interesting Queue is the BlockingQueue, which makes explicit a producer-consumer relationship. Mostly all code already uses something like it, but this implementation is industry strength and very easy to use.
They have hadded ThreadPools with a system of Executors and ExecutorServices. They are very easy to use and configure with factory methods.
Another interesting addition is Future and Callable, which is used to put an execution inside another thread and then get the value when it's finished. But it is not
call-by-need, which was what I first thought. The value will always be calculated, even if it's not needed.
Scheduling primitives have been added, to replace Timer and TimerTask.
There are also some new advanced features for locking. There are some things you just can't do with the synchronized keyword in Java. Hand-over-hand locking is one example. The concurrency library adds Locks, Conditions and Semaphores for more advanced use. They are very complicated so they shouldn't be used if you don't really need them. One good reason for this is that you have to release locks manually.
JUnit next generation
JUnit has many warts and problems. How do you run one test in a test case, for example?
It's not really good for anything but unit testing. It has been very few updates and the
protocol is intrusive. It also uses a very static programming model. And it doesn't use the latest Java features.
TestNG is the new JUnit. It uses annotations for most configuration. There are test groups which can be dynamic depending on your needs.
You also have the possibility to have dependent tests, parallell testing, load testing and partial failures. It has also got a very nice plugin API.
All in all, it looks really good. My personal opinion is that there is no reason not to use TestNG instead of JUnit on all
Java 5 projects.
Restructuring a web application with Hibernate and Spring
This was the first BOF for me this JavaOne and I was very disappointed. The presenters use case was a web application that had been badly written from the beginning. They then decided to rewrite it with Hibernate and Spring, but there really didn't seem to be much better code written for this application. I guess it's a testament to how good Spring and Hibernate are, that they got a really good performance improvement anyway.
Testing a persistence layer
Testing a persistence layer is really hard, for a few different reasons. You want it to be fast, and easy to write. There are many different strategies to testing persistence, and the presentation talked about most of them.
There are a few different kinds of persistence layers. They can be SQL-based, Object/Relational-based or using the ActiveRecord pattern (which the presenter viewed as a special case of SQL-based persistence).
As noted above, there are many strategies available.
You can mock the DAO's for testing the business logic. This is easy if you use a dependency injection framework like Spring to initialize the DAO objects. Otherwise this is the main problem. It's very fast but the scaffolding can get hairy to write.
You can also mock the ORM-framework, to test the DAO's. The presenter have written a utility called ORMUnit to faciliate this testing.
Regarding mock frameworks, the presenter prefers JMock, but is planning on migrating to EasyMock instead.
Another strategy is to test the metadata mapping and schema. This checks for stupid errors like forgetting to map a field. It can also checks that all referenced tables and columns actually exist.
Of course, the standard way of testing persistence, by doing CRUD operations is also available, but it's complex, and very slow. You have to write much code to drop and add data for each test.
Another simple way is to check the generated queries directly, to see that they return correct data.
Using an inprocess database can be very fast, but may also lead to trouble with incomplete SQL implementations.
A final strategy for testing slightly things faster is doing operations but never committing transactions. This has the advantage of being fast and also avoids actually changing the DB.
Summary
All in all, it was a busy first day. Much of it was very interesting, and I've learned many new things. I just regret missing the Scripting Languages BOF at the end of the night, but I was just to beat to manage going to it.
tisdag, maj 16, 2006
JavaOne, day 0: The Fireside Chat
The Fireside Chat is usually an interesting conversation between JavaOne alumni and some of the more Java founders. This year it was James Gosling, Graham Hamilton, Jeff Jackson and a few others. These are more disjointed notes of what I am interested in, than a running commentary on everything that's said.
The format for this chat is that someone in the audience asks one ore more questions, and the panel tries to answer as much as possible. Nothing advanced.
Questions about Java Applets downloading time, is there any plans on having a Java Web Edition? Short answer: no.
Performance, specifically the startup time of the Java engine; Sun have tried, but right now it seems hard to get it any faster without removing significant functionality.
I gathered from some comments that one of Sun's primary priorities right now is good interoperability with Microsoft and .NET.
Regarding deprecated API's, it seems likely they will never disappear, at least not until there is enough good tool support to actually remove and refactor all dependencies on such code.
AJAX: Sun is handling this in a few ways; with toolkits for generating JavaScript with servlets and stuff like that. JSF should create AJAX-aware components without us needing to
have to care about AJAX.
Then the panel got the question what one thing they'd like to remove from Java if they could. Goslings immediate answer was java.awt. He got off on a tangent and talked a little about how the basic feature set of Java was decided, and that he only put in stuff that people really, really needed. The result of this is that he doesn't regret anything with the language. He seemed particularly glad that he didn't try to put in generics or enumerations from the beginning, because he would probably have failed doing it "right".
If he could add something to the language, that would be really lightweight objects. Some kind of structs, for implementing the canonical example of an object oriented number class, for example. Right now the most lightweight classes are still way to heavy. Also, Hamilton would like to improve getters and setters, but he doesn't know how yet.
There seems to be some interesting improvements to JavaBeans in Dolphin.
Hamilton also said that he would like to undeprecate java.util.Date, since it is very much more usable than java.util.Calendar most of the time.
Sun is also working on really good refactoring tools. Gosling has a vision of "lint on steroids". They have a few good prototypes, but nothing close to being released yet.
Another point in planning is better performance for JNI. Make it possible to remove checks and stuff like that. It seemed that arrays were the big problem. You can get quite good speed already, by not using arrays, and having as much NIO as possible.
These were my opinions on the more interesting stuff from the Fireside Chat.
The format for this chat is that someone in the audience asks one ore more questions, and the panel tries to answer as much as possible. Nothing advanced.
Questions about Java Applets downloading time, is there any plans on having a Java Web Edition? Short answer: no.
Performance, specifically the startup time of the Java engine; Sun have tried, but right now it seems hard to get it any faster without removing significant functionality.
I gathered from some comments that one of Sun's primary priorities right now is good interoperability with Microsoft and .NET.
Regarding deprecated API's, it seems likely they will never disappear, at least not until there is enough good tool support to actually remove and refactor all dependencies on such code.
AJAX: Sun is handling this in a few ways; with toolkits for generating JavaScript with servlets and stuff like that. JSF should create AJAX-aware components without us needing to
have to care about AJAX.
Then the panel got the question what one thing they'd like to remove from Java if they could. Goslings immediate answer was java.awt. He got off on a tangent and talked a little about how the basic feature set of Java was decided, and that he only put in stuff that people really, really needed. The result of this is that he doesn't regret anything with the language. He seemed particularly glad that he didn't try to put in generics or enumerations from the beginning, because he would probably have failed doing it "right".
If he could add something to the language, that would be really lightweight objects. Some kind of structs, for implementing the canonical example of an object oriented number class, for example. Right now the most lightweight classes are still way to heavy. Also, Hamilton would like to improve getters and setters, but he doesn't know how yet.
There seems to be some interesting improvements to JavaBeans in Dolphin.
Hamilton also said that he would like to undeprecate java.util.Date, since it is very much more usable than java.util.Calendar most of the time.
Sun is also working on really good refactoring tools. Gosling has a vision of "lint on steroids". They have a few good prototypes, but nothing close to being released yet.
Another point in planning is better performance for JNI. Make it possible to remove checks and stuff like that. It seemed that arrays were the big problem. You can get quite good speed already, by not using arrays, and having as much NIO as possible.
These were my opinions on the more interesting stuff from the Fireside Chat.
Prenumerera på:
Inlägg (Atom)
