Moving away from blogspot

This will be my last post on this blog. For several reasons I like the idea of keeping more in control over my blog and the environment surrounding it. I also have some things I'd like to publish that isn't well suited for the blog format, and moving to another location means that I can keep all my content in the same place. More long term I'm planning on migrating information about my open source projects there to.

But what you need to know is this: This blog ends. A new blog is born. All my old entries have been migrated. The important addresses for the new blog is:

• Blog: http://olabini.com/blog
• Feed: http://feeds.feedburner.com/olabini

And that's it. The new content will obviously be available at http://olabini.com, but right now this site just redirects to the blog.

The blog is dead, long live the blog.

ThoughtWorks Sweden is available

I would like to announce that ThoughtWorks Sweden is now in motion. We have business cards and an office. Everyone is returning from their long lovely Swedish summer vacations.

This means that ThoughtWorks Sweden is ready, and available for work. If you or your business have a project you need help with, don't hesitate to contact me (at obini@thoughtworks.com) or Marcus Ahnve (at mahnve@thoughtworks.com).

We are located in Stockholm, but we are open for work anywhere in the Nordic regions.

So what kind of work are we most suited for? Our sweet spot is in delivery and technical advisory regarding Java, Ruby and JRuby. And if you're interested in understanding how our Agile approach can change your company, we can do organizational transformation projects and also coaching and advisory.

Don't hesitate to get in touch!

JtestR 0.3.1 Released

JtestR allows you to test your Java code with Ruby frameworks.

Homepage: http://jtestr.codehaus.org
Download: http://dist.codehaus.org/jtestr

JtestR 0.3.1 is the current release of the JtestR testing tool. JtestR integrates JRuby with several Ruby frameworks to allow painless testing of Java code, using RSpec, Test/Unit, Expectations, dust and Mocha.

Features:
- Integrates with Ant, Maven and JUnit
- Includes JRuby 1.1, Test/Unit, RSpec, Expectations, dust, Mocha and ActiveSupport
- Customizes Mocha so that mocking of any Java class is possible
- Background testing server for quick startup of tests
- Automatically runs your JUnit and TestNG codebase as part of the build

Getting started: http://jtestr.codehaus.org/Getting+Started

New in the 0.3.1 release is upgrade of JRuby to revision r7479 which includes several new Java Integration features, upgrading of ActiveSupport to 2.1.0, fixing a severe memory leak in the background server and some minor usability features.

New and fixed in this release:
JTESTR-50 Difference in functionality when stubbing a method on a Java class vs a Ruby class using mocha
JTESTR-51 Mocking of classes lacking default constructors results in a NameError
JTESTR-53 Push the JtestR JRuby builds to maven repos
JTESTR-56 Upgrade ActiveSupport
JTESTR-57 Make it possible to use local versions of libraries.
JTESTR-59 No output when no tests found.
JTESTR-60 OutOfMemoryError
JTESTR-61 Documentation improvments - ant test-server
JTESTR-62 Having the jtestr.jar in the base directory doesn't work
JTESTR-63 Update JRuby version

Where is the Net::SSH bug

Yesterday I spent several hours trying to find the problem with our implementation of OpenSSL Cipher, that caused the Net::SSH gem to fail miserable during negotiation and password verification. After various false leads I finally found the reason for the strange behavior. But I really can't decide if it's a bug, and if it's a bug where the bug is. Is it in Ruby's interface to OpenSSL, or is it in Net::SSH?

No matter what cipher suite you use for SSH, you generally end up using a block cipher, mostly something like CBC. That means an IV (initialization vector) is needed, together with a key. The relevant parts of OpenSSL used is the EVP_CipherInit, EVP_CipherUpdate and EVP_CipherFinal family of methods. Nothing really strange there. The Ruby interface matches these methods quite closely; every time you set a key, or an IV, or some other parameter, the CipherInit method is called with the relevant data. When CipherUpdate is called, the actual enciphering or deciphering starts happening, and CipherFinal takes care of the final block.

At the point EVP_CipherFinal is called, nothing more should be done using the specific Cipher context. Specifically, no more Update operations should be used. The man page has this to say about the Final-methods:

After this function is called the encryption operation is finished and no further calls to EVP_EncryptUpdate() should be made.

Now, what I found was that same documentation is not part of the Ruby interface. And Net::SSH is actually reusing the same Cipher object after final has been called on it. Specifically, it continues the conversation, calling update a few times and then final. The general flow for a specific Cipher object in Net::SSH is basically init->update->update->final->update->update->final.

So what is so bad about this then? Well, the question is really this: what IV will the operations after the first final call be using? The assumption I made is that obviously it will use the original IV set on the object. Something else would seem absurd. But indeed, the IV used is actually the last IV-length bytes of encrypted data returned. Is this an obvious or intended effect at some level? Probably not, since the OpenSSL documentation says you shouldn't do it. The reason it works that way is because the temporary buffer used in the Cipher context isn't cleared out at the end of the call to final.

In contrast, the Java Cipher object will call reset() as part of the call to doFinal(). Where reset() will actually reset the internal buffers to use the original IV. So the solution is simple for encryption. Just save away 8 or 16 bytes of the last generated crypto text and set that manually as the IV after the call to doFinal. And what about decryption? Well, here the IV needs to be the last crypto text sent in for deciphering, not the result of the last operation.

So Net::SSH seems to work fine with JRuby now. I'm about to release a new version of JRuby-OpenSSL including these and many other things.

But the question remains. Is it a bug? If it is, is it in the Ruby OpenSSL integration, or in the Net::SSH usages of Ciphers? If it's in the Net::SSH code, why does it actually work correctly when communicating with an SSH server? Or is this behavior of using the last crypto text as IV something documented in the SSH spec?

Enlightenment would be welcome.

Security vs Convenience

I really like Cryptogram and read every issue. It's interesting stuff that talks a lot about how our minds work in conjunction with risk and reward. Today I had a typical example of how security versus convenience is a part of day to day life.

I had just checked out from my hotel, and wanted to store all my luggage (including my laptop bag) in the hotel until my ride out of town arrived. I asked about this, and it was fine, they had a room for this. The person in the reception pointed me to an open room and said it was open and that I could put my stuff there. Feeling uneasy I asked how secure it was, and she answered that the door was usually locked. OK, I said, but can someone take any bag from inside of there? Yes, was the answer. I decided I couldn't store my stuff there. Even if the risk was small, losing my work laptop would be way to bad to risk. But I also decided I couldn't drag my two heavy bags and laptop bag around.

I ended up putting the large bags in the room, and just taking my laptop bag around. I didn't have as much to lose with the large bags, and the price of inconvenience in taking them along was just to high. These considerations go into everything we do in programming and systems engineering. A totally secure system is generally quite inconvenient to use, while an insecure system can be very pleasant to use. The trick is to get the balance right, I guess.

JtestR doesn't start up.

Justin Smestad uncovered an issue with JtestR that can cause some quite unintuitive output, and be hard to debug. Some info can be found here: http://www.evalcode.com/2008/08/jtestr-woes/ and here: http://jira.codehaus.org/browse/JTESTR-62. The issue has been fixed on trunk, but hasn't been released yet. The issue is very simple - just make sure you don't have the jtestr.jar file in the base directory where your project lives (this is usually the same place as the build.xml file). There are two ways to achieve this, either move the file into a directory or rename the file to something else.

Java and mocking

I've just spent my first three days on a project in Leeds. It's a pretty common Java project, RESTful services and some MVC screens. We have been using Mockito for testing which is a first for me. My immediate impression is quite good. It's a nice tool and it allows some very clean testing of stuff that generally becomes quite messy. One of the things I like is how it uses generics and the static typing of Java to make it really easy to make mocks that are actually type checked; like this for example:

Iterator iter = mock(Iterator.class);
stub(iter.hasNext()).toReturn(false);

// Call stuff that starts interaction

verify(iter).hasNext();

These are generally the only things you need to stub stuff out and verify that it was called. The things you don't care about you don't verify. This is pretty good for being Java, but there are some problems with it too. One of the first things I noticed I don't like is that interactions that isn't verified can't be disallowed in an easy way. Optimally this would happen at the creation of the mock, instead of actually calling the verifyNoMoreInteractions() afterwards instead. It's way to easy to forget. Another problem that quite often comes up is that you want to mock out or stub some methods but retain the original behavior of others. This doesn't seem possible, and the alternative is to manually create a new subclass for this. Annoying.

Contrast this to testing the same interaction with Mocha, using JtestR, the difference isn't that much, but there is some missing cruft:

iter = mock(Iterator)
iter.expects(:hasNext).returns(false)

# Call stuff that starts interaction

Ruby makes the checking of interactions happen automatically afterwards, and so you don't have any types you don't need to care about most stuff the way you do in Java. This also shows a few of the inconsistencies in Mockito, that is necessary because of the type system. For example, with the verify method you send the mock as argument and the return value of the verify-method is what you call the actual method on, to verify that it's actually called. Verify is a generic method that returns the same type as the argument you give to it. But this doesn't work for the stub method. Since it needs to return a value that you can call toReturn on, that means it can't actually return the type of the mock, which in turn means that you need to call the method to stub before the actual stub call happens. This dichotomy gets me every time since it's a core inconsistency in the way the library works.

Contrast that to how a Mockito like library might look for the same interaction:

iter = mock(Iterator)
stub(iter).hasNext.toReturn(false)

# Do stuff

verify(iter).hasNext

The lack of typing makes it possible to create a cleaner, more readable API. Of course, these interactions are all based on how the Java code looked. You could quite easily imagine a more free form DSL for mocking that is easier to read and write.

Conclusion? Mockito is nice, but Ruby mocking is definitely nicer. I'm wondering why the current mocking approaches doesn't use the method call way of defining expectations and stubs though, since these are much easier to work with in Ruby.

Also, it was kinda annoying to upgrade from Mockito 1.3 to 1.4 and see half our tests starting to fail for unknown reasons. Upgrade cancelled.

JtestR, RubyGems, and external code

One question I've gotten a few times now that people are starting to use JtestR, is how to make it work with external libraries. This is actually two different questions, masquerading as one. The first one regard the libraries that are already included with JtestR, such as JRuby, RSpec or ActiveSupport. There is an open bug in JIRA for this, called JTESTR-57, but the reason I've been a bit hesitant to add this functionality until now, is because JtestR actually does some pretty hairy things in places. Especially the JRuby integration does ClassLoader magic that can potentially be quite version dependent. The RSpec and Mocha integration is the same. I don't actually modify these libraries, but the code using them is a bit brittle at the moment. I've worked on fixing this by providing patches to the framework maintainers to include the hook functionality I need. This has worked with great success for both Expectations and RSpec.

That said, I will provide something that allows you to use local versions of these libraries, at your own risk. It will probably be part of 0.4, and if you're interested JTESTR-57 is the one to follow.

The second problem is a bit more complicated. You will have seen this problem if you try to do "require 'rubygems'". JtestR does not include RubyGems. There are both tecnnical and non-technical reasons for this. Simply, the technical problem is that RubyGems is coded in such a way that it doesn't interact well with loading things from JAR-packaged files. That means I can't distribute the full JtestR in one JAR-file if I wanted RubyGems, and that's just unacceptable. I need to be able to bundle everything in a way that makes it easy to use.

The non-technical reason is a bit more subtle. If RubyGems can be used in your tests, it encourages locally installed gems. It's a bit less pain to do it that way initially, but remember that as soon as you check the tests in to version control (you are using version control, right?) it will break in unexpected ways if other persons using the code doesn't have the same gems installed, with the same versions.

Luckily, it's quite simple to work provide functionality to JtestR, even if no gems are used. The first step is to create a directory that contains all the third party code. I will call it test_lib and place it in the root of the project. After you have done that you must first unpack your gems:

mkdir test_lib
cd test_lib
jruby -S gem unpack activerecord

When you have the gems you want unpacked in this directory, you can add something like this to your jtestr_config.rb:

Dir["test_lib/*/lib"].each do |dir|
  $LOAD_PATH << dir
end

And finally you can load the libraries you need:

require 'active_record'