måndag, december 31, 2007

Scala unit testing

I wish this could be a happy story. But it's really not. If I have made any factual errors in this little rant, don't hesitate to correct me - I would love to be proved wrong about this.

Actually, I wrote this introduction before I went out to celebrate New Years. Now I'm back to finish the story, and the picture have changed a bit. Not enough yet, but we'll see.

Let's tell it from the start. I have this project I've just started working on. It seemed like a fun and quite large thing that I can tinker on in my own time. It also seemed like a perfect match to implement in Scala. I haven't done anything real in Scala yet, and wanted to have a chance to do it. I like everything I've seen about the language itself. I've said so before and I'll say it again. So I decided to use it.

As you all probably know, the first step in a new project is to set up your basic structure and getting all the simple stuff working together. Right, for me that means a simple Ant script that can compile Java and Scala, package it into a jar file, and run unit tests on the code. This was simple. ... Well, except for the testing bit, that is.

It seems there are a few options for testing in Scala. The ones I found was SUnit (included in the Scala distribution), ScUnit, Rehersal and specs (which is based on ScalaCheck, another framework). So these are our contestants.

First, take SUnit -- this is a very small project, no real support for anything spectacular. The syntax kinda stinks. One class for each test? No way. Also, no integration with Ant. I haven't even tried to run this. Testing should be painless, and in this case I feel that using Java would have been an improvement.

ScUnit looked really, really promising. Quite nice syntax, lots of smarts in the framework. I liked what the documentation showed me. It had a custom Ant task and so on. Very nice. It even worked for a simple Hello, World test case. I thought that this was it. So I started writing the starting points for the first test. For some reasons I needed 20 random numbers for this test. Scala has so many ways of achieving this... I think I've tried almost all of them. But all failed with this nice class loading exception just saying InstantiationException on an anonymous function. Lovely. Through some trial and error, I found out that basically any syntax that causes Scala to generate extra classes, ScUnit will fail to run. I have no idea why.

So I gave up and started on the next framework. Rehersal. I have no idea what the misspelling is about. Anyway, this was a no show quite quickly since the Ant test task didn't even load (it referenced scala.CaseClass, which doesn't seem to be in the distribution anymore). Well then.

Finally I found specs and ScalaCheck. Now, these frameworks look mighty good, but they need better Google numbers. Specs also has the problem of being the plural of a quite common word. Not a good recipe for success. So I tried to get it working. Specs is built on top of ScalaCheck, and I much preferred the specs way of doing things (being an RSpec fan boy and all). Now specs doesn't have Ant integration at all, but it does have a JUnit compatibility layer. So I followed the documentation exactly and tried to run it with the Ant JUnit task. KABOOM. "No runnable methods". This is an error message from JUnit4. But as far as I know, I have been able to run JUnit3 classes as good as JUnit4 classes with the same classpath. Hell, JRuby uses JUnit3 syntax. So obviously I have JUnit4 somewhere on my classpath. For the world of me I cannot find it though.

It doesn't really matter. At that point I had spent several hours getting simple unit testing working. I gave up and integrated JtestR. Lovely. Half of my project will now not be Scala. I imagine I would have learned more Scala by writing tests in it, than with the implementation. Apparently not. JtestR took less than a minute to get up and working.

I am not saying anything about Scala the language here. What I am saying is that things like this need to work. The integration points need to be there, especially with Ant. Testing is the most important thing a software developer does. I mean, seriously, no matter what code you write, how do you know it works correctly unless you test it in a repeatable way? It's the only responsible way of coding.

I'm not saying I'm the worlds best coder in any way. I know the Java and Ruby worlds quite well, and I've seen lots of other stuff. But the fact that I can't get any sane testing framework in Scala up and running in several hours, with Ant, tells me that the Scala ecosystem might not be ready for some time.

Now, we'll see what happens with specs. If I get it working I'll use it to test my code. I would love that to happen. I would love to help make it happen - except I haven't learned enough Scala to actually do it yet. Any way or another, I'm not giving up on using Scala for this project. I will see where this leads me. And you can probably expect a series of these first impression posts from me about Scala, since I have a tendency to rant or rave about my experiences.

Happy New Years, people!

10 kommentarer:

Patrick Logan sa...

I tinkered with scala just a bit a few months ago. I think I tried sunit, can't recall. I do recal scala working fine with junit though. Just define tests the way you usually do in junit, but do it in scala.

Anonym sa...

Why ant? Surely rake.

Unknown sa...

You might want to try using maven. I managed to get specs going with maven and the archetypes from www.liftweb.net. They give some examples in the tutorials section.

The only thing I had to change was the scala library version used from 2.6.0 to 2.6.1 - otherwise I was also getting class not found exceptions.

Eric sa...

Hi Ola,

I'm sorry that I couldn't find in time a solution for you to be able to work with junit4. I wish you didn't have to write that post!

I understand your frustration as writing tests should indeed be at the start of any development.

I've just resumed my search to find a reason why junit4 would refuse to run a junit3 test suite and I hope to be back soon with good news!

If anyone has some idea about that, I would appreciate, thanks.

Eric.
(specs author)

Ola Bini sa...

Eric, I do appreciate all your help, and specs is absolutely first on my list of the frameworks I would like to use. Also, I promise to post an update if we can find something that works.

Villane sa...

I had no problem with specs that I remember. I didn't use Ant, though. Also, just using JUnit directly in Scala should be fine too, as someone already recommended.

Anonym sa...

Hi Ola,

maybe that one can help you (it's using TestNG though):
http://jackcoughonsoftware.blogspot.com/2007/12/using-testng-in-scala-by-example.html

Eric sa...

Hi Ola,

You can try out specs-1.1.3 (http://code.google.com/p/specs/).

It should work with Ant and JUnit4 now. The short story is that JUnit4 was not able to run JUnit3 TestSuites.

Thanks for your future feedback on specs. As you can see in the downloads number, it has not been really battle-tested by hundreds of users, so I expect future issues or enhancement ideas.

Happy specs!

Eric.

Anonym sa...

I'll be back working on ScUnit soon, hopefully in the coming weeks. Just moved countries and started a new job and have been very busy - but I've got your comments here and will look into them.

Thanks!

jcobbers sa...

Have you looked at Scalatest? (http://www.artima.com/scalatest/)