fredag, juli 04, 2008

Java and mocking

I've just spent my first three days on a project in Leeds. It's a pretty common Java project, RESTful services and some MVC screens. We have been using Mockito for testing which is a first for me. My immediate impression is quite good. It's a nice tool and it allows some very clean testing of stuff that generally becomes quite messy. One of the things I like is how it uses generics and the static typing of Java to make it really easy to make mocks that are actually type checked; like this for example:
Iterator iter = mock(Iterator.class);

// Call stuff that starts interaction

These are generally the only things you need to stub stuff out and verify that it was called. The things you don't care about you don't verify. This is pretty good for being Java, but there are some problems with it too. One of the first things I noticed I don't like is that interactions that isn't verified can't be disallowed in an easy way. Optimally this would happen at the creation of the mock, instead of actually calling the verifyNoMoreInteractions() afterwards instead. It's way to easy to forget. Another problem that quite often comes up is that you want to mock out or stub some methods but retain the original behavior of others. This doesn't seem possible, and the alternative is to manually create a new subclass for this. Annoying.

Contrast this to testing the same interaction with Mocha, using JtestR, the difference isn't that much, but there is some missing cruft:
iter = mock(Iterator)

# Call stuff that starts interaction
Ruby makes the checking of interactions happen automatically afterwards, and so you don't have any types you don't need to care about most stuff the way you do in Java. This also shows a few of the inconsistencies in Mockito, that is necessary because of the type system. For example, with the verify method you send the mock as argument and the return value of the verify-method is what you call the actual method on, to verify that it's actually called. Verify is a generic method that returns the same type as the argument you give to it. But this doesn't work for the stub method. Since it needs to return a value that you can call toReturn on, that means it can't actually return the type of the mock, which in turn means that you need to call the method to stub before the actual stub call happens. This dichotomy gets me every time since it's a core inconsistency in the way the library works.

Contrast that to how a Mockito like library might look for the same interaction:
iter = mock(Iterator)

# Do stuff

The lack of typing makes it possible to create a cleaner, more readable API. Of course, these interactions are all based on how the Java code looked. You could quite easily imagine a more free form DSL for mocking that is easier to read and write.

Conclusion? Mockito is nice, but Ruby mocking is definitely nicer. I'm wondering why the current mocking approaches doesn't use the method call way of defining expectations and stubs though, since these are much easier to work with in Ruby.

Also, it was kinda annoying to upgrade from Mockito 1.3 to 1.4 and see half our tests starting to fail for unknown reasons. Upgrade cancelled.

16 kommentarer:

Caleb Powell sa...

JMock will throw an Error if any unspecified methods are invoked on your mock object. For example, the call to will cause a unit test to fail:

//Java 4:
Mock mockIterator = mock(Iterator.class); mockIterator.expects(once()).method("hasNext").will(returnValue(true));

Iterator iterator = (Iterator) mockIterator.proxy();

//Java 5:
final Iterator iterator = mock(Iterator.class);

checking(new Expectations(){{


Of course, Java does not permit you to express this as concisely as Ruby does. And no, you can't retain the behaviour of the original class :-(

Moandji Ezana sa...

I've been using Mockito of late, too, and my impression was that you typically didn't need to verify that stubbed methods were called.

Iirc, the reasoning is that you only stub method calls the code under test cares about, so if it's not called, the test will fail anyway. The Mockito guy explains it more clearly here.

"One of the first things I noticed I don't like is that interactions that isn't verified can't be disallowed in an easy way"

I guess this is a bit of a compromise. If you've used EasyMock, then you know how annoying it is to have to specify every interaction, even the ones you aren't interested in. Overall, I prefer Mockito's choice, as specifically denying interactions can quickly feel like over-specifying.

Szczepan Faber sa...

Hi Ola,

>Mocha, using JtestR

JtestR is an interesting experiment. However, I couldn't give up on superior IDE support (refactoring, finding usages, autocomplete) just to cut few lines of test code. Writing test in a different language than a unit is quite adventurous, though :)

>verifyNoMoreInteractions() afterwards instead.
>It's way to easy to forget.

verifyNoMoreInteractions() is not a framework contract that you have to remember and write for every test method. verifyNoMoreInteractions() is just an assertion from the interaction testing toolkit. Use it only if it is relevant. Have a look at this dummy example:

someMethod() {;;

For above code I wouldn't use verifyNoMoreInteractions().

However in following scenario:

someMethod() {
if (condition) {;
} else {;

it perfectly makes sense to do the following:

shouldFooWhenCondition() {

I use verifyNoMoreInteractions() only if it describes relevant application code.

>Mockito 1.3 to 1.4 and see half our tests starting

We've upgraded the cglib version - perhaps your application already pulls in a different version? You can try with mockito-core.jar from zip distro. BTW, submitting a bug report is a nice way of supporting OSS (you do some OS don't you? :)

Anonym sa...

In jMock you can ignore any methods you don't care about, or even entire objects (and therefore the objects they return and so on). So you can avoid the problems you have with EasyMock of lots of test set-up. You can even ignore methods during set up and then expect them during the test. It's very configurable.

Ola Bini sa...


I kinda expected a response in this style. =)

So first, yeah, the tools question. You know that IntelliJ can refactor Java code that's being used in Ruby, right?

But the longer story is that for me writing tests in Ruby is not to make them a few lines shorter. It's to increase readability and make the test focus on what they are actually testing. The types are seldom under test, and that means that they usually just get in the way. So why not omit them? The Ruby test frameworks are probably the best in the world.

Well, verifyNoMoreInteractions is an assertion that is so common, and so useful, that I would like to have an option to have it checked automatically. I almost _never_ want _any_ more calls than the ones I describe. In your first example, I would probably use verifyNoMoreInteractions, to make sure that the service only gets the two calls expected.

One of the problems I'm seeing in the example code for mocking in general and Mockito in particular, is the focus on procedural code. The problem is, I write in a functional style, and so do most other people who are fluent in other languages beside Java. The TestSpy approach makes it possible to check interactions afterwards, without setting up stubs/expectations, but only if you don't care about the return value. If you write in a functional style, your tests will have to set up the return values. Which means that either your tests gets longer and more complicated, or you drive your code to be more procedural. Both bad things.

I had more time yesterday, and checked why 1.4 broke things. As you say it was the cglib version that interacted badly with XStream. Have you considered doing what we do in JRuby with ASM, and rename all the classes in CGlib, so they don't clash?

Szczepan Faber sa...

>the tools question

For me, the must-have IDE feature is the ability to 'generate' source code from the test (some people call it TDD :P ). It's when I use classes/methods that don't exist yet, then ctrl+1 in Eclipse. If I can do it with JTestR then that's a big, big '+'.

>to increase readability and make the test focus on what they are actually testing

Totally agree, but I guess I don't think that static types or java testing frameworks are the main reason that many tests are unreadable :) Point taken, though.

>I almost _never_ want _any_ more calls than the ones I describe

Me too, but I guess I don't find it worth testing. It's like doing following for state based-testing:

assertEquals(expectedArticle, article);

This way I make sure no other state was changed on the article other than 'read' (if read() method does something extra with a state then the test fails). But hey, no one writes state based tests like that. Is interaction testing really that different that it requires 'extra defensive' style? Think about it and let me know - I'm drafting a blog entry about it.

>the focus on procedural code

You mean that we prefer to "tell, don't ask"? In OO language like Java it works quite well.

>tests get longer and more complicated

When I care about return value it doesn't necessarily mean my tests become longer/complicated. I almost never verify() stubs because I worship the difference between asking and telling :)

>rename all the classes in CGlib, so they don't clash?

Excellent idea!!! Thank you for that!

Interesting post and interesting follow-ups!

Ola Bini sa...

Szczepan: Ok, so the short cut to generate a missing Java class I'm not sure about. I'm pretty sure it's there in IntelliJ, though. And in fact, what's interesting with using Ruby is that you can do TDD much easier: you don't even need to create the class under test before starting testing...

In fact, I generally look at surrounding state too. As a typical example I'm writing a PKCS#7 implementation right now, and for any updating operations I'm making sure that the state in that object is unchanged, except for the state I want to change. That gives me more security, and it also makes the process of updating some behavior mean that I need to update my tests. That's a good thing in my book. So no, I don't think interaction based testing is any different. I'm just testing more defensively than you seem to do.

Procedural vs functional has nothing to do with "tell, don't ask". What I mean with functional code is that most of my invocations return a value that is used in further processing. Typical example is immutable objects - which give your code lots of benefits. So immutable objects need to be returned as the result from interactions, and the chain continues like that. Which means that null is basically never right return value. In fact, returning null is almost always something that should break my code.

So, I need to verify stubs, since I need to make sure that the code under test doesn't break the surrounding code by returning null from the mocked out methods.

Yeah, renaming is good. Do that. We use JarJar Links.

Szczepan Faber sa...

Hi Ola, you're gonna have a nice comment count on this one :)

>In fact, I generally look at surrounding state too.

Fair enough but I still believe that majority of state-based tests with assertions don't protect against 'the unexpected'. Other words, a typical assertion checks only single condition without worrying about surrounding state. Sometimes however - like in your scenario - you really want to be extra defensive which is fine.

>So, I need to verify stubs, since I need to make sure that the code under test doesn't break the surrounding code by returning null from the mocked out methods.

Returning null would break the test anyway because of NullPointerException. Therefore in most cases verifying stubs is redundant.

Ola Bini sa...

So, yeah. I guess our approaches to how much validation we do on both state and behavior differ a bit.

But I think you misunderstood my point about null. The thing is, in a functional test having an interaction return null (since it's only verified, but not stubbed to return something) will mean that the code under test breaks on an NPE. But that will make the test fail, even though the thing I verified worked as it should. So I need to guard against the NPE with a toReturn, and do the verify too.

Szczepan Faber sa...

>our approaches to how much >validation we do on both state and >behavior differ a bit.

Oh yes. I enjoy the level of validation known from typical state testing.

>So I need to guard against the NPE >with a toReturn, and do the verify >too.

Don't you think that not having NPE (because you did toReturn()) is enough proof that there is an interaction? I do. That's why I don't usually verify stubs... I guess this is another difference in our approaches to testing :)

Leandro Silva sa...

Very good post!

I read the your interview in and quite liked.




Hendy sa...

Try JMockit (different than JMock):

It uses Java 5 instrumentation so it's more like the Mocha that we used to love without old-style Java restrictions.

See also:

Anonym sa...

java is a very difficult language. i really cant get it.

Anonym sa...

Interesting, I don't often find myself interested in ensuring an interaction doesn't happen, I'll think think about that some more.

Having used several Java mocking frameworks and then Mockito, I'm now a fan of the latter.

The fundamental difference in my experience is Mockito's use of a TestSpy. This has allowed me to write more focused and concise tests that are simultaneously less brittle (more resilient to change) than a traditional interaction test that uses mocks (ala EasyMock/JMock).

In the Ruby, I'll use Pete Yandell's NotAMock with RSpec for the same benefits. It provides an easy way to create Test Spies and has complimentary Rspec matchers, so your expectations about interaction can be stated just like any other.

Anonym sa...

i do not program using java. so difficult

Anonym sa...

yes. java programming is too difficult. so hard to learn