Vagif Abilov's blog on .NET

Mock framework challenges in F#

Getting mock framework API rigth is uneasy task. Mock framework designers don’t have as much freedom as designers of other libraries: the purpose of a mock framework is not to expose an arbitraty API (unknown at the time of mock framework design) and intercept it in a transarent way, so developers can use mocked object as they were using real instances.

This is why API of modern mock frameworks is full of technique such as extension methods and lambda expressions. In addition these frameworks use so called fluent approach to API. Unfortunately such methods are not easily portable to other, especially non-imperative languages. Being fluent in one language does not mean fluent in the other one.

I decided to try various mock frameworks with F#. I realized that their C#-friendly syntax will look in F# clumsy, but I coudl overcome it by writing small wrappers. What concerned me more is incomatibility of F# functional delegates and expressions comparing to C#. For my tests I’ve chosen a Zoo example listed in an excellent series of blog posts by Richard Banks dedicated to mock framework comparison. Richard ran his comparison on free frameworks:

• Rhino.Mocks
• NSubstitute
• Moq

I’ve extended this set with two commercial products:

• Typemock Isolator
• Telerick JustMock

So far I only tried very basic mocking described in the first post of Richard’s series: faking return value. As I expected, even such a simple operation became a challenge when executed from F# code. I managed to make tests work only for two and half frameworks. Below is a description of what I did and how it was possible to have half of success.

1. Rhino.Mocks: success

Rhino.Mocks is the most popular .NET mock framework. This was the test code that I wanted to port to F#:

[Test]
public void Return()
{
    var monkey = MockRepository.GenerateMock<IMonkey>();
    monkey.Stub(m => m.Name).Return("Spike");

    var actual = monkey.Name;

    Assert.AreEqual("Spike", actual);
}

As you can see, “monkey” is used both as an instance of an object that implements IMoney interface (so we can call money.Name directly) and as an object exposing mocking API (so we can call a Stub method on it). This is achieved by using extension methods, and extension methods are language specific – being defined in C# code they are not available in F# as extensions. But they can be used from a static class where they are originally defined:

[<Test>]
member this.Return() =
    let monkey = MockRepository.GenerateMock<IMonkey>()
    RhinoMocksExtensions.Stub<IMonkey, string>(monkey, fun m -> m.Name).Return("Spike");

    monkey.Name
    |> should equal "Spike"

The test works, but RhinoMocksExtensions class has not been meant to be exposed to general public, so I put it in a little wrapper:

module FsMock.RhinoMocks

open Rhino.Mocks

type mock<'T when 'T : not struct>() =

    let instance = MockRepository.GenerateMock<'T>()

    member this.Object = instance

module Mock =

    let arrange<'T, 'R when 'T : not struct> (f : ('T -> 'R)) (r : 'R) (m : mock<'T>) = 
        RhinoMocksExtensions.Stub<'T, 'R>(m.Object, Function<'T, 'R>(f)).Return(r)

Now I can rewrite the origianl test in a F# style:

[<Test>]
member this.Return() =
    let monkey = mock<IMonkey>()
    monkey
    |> Mock.arrange (fun m -> m.Name) "Spike"
    let monkey = monkey.Object

    monkey.Name
    |> should equal "Spike"

Both the original and rewritten tests passed.

2. NSubstitute: success

NSubstitute is a new kid on the block. The framework prioritizes simplicity over coverage of all possible mocking scenarios. Unlike other frameworks, NSubstitutes avoids using lambda expressions. This makes it easier to use in F#.

Here’s the C# code using NSubstitute:

[Test]
public void Return()
{
    var monkey = Substitute.For<IMonkey>();
    monkey.Name.Returns("Spike");

    var actual = monkey.Name;

    Assert.AreEqual("Spike", actual);
}

Like Rhino.Mocks, NSubstitute also uses extension methods, so porting the above test to F# results in the following code:

[<Test>]
member this.Return() =
    let monkey = Substitute.For<IMonkey>()
    SubstituteExtensions.Returns(monkey.Name, "Spike")

    monkey.Name
    |> should equal "Spike"

Again, I created a little helper to make syntax F# friendly:

module FsMock.NSubstitute

open NSubstitute

type mock<'T when 'T : not struct>() =

    let instance = Substitute.For<'T>()

    member this.Object = instance


module Mock =

    let arrange<'T, 'R when 'T : not struct> (f : ('T -> 'R)) (r : 'R) (m : mock<'T>) = 
        SubstituteExtensions.Returns<'R>(f(m.Object), r)

Now the test code looks identical to Rhino.Mocks example:

[<Test>]
member this.Return() =
    let monkey = mock<IMonkey>()
    monkey
    |> Mock.arrange (fun m -> m.Name) "Spike"
    let monkey = monkey.Object

    monkey.Name
    |> should equal "Spike"

And the test passed!

NB! I have deliberately unified mocking API when creating F# wrappers for different frameworks. Since F# API will be quite different from it’s C# counterpart, I didn’t want to multiply number of API sets. Instead I came up with a common and easy to understand set of names (“mock”, “arrange”) that is used in each wrapper.

3. Moq: half success

Now that sounds strange. A test should either succeed of fail. So what has happenned to Moq? Here’s the story.

Moq is a second most popular framework after Rhino.Mocks, and it has pioneered act-arrange-assert API based on lambda-expressions. This is how the C# test looks when using Moq:

[Test]
public void Return()
{
    var monkey = new Mock<IMonkey>();
    monkey.Setup(m => m.Name).Returns("Spike");

    var actual = monkey.Object.Name;

    Assert.AreEqual("Spike", actual);
}

Although the expression “m => m.Name” looks exactly like in Rhino.Mocks, there is a big difference behind. Rhino.Mocks uses Func<T> delegate, and Moq is based on LINQ expressions.  Working with such expressions in F# requires use of so called quotation expressions that should be converted to LINQ and then downcasted to a generic LINQ expression. The corresponding code looks quite criptic:

[<Test>]
member this.Return() =
    let monkey = new Mock<IMonkey>()
    let expr = (<@@ System.Func<IMonkey, string>(fun (m : IMonkey) -> m.Name) @@>).ToLinq()
    monkey.Setup(expr :?> System.Linq.Expressions.Expression<System.Func<IMonkey, string>>).Returns("Spike");
    let monkey = monkey.Object

    monkey.Name
    |> should equal "Spike"

But this does not work. Here’s the output:

Test 'MoqTests+MoqTests.Return' failed: System.NullReferenceException : Object reference not set to an instance of an object.
	at Moq.MethodCall.SetFileInfo()
	at Moq.MethodCall..ctor(Mock mock, Expression originalExpression, MethodInfo method, Expression[] arguments)
	at Moq.MethodCallReturn..ctor(Mock mock, Expression originalExpression, MethodInfo method, Expression[] arguments)
	at Moq.MethodCallReturn`2..ctor(Mock mock, Expression originalExpression, MethodInfo method, Expression[] arguments)
	at Moq.Mock.<>c__DisplayClass15`2.b__14()
	at Moq.PexProtector.Invoke[T](Func`1 function)
	at Moq.Mock.SetupGet[T1,TProperty](Mock mock, Expression`1 expression)
	at Moq.Mock.<>c__DisplayClass12`2.b__11()
	at Moq.PexProtector.Invoke[T](Func`1 function)
	at Moq.Mock.Setup[T1,TResult](Mock mock, Expression`1 expression)
	at Moq.Mock`1.Setup[TResult](Expression`1 expression)
	C:\Projects\NET\MockComparison\TempTests\MoqTests.fs(20,0): at MoqTests.MoqTests.Return()

So why half success then? Well, if I execute the same code from F# interactive window, it works!

> let monkey = new Mock<IMonkey>()
let expr = (<@@ System.Func<IMonkey, string>(fun (m : IMonkey) -> m.Name) @@>).ToLinq()
monkey.Setup(expr :?> System.Linq.Expressions.Expression<System.Func<IMonkey, string>>).Returns("Spike");
printfn "%s" monkey.Object.Name;;
Spike

val monkey : Mock<IMonkey>
val expr : Expression = m => m.Name

Note the word “Spike” printed right after the line that begins with “printfn”. This is the output.

So I don’t really have a clue why the same code works in F# interactive session but fails being compiled in an assembly. I may need to investigate some more.

4. Typemock Isolator: failure

Ironically, both commercial frameworks failed to be used with F#. The C# code that used to test Typemock Isolator looks like this:

[Test]
public void Return()
{
    var monkey = Isolate.Fake.Instance<IMonkey>();
    Func<string> func = () => monkey.Name;
    Isolate.WhenCalled(func).WillReturn("Spike");

    var actual = monkey.Name;

    Assert.AreEqual("Spike", actual);
}

Here's the corresponding F# code:

[<Test>]
member this.Return() =
    let monkey = Isolate.Fake.Instance<IMonkey>();
    Isolate.WhenCalled(System.Func<string>(fun ignore -> monkey.Name)).WillReturn("Spike");

    monkey.Name
    |> should equal "Spike"

This test fails with the following output:

Test 'TypeMockTests+TypeMocksTests.Return' failed: TypeMock.TypeMockException : 

*** Cannot call Isolate.WhenCalled() with method group fake.Invoke. Try using Isolate.WhenCalled( () => fake.Invoke()) instead
	at ec.a()
	at dm.b(Boolean A_0)
	at im.c(Boolean A_0)
	at im.a(Object A_0, Boolean A_1, Func`1 A_2, Action A_3, Action A_4, Action A_5)
	at im.c(Object A_0)
	at TypeMock.ArrangeActAssert.ExpectationEngine`1.a(TResult A_0)
	C:\Projects\NET\MockComparison\TempTests\TypeMockTests.fs(21,0): at TypeMockTests.TypeMocksTests.Return()

I described the problem to Typemock developers and was advised to try a different approach presented by Roy Osherove in his blog post. Unfortunately the modified test still fails, although with a different exception.

5. JustMock: failure

JustMock is also a commercial product, and it’s API also lacked F# compatibility. This is the original C# code:

[Test]
public void Return()
{
    var monkey = Mock.Create<IMonkey>();
    Mock.Arrange(() => monkey.Name).Returns("Spike");

    var actual = monkey.Name;

    Assert.AreEqual("Spike", actual);
}

JustMock also uses LINQ expressions, so I had to use a similar trick to what I had to do with Moq:

[<Test>]
member this.Return() =
    let monkey = Mock.Create<IMonkey>();
    let expr = (<@@ System.Func<string>(fun ignore -> monkey.Name) @@>).ToLinq()
    Mock.Arrange<string>(expr :?> System.Linq.Expressions.Expression<System.Func<string>>).Returns("Spike");

    monkey.Name
    |> should equal "Spike"

The code compiles and does not throw any expection during the execution. But the “Name” property is not set, so “should equal” assertion fails.

Conclusion: no offence, just exploring possibilities

One thing that I want to get straigh is that these tests say absolutely nothing about general design quality of respective mock frameworks. As I mentioned in the beginning, API design requirements of such frameworks require their developers to apply language-specific tricks to make mocking simple. So the fact that some API fits another programming language is more luck rather than conscious vision. Moreover, I only tried the simplest of a large variety of functions exposed by mock frameworks. A slightly more complicated scenario would fail all frameworks, I am sure.

However, since more developers express interests in F# and start using it in their projects, I believe it’s time for mock frameworks developers to consider this new territory and extend their products to offer full suport for this exciting language.