Vagif Abilov's blog on .NET : unit testing

How to configure AutoTest.Net to use Snarl notifications

Vagif Abilov — Wed, 21 Sep 2011 13:36:11 GMT

Important part of continuous testing is instant and clear notification on test results. Developer should not need to inspect small text in Visual Studio output window, test execution results should flash like traffic lights, especially when things go wrong.

There are several commonly used notification applications for Windows, and AutoTest.Net is integrated with two of them: Growl and Snarl. They provide good visualization of test execution results. Here is how Snarl notification looks:

While integration with Growl works without any adjustments, Snarl needs to be configured to work with AutoTest.Net. Here is how you do it:

1. Open Snarl preference dialog

2. Go to “Network” tab and select “Yes” for “Listen for incoming Growl or Snarl notifications?

3. Start Visual Studio (AutoTest.Net or ContinuousTests a.k.a. MightyMoose must be installed)

4. Find and highlight “AutoTest.Net on 127.0.0.1”

5. Highlight in “Notification Classes” “Run succeeded”, then press “Configure”

6. Select “Yes” for “use custom icon?”

7. Go to AutoTest.Net or ContinousTests installation folder and find in Icons subfolder circleWin.png. You should see the icon with green light:

8. Repeat the previous step for classes “Run succeeded with warnings” and “Run failed” and choose for them icons with yellow and red light.

Now Snarl is configured to receive notifications from AutoTest.Net.

Testing 64-bit Oracle using Visual Studio

Vagif Abilov — Wed, 11 May 2011 20:26:35 GMT

I have installed 64-bit Oracle 11g on my development machine to work on some samples using Entity Framework 4.1 (a.k.a. “code first”). If you ask me why I installed 64-bit version and not 32-bit, I will not answer. I don’t know. Perhaps because Oracle has always associated for me with something big and 64 bits are certainly greater than 32. Now I think it was a mistake, but on the other hand it made me explore something new.

An attempt to connect to the database failed with the following error message: “ ----> System.Exception : Unable to load D:\Oracle\Vagif\product\11.2.0\dbhome_1\bin\oci.dll. Please check that you use 32x version of Oracle client with 32x application.” I checked project platform target. “Any CPU”. I was using TestDriven.NET test runner.

I’ve changed build platform to “64x” and tests passed. I also ran tests built for “Any CPU” using 64-bit NUnit runner, and they passed too. What I wanted now is leave “Any CPU” option for all projects and be able to test them from within Visual Studio.

Resharper was the only once that worked like a charm, no matter what I did. It uses 64-bit task runner on 64-bit platforms, so it just works.

Although TestDriven.NET test runner failed initially, it was easy to re-configure it: just choose 64-bit option for the test runner in its settings page.

~~The only bad guy was built-in Visual Studio test runner: it runs only as 32-bit process, so without unsupported patching it as 64-bit process (described~~ ~~here) it can’t run tests in 64-bit mode.~~

UPDATE. I stand corrected. MsTest also supports 64-bit mode.

Interaction-based testing and stable expectations

Vagif Abilov — Sat, 07 Aug 2010 22:52:48 GMT

Yesterday I participated in a discussion about how to test the following class (it was actually an interview question). Here’s the class under test:

public class ClassUnderTest
{
    private SomeService service;

    public ClassUnderTest(SomeService service)
    {
        this.service = service;
    }

    public void Foo(int fooInput)
    {
        if (fooInput > 0)
            this.service.DoSomething();
    }
}

What SomeService does is not relevant. Just something. As you can see, the ClassUnderTest is not very cooperative for the purpose of testing its functionality: it does not expose any properties, and the only method it contains does not return anything. How can we test it?

It looks like the only validation option that is left is to check that “something” is done, and the only way to check it is through interaction testing. Here is a couple of tests that achieve this (I used “method-condition-outcome” naming convention, but the names could be rephrased in BDD style):

[Test]
public void Foo_PositiveInput_SomethingIsDone()
{
    // Arrange
    var service = MockRepository.GenerateMock<SomeService>();
    var cut = new ClassUnderTest(service);

    // Act
    cut.Foo(1);

    // Assert
    service.AssertWasCalled(s => s.DoSomething());
}

[Test]
public void Foo_NonPositiveInput_SomethingIsNotDone()
{
    // Arrange
    var service = MockRepository.GenerateMock<SomeService>();
    var cut = new ClassUnderTest(service);

    // Act
    cut.Foo(0);

    // Assert
    service.AssertWasNotCalled(s => s.DoSomething());
}

I expressed during the discussion a lack of my enthusiasm about this solution noting that I tend to use state-based testing if possible. But is this possible in the example above? And what is a disadvantage of the interaction-based tests in this case?

Let’s have a closer look. First, the above tests essentially duplicate internal logic of the Foo implementation. If we merged two tests into one, then the logic of the assert part of the code would reflect pretty much what’s inside the implementation of Foo: “for positive input service.DoSomething should be called, otherwise not”.

Since the logic of the tests follow the logic of the code and was probably written by the same developer in the same time, there is very little in tests that validates the actual functionality. If I believe that Foo should call DoSomething for positive input, I will most likely get it right in the test that validates that Foo calls DoSomething for positive input. But what if my assumption is wrong? What if DoSomething should be called for input bigger than 100, and for other input values Foo should call DoSomethingElse? No, these tests can’t validate it. I’d say they can’t validate it by design because they are designed to reflect the internal logic of the method they validate. So the correctness of Foo implementation must be verified using other means, and these tests will ensure the Foo implementation stays correct assuming the original algorithm is right.

And this becomes the main value of such tests: they serve as regression tests protecting correct code from being improperly changed (perhaps by another developer who had to extend the original code and didn’t get it right). However, this sort of protection is not refactoring friendly. Let’s see why.

Imagine that SomeService class is refactored, DoSomething is still there but it’s semantics has changed and no longer fits the implementation of Foo. Instead, Foo is supposed to call a new method DoSomethingElse:

public void Foo(int fooInput)
{
    if (fooInput > 0)
        this.service.DoSomethingElse();
}

The system with the new change works properly, but one of the test for ClassUnderTest now fails! Of course, it has no knowledge about what is right or wrong semantically, it only cares about a certain method to be called. Depending on a system complexity, searching for the failure problem can take some time, but even if it’s an easy fix, we have a test that fails for a wrong reason.

In the article by Steve Freeman et al “Mock Roles, No Objects” they warn about mirroring target code logic in the tests: “Some uses of Mock Objects set up behaviour that shadows the target code exactly, which makes the tests brittle. This isparticularly common in tests that mock third-party libraries. The problem here is that the mock objects are not being used to drive the design, but to work with someone else’s. At some level, mock objects should shadow a scenario for the target code, but only because the design of that code should be driven by the test. Complex mock setup for a test is actually a hint that there is a missing object in the design.”

So the interaction-based tests can be more brittle, as we saw in our example. But what are the choices for that example? If we don’t verify calling DoSomething for positive Foo input, then what else can we verify?

To try to answer this question, let’s first figure out what kind of testing we are dealing with: unit or integation. For black box unit testing I am afraid there is not much to validate. No properties are exposed, the only method is void. If you are a code coverage junkie, you can maybe write a couple of tests to validate that the class can be instantiated and used without raising unexpected exceptions, but the value of such work is outside the scope of this blog post. Let’s focus instead on integration testing. How can we validate that the class does what it should?

As soon as we leave the idealistic constraints of black box unit testing and focus on overall functionality validation, we no longer need to express the validation goal in terms of ClassUnderTest methods, not to say about their internal implementation. We focus on the outcome of Foo in terms of what it does to the rest of the system. Does it change data in a database, send email, draws a graph on a screen?

In most cases business logic operations result in some (persistent or in-memory) state change. Then we can strive to catch these changes and use them in the test assertsions. This will help increasing test trustworthiness and tolerance to refactoring. However, determining state changes that correspond to certain action is not always easy, and even when it is, it may result in higher test complexity – you will have to insert some additional code that retrieves the state that is expected to change. And apart from this, there are still scenarios when the result of a certain action is not easily converted to the state available for validation in a test code. Nat Pryce in his old blog post gave an example of such scenario: tests for a graphical simulator. Really, how would you verify that a cell was drawn on a display using state based testing? Other scenarios that may fall in this category include calling external services, sending messages etc. They may (and usually do) leave the traces that can be used for state based testing, but tests will become complex and overloaded with state retrieval details. Clearly, interaction based tests may have their place here.

So what can we test with interaction-based test? Mark Seemann in his forthcoming book “Dependency Injection in .NET” classify dependencies as either stable or volatile. According to him, stable dependencies “are already there, tend to be very backwards compatible and invoking them have deterministic outcomes”. An example of stable dependency is .NET BCL. “Other examples may include specialized libraries that encapsulate algorithms relevant to your application. If you are developing an application that deals with chemistry, you may reference a third-party library that contains chemistry-specific functionality.” Volatile dependencies, on the other hand, do not provide a sufficient foundation for applications.

I believe separating things on stable and volatile is also helpful when deciding what to test using interaction-based testing, we only need to replace word “dependency” to “expectation”. We can use interaction-based testing as long as our expectations are stable, otherwise test becomes brittle. So coming back to original tests for Foo, we should ask ourselves a question: is DoSomething a stable expectation for a given scenario, or it is only valid for a current implementation and may not survive refactoring? If so, the expectation is volatile. Then we should inspect the call graph, find a stable expectation and use it in interaction-based tests. Examples of stable expectations are calls to external services, external API, output to display. As long as the feature is unchanged, the system will be making same outbound calls and draw same pictures on a screen. These activities are stable and can become foundations for interaction-based tests.

So how the original Foo test should look if we find that DoSomething internally sends an email? It can look like this:

[Test]
public void Foo_Should_Send_Notification_Mail_On_Positive_Input()
{
    // Arrange
    var mailServer = MockRepository.GenerateMock<MailServer>();
    var cut = new ClassUnderTest(new SomeService);

    // Act
    cut.Foo(1);

    // Assert
    mailServer.AssertWasCalled(s => s.SendMail());
}

Note that we no longer care about Foo exact implementation. It’s implementation is volatile. We only expect something that is stable in the scope of the given feature. Like sending mail (more accurate test code could verify that the mail was sent to a right person with a right subject, but we should be careful about not overspecifying the expectations, otherwise they will be no longer stable).

When I was about to finish this post, I found another old blog post, by Jeremy D. Miller, where he also uses email sending as an example of an operation that it worth testing using interaction-based tests: “We could use a state-based strategy to test the email. We could run the test, then run around and ask the expected recipients to check their inbox. […] You could also check an audit trail, but that's not the real functionality being tested. The easiest approach in the case of the email tests is to use interaction testing to verify that the email was sent to the SMTP service.”

So I believe interaction-based testing can be an efficient way to validate that operations result in proper actions, as long as tests don’t focus on volatile interactions and instead select stable expectations related to the feauture itself rather than it’s implementation details.

Impression from Typemock Academy

Vagif Abilov — Thu, 06 May 2010 08:29:56 GMT

I have written a guest post about my impressions from Typemock Academy that was published in Typemock blog.

Don’t use Activator.CreateInstance or ConstructorInfo.Invoke, use compiled lambda expressions

Vagif Abilov — Fri, 02 Apr 2010 15:43:00 GMT

For a long time I’ve been under impression that rule engine that comes with Microsoft Windows Workflow Foundation is slow, very slow. We used it to execute some of our business rules, and soon found out that rule processing slows down application execution. What was strange is that the rules were really simple and it was hard to believe that an industry-strength rule engine uses so long time on processing them.

Eventually I ran a profiler, and it immediately showed where the time was spent. In fact it was not the actual rule validation but preparation for it: instantiation of the WF rule parser. The parser class is not public, so we used a trick to obtain its instance: retrieved its non-public constructor information via reflection and then called Invoke.

This was sloooow. So slow that we noticed it in our integration tests. What made it noticeable is that after we converted some of our business rules to use WF rule engine, we were encouraged by results and move more rules to use the same techinque. In the end rule validation was called many times during a single business operation, and performance deteriorated.

My first approach to this issue was to cache validated rules, and it worked. The total time to execute integration tests was reduced by 50%. However, I was not quite happy with the situation: we have various places where objects are instantiated using reflection, caching is not always possible and in general complicates component design. Recently I reviewed the code of StructureMap (I like reading good code), and remembered that it no longer used reflection and instead instantiated dependencies using compiled lambda expressions.

Roger Alsing in this post presented a generic method that can be used as a replacement for constructor method invocation. The essence of the method is in it’s last three lines:

// Make a NewExpression that calls the ctor with the args we just created
NewExpression newExp = Expression.New(ctor, argsExp);                  

// Create a lambda with the New expression as body and our param object[] as arg
LambdaExpression lambda = Expression.Lambda(typeof(ObjectActivator), newExp, param);              

// Compile it
ObjectActivator compiled = (ObjectActivator)lambda.Compile();

Although lambda expressions are part of LINQ (and you have to import System.Linq.Expressions namespace to manage them), as we can see from this example, they can be very useful to solve core language tasks, such as object instantiation.

But what is the gain? The gain is huge bringing the performance close to the speed of native IL code. I wrote a few tests to measure performance of object creation in different scenarios: by calling ‘new’ constructor, Activator.CreateInstance, ConstructorInfo.Invoke and using compiled lambda expression. Here are the results:

DefaultConstructor_Activator: (0,20 ms per 1000 calls)
DefaultConstructor_CompiledExpression: (0,04 ms per 1000 calls)
DefaultConstructor_Invoke: (1,07 ms per 1000 calls)
DefaultConstructor_New: (0,02 ms per 1000 calls)
DefaultConstructor_NotCompiledExpression: (169,00 ms per 1000 calls)
NonDefaultConstructor_Activator: (3,39 ms per 1000 calls)
NonDefaultConstructor_CompiledExpression: (0,07 ms per 1000 calls)
NonDefaultConstructor_Invoke: (1,57 ms per 1000 calls)
NonDefaultConstructor_New: (0,02 ms per 1000 calls)
NonDefaultConstructor_NotCompiledExpression: (293,00 ms per 1000 calls)

Results says it all, especially when arranged in a graph.

I removed from the graph the results for not compiled lambda expressions, because they make other figures insignificant. But it is important to remember that compilation should be performed only once, so the code should be carefully reviewed to avoid occasional recompilcation of lambdas.

P.S. The title of this blog post may look provocative, and of course I don’t really mean somebody should completely stop using Activator.CreateInstance. Compilation of labmda expressions is expensive enough to limit it’s practical use. But there are certain use cases (like IoC frameworks) when it gives clear performance advantage.

NUnitForVS: integrating NUnit tests into Visual Studio

Vagif Abilov — Sat, 06 Mar 2010 20:23:45 GMT

Roy Osherove listed advantages of NUnit over MsTest but also mentioned one MsTest’s strength that can be crucial for many developers: “the integration with other team system tools and reporting is just beyond compare and the reporting alone helps alot to find recurring breaking tests, code churn vs. new tests and others”. This reminded me about what I recently read in a book by Jeff McWherter and Ben Hall Testing ASP.NET Web Applications where they said that forthcoming release of Visual Studio 2010 will make it possible to configure VS test runner to execute tests that use syntax different from MsTest.

I am a faithful user of TestDriven.Net, but I know that ability to use built-in Visual Studio test runner is an important factor that may affect the choice of unit test framework. So it would be nice to separate these decisions: selection of a unit test framework and selection of a test runner. I asked in Microsoft’s VS 2010 forum it new version of Visual Studio is really that flexible when it comes to configuring it’s test runner. Unfortunately not. Microsoft’s Euan Garden answered: “this was something we wanted to do in the release but never made it into the product.” However, Euan gave a couple of hints: Gallio and NUnit integration CodePlex project.

I checked CodePlex and found NUnitForVS: NUnit integration for Visual Studio 2008. I downloaded an ran the installer and within 5 minutes was able to make Visual Studio development environment to treat NUnit tests as they were native MsTest. The only preparation (in addition to installing NUnitForVS) was to open test project file in a text editor and add the following entry to the first PropertyGroup:

This has to be done to every test project that uses NUnit, but otherwise Visual Studio won’t be able to classify the project as a test project (why should it? – the only type of test projects that it natively recognizes is MsTest). After this change Visual Studio accepts NUnit tests, and you will see all your tests in the Test View window:

So far so good. Next checkpoint is to run and debug some of these tests. This also works well and test results are displayed in a respective window:

Note the very last test in this list, the one that is called “Add(2,3,5)”. This is a parameterized test implemented using TestCase attribute:

[TestCase(2, 3, 5)]
public void Add(int x, int y, int sum)
{
    Calculator calc = new Calculator();
    decimal result = calc.Add(x, y);
    Assert.AreEqual(sum, result, "Incorrect result");
}

So using NUnitForVS we can exploit features of NUnit 2.5 that don’t exist in MsTest, and still Visual Studio correctly treats them. This is good news.

When it comes to collecting code coverage, the situation is a little trickier. When I open code coverage window, I see not very encouraging message “Code Coverage is not enabled for this test run”:

Actually this is correct message: you have to explicitly enable code coverage instrumentation. But how? Apparently our solution sill lacks some piece, but luckily it’s easy to fix. All we need is to add a test configuration file with extension “testrunconfig”. Then we can activate its configuration and enable code coverage collection. One simple way to add such file is to add and then delete a test project to a solution (a “real” MsTest project). The MsTest project will be gone but will a trace in a form of test configuration file:

If you now enable code coverage instrumentation for assemblies in your solution and rerun the tests, you will see the coverate summary:

Pretty useless, isn’t it? The summary displays coverage only for the test assembly and not for the actual code under test. Why?

More googling, and the explanation came from Peter Stephen’s blog post: even though coverage instrumentation is enabled for all assemblies in our solution, one of assemblies was taken from a wrong place: the assembly under the test lies both in its own “bin” folder and in the “bin” folder of the test assembly, and it is there it has to be taken from. To resolve the problem you just need to add the assembly from a right place:

Note 2 copies of UnitTestDemo.dll. The unchecked one is the one that was added initially, the checked one was added manually. And everything works:

Of course, I only tried NUnitForVS on a very simple project. I plan to check it out with more complex code. But so far it looks quite promising opening Visual Studio test runner for NUnit - the most popular unit test framework.

When NUnit stops working on a build machine…

Vagif Abilov — Thu, 10 Dec 2009 12:13:58 GMT

… then it’s time to check the registry entries. Make sure you see something like this:

In case the build runs in the context of a dedicated user, not the one who logs on to a build machine and install applications, then it is likely that the build user does not have correct setup for NUnit assemblies. You can of course import missing registry entries into that user registry part, but the best is just to create them for everyone under LOCAL_MACHINE folder using path \HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\.NETFramework\v2.0.50727\AssemblyFolderEx.

.NET mocking framework comparison

Vagif Abilov — Wed, 14 Oct 2009 14:03:21 GMT

A couple of good sources for those who need to decide:

Capability comparison made by Morgan Soley. Half year old but with a feature comparison table.

Recently updated comparison by Andrew Kazyrevich.

TypeMock Isolator is a clear winner if it fits in your budget. Otherwise Rhino Mocks or Moq seem to be the most mature frameworks. According to a recent Roy Osherove’s pole they are in the lead with Rhino Mocks having 47% and Moq on second place with 34%. If I had to choose a free mocking framework today, I’d probably go with Moq because of Rhino Mocks’ confusing “very large array of syntaxes”. Or wait for Rhino Mocks 4.0 that will be radically simplified and, like Isolator, will go for AAA abd single “fake” concept.

But for mocking without severe limitations there is no alternative to Isolator. Still in the lead.

We Can Mock It Out

Vagif Abilov — Wed, 17 Jun 2009 20:37:00 GMT

Today there was a first day of Norwegian Developers Conference. Great speakers and great sessions. I attended talks by Scott Hanselman, Luca Bologneze, Jeremy Miller, Udi Dahan and Juval Lowy. But that was not all! After hours TypeMock guys arranged Open Spaces discussion, and prior to that that Roy Osherove and Carl Franklin unpacked their guitars and played a few songs. Roy sang his test-driven songs, and Carl appeared to be a great musician and The Beatles lover.

One of the songs that Carl did really well was We Can Work It Out. I think this song has a good potential to join the family of TDD songs. But of course it needs new lyrics :-)

We Can Mock It Out

Try to test it my way,
Do we have to keep it running 'til we can't go on?
While you test it your way,
Run the risk that patience of the boss may soon be gone.

We can mock it out,
We can mock it out.

Think of what you're doing.
You can leave the bugs and still you think that all tests passed.
Think of what I'm doing.
We can mock it out and get it right, or it won't last.

We can mock it out,
We can mock it out.

Life is very short, and there's not time
For reading from database.
I have always thought that it's a crime,
So I will tell you face to face.

Try to test it my way,
Total time will tell if I am right or I am wrong.
While you test it your way
Even on our best machine the tests run far too long.

We can mock it out,
We can mock it out.

True unit tests and cross-domain calls

Vagif Abilov — Wed, 18 Mar 2009 21:25:00 GMT

In our company we recently started marking automated tests as “Unit” and “Integration” for the purpose of running unit tests frequently (and on every check-in). Such separation obviously requires clear definition of what is a unit test. Michael Feathers in his excellent book “Working Effectively with Legacy Code” writes the following:

“Unit tests run fast. If they don't run fast, they aren't unit tests. A test is not unit test if:

It talks to a database
It communicates across network
It touches the file system
It depends on configuration environment”

This list is convincing, however since it defines qualities of non-unit tests, negating them won’t necessarily give us a definition of what unit test is. However, the main Feather’s message is clear: a unit test should only use volatile memory. No databases or network calls, not even reading from configuration files.

But what about use of multiple domains? What if code under test (CUT) creates new AppDomain instances? Should an automated test that executes such code be classified as unit or integration test?

I tend to think that such tests should be considered integration tests. Let’s recall the primary purpose of writing a test: exposing a bug. Now imagine we wrote an automated test for code that uses cross-domain calls and it failed. The reason for failure can have something to do with multiple domains, or failure was caused by an error that had nothing to do with cross-domain communication. If the reason for failure is domain-related, this is an integration issue. Since use of multiple domains is related to assembly probing and loading, any failures in this area are essentially integration failures. But if a test that involves cross-domain calls exposes an error that does not have anything to do with domain setup, such test can and should be re-written to expose the same error within a single domain. Moreover, classifying such test as a unit test may lead to a waste of developers’ precious time: when running tests marked as “unit”, they will also execute tests that consume massive amount of CPU cycles on completely unnecessary operations.

While performance implications of cross-domain calls is usually negligible for business applications, it is huge for CUT that is not supposed to touch non-volatile memory. And if developers want to build a habit of running unit tests often, the speed of unit test execution becomes crucial. If a team has 5000 tests that need 10 minutes to run, chances that they will be run regulary are pretty small. Fit them in one minute – and they will be run often. And running CUT in a different domain can result in more than 10 times slower execution.

So what should be done if an integration test exposes a failure? I believe once an error is identified and located, in case it is an error in application logic, developer should try to write a true unit test that would expose the same problem. Integration test can still be kept unless everything it validates is now fully covered by a new test, and the new test will enrich a set of fast (and therefore frequently) run unit tests.

This reasoning is not AppDomain-specific. In general, if it is possible to write a faster unit test, it should be re-written so it runs faster. To strengthen Michael Feathers’ point: a good unit test validates a given unit of functionallity in a shortest time. A test is not unit test if the same validation can be performed significantly faster.

TypeMock Isolator and matching faked method’s arguments – part 3

Vagif Abilov — Sat, 14 Mar 2009 22:18:26 GMT

Part 1

Part 2

I promised to address matching reference and output arguments using lambda-based syntax that I proposed for TypeMock Isolator. Before getting there I would like to retract my original comment that faked method’s arguments should be matched by default. I still think that, like strong typing, strong matching is a best practice to follow. However I should not forget the main goal of my proposal: increase code simplicity and compactnes. My first examples used very simple method calls consisting of one-two arguments, so an extra effort of typing “(int x, int y)” went unnoticed. Using real API methods immediately showed how inconvenient it can become to faked methods with large number of arguments if developer is forced to list argument types even when no argument match is required. Here how it might look:

Isolate.WhenCalled((string a, Evidence b, byte[] c, AssemblyHashAlgorithm d) => 
        Assembly.LoadFrom(a, b, c, d)).WillReturn(null);

Note where it gets really terrible: you have to type the whole argument list of LoadFrom: “string a, Evidence b, byte[] c, AssemblyHashAlgorithm d”. All this without help from Intellisense! The argument list comes before the method name.

Now compare it with current use of TypeMock API:

Isolate.WhenCalled(() => Assembly.LoadFrom(null, null, null, 0)).WillReturn(null);

What would you choose? With all respect to strong matching the choice is clear.

Eli suggested defining a different method for faking a method a method when arguments matter, and now I came to the same conclusion. So I will leave WhenCalled alone and use a different method name, let it be called WhenCalledArgs (I am not sure about using word “Exact” in such method name, because this method will support partial match, and in case you only match one argument out of five, the word “exact” can be confusing).

Now back to agenda: matching reference and output arguments. In my previous post for each size of argument list I defined two overloads (I will be using new name WhenCalledArgs):

public static IPublicNonVoidMethodHandler WhenCalledArgs(Func func);
public static IPublicNonVoidMethodHandler WhenCalledArgs(Func func, Func predicate);

The definitions above cover the case of a method with single argument. The first overload corresponds to a call with no argument match and no checker. The second overload can be used to define a custom checker for an argument. What is missing here is support for reference and output arguments, when the passed value is assigned to an argument marked as “ref” or “out” on method return. Fortunately, this is easy to achive using the same notation. It just requires new oveloads.

To assign reference and output value we need to extend WhenCalledArgs with a new parameter: an Action<T> delegate:

public static IPublicNonVoidMethodHandler WhenCalledArgs(Func func, Action output);

And to enable specifying both custom checker and output delegate, we will have an overload that combines them all:

public static IPublicNonVoidMethodHandler WhenCalledArgs(Func func, Func predicate, Action output);

Similar overloads we will need to provide for void methods (in such case WhenCalledArgs will be returning IVoidActionHandler instead of IPublicNonVoidMethodHandler). Now let’s look at the test code. For the sake of simplicity we assume to be dealing with a static class called MyClass, and the method that we are going to fake has the following signature:

public static void LookupCustomer(int customerId, out string firstName, out string lastName);

First we will fake its call but assign on return the values “John” and Smith” to the firstName and lastName respectively. Here how the code will look:

Isolate.WhenCalledArgs((int x, string y, string z) => 
        MyClass.LookupCustomer(x, out y, out z), (x, y, z) => { y = "John"; z = "Smith"; }).IgnoreCall();

Note that the customerId argument will not be matched. If we want to match it we slightly change the code:

Isolate.WhenCalledArgs((string x, string y) => 
        MyClass.LookupCustomer(123, out x, out y), (x, y) => { x = "John"; y = "Smith"; }).IgnoreCall();

What happens now is that the method LookupCustomer will only be faked for customerId equals 123, and upon return firstName and lastName arguments will be assigned values “John” and “Smith”.

Using “ref” instead of “out” does not really change much. It just gives an opportunity to assign (and match) a value to an argument and in addition specify another one on return.

And of course we can combine assignment of ret/out values with custom argument checker. Here’s an example:

Isolate.WhenCalledArgs((int x, string y, string z) => 
        MyClass.LookupCustomer(x, out y, out z), (x, y, z) => x > 100, 
        (x, y, z) => { y = "John"; z = "Smith"; }).IgnoreCall();

This is quite compact notation, however it does various things:

It fakes calls to MyClass.LookupCustomer without argument match
Even though arguments are not matched, only calls with customerId greater than 100 will be faked, as speficied in custom checker predicate
On return output arguments firstName and lastName will be assigned values “John” and “Smith” respectively.

Conclusion

In these blogs posts I tried to demonstrate how we can take advantage of C# 3.0 language features and provide compact notation for TypeMock Isolator AAA syntax. I am pretty sure there is someting uncovered, and I haven’t even thought about how this approach can be applied to VB (I am afraid it can’t, at least not until Visual Basic gets action-based lambda expression support). However I believe this notation can be successfully applied to Isolator AAA syntax, as it is based on the same principles and can further improve compactness and readability of the code.

TypeMock Isolator and matching faked method’s arguments – part 2

Vagif Abilov — Sat, 14 Mar 2009 10:49:25 GMT

I was glad to see that my thoughts about figuring out the best syntax to control the match of faked method’s arguments haven’t been unnoticed by TypeMockians. Eli Lopian raised a few questions, and addressing them seems to be crucial for materialization of the whole idea.

1. Backward compatibility with existing tests. If WhenCalled(() => some_method(a, b, c)) will be interpreted as a call with exact argument match, existing tests will fail. Parameterless delegates in such calls will have to be replaced with parameterized ones, e.g. WhenCalled((x, y, z) => some_method(x, y, z)).

I was quick enough to respond that I would be willing to update all tests just to get the syntax right, but I admit such response is childish. You can’t force your customers to use many hours on fixing their tests just because you discovered great syntax. Of course you need to provide a compatibility mode. There are different ways to achieve that. The simplest is to use a global flag:

Isolate.DefaultArgumentMatchBehavior = ArgumentBehavior.ExactMatch;

Exact match can be used as default for backward compatibility, at least during transition period. Or argument match mode can be implemented as an attribute that can later be deprecated. I don’t think this looks very elegant (global flags can’t be elegant), this is more like a temporary solution to let tests survive until the next major TypeMock release.

If this sounds too messy, then alternative approach might be to place functionality based on lambda syntax in a new namespace. Perhaps this is cleaner. I can compare it with approach taken by LINQ developers that divided LINQ classes between two namespaces and placed Expression and its derivations in System.Linq.Expressions. Later in this post I will show new extensions that solve custom checkers, and the more such extensions come, the more it is justified to define a dedicated namespace for lambda-based argument match control. What can work best, I don’t really know. Anyway, I believe resolving backward compatibility is more like configuration issue that should not compromise syntax selection.

What I find however important is to let a developer set up the tests so the arguments would be matched by default. Already now indexers are matched, this change broke many of our tests (I like defining custom indexers), but I think this was a right thing. When mock framework encounters ar[1], I believe it should respect the index value unless tests are specifically configured to ignore it. The same behavior I expect from faked methods: mock framework should not ignore that a method DoSomething(a, b, c) is expected to be faked with arguments (a, b, c). I do agree that this may not be turned on by default for backward compatibility purpose, but I would like to have a way to turn it on – at least to unifier indexers’ and methods’ behaviour.

2. Support for custom checkers. This is perhaps the most important requirement. If we find binary approach to argument matchings (match/nomatch) not sufficient because in some cases we expect partial argument match, we should also expect support for matching with custom checkers, for example when a string value begins with certain pattern, numerical value is within a certain range etc. I agree that without providing an easy way to define custom checkers the idea that I described in my previous post will stay a purely academical excersise. So I started playing with example code.

MyIsolate.WhenCalled((int x, int y) => MyClass.GetSum(x, y)).WillReturn(6);

So how we can extend this code to support custom argument checkers, for example to fake a method only if “x” is greater than 0? Something like this:

MyIsolate.WhenCalled((int x, int y) => MyClass.GetSum(x, y) where x > 0).WillReturn(6);

Of course this code will never compile, but I admit I imported System.Linq namespace and spent some time shuffling “where” keyword around. Not to resolve it using LINQ – just to get some inspiration. Making mock framework depedent on LINQ would be really mad (and would never work), not to say that both “where” clause is just a syntactic sugar around Where method that requires support for IEnumerable. And what IEnumerable can have with argument matching?

However turning inspecting implementation of “where” paid of, because the its core part is a predicate delegate Func<T, bool>. This is what we need to extend our WhenCalled with support for custom checkers.

Remember our new WhenCalled overloads listed in the previous post:

public static IPublicNonVoidMethodHandler WhenCalled(Func func)
public static IPublicNonVoidMethodHandler WhenCalled(Func func); 
public static IPublicNonVoidMethodHandler WhenCalled(Func func);
…

If for each parameterized Func delegate we provide an additional overload that takes a predicate delegate, then we enable custom checkers.

public static IPublicNonVoidMethodHandler WhenCalled(Func func);
public static IPublicNonVoidMethodHandler WhenCalled(Func func)
public static IPublicNonVoidMethodHandler WhenCalled(Func func, Func predicate)
public static IPublicNonVoidMethodHandler WhenCalled(Func func)
public static IPublicNonVoidMethodHandler WhenCalled(Func func, Func predicate)
…

Now look at that tests we can write now:

[Test, Isolated]
public void TestWithCustomCheckers()
{
    Isolate.WhenCalled((int x, int y) => MyClass.GetSum(x, y), (x, y) => x > 0).WillReturn(6);
    Assert.AreEqual(6, MyClass.GetSum(1, 2));
    Assert.AreEqual(1, MyClass.GetSum(-1, 2));

    Isolate.WhenCalled((int x, int y) => MyClass.GetSum(x, y), (x, y) => x > 10 && y > 10).WillReturn(6);
    Assert.AreEqual(6, MyClass.GetSum(20, 20));
    Assert.AreEqual(10, MyClass.GetSum(5, 5));
 
    Isolate.WhenCalled((int y) => MyClass.GetSum(1, y), y => y == 3).WillReturn(7);
    Assert.AreEqual(7, MyClass.GetSum(1, 3));
    Assert.AreEqual(3, MyClass.GetSum(1, 2));
 
    Isolate.WhenCalled((int x) => MyClass.Elements[x], x => x == 10).WillReturn(9);
    Assert.AreEqual(9, MyClass.Elements[10]);
 
    Isolate.WhenCalled((string x, string y) => MyClass.Concatenate(x, y), (x, y) => x.StartsWith("abc") && y.EndsWith("xyz")).WillReturn("bingo!");
    Assert.AreEqual("bingo!", MyClass.Concatenate("abc", "xyz"));
    Assert.AreEqual("ab", MyClass.Concatenate("a", "b"));
}

What do we gain with this syntax?

1. I think it is more readable. In fact, I think it is much more readable. Compare the line 19 with something like Isolate.WhenCalled(MyClass.Concatenate(null, null)).AndArgumentsMatch(1, Arg.StartsWith("abc")).AndArgumentsMatch(2, Arg.EndsWith("xyz")).WillReturn("bingo!").

2. I believe when we extend functionality, we should prefer using built-in language constructions over new methods definition. The advanage of Func<T, bool> delegate is that it is based on generics and if we use it to define predicates then the predicates will naturally support methods and properties of respective types. You can simply write “x == 10” or “x.StartsWith(“abc”)”.

3. You can take advantage of boolean logic using built-in language operators. Writing “x > 0 && y > 0” is equally simple as writing “x > 0 || y > 0”, but how would you specify the latter checker using method-based notation? AndArgumentsMatch(1, Arg.GreaterThan(0)).OrArgumentsMatch(2, Arg.GreaterThan(0)?

There is one unanswered question from Eli: handing “ref” and “out” arguments. I need to think how this can be resolved.

How I would match faked method's arguments if I was TypeMock Isolator

Vagif Abilov — Thu, 12 Mar 2009 20:43:00 GMT

I enjoy compactness and clarity of TypeMock Isolator AAA syntax. But it still lacks certain features in comparison with the original RecordExpectation-based syntax. One of such features is support for exact match of method's attributes. Here's an example (I assume that MyClass is a static class):

Isolate.WhenCalled(() => MyClass.GetSum(1, 2)).WillReturn(10);
Isolate.WhenCalled(() => MyClass.GetSum(3, 4)).WillReturn(20);

At the time of writing (TypeMock Isolator version 5.2.2) this will not trigger return value of 10 on all calls to GetSum with arguments 1 and 2. Argument values are currently ignored. However, recently Isolator was enhanced with exact argument match for indexers:

Isolate.WhenCalled(() => MyClass.Elements[1]).WillReturn(10);
Isolate.WhenCalled(() => MyClass.Elements[2]).WillReturn(20);

In the case above any subsequent call to MyClass.Elements with an index value of 2 will return 20.

Needless to say that support for exact argument match is one of the most requested features for AAA syntax. And the wait is soon over - there will be new methods that will make it possible to require exact argument matching. But is it really necessary to define new methods for this purpose? And what about partial argument match? I think there is another way to go.

Before I move on to my proposal, let me express first what does not make me fully satisfied with current TypeMock approach.

1. Indexers' and methods' arguments should not be handled differently.

I understand what was the motivation for exact match of arguments on indexers. Indexers are mostly used with arrays, so when you set different expectations on ar[1] and ar[2], you will most likely want Isolator to map indexes to corresponding values and always return values depending on the passed index. The problem however is that C# makes it very easy to implement indexers as alternatives for getters and setters. So calls MyClass.GetValue(1) and MyClass.Values[1] can mean the same, but when you set expectations on GetValue and Values, the first one will ignore the argument value, while the second one will memorize it.

2. All or nothing is not enough.

Alright, we are soon getting support for exact match of arguments. All arguments. But will it cover all needs? If you can come up with scenario when you need to fake a method with exact argument match, and you can come up with scenario when you need to fake a method with no argument match, then why can't you have a scenario when you need to match just certain method arguments? I can clearly see some typical scenarios: for example, you may fake a method that receives as part of its input current time, generated GUIDs etc. We can of course ask if this is a good practice to make tests dependent on non-deterministic data, but don't forget that Isolator can be successfully used not just for pure unit tests. Smoke tests and even integration tests can also be simplified by isolating some aspects, and once there is support for both exact match and no match of method arguments, there is nothing wrong with requesting a partial match.

But how can it be provided? Extend ExactMatch or whatever the new method will be called with an overload that would take an array of argument ordinals? Cumbersome and hard to read and maintain.

Let's use lambda expressions at its full strength!

One thing that makes TypeMock Isolator AAA syntax charming (although if I am not mistaken it was Moq that introduced it first) is its use of lambda expressions:

Isolate(() => SomeCall(...)).WillReturn(...);

First time I saw this syntax, I spent some time trying to understand what I can use instead of "()" to the left from "=>". Maybe depending on the faked method's signature I could sometimes write (x)? Or even (x,y)? Then I realized that this was not an option: static class Isolate supports two overloads of WhenCalled:

public static IVoidActionHandler WhenCalled(Action action); 
public static IPublicNonVoidMethodHandler<TResult> WhenCalled<TResult>(Func<TResult> func)

So no matter whether you fake a void method or a method that returns some value, in both cases you pass a parameterless delegate. You can't have unbound variables: all method's arguments must be resolved.

But what if WhenCalled supported overload for each form of Action and Func? Like this:

public static IPublicNonVoidMethodHandler WhenCalled(Func func) {...}
public static IPublicNonVoidMethodHandler WhenCalled(Func func)  {...}
public static IPublicNonVoidMethodHandler WhenCalled(Func func) {...} 
public static IPublicNonVoidMethodHandler WhenCalled(Func func) {...} 
public static IPublicNonVoidMethodHandler WhenCalled(Func func) {...}

This would open quite interesting way of setting expectations with exact match of only selected arguments! Because then we could write Isolate.WhenCalled(() => ..., Isolate.WhenCalled((x) => ..., Isolate.WhenCalled((x, y) => .... And use these variables in a method call to indicate the unspecified arguments that will not be matched exactly.

Let me illustrate this with an example. Let's say we have a static class MyClass with a method GetSum(int x, int y). In TypeMock Isolator the following code will work fine:

Isolate.WhenCalled(() => MyClass.GetSum(1, 2)).WillReturn(4);
Assert.AreEqual(4, MyClass.GetSum(1, 2));
Assert.AreEqual(4, MyClass.GetSum(10, 20));

The code above uses currently supported syntax, except that I believe it's semantics should be identical to the one used by indexers: method arguments should be matched exactly.

And here how we can disregard argument values:

Isolate.WhenCalled((int x, int y) => MyClass.GetSum(x, y)).WillReturn(6);
Assert.AreEqual(6, MyClass.GetSum(1, 2));
Assert.AreEqual(6, MyClass.GetSum(2, 1));

In the code above it does not matter what arguments are passed to GetSum: we used a new overload of WhenCalled (unfortunately only supported in this blog entry) that let us speficy what arguments should be treated as unbound.

And what about partial argument match? Now it's easy:

Isolate.WhenCalled((int y) => MyClass.GetSum(1, y)).WillReturn(7);
Assert.AreEqual(7, MyClass.GetSum(1, 2));
Assert.AreEqual(4, MyClass.GetSum(2, 2));

Here Isolator is instructed to set expectations only on calls with first argument equal to 1. So GetSum(2,2) will not be faked.

Finally setting expectations on an indexer:

Isolate.WhenCalled(() => MyClass.Elements[0]).WillReturn(8);
Assert.AreEqual(8, MyClass.Elements[0]);
Assert.AreNotEqual(8, MyClass.Elements[123]);
Isolate.WhenCalled((int x) => MyClass.Elements[x]).WillReturn(9);
Assert.AreEqual(9, MyClass.Elements[123]);

What do we gain by this?

1. Consistent semantics for argument match on indexers and methods.

2. Support for partial argument match.

3. No need to introduce new method names - just overloads to WhenCalled.

4. And isn't such use of labmda expressions cool?

To demonstrate the proposed syntax, I wrote a simple example with unit tests that include the code I showed above. Of course I could not use real TypeMock Isolator in these tests: it does not support (yet!) proposed WhenCalled overloads. So I cheated a little and implemented a toy MyIsolate class with just enough overloaded methods to make these example work. The test project can be downloaded from this post. You can compile and run the tests - they will pass as if they were real. I wish they were.

Part 2

Part 3