Vagif Abilov's blog on .NET

Expose MongoDB data as OData feed using MongOData provider

With growing popularity of MongoDB grows the variety of tools and technologies that expose data stored in MongoDB databases. Since I am using both MongoDB and OData in my projects, I decided to cross them and enable exposure MongoDB collections as OData feeds. So I started a project MongOData hosted at GitHub. Although I was using MongOData internally for some time for data retrieval, I was reluctant to announce it until I implement support for update. Update is now supported (version 0.4.0), so I feel it’s time to write a blog post presenting how MongOData works and how it was built.

MongOData highlights

• MongOData is a MongoDB OData provider. It exposes each MongoDB collection as an OData resource set. Current release assumes that all data inside the collection have the same structure, so it builds metadata information based on the first row in each collection. I plan to support different metadata build strategies in future releases.
• MongOData does not use OData open types, although open types seem to be a good fit for schemaless databases. Most of OData client tools don’t have support for open types and rely on types exposed via metadata, so I took an approach that is compatible with traditional OData consumers.
• There are two ways to install MongOData: using MSI installer and using NuGet package. MSI installer is an easiest option if you only need to publish OData feed for one or all MongoDB databases. NuGet MongOData package should be used to add MongoDB WCF Data service to an existing Web project.
• MongOData uses modified code from OData Provider Toolkit that helps building WCF Data Services custom providers. OData Provider Toolkit uses LINQ to Objects to execute queries, so it only supports in-memory queries and therefore requires all data to be read in memory upfront. Such implementation is of course is not suitable for large databases, so I revised the original code to support two strategies: in-memory and queryable providers. MongOData uses queryable provider and LINQ provider from the official MongoDB C# driver.

Installing MongOData using MSI installer

This is the easiest option to start using MongOData with MongoDB. Simply download the installer and run it on a machine with MongoDB and IIS installed. During the installation you can configure Web server virtual directory and MongoDB connection string:

Note that default connection string contains the wild card ‘*’. This syntax is not supported by MongoDB, it indicates that MongOData should expose all MongoDB databases available on the server. The specific database can then be selected by specifying its name in an URL, e.g.:

http://localhost/MongOData/Northwind/$metadata

http://localhost/MongOData/zips/zips

In the examples above we read data from “Northwind” and “zips” database.

In case you don’t want to open OData access to all configured MongoDB databases, you replace the wild card in the installation screen with the name of the specific database, e.g.: mongodb://localhost/Northwind?strict=true. Then only this specific database will be exposed as OData feed.

Installing MongOData using NuGet package

If you want to embed MongoDB OData feed into your Web service or application, you can use MongOData NuGet package. NuGet package adds custom WCF Data Service to your Web component, so you only need to specify the MongoDB connection string (MongOData NuGet package does not support wild cards, you will have to refer to a specific database). Here is what you need to do.

1. Create a Web project using one of Visual Studio templates, for example, “ASP.NET MVC3 Web Application” (almost any generic ASP.NET Web application template will do).
   
2. Open Package Manager Console window and write the following command:

   Install-Package MongOData

   NuGet package manager will install the latest MongOData component along with its dependency mongocsharpdriver, so you will see the package installation report similar to this:

   Attempting to resolve dependency 'mongocsharpdriver (≥ 1.4.1)'.
   Successfully installed 'mongocsharpdriver 1.4.2'.
   Successfully installed 'MongOData 0.4.0'.
   Successfully added 'mongocsharpdriver 1.4.2' to MongODataTest.
   Successfully added 'MongOData 0.4.0' to MongODataTest.

   If you check the project content, you will see that there are references to MongoDB driver and MongOData assembly files, and two new source files MongoDataService.cs and MongoDataService.svc. Here is the content of the newly added C# source file:

   public class MongoDataService : MongoQueryableDataService
   {
       public MongoDataService()
           : base(ConfigurationManager.ConnectionStrings["MongoDB"].ConnectionString)
       {
       }
   
       // This method is called only once to initialize service-wide policies.
       public static void InitializeService(DataServiceConfiguration config)
       {
           // TODO: set rules to indicate which entity sets and service operations are visible, updatable, etc.
           config.SetEntitySetAccessRule("*", EntitySetRights.AllRead);
           config.SetServiceOperationAccessRule("*", ServiceOperationRights.All);
           config.DataServiceBehavior.MaxProtocolVersion = DataServiceProtocolVersion.V2;
           config.DataServiceBehavior.AcceptCountRequests = true;
           config.DataServiceBehavior.AcceptProjectionRequests = true;
           config.UseVerboseErrors = true;
       }
   }

   Note that MongoDataService refers to a connection string named as "MongoDB". If you are comfortable with this, you don't need to do any modification to the source files, just update the connection string in web.config file so it points to the actual MongoDB database that you are going to expose as OData feed.
   
3. Build the project and go the MongoDataService.svc page, you will see something like this:
   
   

You're done! Now you can query your MongoDB OData feed using standard OData URI convention.

Expanding results with associated entries using Simple.Data OData adapter

In the post where I introduced Simple.Data OData adapter I showed a few examples of retrieving single or multiple entities. What was common with those examples is that they presented technique to read just the requested resource type, even though there in can be useful to read associated entities at once. Soon after I wrote that blog post, Mark Rendle extended Simple.Data with support to retrieve related data using “With” clause. This was exactly what Simple.Data OData adapter lacked in order to implement its own support to expand results with associated entities. Now I am going to show how it works.

We will continue using Northwind example, since every .NET developer must have learned by heart all its entity types. First, we will retrieve a single category by its name and expand it with associated products. This is how the code will look (I will also show the test assertions to make it clear how to deal with request results):

var category = _db.Category.WithProducts().FindByCategoryName("Beverages");
Assert.Equal("Beverages", category.CategoryName);
Assert.True(category.Products.Count > 0);

And here is the generated request URI:

Categories?expand=Products&$filter=CategoryName+eq+'Beverages'

Now let’s retrieve all categories and expand its products. Here is the code

var categories = _db.Category.All().WithProducts().ToList();
Assert.True(categories.Count > 0);
Assert.True(categories[0].Products.Count > 0);

The request URI will be as follows:

Categories?expand=Products

If we swap the master resource type and start retrieving products with associated categories, the relationship type will change from one-to-many to many-to-one, as you see from this example:

var products = _db.Products.All().WithCategory().ToList();
Assert.True(products.Count > 0);
Assert.Equal("Beverages", products[0].Category.CategoryName);

The request URI for the example above:

Products?expand=Category

Finally, we will look at self-joins. In Northwind database an employee may have a superior who is also an employee, so for each Employee entity there is a corresponding Employee entity associated via “Superior” relationship (can be null), and a set of Employee entities associated via “Subordinates” relationship (can be empty). Here’s how to retrieve a given employee with its superior:

var employee = _db.Employees.WithSuperior().FindByFirstNameAndLastName("Nancy", "Davolio");
Assert.NotNull(employee);
Assert.NotNull(employee.Superior);

And here’s the request URI:

Employees?$expand=Superior&$filter=FirstName+eq+%27Nancy%27 and LastName+eq+%27Davolio%27

And in case we need to expand an employee with its subordinates, the following code will do the trick:

var employees = _db.Employees.All().WithSubordinates().ToList();
Assert.True(employees.Count > 0);
Assert.True(employees[0].Subordinates.Count > 0);

The corresponding request URI:

Employees?$expand=Superior&$filter=FirstName+eq+%27Nancy%27 and LastName+eq+%27Davolio%27

Simple.Data OData adapter source code is available at GitHub, and the binaries can be obtained from NuGet using package ID “Simple.Data.OData”.

Introducing Solution Maker 2.0: new features and support for Visual Studio 11

A couple of years ago I wrote a utility that could scan directories and generate solution files (I blogged about it here and here). I called it Solution Maker and later made its source available at BitBucket. I received feedback from users that helped me come with new features. And since developers are already using Visual Studio 11 preview, it was good time to add support for it. So this post is about new version of Solution Maker that can be downloaded from Visual Studio Gallery (and of course from BitBucket).

To show what can be achieved using Solution Maker, I’ve chosen a large project that builds assemblies familiar to every .NET developer: ASP.NET. Microsoft recently turned it into an open-source project and uploaded to CodePlex. Let’s point Solution Maker to its root folder and see what we can do.

When you start Solution Maker, it comes with a screen showing setting for a solution to be generated. We want to create a solution for ASP.NET Web Stack, so we will choose the name for a new solution (“AspNetWebStack”) and browse for a project root folder:

If we just want a flat project structure (and for small solutions this may sufficient), we don’t need to do anything else and can just click on “Create Solution” button. However, solutions tend to grow, and this where solution folders help splitting projects into logical groups. Solution Maker can automate this process by analyzing project and output assembly names. Let’s start however with a flat structure just to show what we will get in case we don’t want to create solution folders. Click on “Preview” button and Solution Maker will display a new window with a solution preview.

What we see here is 44 projects without any grouping. Not a very convenient structure. Let’s try to improve it a bit: set “Solution folder levels” option to 1, so Solution Maker will parse project names and create solution folders based on the leftmost word in a name of a project.

Looks better, doesn’t it? Note that ASP.NET Web Stack projects are named after the output of their respective assemblies, so you can leave a default naming strategy “Create solution folders based on project file names”. If this choice results in non-informative folder names (for example, if a project for assembly “System.Web.Http” was called just “Http”, then you wouldn’t have a chance to place it in a solution folder “System”), you may try an option “Create solution folders based on project output assembly names”. You can even choose the best of breeds: an option “Create solution folders based on most qualified names” is useful when projects don’t have single naming strategy. Then Solution Maker will analyze their names and assemblies and choose the one that gives the most informative name.

Now we have 44 projects divided into 4 solution folders, but folder “System” contains 27 projects which may be still too many. Let’s increase the number of solution folder levels. Here is what we will get:

What if you only want a subset of projects? This can be achieved using filters. For example, if we want a solution without test projects, we can specify a filter string “*.Test;*.Test.*” in “Project exclude filter” edit box. This will instruct Solution Maker to exclude 25 test projects:

So far so good. But until we assumed that the project root contains all project dependencies, and this is not necessarily the case. It can be easily demonstrated by moving the project root folder to one of subdirectories, for example, to “Src\Microsoft.Web.Mvc”. Imagine that we want to create a solution only for MVC stuff. If we now generate a solution file, it will contain just one project:

This is rather silly. Obviously, Microsoft.Web.Mvc has some dependencies, and we want them in the solution. This is the time to enable “Include referenced projects” option:

Finally, if you prefer command line or need to integrate solution generation into a build workflow, all Solution Maker options can be specified at the command line:

Solution Maker is a free open-source projects, so it’s easy to extend it or adjust for your specific needs.

Introducing Simple.Data OData provider

Mark Rendle’s Simple.Data has quickly become a popular choice to retrieve data from various data sources. The number of Simple.Data providers includes today SQL Server, Oracle, Mysql, MongoDB and others. When I saw Simple.Data in action, I became enthusiastic about extending its list of providers with OData. I believe dynamic nature of Simple.Data matches very well RESTful nature of OData. Currently Visual Studio offers creation of a client context from an OData feed using “Add Service Reference” dialog which treats an OData service like it was a traditional SOAP service. This is unfortunate because OData communication does not need an interface proxy: communication is based on HTTP methods and can be implemented without context creation ceremony.

Welcome to Simple.Data OData provider! When I asked Mark Rendle on Twitter if he had thought about writing a Simple.Data OData provider, he replied that he was working on Windows Azure table provider that also uses OData protocol, and he offered me to join the project and focus on generic OData client implementation. This gave me an exciting opportunity, and today I am glad to announce that Simple.Data OData provider preview is available, and it’s Nuget feed is published.

Simple.Data OData provider is hosted at GitHub. The source code based is shared by OData and Azure provider implementation, but the Nuget packages are different.

Getting started with Simple.Data OData provider

The easiest way to start using Simple.Data OData provider is to install it’s Nuget package. In Visual Studio open Package Manager console and type the following:

Install-Package Simple.Data.OData

You will see output similar to this:

Attempting to resolve dependency 'Simple.Data.Core (≥ 0.12.2.2)'.
Successfully installed 'Simple.Data.OData 0.1.1'.
Successfully added 'Simple.Data.OData 0.1.1' to SimpleODataTest.

In the source file where you will be using OData provider import namespaces:

using Simple.Data; 
using Simple.Data.OData;

Create an instance of dynamic database object that will be used to communication with OData feed:

dynamic db = Database.Opener.Open("http://packages.nuget.org/v1/FeedService.svc/");

Now you can start retrieving data from the OData server using Simple.Data syntax:

var package = db.Packages.FindByTitle("Simple.Data.OData"); 
Console.WriteLine(package.Title);

Reading data using Simple.Data OData provider

package = db.Packages.FindByTitleAndVersion("Simple.Data.OData", "0.1.1.2");
Console.WriteLine(package.Title);

Find multiple results:

IEnumerable<dynamic> packages = db.Packages.FindAllByTitle("Simple.Data.OData");
packages.ToList().ForEach(x => Console.WriteLine(x.Title + " " + x.Version));

packages = db.Packages.FindAll(db.Packages.Version == "1.0.0.0");
packages.ToList().ForEach(x => Console.WriteLine(x.Title + " " + x.Version));

Select column subset:

IEnumerable<dynamic> packages = db.Packages
    .FindAllByTitle("Simple.Data.OData")
    .Select(db.Packages.Version);

Limit number of results using Skip and Take:

IEnumerable<dynamic> packages = packages = db.Packages
    .FindAllByTitle("Simple.Data.OData")
    .Take(1); 

packages = db.Packages
    .FindAllByTitle("Simple.Data.OData")
    .Skip(1)
    .Take(1);

Retrieve results in specific order:

IEnumerable<dynamic> packages = db.Packages
    .FindAllByTitle("Simple.Data.OData")
    .OrderByDescending(db.Packages.Version);

What’s next

OData provider already supports Insert,Update and Delete operations (more about it in a next blog post), but it does not support associations. The implementation of the new functionality requires some changes to Simple.Data.Core library that is on its way.

How to configure AutoTest.Net to use Snarl notifications

Important part of continuous testing is instant and clear notification on test results. Developer should not need to inspect small text in Visual Studio output window, test execution results should flash like traffic lights, especially when things go wrong.

There are several commonly used notification applications for Windows, and AutoTest.Net is integrated with two of them: Growl and Snarl. They provide good visualization of test execution results. Here is how Snarl notification looks:

While integration with Growl works without any adjustments, Snarl needs to be configured to work with AutoTest.Net. Here is how you do it:

1. Open Snarl preference dialog

2. Go to “Network” tab and select “Yes” for “Listen for incoming Growl or Snarl notifications?

3. Start Visual Studio (AutoTest.Net or ContinuousTests a.k.a. MightyMoose must be installed)

4. Find and highlight “AutoTest.Net on 127.0.0.1”

5. Highlight in “Notification Classes” “Run succeeded”, then press “Configure”

6. Select “Yes” for “use custom icon?”

7. Go to AutoTest.Net or ContinousTests installation folder and find in Icons subfolder circleWin.png. You should see the icon with green light:

8. Repeat the previous step for classes “Run succeeded with warnings” and “Run failed” and choose for them icons with yellow and red light.

Now Snarl is configured to receive notifications from AutoTest.Net.

LINQ-based instance and set comparison in SpecFlow

A few weeks ago I blogged about a proposed feature: enhance SpecFlow Assist comparison helpers with support for different comparison modes. The enchancment was based on introduction of an enumerator that would specify how a table should be compared with an instance or set: TheTableAndSetMatchWithoutRegardForOrder, TheTableAndSetAreAnExactMatch etc. In a discussion in SpecFlow Google group there came another suggestion: use LINQ instead of enumerator. It would open for use of any set transformation operation supported by LINQ.

Now that the feature implementation has been accepted and will become a part of next SpecFlow release, I would like first to show how it works and then explain how it is implemented.

How it works

So let’s look at the following scenario (you will never want to mix so many validations in a single scenario but I just want to take all at once):

CompareToSet Table extension method only checks for equivalence of collections which is a reasonable default. But with LINQ-based operations each of the above comparisons can be expressed using a single line of code:

[When(@"I have a collection")]
public void WhenIHaveACollection(Table table)
{
    var collection = table.CreateSet<Item>();
    ScenarioContext.Current.Add("Collection", collection);
}

[Then(@"it should match")]
public void ThenItShouldMatch(Table table)
{
    var collection = ScenarioContext.Current["Collection"] as IEnumerable<Item>;
    Assert.IsTrue(table.RowCount == collection.Count() && table.ToProjection<Item>().Except(collection.ToProjection()).Count() == 0);
}

[Then(@"it should exactly match")]
public void ThenItShouldExactlyMatch(Table table)
{
    var collection = ScenarioContext.Current["Collection"] as IEnumerable<Item>;
    Assert.IsTrue(table.ToProjection<Item>().SequenceEqual(collection.ToProjection()));
}

[Then(@"it should not match")]
public void ThenItShouldNotMatch(Table table)
{
    var collection = ScenarioContext.Current["Collection"] as IEnumerable<Item>;
    Assert.IsFalse(table.RowCount == collection.Count() && table.ToProjection<Item>().Except(collection.ToProjection()).Count() == 0);
}

[Then(@"it should not exactly match")]
public void ThenItShouldNotExactlyMatch(Table table)
{
    var collection = ScenarioContext.Current["Collection"] as IEnumerable<Item>;
    Assert.IsFalse(table.ToProjection<Item>().SequenceEqual(collection.ToProjection()));
}

In a similar way we can implement containment validation:

And here are corresponding step definitions:

[Then(@"it should contain all items")]
public void ThenItShouldContainAllItems(Table table)
{
    var collection = ScenarioContext.Current["Collection"] as IEnumerable<Item>;
    Assert.IsTrue(table.ToProjection<Item>().Except(collection.ToProjection()).Count() == 0);
}

[Then(@"it should not contain all items")]
public void ThenItShouldNotContainAllItems(Table table)
{
    var collection = ScenarioContext.Current["Collection"] as IEnumerable<Item>;
    Assert.IsFalse(table.ToProjection<Item>().Except(collection.ToProjection()).Count() == 0);
}

[Then(@"it should not contain any of items")]
public void ThenItShouldNotContainAnyOfItems(Table table)
{
    var collection = ScenarioContext.Current["Collection"] as IEnumerable<Item>;
    Assert.IsTrue(table.ToProjection<Item>().Except(collection.ToProjection()).Count() == table.RowCount);
}

ToProjection, ToProjectionOfSet and ToProjectionOfInstance

It’s so easy to compare tables to collections using CompareToSet<T>, why do we need to call ToProjection both for table and collection? The answer becomes clear once you take a look at Table class definition. A Table class does not implement IEnumerable, it is a composite of a header and collection of rows. So it’s impossible to call Except or SequenceEqual directly on an instance of a Table. But it’s not only support for IEnumerable that requires introduction of an adapter class: a collection to be compared to does not need to be a set of items of a type known at compile time. It can be generated as a result of execution of a LINQ statement, so collection elements will have anonymous type.

What if Artist and Album are properties of different entities? Look at this piece of code:

    var collection = from x in ctx.Artists 
                     where x.Name == "Muse"
                     join y in ctx.Albums on
                     x.ArtistId equals y.ArtistId
                     select new
                     {
                         Artist = x.Name,
                         Album = y.Name
                     };

A “collection” object represents a projection of a join of two tables, so if we want to compare a Table instance to this collection, we should be able to compare Table to a set of anonymous classes. This makes it tricky to implement an adapter class: we define a generic class of “T”, but T in the example above is an anonymous type.

So here is how it works. SpecFlow.Assist has a new generic class EnumerableProjection<T>. If a type “T” is known at compile time, “ToProjection” method converts a table or a collection straight to an instance of EnumerableProjection:

    table.ToProjection<Item>();

But if we need to compare a table with the collection of anonymous types from the example above, we need to express this type in some way so ToProjection will be able to build an instance of specialized EnumerableProjection. This is done by sending a collection as an argument to ToProjection. And to support both sets and instances and avoid naming ambiguity, corresponding methods are called ToProjectionOfSet and ToProjectionOfInstance:

    table.ToProjectionOfSet(collection);
    table.ToProjectionOfInstance(instance);

Here are the definitions of SpecFlow Table extensions methods that convert tables and collections of IEnumerables to EnumerableProjection:

    public static IEnumerable<Projection<T>> ToProjection<T>(this IEnumerable<T> collection, Table table = null)
    {
        return new EnumerableProjection<T>(table, collection);
    }

    public static IEnumerable<Projection<T>> ToProjection<T>(this Table table)
    {
        return new EnumerableProjection<T>(table);
    }

    public static IEnumerable<Projection<T>> ToProjectionOfSet<T>(this Table table, IEnumerable<T> collection)
    {
        return new EnumerableProjection<T>(table);
    }

    public static IEnumerable<Projection<T>> ToProjectionOfInstance<T>(this Table table, T instance)
    {
        return new EnumerableProjection<T>(table);
    }

Note that last arguments of ToProjectionOfSet and ToProjectionOfInstance methods are not used in method implementation! Their only purpose is to bring information about “T”, so the EnumerableProjection adapter class can be built properly. Now we can perform the following comparisons with anomymous types collections and instances:

[Test]
public void Table_with_subset_of_columns_with_matching_values_should_match_collection()
{
    var table = CreateTableWithSubsetOfColumns();
    table.AddRow(1.ToString(), "a");
    table.AddRow(2.ToString(), "b");

    var query = from x in testCollection
                select new { x.GuidProperty, x.IntProperty, x.StringProperty };

    Assert.AreEqual(0, table.ToProjectionOfSet(query).Except(query.ToProjection()).Count());
}

[Test]
public void Table_with_subset_of_columns_should_be_equal_to_matching_instance()
{
    var table = CreateTableWithSubsetOfColumns();
    table.AddRow(1.ToString(), "a");

    var instance = new { IntProperty = testInstance.IntProperty, StringProperty = testInstance.StringProperty };

    Assert.AreEqual(table.ToProjectionOfInstance(instance), instance);
}

Powerful helpers, simple step definitions

SpecFlow is becoming preferred BDD tool for many .NET developers, it has been easy to use from the beginning, and recent enhancements like Intellisense suport for Gherkin makes it really addictive. SpecFlow.Assist table management helpers add to the efficiency of the product by simplifying collection instantiation and comparison. I believe adding LINQ support to Table extension methods will help developers write more compact scenario step definitions.

Testing 64-bit Oracle using Visual Studio

I have installed 64-bit Oracle 11g on my development machine to work on some samples using Entity Framework 4.1 (a.k.a. “code first”). If you ask me why I installed 64-bit version and not 32-bit, I will not answer. I don’t know. Perhaps because Oracle has always associated for me with something big and 64 bits are certainly greater than 32. Now I think it was a mistake, but on the other hand it made me explore something new.

An attempt to connect to the database failed with the following error message: “ ----> System.Exception : Unable to load D:\Oracle\Vagif\product\11.2.0\dbhome_1\bin\oci.dll. Please check that you use 32x version of Oracle client with 32x application.” I checked project platform target. “Any CPU”. I was using TestDriven.NET test runner.

I’ve changed build platform to “64x” and tests passed. I also ran tests built for “Any CPU” using 64-bit NUnit runner, and they passed too. What I wanted now is leave “Any CPU” option for all projects and be able to test them from within Visual Studio.

Resharper was the only once that worked like a charm, no matter what I did. It uses 64-bit task runner on 64-bit platforms, so it just works.

Although TestDriven.NET test runner failed initially, it was easy to re-configure it: just choose 64-bit option for the test runner in its settings page.

The only bad guy was built-in Visual Studio test runner: it runs only as 32-bit process, so without unsupported patching it as 64-bit process (described here) it can’t run tests in 64-bit mode.

UPDATE. I stand corrected. MsTest also supports 64-bit mode.

ConstructorInfo.Invoke over Activator.CreateInstance

About a year ago I blogged about different methods of object instantiation and that compiled lambda expressions should be preferred in case same type is instantiated frequently and instantiation performance is critical. However, on many occasions dealing with expression compilation is not worth it, and creating an instance using reflection is simple and usually good enough. Then Activator.CreateInstance seems to be a default choice for such operations. What can be simpler than this call?

var instance = Activator.CreateInstance<SomeType>();

The problem with this approach is its false sense of simplicity: it gives an impression that the instance will be created without any additional information, for example initialization parameters.

To prove how misleading such simplicity can be you just need to try creating an instance of a string:

var instance = Activator.CreateInstance<string>();

The line above causes MissingMethodException with message “No parameterless constructor defined for this object”. And of course, System.String does not have a default constructor.

So what to do? We can of course say that if a class does not have default constructor, then instantiating it in such way is a bad idea, and those classes are just not intended to be used with code trying to create class instances in such manner. However, I bet that more often than not the semantics behind the code “Activator.CreateInstance<T>” is “give me some instance of T, I don’t care about how you create it”. Under such conditions invoking an arbitrary class constructor is preferred over dealing with MissingMethodException. So why not using ConstructorInfo.Invoke then?

Of course, once you settle for ConstructorInfo, you have to decide what constructor overload to use. But this is not really different from choosing parameterless overload of Activator.CreateInstance. In many cases you don’t choose default constructor because this the state of the object you need – you choose it because you don’t care about the state and thus would like to run the simplest form of creation. Applying the same logic to classes without default constructor we can choose a constructor overload with fewest number of parameters:

var constructor = typeof(T).GetConstructors().OrderBy(c => c.GetParameters().Length).FirstOrDefault();
if (constructor == null)
    throw new MissingMethodException("No public constructor defined for this object");
var instance = constructor.Invoke(new object[constructor.GetParameters().Length]);

One area when this approach can be useful is if there is a need to create an instance of an anonymous type (I know it’s a very special need – why on earth will you want to create an instance of an anonymous type? – but I’ve come across a situation):

var o = new { A = "a", B = 1 };
var constructor = o.GetType().GetConstructors()[0];
var instance = constructor.Invoke(new object[constructor.GetParameters().Length]);

Anonymous types don’t get a parameterless constructor, but there is a constructor with a number of arguments equal to the number of type’s properties. I would however use syntax GetConstructors().OrderBy(c => c.GetParameters().Length).FirstOrDefault() that can be applied in general case.

Enhancing SpecFlow set comparison methods

SpecFlow has an excellent set of Table extension methods contributed by Darren Cauthon (he also made a video available on TekPub). Methods such as CreateInstance, CreateSet or CompareSet enable conversion of SpecFlow tables to strong-typed item collections and comparing tables to IEnumerable generic types. Here is an example of such comparison.

First we create a feature scenario:

And here are the step definitions:

[When(@"I have a collection")]
public void WhenIHaveACollection(Table table)
{
    var collection = table.CreateSet();
    ScenarioContext.Current.Add("Collection", collection);
}

[Then(@"it should match")]
public void ThenItShouldMatch(Table table)
{
    var collection = ScenarioContext.Current["Collection"] as IEnumerable;
    table.CompareToSet(collection);
}

Use of CompareToSet is a great time-saver, but it current implementation (1.6) covers only collection equivalence: two collections are considered to be equivalent if they have the same items without regard for order. But there may be scenarios when collections need to be compared differently. If you check MsTest helper class CollectionAssert, you will find methods like IsSubsetOf, Contains, DoesNotContain etc. Comparison criteria can be divided in two groups, with four modes in each. Positive comparison group include the following criteria:

• Equivalence (same items in both collections, order is insignificant, corresponds to CompareToSet implementation in SpecFlow 1.6 and earlier versions);
• Equality (same items in the same order);
• Subset (one collection is a subset of another collection);
• Intersection (collections have common items).

Negative comparison group consists of the same comparison criteria, but in negated form: not equivalent, not equal, not subset and empty intersection.

CompareToSet<T> can be extended with these criteria, it just needs extra optional parameters and enumerator type that lists comparison modes:

public enum TableComparison
{
    TheTableAndSetMatchWithoutRegardForOrder,
    TheTableAndSetAreAnExactMatch,
    AllItemsInTheTableCanBeFoundWithinTheSet,
    TheTableAndSetHaveCommonItems
}

public static void CompareToSet(this Table table, IEnumerable set, 
    TableComparison comparisonMode = TableComparison.TheTableAndSetMatchWithoutRegardForOrder, 
    bool expectedMatch = true)

Calling CompareToSet without specifying optional parameters will result in test for equivalence, while specifying explicit enumerator value can choose other criteria. In addition, passing “false” ad expectedMatch will assert for comparison failure. Here are examples of using enhanced version of CompareToSet:

Scenario: Match
When I have a collection
| Artist | Album |
| Beatles | Rubber Soul |
| Pink Floyd | Animals |
| Muse | Absolution |
Then it should match
| Artist | Album |
| Beatles | Rubber Soul |
| Pink Floyd | Animals |
| Muse | Absolution |
And it should match
| Artist | Album |
| Beatles | Rubber Soul |
| Muse | Absolution |
| Pink Floyd | Animals |
And it should exactly match
| Artist | Album |
| Beatles | Rubber Soul |
| Pink Floyd | Animals |
| Muse | Absolution |
But it should not match
| Artist | Album |
| Beatles | Rubber Soul |
| Queen | Jazz |
| Muse | Absolution |
And it should not match
| Artist | Album |
| Beatles | Rubber Soul |
| Muse | Absolution |
And it should not exactly match
| Artist | Album |
| Beatles | Rubber Soul |
| Muse | Absolution |
| Pink Floyd | Animals |

Scenario: Containment
When I have a collection
| Artist | Album |
| Beatles | Rubber Soul |
| Pink Floyd | Animals |
| Muse | Absolution |
Then it should contain all items
| Artist | Album |
| Beatles | Rubber Soul |
| Muse | Absolution |
But it should not contain all items
| Artist | Album |
| Beatles | Rubber Soul |
| Muse | Resistance |
And it should not contain any of items
| Artist | Album |
| Beatles | Abbey Road |
| Muse | Resistance |

And here are the step definitions:

[Binding]
public class StepDefinitions
{
    [When(@"I have a collection")]
    public void WhenIHaveACollection(Table table)
    {
        var collection = table.CreateSet();
        ScenarioContext.Current.Add("Collection", collection);
    }

    [Then(@"it should match")]
    public void ThenItShouldMatch(Table table)
    {
        Compare(table);
    }

    [Then(@"it should exactly match")]
    public void ThenItShouldExactlyMatch(Table table)
    {
        Compare(table, TableComparison.TheTableAndSetAreAnExactMatch);
    }

    [Then(@"it should not match")]
    public void ThenItShouldNotMatch(Table table)
    {
        Compare(table, TableComparison.TheTableAndSetMatchWithoutRegardForOrder, false);
    }

    [Then(@"it should not exactly match")]
    public void ThenItShouldNotExactlyMatch(Table table)
    {
        Compare(table, TableComparison.TheTableAndSetAreAnExactMatch, false);
    }

    [Then(@"it should contain all items")]
    public void ThenItShouldContainAllItems(Table table)
    {
        Compare(table, TableComparison.AllItemsInTheTableCanBeFoundWithinTheSet);
    }

    [Then(@"it should not contain all items")]
    public void ThenItShouldNotContainAllItems(Table table)
    {
        Compare(table, TableComparison.AllItemsInTheTableCanBeFoundWithinTheSet, false);
    }

    [Then(@"it should not contain any of items")]
    public void ThenItShouldNotContainAnyOfItems(Table table)
    {
        Compare(table, TableComparison.TheTableAndSetHaveCommonItems, false);
    }

    private void Compare(Table table, TableComparison comparisonMode = TableComparison.TheTableAndSetMatchWithoutRegardForOrder, bool expectedMatch = true)
    {
        var collection = ScenarioContext.Current["Collection"] as IEnumerable;
        table.CompareToSet(collection, comparisonMode, expectedMatch);
    }
}

Currently this extensions are a proposition that I created in a forked branch of SpecFlow, available here.

YAGNI fascism

I am at QCon London now, and yesterday I went to Dan North's workshop "Secrets of Agile Architecture". One of the good points he made was about overuse of YAGNI principle in agile development world. We frequently hear "you ain't gonna need it" as an argument for not doing certain things. Dan warned against what he called "YAGNI fascism" and gave an example from aviation industry. Sometimes plane manufacturer orders the same type of component simultaneously from different manufacturers - for example, several manufacturers may be contracted to design plane wings. Not just compete to win the contract, but to actually fully design wings for a new plane. This sounds like a harsh violation of YAGNI principle - how many pairs of wings a plane needs? - but wings are so crucial part of a plane that it's worth to compare design made by several independent teams that don't communicate with each other, to reduce the chances of the overall best design having flaws.

This is a good example of why YAGNI principle should at least not be interpreted as “you ain’t gonna need it in production”. As you see, even the statement “you ain’t gonna need several production quality implementations” can be false. I also believe that YAGNI argument is too often used to  oversimplify the situation, and therefore in many cases it’s better to go for more accurate and problem specific reasoning than just cut the discussion with “you ain’t gonna need it” remark.

Migrating database schema from Microsoft SQL Server to Oracle using Entity Framework and Devart dotConnect

This walkthrough covers the migration of tables and primary/foreign keys that is a reasonable assumption when accessing data using ORM.

Prerequisites

• Visual Studio 2010 (should also work with Visual Studio 2008 with .NET 3.5 SP1 applied).
• Devart dotConnect for Oracle (formely known as OraDirect.NET) that supports Entity Framework capabilities.

Steps to migrate data schema

1. Create a C# class library project using Visual Studio 2010.

2. Delete a class created in a projects (“Class1”) and add a new item using “ADO.NET Entity Data Model” template.

3. Select “Generate from database” and choose SQL Server database that contains the schema to be migrated.

4. Select “Tables” in the next screen (“Choose Your Database Objects”), and Visual Studio will generate a model based on the selected database schema.

Now if you right-click on a model diagram, Visual Studio will display a menu with an option “Generate Database from Model…”

If you choose database generation, Entity Designer will generate a script for Microsoft SQL Server.

4. Assuming Devart dotConnect is installed, it should be possible to change a template that is used to generate database script. Open “DDL Generation Template” combo box in Entity Designer Properties window and select “Devart SSDLToOracle.tt (VS)”.

5. After the change of DDL generation template execute “Generate Database from Model” and you should get a script for Oracle data schema:

Voila!

Test post to check syntax highlighter

This post has been written using Windows Live Writer 2011 with FiftyEightBits PreCode Snippet. This is C# code sample:

public void Test()
{
    Console.WriteLine("Hello");
}

And this is F# code:

let rec search m x y f accu = match x, y with
  | x, y when x = size2 -> search m 0 (y + 1) f accu
  | 0, y when y = size2 -> f accu
  | x, y when m.(y).(x) <> 0 -> search m (x + 1) y f accu
  | x, y ->
      let aux accu n =
        if invalid m 0 x y n then accu else begin
          m.(y).(x) <- n;
          let accu = search m (x + 1) y f accu in
          m.(y).(x) <- 0;
          accu
        end in
      fold aux accu 1 10

Well, actually I used C# brush for F# code. Hope that the server will be update with F# brush soon.

UPDATE. Dennis van der Stelt who is in charge for BloggingAbout.Net updated the server with new brushes, so I can now write as much F# code as I want. Thank you very much, Dennis!

Interaction-based testing and stable expectations

Yesterday I participated in a discussion about how to test the following class (it was actually an interview question). Here’s the class under test:

public class ClassUnderTest
{
    private SomeService service;

    public ClassUnderTest(SomeService service)
    {
        this.service = service;
    }

    public void Foo(int fooInput)
    {
        if (fooInput > 0)
            this.service.DoSomething();
    }
}

What SomeService does is not relevant. Just something. As you can see, the ClassUnderTest is not very cooperative for the purpose of testing its functionality: it does not expose any properties, and the only method it contains does not return anything. How can we test it?

It looks like the only validation option that is left is to check that “something” is done, and the only way to check it is through interaction testing. Here is a couple of tests that achieve this (I used “method-condition-outcome” naming convention, but the names could be rephrased in BDD style):

[Test]
public void Foo_PositiveInput_SomethingIsDone()
{
    // Arrange
    var service = MockRepository.GenerateMock<SomeService>();
    var cut = new ClassUnderTest(service);

    // Act
    cut.Foo(1);

    // Assert
    service.AssertWasCalled(s => s.DoSomething());
}

[Test]
public void Foo_NonPositiveInput_SomethingIsNotDone()
{
    // Arrange
    var service = MockRepository.GenerateMock<SomeService>();
    var cut = new ClassUnderTest(service);

    // Act
    cut.Foo(0);

    // Assert
    service.AssertWasNotCalled(s => s.DoSomething());
}

I expressed during the discussion a lack of my enthusiasm about this solution noting that I tend to use state-based testing if possible. But is this possible in the example above? And what is a disadvantage of the interaction-based tests in this case?

Let’s have a closer look. First, the above tests essentially duplicate internal logic of the Foo implementation. If we merged two tests into one, then the logic of the assert part of the code would reflect pretty much what’s inside the implementation of Foo: “for positive input service.DoSomething should be called, otherwise not”.

Since the logic of the tests follow the logic of the code and was probably written by the same developer in the same time, there is very little in tests that validates the actual functionality. If I believe that Foo should call DoSomething for positive input, I will most likely get it right in the test that validates that Foo calls DoSomething for positive input. But what if my assumption is wrong? What if DoSomething should be called for input bigger than 100, and for other input values Foo should call DoSomethingElse? No, these tests can’t validate it. I’d say they can’t validate it by design because they are designed to reflect the internal logic of the method they validate. So the correctness of Foo implementation must be verified using other means, and these tests will ensure the Foo implementation stays correct assuming the original algorithm is right.

And this becomes the main value of such tests: they serve as regression tests protecting correct code from being improperly changed (perhaps by another developer who had to extend the original code and didn’t get it right). However, this sort of protection is not refactoring friendly. Let’s see why.

Imagine that SomeService class is refactored, DoSomething is still there but it’s semantics has changed and no longer fits the implementation of Foo. Instead, Foo is supposed to call a new method DoSomethingElse:

public void Foo(int fooInput)
{
    if (fooInput > 0)
        this.service.DoSomethingElse();
}

The system with the new change works properly, but one of the test for ClassUnderTest now fails! Of course, it has no knowledge about what is right or wrong semantically, it only cares about a certain method to be called. Depending on a system complexity, searching for the failure problem can take some time, but even if it’s an easy fix, we have a test that fails for a wrong reason.

In the article by Steve Freeman et al “Mock Roles, No Objects” they warn about mirroring target code logic in the tests: “Some uses of Mock Objects set up behaviour that shadows the target code exactly, which makes the tests brittle. This isparticularly common in tests that mock third-party libraries. The problem here is that the mock objects are not being used to drive the design, but to work with someone else’s. At some level, mock objects should shadow a scenario for the target code, but only because the design of that code should be driven by the test. Complex mock setup for a test is actually a hint that there is a missing object in the design.”

So the interaction-based tests can be more brittle, as we saw in our example. But what are the choices for that example? If we don’t verify calling DoSomething for positive Foo input, then what else can we verify?

To try to answer this question, let’s first figure out what kind of testing we are dealing with: unit or integation. For black box unit testing I am afraid there is not much to validate. No properties are exposed, the only method is void. If you are a code coverage junkie, you can maybe write a couple of tests to validate that the class can be instantiated and used without raising unexpected exceptions, but the value of such work is outside the scope of this blog post. Let’s focus instead on integration testing. How can we validate that the class does what it should?

As soon as we leave the idealistic constraints of black box unit testing and focus on overall functionality validation, we no longer need to express the validation goal in terms of ClassUnderTest methods, not to say about their internal implementation. We focus on the outcome of Foo in terms of what it does to the rest of the system. Does it change data in a database, send email, draws a graph on a screen?

In most cases business logic operations result in some (persistent or in-memory) state change. Then we can strive to catch these changes and use them in the test assertsions. This will help increasing test trustworthiness and tolerance to refactoring. However, determining state changes that correspond to certain action is not always easy, and even when it is, it may result in higher test complexity – you will have to insert some additional code that retrieves the state that is expected to change. And apart from this, there are still scenarios when the result of a certain action is not easily converted to the state available for validation in a test code. Nat Pryce in his old blog post gave an example of such scenario: tests for a graphical simulator. Really, how would you verify that a cell was drawn on a display using state based testing? Other scenarios that may fall in this category include calling external services, sending messages etc. They may (and usually do) leave the traces that can be used for state based testing, but tests will become complex and overloaded with state retrieval details. Clearly, interaction based tests may have their place here.

So what can we test with interaction-based test? Mark Seemann in his forthcoming book “Dependency Injection in .NET” classify dependencies as either stable or volatile. According to him, stable dependencies “are already there, tend to be very backwards compatible and invoking them have deterministic outcomes”. An example of stable dependency is .NET BCL. “Other examples may include specialized libraries that encapsulate algorithms relevant to your application. If you are developing an application that deals with chemistry, you may reference a third-party library that contains chemistry-specific functionality.” Volatile dependencies, on the other hand, do not provide a sufficient foundation for applications.

I believe separating things on stable and volatile is also helpful when deciding what to test using interaction-based testing, we only need to replace word “dependency” to “expectation”. We can use interaction-based testing as long as our expectations are stable, otherwise test becomes brittle. So coming back to original tests for Foo, we should ask ourselves a question: is DoSomething a stable expectation for a given scenario, or it is only valid for a current implementation and may not survive refactoring? If so, the expectation is volatile. Then we should inspect the call graph, find a stable expectation and use it in interaction-based tests. Examples of stable expectations are calls to external services, external API, output to display. As long as the feature is unchanged, the system will be making same outbound calls and draw same pictures on a screen. These activities are stable and can become foundations for interaction-based tests.

So how the original Foo test should look if we find that DoSomething internally sends an email? It can look like this:

[Test]
public void Foo_Should_Send_Notification_Mail_On_Positive_Input()
{
    // Arrange
    var mailServer = MockRepository.GenerateMock<MailServer>();
    var cut = new ClassUnderTest(new SomeService);

    // Act
    cut.Foo(1);

    // Assert
    mailServer.AssertWasCalled(s => s.SendMail());
}

Note that we no longer care about Foo exact implementation. It’s implementation is volatile. We only expect something that is stable in the scope of the given feature. Like sending mail (more accurate test code could verify that the mail was sent to a right person with a right subject, but we should be careful about not overspecifying the expectations, otherwise they will be no longer stable).

When I was about to finish this post, I found another old blog post, by Jeremy D. Miller, where he also uses email sending as an example of an operation that it worth testing using interaction-based tests: “We could use a state-based strategy to test the email. We could run the test, then run around and ask the expected recipients to check their inbox. […] You could also check an audit trail, but that's not the real functionality being tested. The easiest approach in the case of the email tests is to use interaction testing to verify that the email was sent to the SMTP service.”

So I believe interaction-based testing can be an efficient way to validate that operations result in proper actions, as long as tests don’t focus on volatile interactions and instead select stable expectations related to the feauture itself rather than it’s implementation details.

Mock framework challenges in F#

Getting mock framework API rigth is uneasy task. Mock framework designers don’t have as much freedom as designers of other libraries: the purpose of a mock framework is not to expose an arbitraty API (unknown at the time of mock framework design) and intercept it in a transarent way, so developers can use mocked object as they were using real instances.

This is why API of modern mock frameworks is full of technique such as extension methods and lambda expressions. In addition these frameworks use so called fluent approach to API. Unfortunately such methods are not easily portable to other, especially non-imperative languages. Being fluent in one language does not mean fluent in the other one.

I decided to try various mock frameworks with F#. I realized that their C#-friendly syntax will look in F# clumsy, but I coudl overcome it by writing small wrappers. What concerned me more is incomatibility of F# functional delegates and expressions comparing to C#. For my tests I’ve chosen a Zoo example listed in an excellent series of blog posts by Richard Banks dedicated to mock framework comparison. Richard ran his comparison on free frameworks:

• Rhino.Mocks
• NSubstitute
• Moq

I’ve extended this set with two commercial products:

• Typemock Isolator
• Telerick JustMock

So far I only tried very basic mocking described in the first post of Richard’s series: faking return value. As I expected, even such a simple operation became a challenge when executed from F# code. I managed to make tests work only for two and half frameworks. Below is a description of what I did and how it was possible to have half of success.

1. Rhino.Mocks: success

Rhino.Mocks is the most popular .NET mock framework. This was the test code that I wanted to port to F#:

[Test]
public void Return()
{
    var monkey = MockRepository.GenerateMock<IMonkey>();
    monkey.Stub(m => m.Name).Return("Spike");

    var actual = monkey.Name;

    Assert.AreEqual("Spike", actual);
}

As you can see, “monkey” is used both as an instance of an object that implements IMoney interface (so we can call money.Name directly) and as an object exposing mocking API (so we can call a Stub method on it). This is achieved by using extension methods, and extension methods are language specific – being defined in C# code they are not available in F# as extensions. But they can be used from a static class where they are originally defined:

[<Test>]
member this.Return() =
    let monkey = MockRepository.GenerateMock<IMonkey>()
    RhinoMocksExtensions.Stub<IMonkey, string>(monkey, fun m -> m.Name).Return("Spike");

    monkey.Name
    |> should equal "Spike"

The test works, but RhinoMocksExtensions class has not been meant to be exposed to general public, so I put it in a little wrapper:

module FsMock.RhinoMocks

open Rhino.Mocks

type mock<'T when 'T : not struct>() =

    let instance = MockRepository.GenerateMock<'T>()

    member this.Object = instance

module Mock =

    let arrange<'T, 'R when 'T : not struct> (f : ('T -> 'R)) (r : 'R) (m : mock<'T>) = 
        RhinoMocksExtensions.Stub<'T, 'R>(m.Object, Function<'T, 'R>(f)).Return(r)

Now I can rewrite the origianl test in a F# style:

[<Test>]
member this.Return() =
    let monkey = mock<IMonkey>()
    monkey
    |> Mock.arrange (fun m -> m.Name) "Spike"
    let monkey = monkey.Object

    monkey.Name
    |> should equal "Spike"

Both the original and rewritten tests passed.

2. NSubstitute: success

NSubstitute is a new kid on the block. The framework prioritizes simplicity over coverage of all possible mocking scenarios. Unlike other frameworks, NSubstitutes avoids using lambda expressions. This makes it easier to use in F#.

Here’s the C# code using NSubstitute:

[Test]
public void Return()
{
    var monkey = Substitute.For<IMonkey>();
    monkey.Name.Returns("Spike");

    var actual = monkey.Name;

    Assert.AreEqual("Spike", actual);
}

Like Rhino.Mocks, NSubstitute also uses extension methods, so porting the above test to F# results in the following code:

[<Test>]
member this.Return() =
    let monkey = Substitute.For<IMonkey>()
    SubstituteExtensions.Returns(monkey.Name, "Spike")

    monkey.Name
    |> should equal "Spike"

Again, I created a little helper to make syntax F# friendly:

module FsMock.NSubstitute

open NSubstitute

type mock<'T when 'T : not struct>() =

    let instance = Substitute.For<'T>()

    member this.Object = instance


module Mock =

    let arrange<'T, 'R when 'T : not struct> (f : ('T -> 'R)) (r : 'R) (m : mock<'T>) = 
        SubstituteExtensions.Returns<'R>(f(m.Object), r)

Now the test code looks identical to Rhino.Mocks example:

[<Test>]
member this.Return() =
    let monkey = mock<IMonkey>()
    monkey
    |> Mock.arrange (fun m -> m.Name) "Spike"
    let monkey = monkey.Object

    monkey.Name
    |> should equal "Spike"

And the test passed!

NB! I have deliberately unified mocking API when creating F# wrappers for different frameworks. Since F# API will be quite different from it’s C# counterpart, I didn’t want to multiply number of API sets. Instead I came up with a common and easy to understand set of names (“mock”, “arrange”) that is used in each wrapper.

3. Moq: half success

Now that sounds strange. A test should either succeed of fail. So what has happenned to Moq? Here’s the story.

Moq is a second most popular framework after Rhino.Mocks, and it has pioneered act-arrange-assert API based on lambda-expressions. This is how the C# test looks when using Moq:

[Test]
public void Return()
{
    var monkey = new Mock<IMonkey>();
    monkey.Setup(m => m.Name).Returns("Spike");

    var actual = monkey.Object.Name;

    Assert.AreEqual("Spike", actual);
}

Although the expression “m => m.Name” looks exactly like in Rhino.Mocks, there is a big difference behind. Rhino.Mocks uses Func<T> delegate, and Moq is based on LINQ expressions.  Working with such expressions in F# requires use of so called quotation expressions that should be converted to LINQ and then downcasted to a generic LINQ expression. The corresponding code looks quite criptic:

[<Test>]
member this.Return() =
    let monkey = new Mock<IMonkey>()
    let expr = (<@@ System.Func<IMonkey, string>(fun (m : IMonkey) -> m.Name) @@>).ToLinq()
    monkey.Setup(expr :?> System.Linq.Expressions.Expression<System.Func<IMonkey, string>>).Returns("Spike");
    let monkey = monkey.Object

    monkey.Name
    |> should equal "Spike"

But this does not work. Here’s the output:

Test 'MoqTests+MoqTests.Return' failed: System.NullReferenceException : Object reference not set to an instance of an object.
	at Moq.MethodCall.SetFileInfo()
	at Moq.MethodCall..ctor(Mock mock, Expression originalExpression, MethodInfo method, Expression[] arguments)
	at Moq.MethodCallReturn..ctor(Mock mock, Expression originalExpression, MethodInfo method, Expression[] arguments)
	at Moq.MethodCallReturn`2..ctor(Mock mock, Expression originalExpression, MethodInfo method, Expression[] arguments)
	at Moq.Mock.<>c__DisplayClass15`2.b__14()
	at Moq.PexProtector.Invoke[T](Func`1 function)
	at Moq.Mock.SetupGet[T1,TProperty](Mock mock, Expression`1 expression)
	at Moq.Mock.<>c__DisplayClass12`2.b__11()
	at Moq.PexProtector.Invoke[T](Func`1 function)
	at Moq.Mock.Setup[T1,TResult](Mock mock, Expression`1 expression)
	at Moq.Mock`1.Setup[TResult](Expression`1 expression)
	C:\Projects\NET\MockComparison\TempTests\MoqTests.fs(20,0): at MoqTests.MoqTests.Return()

So why half success then? Well, if I execute the same code from F# interactive window, it works!

> let monkey = new Mock<IMonkey>()
let expr = (<@@ System.Func<IMonkey, string>(fun (m : IMonkey) -> m.Name) @@>).ToLinq()
monkey.Setup(expr :?> System.Linq.Expressions.Expression<System.Func<IMonkey, string>>).Returns("Spike");
printfn "%s" monkey.Object.Name;;
Spike

val monkey : Mock<IMonkey>
val expr : Expression = m => m.Name

Note the word “Spike” printed right after the line that begins with “printfn”. This is the output.

So I don’t really have a clue why the same code works in F# interactive session but fails being compiled in an assembly. I may need to investigate some more.

4. Typemock Isolator: failure

Ironically, both commercial frameworks failed to be used with F#. The C# code that used to test Typemock Isolator looks like this:

[Test]
public void Return()
{
    var monkey = Isolate.Fake.Instance<IMonkey>();
    Func<string> func = () => monkey.Name;
    Isolate.WhenCalled(func).WillReturn("Spike");

    var actual = monkey.Name;

    Assert.AreEqual("Spike", actual);
}

Here's the corresponding F# code:

[<Test>]
member this.Return() =
    let monkey = Isolate.Fake.Instance<IMonkey>();
    Isolate.WhenCalled(System.Func<string>(fun ignore -> monkey.Name)).WillReturn("Spike");

    monkey.Name
    |> should equal "Spike"

This test fails with the following output:

Test 'TypeMockTests+TypeMocksTests.Return' failed: TypeMock.TypeMockException : 

*** Cannot call Isolate.WhenCalled() with method group fake.Invoke. Try using Isolate.WhenCalled( () => fake.Invoke()) instead
	at ec.a()
	at dm.b(Boolean A_0)
	at im.c(Boolean A_0)
	at im.a(Object A_0, Boolean A_1, Func`1 A_2, Action A_3, Action A_4, Action A_5)
	at im.c(Object A_0)
	at TypeMock.ArrangeActAssert.ExpectationEngine`1.a(TResult A_0)
	C:\Projects\NET\MockComparison\TempTests\TypeMockTests.fs(21,0): at TypeMockTests.TypeMocksTests.Return()

I described the problem to Typemock developers and was advised to try a different approach presented by Roy Osherove in his blog post. Unfortunately the modified test still fails, although with a different exception.

5. JustMock: failure

JustMock is also a commercial product, and it’s API also lacked F# compatibility. This is the original C# code:

[Test]
public void Return()
{
    var monkey = Mock.Create<IMonkey>();
    Mock.Arrange(() => monkey.Name).Returns("Spike");

    var actual = monkey.Name;

    Assert.AreEqual("Spike", actual);
}

JustMock also uses LINQ expressions, so I had to use a similar trick to what I had to do with Moq:

[<Test>]
member this.Return() =
    let monkey = Mock.Create<IMonkey>();
    let expr = (<@@ System.Func<string>(fun ignore -> monkey.Name) @@>).ToLinq()
    Mock.Arrange<string>(expr :?> System.Linq.Expressions.Expression<System.Func<string>>).Returns("Spike");

    monkey.Name
    |> should equal "Spike"

The code compiles and does not throw any expection during the execution. But the “Name” property is not set, so “should equal” assertion fails.

Conclusion: no offence, just exploring possibilities

One thing that I want to get straigh is that these tests say absolutely nothing about general design quality of respective mock frameworks. As I mentioned in the beginning, API design requirements of such frameworks require their developers to apply language-specific tricks to make mocking simple. So the fact that some API fits another programming language is more luck rather than conscious vision. Moreover, I only tried the simplest of a large variety of functions exposed by mock frameworks. A slightly more complicated scenario would fail all frameworks, I am sure.

However, since more developers express interests in F# and start using it in their projects, I believe it’s time for mock frameworks developers to consider this new territory and extend their products to offer full suport for this exciting language.

Exploring Amazon S3 with F#

Introduction

Traditional imperative languages teach developers that program code requires ceremony. You can’t just write a line and expect it to do something. When programming book writers explain how easy is it to display "Hello world" string in their favorite language, they actually start with an explanation that every call should be wrapped in a method, and the method typically takes some arguments etc. And of course they have to let beginners know how to compile the resulting program and how to execute it. Without all these steps the world won’t get its greeting.

All these steps have valid reasons, but as a result development environment based on major imperative languages does not really encourage exploration. It’s best suited towards large scale development, but if a developer wants to have a quick look on something new, and he can’t proceed without creating a new project, then he will often postpone this work until he has enough time or enough reasons to go ahead.

Luckily, there’ve always been alternatives to languages like C# or Java that worked better for rapid prototyping or technology exploration. One of the late additions is F#, and when I recently introduced myself to Amazon Simple Storage Service, I realized that if I need to get a quick overview of its programming facilities, using F# would be much more efficient than if write a test program in C#.

The purpose of this article is to show how fast you can dig into low level details of an unknown technology using F#, and how little code you need to write. Therefore I keep the text of the article short, focusing just on the required steps. If you need more information about F# or Amazon S3, there’s plenty of books and articles available. Just google for them. Instead of numbering steps that I’ve gone through, I’ll provide timing information: I started the exercise at about 17:00 CET on July 15th. It took me about 45 minutes to display a picture extracted from my S3 bucket. And all code was written in F#.

17:00. Getting Amazon Web Services SDK

Amazon Web Services SDK, also known as AWS SDK is available at http://aws.amazon.com/sdkfornet. Once downloaded, it’s installer copies binaries and samples, and optionally registers AWSSDK.dll in the GAC.

17:10. You can’t go wrong with Reflector

Before firing a Visual Studio F# session, I checked the content of the installed binaries and loaded into Red Gate .NET Reflector the only .NET assembly that I found: AWSSDK.dll. I could probably manage without it, but it would take longer time. After I created an S3 client session, I mostly relied on Intellisense support for F# in Visual Studio, but spending few minutes looking at namespaces and classes hinted me about entry points. For example, I assumed that I would need to create an instance of AmazonS3Client and probably use classes from Amazon.S3.Model.

17:15. Opening F# Interactive session

I started Visual Studio and immediately was able to use comparative advantage of F# environment. I didn’t have to create a project – in fact I didn’t know what kind of project I would need to create if I had to: should be a console application? Or maybe a Windows Form program? I didn’t know yet what to expect, all I wanted to get from an IDE is a notepad where I could write code lines and execute them one by one. So F# was a very good fit for my intention: I created a new F# script file and started sending individual lines to an FSI window. The first lines referenced AWSSDK and imported Amazon.S3 namespace that I expected to be useful (thanks Reflector):

#r "AWSSDK"
open Amazon.S3

Later I imported another namespace (Amazon.S3.Model), but these two lines above were enough to connect to Amazon S3 storage.

17:20. Connecting to Amazon S3 and accessing a bucket

The fun part began. My guess was that since AmazonS3Client class has a constructor with two string parameters, this is the one I should use and assign parameters to AWS key ID and secret key. F# Intellisense confirmed it. So here’s my first call to Amazon S3 in F# (I removed actual access key values).

let s3Client = new AmazonS3Client("KEY-ID", "SECRET-KEY")

After a few seconds F# Interactive came with a response:

val s3Client : AmazonS3Client

Now I had a mighty client with many interesting methods to try. The next logical step would be to list available buckets. I had only one, so I could retrieve only the first element of the list.

let response = s3Client.ListBuckets()
response.Buckets

Here’s an FSI output:

val it : System.Collections.Generic.List =
  seq
    [Amazon.S3.Model.S3Bucket {BucketName = "abilov.com";
                               CreationDate = "to, 15 jul 2010 10:17:18 GMT";}]

Great! This looked like a real thing. Encouraged with a quick success, I wanted to start retrieving the bucket contents.

17:25. Failed attempt to retrieve bucket objects

The bucket collection was wrapped in a sequence, so I only needed to get it’s head to obtain the only bucket I had at S3:

let bucket = Seq.head response.Buckets

I thought in order to retrieve bucket objects, I had to send the obtained bucket to some method, but I was wrong. Actually I didn’t need to obtain a bucket at all – since I knew the name of the bucket, I could construct a ListObjectRequest from the name and send it to a ListObjects. But I didn’t know about it 5 minutes ago. Anyway, here’re new calls, to retrieve bucket objects – data files with images from our home photo gallery.

let request = new ListObjectsRequest(BucketName = bucket.BucketName)
let objects = s3Client.ListObjects(request)

Bang!!! Exception!

System.Net.WebException: The underlying connection was closed: Could not establish trust relationship for the SSL/TLS secure channel.
---> System.Security.Authentication.AuthenticationException: The remote certificate is invalid according to the validation procedure.
   at System.Net.Security.SslState.StartSendAuthResetSignal(ProtocolToken message, AsyncProtocolRequest asyncRequest, Exception exception)
   at System.Net.Security.SslState.CheckCompletionBeforeNextReceive(ProtocolToken message, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.StartSendBlob(Byte[] incoming, Int32 count, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.ProcessReceivedBlob(Byte[] buffer, Int32 count, AsyncProtocolRequest asyncRequest)
   at System.Net.Security.SslState.StartReadFrame(Byte[] buffer, Int32 readBytes, AsyncProtocolRequest asyncRequest)

(The rest of the call stack is skipped)

Now when I am writing this article, I am in fact glad that I had this problem. If everything went smooth, you might look at my timing information with a grain of disbelief: the guy probably knew what he was doing. No I didn’t! So I had to check around. Since the error came after so basic steps, I hoped that I wasn’t alone who experienced it, and luckily I was not. Several people complained at Amazon forum, and they were recommended a simple workaround in case they would accept using HTTP instead of HTTPS. And for the purpose of exploration it was just fine.

17:30. Connection to Amazon S3 revisited

Here’s an updated version of connection code, taking now two lines instead of one.

let s3Config = new AmazonS3Config(CommunicationProtocol = Protocol.HTTP)
let s3Client = new AmazonS3Client("KEY-ID", "SECRET-KEY", s3Config)

Then I replayed the other two lines requesting bucket objects. And now I successfully received results:

    {AmazonId2 = "//VkK6Mb7Xe26pNPG7nt40GkYcp4GDyNV45I1E1mfZVCL67aak9/0Txmrk8X+JAE";
     CommonPrefix = seq [];
     CommonPrefixes = seq [];
     Delimiter = null;
     Headers = seq
                 ["x-amz-id-2"; "x-amz-request-id"; "Transfer-Encoding";
                  "Content-Type"; ...];
     IsTruncated = true;
     MaxKeys = "1000";
     Metadata = seq [];
     Name = "abilov.com";
     NextMarker = "private/photo/2005/2005-06-10-12 Elverum/IMG_6803.JPG";
     Prefix = "";
     RequestId = "3A27D1A4495A747A";
     ResponseStream = null;
     S3Objects = seq {…}

     (The rest of results is skipped)

17:35. Retrieving object data

I inspected the response and found the property that I needed: S3Objects. It was a collection of bucket objects representing the photo gallery files I uploaded to S3. Taking a further look at available AmazonS3Client methods I’ve found a method that I should use to retrieve an individual object: GetObject. It took an instance of GetObjectRequest. I first created a GetObjectRequest instance without assigning a Key property. But after receiving exception with error message about missing a key I corrected my mistake. Here’s a code that worked:

let objectRequest = new GetObjectRequest(BucketName=bucket.BucketName, Key="private/photo/2005/2005-01 Kolbotn/IMG_5645.JPG")
let obj = s3Client.GetObject(objectRequest)

The "Key" property is just a file path within the bucket, so it’s easy to figure it out.

I sent "obj" to an FSI window, and it printed it’s properties:

{AmazonId2 = "zUH8IaRglPSPQuC+8m5pzAR59paJTi/L5M1DnxNympp9HcU5RigrntS7NGvpxQQf";
     ContentLength = 1138630L;
     ContentType = "image/jpeg";
     ETag = ""ea02192ee842d3b1987a8de4ac879595"";
     Headers = ?;
     Metadata = seq ["x-amz-meta-cb-modifiedtime"];
     RequestId = "A93CF70A966E4A56";
     ResponseStream = System.Net.ConnectStream;
     ResponseXml = null;
     VersionId = null;}

One of the properties was called ReponseStream. This sounded very promising – perhaps I could try to create an image out of this stream and display it in a Windows Form?

17:40. Displaying an image

What should you do if you want to display an image in a Windows Form in C#? You have to start from creating a new project using Windows Forms Application template. Then you can edit form class and add an Image control to it in a Form designer. How do you do it in F#? You simply create a form with an image right from the script. Here’s the code:

#r "System.Drawing"
#r "System.Windows.Forms"
 
open System.Windows.Forms
open System.Drawing

let img = Image.FromStream(obj.ResponseStream)
let frm = new Form(ClientSize = img.Size)
frm.Paint.Add(fun e -> e.Graphics.DrawImage(img, new Point(0,0)))
frm.Show()

Note that first four lines are just to reference required DLLs and import namespaces. The rest of the code is equally short and complete the task. Look at the Image.FromStream call: I wasn’t quite sure I could do this, but the property name "ResponseStream" was too tempting not to give it a try. And suddenly I saw this window:

This is our house cat Figaro (unfortunately not with us anymore). And no, I didn’t specifically select this photo for my test. This was a random choice, perhaps hinting about what kind of pictures some families tend to take.

17:45. Done!

Yes, we are! Perhaps just to show my environmentalist attitude, I’ll make one last call:

s3Client.Dispose()

Conclusion

So it was three quarters of exploration. Completely new API, combined with my relatively poor experience with Amazon S3, but a picture from a family photo gallery retrieved from an S3 bucket and shown in a Windows form proves that we managed the full stack of operations. All done in F#. This experiment does not expose other (and far more important) language qualities, but I deliberately wanted to focus on a different topic: language efficiency for the purpose of technology exploration and rapid prototyping. Being .NET language, F# can access any .NET area or feature, and its REPL support ensures that developer’s time is spent in a most optimal way.