Vagif Abilov's blog on .NET

Announcing Simple.OData.Client NuGet package for Mono iOS/Android

Release of NuGet 2.5 opens possibilities for targeting Mono iOS/Android platform. The Simple.OData.Client that I maintain is a portable class library, so it was natural step to extend the number of supported frameworks with Xamarin Mono offerings. But as it often happens, I hit a few problems before I could publish the updated package (version 0.15).

One problem I had to deal with was forwarding of the types that are packaged differently in Windows and .NET. Luckily, this problem was addressed earlier by other people, so big thanks to Stuart Lodge (@slodge) and Daniel Plastied (@dsplaisted) for investigating this topic and describing workarounds. Those interested in more details should read this article at StackOverflow. I am now using the same approach that Stuart implemented in MvvmCross, and here is how the file sets for Mono platforms look:

<!—Droid –>
<file src="Simple.OData.Client.Core\bin\Release\Simple.OData.Client.Core.dll" 
      target="lib\MonoAndroid16\Simple.OData.Client.Core.dll" />
<file src="lib\Droid\System.Net.dll" target="lib\MonoAndroid16\System.Net.dll" />

<!-- Touch –>
<file src="Simple.OData.Client.Core\bin\Release\Simple.OData.Client.Core.dll" 
      target="lib\MonoTouch40\Simple.OData.Client.Core.dll" />
<file src="lib\Touch\System.Net.dll" target="lib\MonoTouch40\System.Net.dll" />

Next challenge came from a couple of places in my code where I was using async/await pattern. Since Simple.OData.Client is targeting wide range of platforms including .NET 4.x and Silverlight, I installed Microsoft.Bcl.Async NuGet package that enabled use of async/await keywords when targeting legacy platforms. However, for the time being the files from this package can not be deployed on iOS and Android devices (legal reasons), and Xamarin async/await support is a work in progress. After struggling for a few days with various async issues (only compile-time issues, I haven’t had a single runtime problem with asynchronous operations), I reviewed the code and figured out that it’s so few places where I was using await, that I can rewrite this code replacing await with TPL API almost in no time. So now Simple.OData.Client does not bear any dependency on Microsoft.Bcl.Async.

Finally, I had to write new tests to verify functionality on new platforms. I used an excellent little helper library MonoDroidUnitTest for Android tests, and for iOS there was a Unit Test project template in Xamarin tools. Since I don’t have Mac, I signed in to MacInCloud to test using iOS simulator. It kind of works, but the performance is sluggish so I don’t think this is a good alternative for a serious iOS development (but OK for occasional validation). It was easier with Android – I could simply start a Droid simulator, loaded an app with a few tests, and in a few seconds (well, actually minutes) I could see the following picture:

So in case you are doing OData development and need a library that you can use on multiple platforms, have a look at Simple.OData.Client. Now with Mono support.

Cross-platform design-time view models using portable class libraries

Stuart Lodge is working on a fantastic series of videos and blog posts showing how to build cross-platform mobile applications with MvvmCross. One of his tips is about exposing design-time data. His method is straightforward and efficient, but in case you don’t want to copy sample files to a location inside Windows Program Files folder, you may consider other alternatives, such as storing design-time information as project content files or embeding it into an assembly.

This task becomes more complicated in case you want to show some images at design-time, and BitmapImage class is not part of portable class libraries. I used to solve it by representing view model image properties using opaque object class and setting its values in platform-specific part of the view models. The unfortunate consequence of this approach was declaring a view model per platform serving design-time purposes. The platform view models inherited from a common portable view model and only added a tiny bit of non-portable logic – like reading image resources. Here is how I used to show design-time data in my ODataPad application:

public class ServiceViewModel 
{ 
    public string Name { get; set; } 
    public string Description { get; set; } 
    public string Url { get; set; } 
    public object Image { get; set; } 
} 

public class WinRTDesignHomeViewModel : DesignHomeViewModel 
{ 
    public WinRTDesignHomeViewModel() 
    { 
        foreach (var service in this.Services) 
        { 
            service.Image = new BitmapImage(
                new Uri("ms-appx:///Samples/" + service.Name + ".png")); 
        } 
    } 
} 

public class Net45DesignHomeViewModel : DesignHomeViewModel 
{ 
    public Net45DesignHomeViewModel() 
    { 
        foreach (var service in this.Services) 
        { 
            service.Image = new BitmapImage(
                new Uri(@"pack://application:,,,/ODataPad.UI.Net45;component/Samples/" + service.Name + ".png")); 
        } 
    } 
}

<Image Source="{Binding Image}" Stretch="UniformToFill" />

Here are the screenshots of design-time views:

This method works but as I already pointed out, it feels heavy because each design-time data set is backed with its own design-time view model – due to non-portability of image .NET types and image management methods.

But there is another way – storing image properties using portable data types, and what can be more portable than a standard string? Enter base64 strings.

Here’s a revised ServiceViewModel class with image data stored as strings:

public class ServiceViewModel 
{ 
    public string Name { get; set; } 
    public string Description { get; set; } 
    public string Url { get; set; } 
    public string ImageBase64 { get; set; } 
}

We no longer need WinRTDesignHomeViewModel  and Net45DesignHomeViewModel classes. Instead we will store all design-time information in a portable class library as an embedded resource. Not that Assembly.GetManifestResourceStream method is part of most of portable class libraries profiles, so we can share the following PCL code between all platforms:

public DesignHomeViewModel()
{
    IEnumerable services = null;
    var namespaceName = typeof(DesignHomeViewModel).Namespace;

    var stream = typeof (DesignHomeViewModel).Assembly
        .GetManifestResourceStream(string.Join(".", namespaceName, "SampleServices.xml"));
    using (var reader = new StreamReader(stream))
    {
        this.Services = SamplesService.ParseSamplesXml(reader.ReadToEnd());
    }
}

Wait, but that can’t be sufficient – our XAML views contain Image elements, we can’t just throw base64 strings at them. Well, almost: it’s all about converters.

Here’s revised XAML code:

<Image Source="{Binding ImageBase64, Converter={StaticResource Base64ToImage}}" Stretch="UniformToFill"/>

namespace ODataPad.UI.WinRT.Common
{
    public sealed class Base64ImageConverter : IValueConverter
    {
        public object Convert(object value, Type targetType, object parameter, string language)
        {
            return ConvertAsync(value, targetType, parameter, language).Result;
        }

        public object ConvertBack(object value, Type targetType, object parameter, string language)
        {
            return null;
        }

        public async Task ConvertAsync(object value, Type targetType, object parameter, string language)
        {
            var image = new BitmapImage();

            if (value != null)
            {
                var bytes = System.Convert.FromBase64String((string)value);

                var ras = new InMemoryRandomAccessStream();
                using (var writer = new DataWriter(ras.GetOutputStreamAt(0)))
                {
                    writer.WriteBytes(bytes);
                    await writer.StoreAsync();
                }

                image.SetSource(ras);
            }
            return image;
        }
    }
}

namespace ODataPad.UI.Net45.Common
{
    public class Base64ImageConverter : IValueConverter
    {
        public object Convert(object value, Type targetType, object parameter, CultureInfo culture)
        {
            var image = new BitmapImage();
            if (value != null)
            {
                var bytes = System.Convert.FromBase64String((string)value);

                image.BeginInit();
                image.StreamSource = new MemoryStream(bytes);
                image.EndInit();
            }
            return image;
        }

        public object ConvertBack(object value, Type targetType, object parameter, CultureInfo culture)
        {
            return null;
        }
    }
}

namespace ODataPad.UI.WP8.Common
{
    public class Base64ImageConverter : IValueConverter
    {
        public object Convert(object value, Type targetType, object parameter, CultureInfo culture)
        {
            var image = new BitmapImage();
            if (value != null)
            {
                var bytes = System.Convert.FromBase64String((string)value);
                image.SetSource(new MemoryStream(bytes));
            }
            return image;
        }

        public object ConvertBack(object value, Type targetType, object parameter, CultureInfo culture)
        {
            return null;
        }
    }
}

Now all my design-time data are stored in a single portable class library and I display them using a single view model also declared in a PCL. All platform-specific data such as images are stored as strings and only converted to bitmaps when rendering views using value converters.

Introducing Simple.OData.Client: a portable OData client library for .NET4.x, Windows Store, Silverlight 5 and Windows Phone 8

Last week I was busy with my home project: creating an OData portable class library. This project originated from the Simple.Data OData adapter when I needed a library for Windows Store application and didn’t find any suitable one. And since Simple.Data only supports .NET 4.x platforms, I started extracting parts of the adapter to make a PCL.

As it always happens, I spent far longer time to complete this task, but it was worth it even the only outcome was code cleanup. I thought the quality of Simple.Data OData adapter code was pretty good, but when I was forced to separate non-portable portions of it, I realized that this was a separation of concerns. When you are writing code that targets multiple platforms, it’s no longer just your subjective decision of how much responsibility you can delegate to a certain class or module. If your occasionally brought stuff that does not meet portability criteria – logging, creating user credentials, external data access – you will have to remove it. Otherwise your code won’t compile.

Of course running the code through portability check won’t expose all violations of clean code principles, moreover – there may be plenty of reasons for code to not be portable. But in many cases it’s a good check, and we should probably ask yourself a question “why this code is not portable?” more often.

But enough for lessons and principles. The portable OData client library is here, and Mark Rendle was kind enough to let me keep the “Simple” prefix in the library name, although this word is kind of his trademark now. (His remark was “as far as I'm concerned it all grows the brand “). And I hope that simplicity of Simple.OData.Client API is on the level of other members of the family of Simple frameworks.

Simple.OData.Client is an open source library, available at GitHub. In addition, it has a NuGet package. If you install it from NuGet, depending on your project target framework you will get a reference to one of two assemblies:

• Simple.OData.Client.Net40.dll: for target frameworks .NET 4.0, .NET 4.0.3, .NET 4.5;
• Simple.OData.Client.Core.dll: for portable libraries targeting .NET 4.0.3, .NET 4.5, Windows Store, Silvelight 5, Windows Phone 8 and for target frameworks Windows Store, Silverlight 5 and Windows Phone 8.

Support for Android and iOS (via Xamarin Mono) is in my plans (in fact, AFAIK the portable version may be used to target Mono for Android).

The project has Wiki pages with many examples, it has two API flavors: basic and fluent. Below is an a example of using its fluent API to retrieve data from NuGet OData feed:

var client = new ODataClient("http://packages.nuget.org/v1/FeedService.svc/");
var x = ODataFilter.Expression;
var packages = client
    .For("Packages)
    .Filter(x.Title == "Simple.OData.Client")
    .FindEntries();

foreach (var package in packages)
{
    Console.WriteLine(package["Title"]);
}

Simple.OData.Client supports all HTTP methods used by OData protocol (GET/POST/PUT/MERGE/DELETE) including support for batch requests.

Converting a Git repository to TFS

Recently we had a task of converting a Git repo to a TFS. The original source code repository was managed under TFS, then during a trial period we ran it using Git (exported from TFS). However, due to organizational policies we had to switch back to TFS. At the same time our project was assigned a new name in TFS, so we had to create a new TFS source repository and preserve Git commit history. Below is a description of how we managed to achieve this. We used Git-Tf open-source tool released by Microsoft.

1. Cloning an existing Git repo

A new copy of a Git repo was created to be used as an import source. This was done using a trivial Git command:

git clone //gitserver/Source/OurProject.git OurProject.Git.Tfs

So the OurProject.Git.Tfs is the name of a new repository that will be converted to TFS.

2. Configuring Git-Tf to establish a link between Git and TFS repositories

The following command instructs Git-Tf to establish a link between the Git and TFS repos:

git-tf configure http://tfsserver:8080/tfs $/OurProject/Master

3. Importing Git commit history… if it works

The following command should convert to TFS all your Git commits, assuming they have a linear form:

git-tf checkin –deep

Most likely this won’t work for any Git repo with non-rebased branches. Then you have a choice of either import all Git commits as a single changeset (losing the history) or add some more efforts to transform the history into a shape suitable for TFS export. But first…

4. Ensuring Git commit messages are not empty

If you try to rebase a Git repo where some commit messages are empty (bad practice!), you are going to fail. But you can fix the messages by running a script similar to the one I’ve found on StackOverflow:

git filter-branch -f --msg-filter '
read msg
if [ -n "$msg" ] ; then
    echo "$msg"
else
    echo "The commit message was empty"
fi'

If your repository is large, it will take some time, but after that you can move on to rebase step.

5. Rebasing a Git repo to make a linear commit history

Git and TFS have significant differences in its architecture, so in order for Git commit history to be swallowed by TFS branch, it has to have a linear form. This can be achieved by finding a last commit prior the first branch (i.e. find the very first (oldest) branch and take a commit prior to that) and using its hash as a parameter to an interactive rebase command:

git rebase -i 5fa13f88cf61b37fea760d24e78819383a8df8ca

This method was also suggested at StackOverflow.

Git interactive rebase tool will popup with a long list of actions. Don’t change anything there, just save them, and the rebase commences. It may interrupt quite a few times complaining about merge conflicts. Resolving the conflicts is on you, but usually they are trivial and can be resolved just by invoking “git add .” or “git add add –u” followed by a “git commit”. In the end you will have a rebased Git repo that is ready to be exported to TFS.

6. Importing Git commit history… second attempt

So now you type again the checkin command:

git-tf checkin –deep

Worked? Then you’re lucky. Because in our case it didn’t due to a bug in the current version of Git-Tf tool. If your commits include renames with moving files between directories, the tool gets confused and stops with a message like this:

Git-tf: failed to pend changes to TFS due to the following errors. Please fix the errors and retry check in.
The item $/OurProject/Master2/Backend/Client/Service References/Entities already exists.

We contacted Microsoft team, they admitted it is a bug in the Git-Tf tool but suggested a workaround:

git tf checkin --deep --renamemode=none

This workaround causes renames to be handled as deletions followed by insertions – not quite right but acceptable in our case, because we only need to preserve historical code snapshots and not how they were reached.

After the last step a TFS repository with full commit history should be established, and developers may either use it directly or keep Git-Tf to use Git locally and send it to TFS using this tool.

Check your code portability with PCL Compliance Analyzer

I am extracting parts of my Simple.Data OData adapter to make a portable class library (PCL). The goal is to create an OData library available for desktop .NET platforms, Windows Store, Silverlight, Windows Phone and even Android/iOS (using Xamarin Mono). To study platform-specific PCL capabilities I used an Excel worksheet provided by Daniel Plaisted (@dsplaisted). It’s a very helpful document, but it would be much easier if I could simply point some tool to an assembly file and it would show its portable and non-portable calls.

I haven’t found such tool, so I wrote one. It’s called PCL Compliance Analyzer. Select an assembly file, set of platforms you want to target, and it will show you if the assembly is PCL compliant and what calls are not. Here are the results for Mono.Cecil (that I used to scan assembly calls):

As you can see, only about 10% of Mono.Cecil is not portable to any of the .NET platforms.

Let’s try something completely different. Here’s RavenDB.

Note that non-portable calls include calls to third-party libraries, like NLog. This is not interesting because those third-party assemblies are subject to a separate check, and it’s reasonable to limit portability compliance check to only system API. So we check “Exclude third party libraries” check box and repeat the scan. Here’s the new result:

Exclusion of third-party libs reduce the number of non-portable calls from 1642 to 674. But we are targetting all platforms (even Xbox 360), and this is perhaps not a very good idea for a NoSQL database with LINQ support. If we limit supported platforms to only .NET 4.5, we will end up with this picture:

Of course the choice of one of RavenDB files is completely random here. I just had an old project folder with files left from Rob Ashton’s workshop on RavenDB, so I used one of the assembly files to demonstrate what you can do with PCL Compliance Analyzer.

Finally, if you point it to a true PCL assembly and select right set of target platforms, PCL Analyzer will confirm that all calls are portable. Like it did in case of my Simple.OData.Client:

PCL Compliance Analyzer is an open-source project hosted at GitHub, and if you only need its binaries, you can download them here.

Managing type inheritance with Simple.Data OData adapter

Type inheritance is not frequently implemented in OData services. I’d say that REST services is not a very good fit for OO paradigm and there are other means to expose information in a REST service than to open up the whole object hierarchy. However, today the easiest way to build an OData service is to write a tiny wrapper around Entity Framework model using WCF Data Services. So if OData didn’t have support for inheritance, it would leave out a wide range of real-world data models. Therefore OData and WCF Data Services has full support for type inheritance, and so do the Simple.Data OData adapter. Initially inheritance was added in the version 0.11 at request from Gianni Mellon (thanks for a valuable feedback), and the version 0.12 fixed a few bugs related to update and deletion of inherited type instances.

Consider the following type hierarchy:

“Transport” is an abstract type, and we want to be able to retrieve and modify both derived types: “Ships” and “Trucks”. Here’s how you can do it with Simple.Data and its OData adapter.

1. Retrieving all transport instances (both ships and trucks)

IEnumerable<dynamic> transports = _db.Transport.All();

Then you can traverse the resulting collection and access either ShipName or TruckNumber properties depending on the concrete type of each element.

2. Retrieving only derived type (ships) instances

IEnumerable<dynamic> ships = _db.Ships.All();

Simple, isn’t? It’s Simple.Data. Now let’s look up just one ship:

var ship = _db.Ships.Find(_db.Ships.ShipName == "Titanic");

3. Creating new entries

var ship = _db.Ships.Insert(ShipName: "Test1");

4. Updating entries

_db.Ships.UpdateByTransportID(TransportID: ship.TransportID, ShipName: "Test2");

or

var ship = _db.Ships.Find(_db.Ships.ShipName == "Titanic");
ship.ShipName = "Not Titanic";
_db.Ships.Update(ship);

5. Deleting entries

_db.Transport.DeleteByTransportID(ship.TransportID);

or

var ship = _db.Ships.Find(_db.Ships.ShipName == "Titanic");
_db.Ships.Delete(ship);

The latest Simple.Data.OData package is available from NuGet.

Simple.Data OData adapter: approaching version 1.0

It’s January the first, but when I looked at my Twitter feed I realized I am lazy. People are discussing programming bugs, API design flaws and book chapters they are writing. I need to pretend I’m one of them and do something useful. And it’s a good opportunity for me to say a few words about the current status of Simple.Data OData adapter, an open source library that I’ve been developing in my free time whole last year. So here we go.

Is Simple.Data OData adapter finished?

It’s not a good idea to use the word “finished” when talking about software, but if you mean “production ready”, then yes. Although it’s current NuGet package version is 0.8.4, it will be updated to 1.0 as soon as its master project Simple.Data version 1.0 will be released. Current OData adapter is a release candidate and I am not aware of any issues with it.

What about documentation then?

I was waiting for this question, and I am well prepared! You will find a set of Wiki pages at GitHub describing every single scenario of using the adapter. I am copying here the table of content:

Getting started with Simple.Data OData provider

Retrieving data

Retrieving all data from a collection
Retrieving all data matching search criteria
Retrieving all data matching search criteria specified in a method name
Retrieving a single row matching search criteria
Retrieving a single row matching search criteria specified in a method name
Retrieving a single row by key
Expanding results with linked entries
Retrieving linked entries without fetching its owners
Including standard functions in search criteria
Results projection, paging and ordering

Modifying data

Adding entries
Adding entries with links
Updating entries
Updating entries with links
Deleting entries
Linking and unlinking entries
Batch updates

What’s next?

Recently I have split the OData adapter project in two parts: one (Simple.OData.Client) is the actual OData client with both C# dynamic and more traditional fluent API, and the second (Simple.Data.OData) is a thin wrapper around the first one to expose Simple.Data client API. The motivation for this change was the work on a Windows Store application with OData client support that couldn’t use Simple.Data because it does not run on WinRT platform. To make this happen I began working on Simple.OData.Client aiming to release it as a portable class library (PCL) that could be run on various platforms including Windows Phone. At the present time there’s only a NET 4.0 Simple.OData.Client NuGet package, but WinRT, NET 4.5 and Windows Phone versions will follow soon.

Using OData protocol V3 with MongOData OData provider

In a recent post I showed how MongOData (it’s a MongoDB OData provider that I wrote and maintain) exposes BsonArray MongoDB properties. Support for arrays of primitive and complex types has been added to OData protocol in its latest version 3, but many clients and code generation tools haven’t been enhanced with this feature, so they will fail accessing a MongoDB collection where some items are exposed as arrays.

Moreover, due to the nature of document databases, array is a natural way of exposing collections of items, versus one-to-many relationship used in relational database. So the chances are pretty high that an arbitrary MongoDB repository will contain documents that will break old generation client tools.

Let me show you this on a simple example. I’ve been working on a set of MongOData samples that would illustrate how to set up and use the provider, and once I tried to generate a WCF Data Services client proxy using Visual Studio 2010, it showed me the following error message:

The offending part was the Product class definition that contained an array of Suppliers:

public class Supplier
{
    public string Name { get; set; }
    public Address[] Addresses { get; set; }
}

So what can you do to overcome such limitation? It’s really silly not to be able to retrieve data structure that both service can expose and your code can consume when all that stops you is a man in the middle: a proxy generation tool.

My answer to this is simple: bypass proxy generation. OData protocol follows REST principles, and proxy generation is so SOAP. Just use C# dynamics so the service metadata will be evaluated at runtime.

One simple approach to achieve this is to use Simple.Data OData adapter. Here’s the code that retrieves the content of MongoDB collection (containing arrays) using Simple.Data:

Console.WriteLine("Connecting to MongoDB sample OData service...");
dynamic context = Database.Opener.Open("http://localhost:50336/MongoDataService.svc/");

Console.WriteLine("Retrieving categories...");
Console.WriteLine();
foreach (var category in context.Categories.All().ToList())
{
    Console.WriteLine("Category: ID=[{0}], Name=[{1}]",
        category.ID,
        category.Name);
}
Console.WriteLine();

Console.WriteLine("Retrieving products...");
Console.WriteLine();
foreach (var product in context.Products.All().ToList())
{
    Console.WriteLine("Product: ID=[{0}], Name=[{1}], Category=[{2}], Quantity=[{3} {4}], " +
                      "ReleaseDate=[{5}], DiscontinueDate=[{6}], Suppier=[{7}]",
        product.ID,
        product.Name,
        product.Category.Name,
        product.Quantity.Value,
        product.Quantity.Units,
        product.ReleaseDate,
        product.DiscontinueDate,
        product.Supplier == null ? null : product.Supplier.Name);
}

I created MongOData.Samples solution with the following projects:

• CreateSampleData – creates a sample MongoDB database (default connection string is set to mongodb://localhost/mongodatasamples, you can of course change it), during creation is asks if you want to enable OData protocol V3 support;
• SampleService – WCF Data Service provider that uses MongOData to connect to a MongoDB instance;
• SampleWcfDataServiceClient – OData client that uses proxy classes generated by Visual Studio, so it can not consume services that use OData V3 features such as arrays;
• SampleDynamicClient – proxy-less client that uses Simple.Data OData adapter and support OData V3 features.

You can download sample solution here.

Book review: “Using MSBuild and Team Foundation Build”

I’ve had the book by Sayed Hashimi and William Bartholomew “Inside the Microsoft Build Engine: Using MSBuild and Team Foundation Build” for more than a year and had to refer to it every time I needed to tailor MSBuild scripts. There are very few books on this subject, and for the time being you have a choice of two books dedicated to MSBuild / Team Build: this one and “MSBuild Trickery” by Brian Kretzler. I haven’t read Brian’s book but judging from the table of contents it’s an MSBuild recipe book. Sayed & William’s book also has various guidelines about performing specific tasks, but in addition it gives a good foundation of MSBuild basic concepts and architecture, providing a deep dive into build targets, properties, use of item groups, environment variables etc. I also own a previous Sayed Hashimi’s book “Deploying .NET Applications: Learning MSBuild and ClickOnce” and used it on different occasions to set up a continuous build and test in my former project, so I knew I wouldn’t be disappointed with the new edition. And I was not.

I think the most important book’s parts are  the second (Customizing MSBuild) the third (Advanced MSBuild topics) and the forth (MSBuild cookbook). If you master MSBuild on the level presented in these parts, you will be able to solve difficult project build scenarios. And MSBuild is known for its steep learning curve, so plan to use some time. I also have to admit that whatever I learn about MSBuild I forget within a relatively short time. It’s either something with my memory or MSBuild technology, but in any case I need to have this book around every time I need to work with build scripts. And the book provide a series of recipes that are ready to be used in your project, for example:

• Automatically assigning the assembly version
• Replacing configuration file section values
• Starting and stopping Windows services
• Uploading files to FTP site on successful build
• Deploying Web projects

New in this edition are chapters about building C++ projects (that I didn’t read since I don’t work with C++ these days) and Team Foundation Build. I am not a big fan of Team Foundation Build that in my opinion is over-engineered (look at this 16-pages description of how to override OutDir variable in Team Build), so in case you need to tailor TFS build for your needs, you will most likely have to struggle. I am not sure if a chapter “Team Build Deep Dive” will solve your challenges – in my experience you have to google a lot in order to sort out Team Build related problems – but Sayed and William’s book provides enough to manage a default build set up.

Microsoft recently announced some attractive Azure features to set up a build in the cloud, so I guess we should expect the third edition of this book to cover these topics. However, if you want a sold introduction in MSBuild technology and continuous build on-premises, “Inside the Microsoft Build Engine” is a very good reference book on this topic.

MongoDB OData provider now supports arrays and nested collections

It’s been a while since I blogged about MongOData – a MongoDB OData provider that I wrote to cross MongoDB and OData protocol. Even though the provider has not reached it’s “Version 1.0” state, the response was quite encouraging: I received suggestions and bug reports, and this was indeed a good motivation factor. One of requests was to add support for collections. Property collections haven’t been supported by OData until the most recent protocol version (version 3), and such late attention to this topic is partly explained by the fact that property collections are not currently supported by Entity Framework which is often used to create OData services using WCF Data Services classes. But this should not be a limiting factor for document databases where arrays are used instead of relations in traditional SQL databases. To fill the gap I have upgraded OData protocol used by MongOData to version 3 and added creation of metadata for BsonArray properties.

Let me show how it works using a simple example. Imagine we have a JSON document with a root element colorsArray that consists of a collection of pairs (colorName,hexValue). Here’s the sample document:

{
    "colorsArray":[{
            "colorName":"red",
            "hexValue":"#f00"
        },
        {
            "colorName":"green",
            "hexValue":"#0f0"
        },
        {
            "colorName":"blue",
            "hexValue":"#00f"
        },
        {
            "colorName":"cyan",
            "hexValue":"#0ff"
        },
        {
            "colorName":"magenta",
            "hexValue":"#f0f"
        },
        {
            "colorName":"yellow",
            "hexValue":"#ff0"
        },
        {
            "colorName":"black",
            "hexValue":"#000"
        }
    ]
}

How will MongOData expose it if we create a MongoDB collection “Colors” and import there the document above? Let’s have a look at generated metadata for entity and complex type:

<EntityType Name="Colors">
    <Key><PropertyRef Name="db_id" /></Key>
    <Property Name="db_id" Type="Edm.String" Nullable="false" />
    <Property Name="colorsArray" Type="Collection(Mongo.Colors__colorsArray)" Nullable="false" />
</EntityType>
<ComplexType Name="Colors__colorsArray">
    <Property Name="colorName" Type="Edm.String" />
    <Property Name="hexValue" Type="Edm.String" />
</ComplexType>

Note the type name assigned to a property colorsArray. It’s a collection of items of a type Mongo.Colors__colorsArray – MongOData generates it’s type definition when it reads the first element. It is assumed that all array elements have the same structure – a reasonable assumption if we expect to generate the collection’s metadata.

Now let’s see how data exposed by this OData service can be consumed from the client. On a client side I am using Simple.Data OData adapter – a library that I also happened to maintain.

[Test]
public void Colors()
{
    var result = ctx.Colors.All().First();
    Assert.AreEqual(7, result.colorsArray.Count);
    Assert.AreEqual("red", result.colorsArray[0].colorName);
    Assert.AreEqual("#f00", result.colorsArray[0].hexValue);
    Assert.AreEqual("black", result.colorsArray[6].colorName);
    Assert.AreEqual("#000", result.colorsArray[6].hexValue);
}

MongOData can be downloaded as an MSI installer or as a NuGet package.

DRY – one of the many reasons to upgrade to SpecFlow 1.9

Having been using SpecFlow for almost two years, I was very excited about the (long awaited) release of SpecFlow 1.9. It’s packed with so many goodies that I really think it should have been called 2.0. But Gaspar Nagy is a modest man

There is no point of listing here all new SpecFlow features – you can read it here. In this blog I’d like to focus on one particular improvement that made me upgrade my projects immediately. Because it helps against text duplication. Let’s see how.

Original Cucumber approach to variables is to represent them in step definitions using regular expressions, so they can be matched and replaced with instance data. Even hardcore regex lovers should admit that regular expressions don’t really fit in environment where the goal is to present specifications and tests using natural language. So there’s been numerous attempts in various Cucumber ports to provide alternative conventions.

Here are a few step definitions from a project I am working on. You don’t need to pay attention to service names and other domain-specific stuff – this time we’ll look at the ceremony around it, or in fact reducing this ceremony. So let’s look at how the step definitions were written prior release of SpecFlow 1.9.

I have marked SpecFlow binding attributes – they are just Gherkin phrases except for the variables represented by regular expressions. They are attached to method with names that are also (at least when generated by SpecFlow) same Gherkin phrases with underscores replacing spaces (you can keep spaces when writing step definitions in F#) and variables moved to a method parameter list.

So we have three variations of the same Gherkin phrase: first in the original feature file, second in a method name and third in the attached .NET attribute (Given, When or Then). We have to keep two of them in order to bind the feature and the steps, but can we get rid of the third one?

With SpecFlow 1.9 we can – unless we have special reasons to match variables using regular expressions, and in most of the specifications I’ve seen there were no such reasons. The feature text can be mapped directly to the method names if we add a simple convention to represent variables.

We’ll begin by removing text from the Given/When/Then attributes, as you can see below.

If you were systematic about step definitions method names so they matched related Gherkin text and if .NET attribute texts that you removed didn’t contain regular expressions, SpecFlow will be able to bind methods to Gherkin phrases. If method names were different or if step definitions used regular expressions, SpecFlow will display unmatched steps in purple. In our example this will be a method that used regular expression matching return code:

We need to fix unmatched steps. This is as easy as extending a method name with a variable name written in capitalized form and separated with underscores. So if your original Gherkin phrase was “Service should return OK” and you want “OK” to be an instance of a variable, then in a step definition you should declare such variable as you do in standard Cucumber approach, let’s say it’s called “returnCode” and call a method “Service_should_return_RETURNCODE”. This will do the trick.

After saving the step definition file we can check the feature file and verify that the purple color is gone: SpecFlow restored the step binding:

If you are like me and hate typing same text multiple times, you may want to start refactoring your step definitions immediately. Bear in mind however a couple of things to ensure the transition to regex-free step definitions will be smooth:

1. Your step definition method names should match Gherkin steps, otherwise SpecFlow won’t be able to bind them. If you accepted SpecFlow automatic method generation and updated method names whenever you changed Gherkin steps, you will be rewarded. But even if you didn’t, there is not much to worry about: as you saw, SpecFlow will kindly display in purple all unmatched steps, so a quick feature definition review will uncover all missing bindings.
2. If your variables in your step definitions used special formatting, for example, were enclosed in double-quotes like some of the variables in the example above, you will have to remove special formatting or keep regular expressions. Regex-free approach interprets the whole string as a variable, so spend some time on regression testing after the refactoring. I first didn’t notice double-quotes in my Gherkin steps, and most of the related tests failed. Fortunately, this is usually easy to fix.

Expose MongoDB data as OData feed using MongOData provider

With growing popularity of MongoDB grows the variety of tools and technologies that expose data stored in MongoDB databases. Since I am using both MongoDB and OData in my projects, I decided to cross them and enable exposure MongoDB collections as OData feeds. So I started a project MongOData hosted at GitHub. Although I was using MongOData internally for some time for data retrieval, I was reluctant to announce it until I implement support for update. Update is now supported (version 0.4.0), so I feel it’s time to write a blog post presenting how MongOData works and how it was built.

MongOData highlights

• MongOData is a MongoDB OData provider. It exposes each MongoDB collection as an OData resource set. Current release assumes that all data inside the collection have the same structure, so it builds metadata information based on the first row in each collection. I plan to support different metadata build strategies in future releases.
• MongOData does not use OData open types, although open types seem to be a good fit for schemaless databases. Most of OData client tools don’t have support for open types and rely on types exposed via metadata, so I took an approach that is compatible with traditional OData consumers.
• There are two ways to install MongOData: using MSI installer and using NuGet package. MSI installer is an easiest option if you only need to publish OData feed for one or all MongoDB databases. NuGet MongOData package should be used to add MongoDB WCF Data service to an existing Web project.
• MongOData uses modified code from OData Provider Toolkit that helps building WCF Data Services custom providers. OData Provider Toolkit uses LINQ to Objects to execute queries, so it only supports in-memory queries and therefore requires all data to be read in memory upfront. Such implementation is of course is not suitable for large databases, so I revised the original code to support two strategies: in-memory and queryable providers. MongOData uses queryable provider and LINQ provider from the official MongoDB C# driver.

Installing MongOData using MSI installer

This is the easiest option to start using MongOData with MongoDB. Simply download the installer and run it on a machine with MongoDB and IIS installed. During the installation you can configure Web server virtual directory and MongoDB connection string:

Note that default connection string contains the wild card ‘*’. This syntax is not supported by MongoDB, it indicates that MongOData should expose all MongoDB databases available on the server. The specific database can then be selected by specifying its name in an URL, e.g.:

http://localhost/MongOData/Northwind/$metadata

http://localhost/MongOData/zips/zips

In the examples above we read data from “Northwind” and “zips” database.

In case you don’t want to open OData access to all configured MongoDB databases, you replace the wild card in the installation screen with the name of the specific database, e.g.: mongodb://localhost/Northwind?strict=true. Then only this specific database will be exposed as OData feed.

Installing MongOData using NuGet package

If you want to embed MongoDB OData feed into your Web service or application, you can use MongOData NuGet package. NuGet package adds custom WCF Data Service to your Web component, so you only need to specify the MongoDB connection string (MongOData NuGet package does not support wild cards, you will have to refer to a specific database). Here is what you need to do.

1. Create a Web project using one of Visual Studio templates, for example, “ASP.NET MVC3 Web Application” (almost any generic ASP.NET Web application template will do).
2. Open Package Manager Console window and write the following command:

   Install-Package MongOData

   NuGet package manager will install the latest MongOData component along with its dependency mongocsharpdriver, so you will see the package installation report similar to this:

   Attempting to resolve dependency 'mongocsharpdriver (≥ 1.4.1)'.
   Successfully installed 'mongocsharpdriver 1.4.2'.
   Successfully installed 'MongOData 0.4.0'.
   Successfully added 'mongocsharpdriver 1.4.2' to MongODataTest.
   Successfully added 'MongOData 0.4.0' to MongODataTest.

   If you check the project content, you will see that there are references to MongoDB driver and MongOData assembly files, and two new source files MongoDataService.cs and MongoDataService.svc. Here is the content of the newly added C# source file:

   public class MongoDataService : MongoQueryableDataService
   {
       public MongoDataService()
           : base(ConfigurationManager.ConnectionStrings["MongoDB"].ConnectionString)
       {
       }
   
       // This method is called only once to initialize service-wide policies.
       public static void InitializeService(DataServiceConfiguration config)
       {
           // TODO: set rules to indicate which entity sets and service operations are visible, updatable, etc.
           config.SetEntitySetAccessRule("*", EntitySetRights.AllRead);
           config.SetServiceOperationAccessRule("*", ServiceOperationRights.All);
           config.DataServiceBehavior.MaxProtocolVersion = DataServiceProtocolVersion.V2;
           config.DataServiceBehavior.AcceptCountRequests = true;
           config.DataServiceBehavior.AcceptProjectionRequests = true;
           config.UseVerboseErrors = true;
       }
   }

   Note that MongoDataService refers to a connection string named as "MongoDB". If you are comfortable with this, you don't need to do any modification to the source files, just update the connection string in web.config file so it points to the actual MongoDB database that you are going to expose as OData feed.
3. Build the project and go the MongoDataService.svc page, you will see something like this:
   
   

You're done! Now you can query your MongoDB OData feed using standard OData URI convention.

Expanding results with associated entries using Simple.Data OData adapter

In the post where I introduced Simple.Data OData adapter I showed a few examples of retrieving single or multiple entities. What was common with those examples is that they presented technique to read just the requested resource type, even though there in can be useful to read associated entities at once. Soon after I wrote that blog post, Mark Rendle extended Simple.Data with support to retrieve related data using “With” clause. This was exactly what Simple.Data OData adapter lacked in order to implement its own support to expand results with associated entities. Now I am going to show how it works.

We will continue using Northwind example, since every .NET developer must have learned by heart all its entity types. First, we will retrieve a single category by its name and expand it with associated products. This is how the code will look (I will also show the test assertions to make it clear how to deal with request results):

var category = _db.Category.WithProducts().FindByCategoryName("Beverages");
Assert.Equal("Beverages", category.CategoryName);
Assert.True(category.Products.Count > 0);

And here is the generated request URI:

Categories?expand=Products&$filter=CategoryName+eq+'Beverages'

Now let’s retrieve all categories and expand its products. Here is the code

var categories = _db.Category.All().WithProducts().ToList();
Assert.True(categories.Count > 0);
Assert.True(categories[0].Products.Count > 0);

The request URI will be as follows:

Categories?expand=Products

If we swap the master resource type and start retrieving products with associated categories, the relationship type will change from one-to-many to many-to-one, as you see from this example:

var products = _db.Products.All().WithCategory().ToList();
Assert.True(products.Count > 0);
Assert.Equal("Beverages", products[0].Category.CategoryName);

The request URI for the example above:

Products?expand=Category

Finally, we will look at self-joins. In Northwind database an employee may have a superior who is also an employee, so for each Employee entity there is a corresponding Employee entity associated via “Superior” relationship (can be null), and a set of Employee entities associated via “Subordinates” relationship (can be empty). Here’s how to retrieve a given employee with its superior:

var employee = _db.Employees.WithSuperior().FindByFirstNameAndLastName("Nancy", "Davolio");
Assert.NotNull(employee);
Assert.NotNull(employee.Superior);

And here’s the request URI:

Employees?$expand=Superior&$filter=FirstName+eq+%27Nancy%27 and LastName+eq+%27Davolio%27

And in case we need to expand an employee with its subordinates, the following code will do the trick:

var employees = _db.Employees.All().WithSubordinates().ToList();
Assert.True(employees.Count > 0);
Assert.True(employees[0].Subordinates.Count > 0);

The corresponding request URI:

Employees?$expand=Superior&$filter=FirstName+eq+%27Nancy%27 and LastName+eq+%27Davolio%27

Simple.Data OData adapter source code is available at GitHub, and the binaries can be obtained from NuGet using package ID “Simple.Data.OData”.

Introducing Solution Maker 2.0: new features and support for Visual Studio 11

A couple of years ago I wrote a utility that could scan directories and generate solution files (I blogged about it here and here). I called it Solution Maker and later made its source available at BitBucket. I received feedback from users that helped me come with new features. And since developers are already using Visual Studio 11 preview, it was good time to add support for it. So this post is about new version of Solution Maker that can be downloaded from Visual Studio Gallery (and of course from BitBucket).

To show what can be achieved using Solution Maker, I’ve chosen a large project that builds assemblies familiar to every .NET developer: ASP.NET. Microsoft recently turned it into an open-source project and uploaded to CodePlex. Let’s point Solution Maker to its root folder and see what we can do.

When you start Solution Maker, it comes with a screen showing setting for a solution to be generated. We want to create a solution for ASP.NET Web Stack, so we will choose the name for a new solution (“AspNetWebStack”) and browse for a project root folder:

If we just want a flat project structure (and for small solutions this may sufficient), we don’t need to do anything else and can just click on “Create Solution” button. However, solutions tend to grow, and this where solution folders help splitting projects into logical groups. Solution Maker can automate this process by analyzing project and output assembly names. Let’s start however with a flat structure just to show what we will get in case we don’t want to create solution folders. Click on “Preview” button and Solution Maker will display a new window with a solution preview.

What we see here is 44 projects without any grouping. Not a very convenient structure. Let’s try to improve it a bit: set “Solution folder levels” option to 1, so Solution Maker will parse project names and create solution folders based on the leftmost word in a name of a project.

Looks better, doesn’t it? Note that ASP.NET Web Stack projects are named after the output of their respective assemblies, so you can leave a default naming strategy “Create solution folders based on project file names”. If this choice results in non-informative folder names (for example, if a project for assembly “System.Web.Http” was called just “Http”, then you wouldn’t have a chance to place it in a solution folder “System”), you may try an option “Create solution folders based on project output assembly names”. You can even choose the best of breeds: an option “Create solution folders based on most qualified names” is useful when projects don’t have single naming strategy. Then Solution Maker will analyze their names and assemblies and choose the one that gives the most informative name.

Now we have 44 projects divided into 4 solution folders, but folder “System” contains 27 projects which may be still too many. Let’s increase the number of solution folder levels. Here is what we will get:

What if you only want a subset of projects? This can be achieved using filters. For example, if we want a solution without test projects, we can specify a filter string “*.Test;*.Test.*” in “Project exclude filter” edit box. This will instruct Solution Maker to exclude 25 test projects:

So far so good. But until we assumed that the project root contains all project dependencies, and this is not necessarily the case. It can be easily demonstrated by moving the project root folder to one of subdirectories, for example, to “Src\Microsoft.Web.Mvc”. Imagine that we want to create a solution only for MVC stuff. If we now generate a solution file, it will contain just one project:

This is rather silly. Obviously, Microsoft.Web.Mvc has some dependencies, and we want them in the solution. This is the time to enable “Include referenced projects” option:

Finally, if you prefer command line or need to integrate solution generation into a build workflow, all Solution Maker options can be specified at the command line:

Solution Maker is a free open-source projects, so it’s easy to extend it or adjust for your specific needs.

Introducing Simple.Data OData provider

Mark Rendle’s Simple.Data has quickly become a popular choice to retrieve data from various data sources. The number of Simple.Data providers includes today SQL Server, Oracle, Mysql, MongoDB and others. When I saw Simple.Data in action, I became enthusiastic about extending its list of providers with OData. I believe dynamic nature of Simple.Data matches very well RESTful nature of OData. Currently Visual Studio offers creation of a client context from an OData feed using “Add Service Reference” dialog which treats an OData service like it was a traditional SOAP service. This is unfortunate because OData communication does not need an interface proxy: communication is based on HTTP methods and can be implemented without context creation ceremony.

Welcome to Simple.Data OData provider! When I asked Mark Rendle on Twitter if he had thought about writing a Simple.Data OData provider, he replied that he was working on Windows Azure table provider that also uses OData protocol, and he offered me to join the project and focus on generic OData client implementation. This gave me an exciting opportunity, and today I am glad to announce that Simple.Data OData provider preview is available, and it’s Nuget feed is published.

Simple.Data OData provider is hosted at GitHub. The source code based is shared by OData and Azure provider implementation, but the Nuget packages are different.

Getting started with Simple.Data OData provider

The easiest way to start using Simple.Data OData provider is to install it’s Nuget package. In Visual Studio open Package Manager console and type the following:

Install-Package Simple.Data.OData

You will see output similar to this:

Attempting to resolve dependency 'Simple.Data.Core (≥ 0.12.2.2)'.
Successfully installed 'Simple.Data.OData 0.1.1'.
Successfully added 'Simple.Data.OData 0.1.1' to SimpleODataTest.

In the source file where you will be using OData provider import namespaces:

using Simple.Data; 
using Simple.Data.OData;

Create an instance of dynamic database object that will be used to communication with OData feed:

dynamic db = Database.Opener.Open("http://packages.nuget.org/v1/FeedService.svc/");

Now you can start retrieving data from the OData server using Simple.Data syntax:

var package = db.Packages.FindByTitle("Simple.Data.OData"); 
Console.WriteLine(package.Title);

Reading data using Simple.Data OData provider

package = db.Packages.FindByTitleAndVersion("Simple.Data.OData", "0.1.1.2");
Console.WriteLine(package.Title);

Find multiple results:

IEnumerable<dynamic> packages = db.Packages.FindAllByTitle("Simple.Data.OData");
packages.ToList().ForEach(x => Console.WriteLine(x.Title + " " + x.Version));

packages = db.Packages.FindAll(db.Packages.Version == "1.0.0.0");
packages.ToList().ForEach(x => Console.WriteLine(x.Title + " " + x.Version));

Select column subset:

IEnumerable<dynamic> packages = db.Packages
    .FindAllByTitle("Simple.Data.OData")
    .Select(db.Packages.Version);

Limit number of results using Skip and Take:

IEnumerable<dynamic> packages = packages = db.Packages
    .FindAllByTitle("Simple.Data.OData")
    .Take(1); 

packages = db.Packages
    .FindAllByTitle("Simple.Data.OData")
    .Skip(1)
    .Take(1);

Retrieve results in specific order:

IEnumerable<dynamic> packages = db.Packages
    .FindAllByTitle("Simple.Data.OData")
    .OrderByDescending(db.Packages.Version);

What’s next

OData provider already supports Insert,Update and Delete operations (more about it in a next blog post), but it does not support associations. The implementation of the new functionality requires some changes to Simple.Data.Core library that is on its way.