Vagif Abilov's blog on .NET

Expose MongoDB data as OData feed using MongOData provider

With growing popularity of MongoDB grows the variety of tools and technologies that expose data stored in MongoDB databases. Since I am using both MongoDB and OData in my projects, I decided to cross them and enable exposure MongoDB collections as OData feeds. So I started a project MongOData hosted at GitHub. Although I was using MongOData internally for some time for data retrieval, I was reluctant to announce it until I implement support for update. Update is now supported (version 0.4.0), so I feel it’s time to write a blog post presenting how MongOData works and how it was built.

MongOData highlights

  • MongOData is a MongoDB OData provider. It exposes each MongoDB collection as an OData resource set. Current release assumes that all data inside the collection have the same structure, so it builds metadata information based on the first row in each collection. I plan to support different metadata build strategies in future releases.
  • MongOData does not use OData open types, although open types seem to be a good fit for schemaless databases. Most of OData client tools don’t have support for open types and rely on types exposed via metadata, so I took an approach that is compatible with traditional OData consumers.
  • There are two ways to install MongOData: using MSI installer and using NuGet package. MSI installer is an easiest option if you only need to publish OData feed for one or all MongoDB databases. NuGet MongOData package should be used to add MongoDB WCF Data service to an existing Web project.
  • MongOData uses modified code from OData Provider Toolkit that helps building WCF Data Services custom providers. OData Provider Toolkit uses LINQ to Objects to execute queries, so it only supports in-memory queries and therefore requires all data to be read in memory upfront. Such implementation is of course is not suitable for large databases, so I revised the original code to support two strategies: in-memory and queryable providers. MongOData uses queryable provider and LINQ provider from the official MongoDB C# driver.

Installing MongOData using MSI installer

This is the easiest option to start using MongOData with MongoDB. Simply download the installer and run it on a machine with MongoDB and IIS installed. During the installation you can configure Web server virtual directory and MongoDB connection string:

image

Note that default connection string contains the wild card ‘*’. This syntax is not supported by MongoDB, it indicates that MongOData should expose all MongoDB databases available on the server. The specific database can then be selected by specifying its name in an URL, e.g.:

http://localhost/MongOData/Northwind/$metadata

http://localhost/MongOData/zips/zips

In the examples above we read data from “Northwind” and “zips” database.

In case you don’t want to open OData access to all configured MongoDB databases, you replace the wild card in the installation screen with the name of the specific database, e.g.: mongodb://localhost/Northwind?strict=true. Then only this specific database will be exposed as OData feed.

Installing MongOData using NuGet package

If you want to embed MongoDB OData feed into your Web service or application, you can use MongOData NuGet package. NuGet package adds custom WCF Data Service to your Web component, so you only need to specify the MongoDB connection string (MongOData NuGet package does not support wild cards, you will have to refer to a specific database). Here is what you need to do.

  1. Create a Web project using one of Visual Studio templates, for example, “ASP.NET MVC3 Web Application” (almost any generic ASP.NET Web application template will do).
  2. Open Package Manager Console window and write the following command:
    Install-Package MongOData
    NuGet package manager will install the latest MongOData component along with its dependency mongocsharpdriver, so you will see the package installation report similar to this:
    Attempting to resolve dependency 'mongocsharpdriver (≥ 1.4.1)'.
    Successfully installed 'mongocsharpdriver 1.4.2'.
    Successfully installed 'MongOData 0.4.0'.
    Successfully added 'mongocsharpdriver 1.4.2' to MongODataTest.
    Successfully added 'MongOData 0.4.0' to MongODataTest.
    If you check the project content, you will see that there are references to MongoDB driver and MongOData assembly files, and two new source files MongoDataService.cs and MongoDataService.svc. Here is the content of the newly added C# source file:
    public class MongoDataService : MongoQueryableDataService
    {
        public MongoDataService()
            : base(ConfigurationManager.ConnectionStrings["MongoDB"].ConnectionString)
        {
        }
    
        // This method is called only once to initialize service-wide policies.
        public static void InitializeService(DataServiceConfiguration config)
        {
            // TODO: set rules to indicate which entity sets and service operations are visible, updatable, etc.
            config.SetEntitySetAccessRule("*", EntitySetRights.AllRead);
            config.SetServiceOperationAccessRule("*", ServiceOperationRights.All);
            config.DataServiceBehavior.MaxProtocolVersion = DataServiceProtocolVersion.V2;
            config.DataServiceBehavior.AcceptCountRequests = true;
            config.DataServiceBehavior.AcceptProjectionRequests = true;
            config.UseVerboseErrors = true;
        }
    }
    Note that MongoDataService refers to a connection string named as "MongoDB". If you are comfortable with this, you don't need to do any modification to the source files, just update the connection string in web.config file so it points to the actual MongoDB database that you are going to expose as OData feed.
  3. Build the project and go the MongoDataService.svc page, you will see something like this:

    MongOData02

You're done! Now you can query your MongoDB OData feed using standard OData URI convention.

Comments

edi said:

Hello,

I have a problem with connection string. The error message is:

"Parameter name: Database name '/ip:port/ShopOffers' is not valid. The character 0x002f '/' is not allowed in database names.". The connection string:

<add name="MongoDB" connectionString="mongodb://ip:port/ShopOffers?strict=true" />

What is wrong?

# May 28, 2012 12:44 PM

edi said:

Hi,

I discovered a problem. I can't set connection string as ip:port. If I change to 'localhost' then everything is correct.

Can you fix this issue?

Thanks.

# May 28, 2012 12:57 PM

Vagif Abilov said:

Hi edi,

Thank you for the bug report. Yes, I managed to reproduce the issue. I'll try to fix it today and let you know.

Vagif

# May 28, 2012 1:06 PM

Vagif Abilov said:

Hi again Edi,

I just uploaded a fix (version 0.4.0.2) that is available both as a NuGet package and MSI installer. Please try.

Best regards

Vagif

# May 28, 2012 2:41 PM

edi said:

Hi Vagif,

Thank you for your quick reaction. It's very promising project.

So I have another problem. I've got error:

"The serialized resource has a null value in key member 'db_id'. Null values are not supported in key members."

My Mongo Db has strong typed collections with the not nullable Id field (int, string or Guid) like this:

public class Shop

   {

       public int Id { get; set; }

       public string Domain { get; set; }

   }

Do I have to do something more?

# May 29, 2012 8:44 AM

Vagif Abilov said:

Hi edi,

You will get a mail from me soon.

Vagif

# May 29, 2012 9:26 AM

Vagif Abilov said:

Oh no, in fact I don't have your mail. May I have your email address so we can bypass blog post comments? You can send it at vagif.abilov@gmail.com

If you have a github account you also open an issue at github.com/.../MongOData.

Regarding your problem. Do I have a small sample of your data or a script (maybe sample code) that creates a database, so I can try it myself? That will be the quickest way for me to test it.

Vagif

# May 29, 2012 9:31 AM

Karl said:

Hi Vagif,

Many thanks for this brilliant and useful bit of essential plumbing, this is coming in very handy for a project I've been working on and it looks like you've put a lot of effort into it...the QueryExpressionVisitor alone is a non-trivial (and formidable) bit of code!

Since you have called the current version "0.5.1", does this imply that you are planning to continue updating this, and if so can you talk a bit about what direction you are thinking about next? I notice that both Bson Arrays and nested (complex) types seem to be unimplemented, and was wondering whether you were giving these any thought. Also curious if you had any thoughts about eTags and optimistic concurrency handling.

I'm guessing you may be on summer holiday in which case hope you are having a good one. Thanks again!

# August 7, 2012 12:14 AM

Vagif Abilov said:

Hi Karl,

Thank you for the feedback. Yes, I am actively working on a project (took some time off due to summer and other activities), so your input is very valuable. In fact I am finishing now complex types and will soon publish a new version. You can follow it on GitHub or check Nuget updates. In case you have some suggestions, I will appreciated if you open issue on GitHub or just mail me to vagif dot abilov at gmail dot com.

Best regards

Vagif

# August 7, 2012 11:40 AM

NIckvh said:

HI quick question, and thank you for this service btw, does the driver provide support for a list within a list within an object, so like

Person{

Orders[]

  Products[]

}

Where Products is a list within the orders list.

thanks

# September 18, 2012 2:07 AM

Vagif Abilov said:

Hi Nickvh,

I am now testing the implementation that supports nested BsonArrays (and your example will use internally BsonArray). I will write a blog post once I am done with it.

Best regards

Vagif

# September 18, 2012 10:16 AM

NIckvh said:

Hi Vagif - if you need some additional testing, please loop me in.

# September 19, 2012 9:29 PM

Vagif Abilov said:

Hi NIckvh,

Sure this can be helpful. I plan to do a release at GitHub and Nuget within a few days.

# September 19, 2012 10:58 PM

Vagif Abilov said:

Hi NIckvh,

I just pushed both Nuget package and installer (at GitHub) for MongOData version 0.6 that supports arrays and nested arrays. Give it a try!

# October 10, 2012 10:03 PM

Vagif Abilov's blog on .NET said:

It’s been a while since I blogged about MongOData – a MongoDB OData provider that I wrote to cross MongoDB

# October 11, 2012 12:08 AM

Gabriel de Fombelle said:

I get the error below every time the constructor is invoked, but the first one.

Config is: VS 2012, Windows 8, IIS Express

Thanks for your help.

"An element with the same key has already been added"

# November 27, 2012 3:32 PM

Vagif Abilov said:

Hi Gabriel,

How are you using MongOData? Did you install it using MSI installer or via NuGet package? Is it the latest version? I'd like to have some more info to understand where the problem lies.

Thanks

# November 27, 2012 3:51 PM

Sean said:

I get the same error as Gabriel, here is a stack trace:

mscorlib.dll!System.ThrowHelper.ThrowArgumentException(System.ExceptionResource resource) + 0x35 bytes

mscorlib.dll!System.Collections.Generic.Dictionary<string,System.Data.Services.Providers.ResourceType>.Insert(string key, System.Data.Services.Providers.ResourceType value, bool add) + 0x2cc bytes

mscorlib.dll!System.Collections.Generic.Dictionary<System.__Canon,System.__Canon>.Add(System.__Canon key, System.__Canon value) + 0xb bytes

DataServiceProvider.dll!DataServiceProvider.DSPMetadata.AddEntityType(string name) + 0xd1 bytes

Mongo.Context.dll!Mongo.Context.MongoMetadata.AddDocumentType(Mongo.Context.MongoContext context, string collectionName, MongoDB.Bson.BsonDocument document, System.Data.Services.Providers.ResourceTypeKind resourceTypeKind) + 0x76 bytes

Mongo.Context.dll!Mongo.Context.MongoMetadata.AddResourceSet(Mongo.Context.MongoContext context, string collectionName, MongoDB.Bson.BsonDocument document) + 0x36 bytes

Mongo.Context.dll!Mongo.Context.MongoMetadata.PopulateMetadataFromCollection(Mongo.Context.MongoContext context, string collectionName, System.Data.Services.Providers.ResourceSet resourceSet) + 0x18e bytes

Mongo.Context.dll!Mongo.Context.MongoMetadata.PopulateMetadata(Mongo.Context.MongoContext context) + 0x102 bytes

Mongo.Context.dll!Mongo.Context.MongoMetadata.MongoMetadata(string connectionString, Mongo.Context.MongoConfiguration.Metadata metadata) + 0x191 bytes

Mongo.Context.dll!Mongo.Context.MongoDataServiceBase<DataServiceProvider.DSPQueryableContext,Mongo.Context.Queryable.MongoDSPResourceQueryProvider>..ctor.AnonymousMethod__1(string x) + 0xa0 bytes

Mongo.Context.dll!Mongo.Context.MongoDataServiceBase<DataServiceProvider.DSPQueryableContext,Mongo.Context.Queryable.MongoDSPResourceQueryProvider>.MongoDataServiceBase(string connectionString, Mongo.Context.MongoConfiguration mongoConfiguration) + 0x162 bytes

Mongo.Context.dll!Mongo.Context.Queryable.MongoQueryableDataService.MongoQueryableDataService(string connectionString, Mongo.Context.MongoConfiguration mongoConfiguration) + 0x34 bytes

> MongODataMvc3.dll!MongODataMvc3.MongoDataService.MongoDataService() Line 12 + 0x3c bytes C#

[Lightweight Function]

System.ServiceModel.dll!System.ServiceModel.Dispatcher.InstanceProvider.GetInstance(System.ServiceModel.InstanceContext instanceContext, System.ServiceModel.Channels.Message message) + 0xb bytes

System.ServiceModel.dll!System.ServiceModel.Dispatcher.InstanceBehavior.GetInstance(System.ServiceModel.InstanceContext instanceContext, System.ServiceModel.Channels.Message request) + 0x18 bytes

System.ServiceModel.dll!System.ServiceModel.InstanceContext.GetServiceInstance(System.ServiceModel.Channels.Message message) + 0x6a bytes

System.ServiceModel.dll!System.ServiceModel.Dispatcher.InstanceBehavior.EnsureServiceInstance(ref System.ServiceModel.Dispatcher.MessageRpc rpc) + 0x2e bytes

System.ServiceModel.dll!System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage41(ref System.ServiceModel.Dispatcher.MessageRpc rpc) + 0x45 bytes

System.ServiceModel.dll!System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage4(ref System.ServiceModel.Dispatcher.MessageRpc rpc) + 0x9f bytes

System.ServiceModel.dll!System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage31(ref System.ServiceModel.Dispatcher.MessageRpc rpc) + 0x59 bytes

System.ServiceModel.dll!System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage3(ref System.ServiceModel.Dispatcher.MessageRpc rpc) + 0x3b bytes

System.ServiceModel.dll!System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage2(ref System.ServiceModel.Dispatcher.MessageRpc rpc) + 0x4e bytes

System.ServiceModel.dll!System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage11(ref System.ServiceModel.Dispatcher.MessageRpc rpc) + 0x118 bytes

System.ServiceModel.dll!System.ServiceModel.Dispatcher.ImmutableDispatchRuntime.ProcessMessage1(ref System.ServiceModel.Dispatcher.MessageRpc rpc) + 0x34 bytes

System.ServiceModel.dll!System.ServiceModel.Dispatcher.MessageRpc.Process(bool isOperationContextSet) + 0xf7 bytes

System.ServiceModel.dll!System.ServiceModel.Dispatcher.ChannelHandler.DispatchAndReleasePump(System.ServiceModel.Channels.RequestContext request, bool cleanThread, System.ServiceModel.OperationContext currentOperationContext) + 0x574 bytes

System.ServiceModel.dll!System.ServiceModel.Dispatcher.ChannelHandler.HandleRequest(System.ServiceModel.Channels.RequestContext request, System.ServiceModel.OperationContext currentOperationContext) + 0x122 bytes

System.ServiceModel.dll!System.ServiceModel.Dispatcher.ChannelHandler.AsyncMessagePump(System.IAsyncResult result) + 0x3f bytes

System.ServiceModel.dll!System.ServiceModel.Dispatcher.ChannelHandler.OnAsyncReceiveComplete(System.IAsyncResult result) + 0x3f bytes

System.ServiceModel.Internals.dll!System.Runtime.Fx.AsyncThunk.UnhandledExceptionFrame(System.IAsyncResult result) + 0x2c bytes

System.ServiceModel.Internals.dll!System.Runtime.AsyncResult.Complete(bool completedSynchronously) + 0x11f bytes

System.ServiceModel.Internals.dll!System.Runtime.InputQueue<System.ServiceModel.Channels.RequestContext>.AsyncQueueReader.Set(System.Runtime.InputQueue<System.ServiceModel.Channels.RequestContext>.Item item) + 0x44 bytes

System.ServiceModel.Internals.dll!System.Runtime.InputQueue<System.ServiceModel.Channels.RequestContext>.EnqueueAndDispatch(System.Runtime.InputQueue<System.ServiceModel.Channels.RequestContext>.Item item, bool canDispatchOnThisThread) + 0x1b0 bytes

System.ServiceModel.Internals.dll!System.Runtime.InputQueue<System.ServiceModel.Channels.RequestContext>.EnqueueAndDispatch(System.ServiceModel.Channels.RequestContext item, System.Action dequeuedCallback, bool canDispatchOnThisThread) + 0x60 bytes

System.ServiceModel.dll!System.ServiceModel.Channels.SingletonChannelAcceptor<System.ServiceModel.Channels.IReplyChannel,System.ServiceModel.Channels.ReplyChannel,System.ServiceModel.Channels.RequestContext>.Enqueue(System.ServiceModel.Channels.RequestContext item, System.Action dequeuedCallback, bool canDispatchOnThisThread) + 0x6b bytes

System.ServiceModel.dll!System.ServiceModel.Channels.HttpPipeline.EnqueueMessageAsyncResult.CompleteParseAndEnqueue(System.IAsyncResult result) + 0x78 bytes

System.ServiceModel.dll!System.ServiceModel.Channels.HttpPipeline.EnqueueMessageAsyncResult.HandleParseIncomingMessage(System.IAsyncResult result) + 0x33 bytes

System.ServiceModel.Internals.dll!System.Runtime.AsyncResult.SyncContinue(System.IAsyncResult result) + 0x32 bytes

System.ServiceModel.dll!System.ServiceModel.Channels.HttpPipeline.EnqueueMessageAsyncResult.EnqueueMessageAsyncResult(System.ServiceModel.Channels.ReplyChannelAcceptor acceptor, System.Action dequeuedCallback, System.ServiceModel.Channels.HttpPipeline pipeline, System.AsyncCallback callback, object state) + 0x7b bytes

System.ServiceModel.dll!System.ServiceModel.Channels.HttpPipeline.EmptyHttpPipeline.BeginProcessInboundRequest(System.ServiceModel.Channels.ReplyChannelAcceptor replyChannelAcceptor, System.Action dequeuedCallback, System.AsyncCallback callback, object state) + 0x30 bytes

System.ServiceModel.dll!System.ServiceModel.Channels.HttpChannelListener<System.ServiceModel.Channels.IReplyChannel>.HttpContextReceivedAsyncResult<System.ServiceModel.Channels.IReplyChannel>.ProcessHttpContextAsync() + 0xe0 bytes

System.ServiceModel.dll!System.ServiceModel.Channels.HttpChannelListener<System.ServiceModel.Channels.IReplyChannel>.BeginHttpContextReceived(System.ServiceModel.Channels.HttpRequestContext context, System.Action acceptorCallback, System.AsyncCallback callback, object state) + 0x5a bytes

System.ServiceModel.Activation.dll!System.ServiceModel.Activation.HostedHttpTransportManager.HttpContextReceived(System.ServiceModel.Activation.HostedHttpRequestAsyncResult result) + 0x126 bytes

System.ServiceModel.Activation.dll!System.ServiceModel.Activation.HostedHttpRequestAsyncResult.HandleRequest() + 0x1d8 bytes

System.ServiceModel.Activation.dll!System.ServiceModel.Activation.HostedHttpRequestAsyncResult.BeginRequest() + 0x1f bytes

System.ServiceModel.Activation.dll!System.ServiceModel.Activation.HostedHttpRequestAsyncResult.OnBeginRequest(object state) + 0x40 bytes

System.ServiceModel.Internals.dll!System.Runtime.IOThreadScheduler.ScheduledOverlapped.IOCallback(uint errorCode, uint numBytes, System.Threading.NativeOverlapped* nativeOverlapped) + 0x72 bytes

System.ServiceModel.Internals.dll!System.Runtime.Fx.IOCompletionThunk.UnhandledExceptionFrame(uint error, uint bytesRead, System.Threading.NativeOverlapped* nativeOverlapped) + 0x32 bytes

mscorlib.dll!System.Threading._IOCompletionCallback.PerformIOCompletionCallback(uint errorCode, uint numBytes, System.Threading.NativeOverlapped* pOVERLAP) + 0x6e bytes

[Native to Managed Transition]

[Appdomain Transition]

[Native to Managed Transition]

# April 10, 2013 12:10 AM

Vagif Abilov said:

Hello Sean,

I need more info to be able to investigate the problem. I just released the version 0.10 of the provider, and all tests there passed. Obviousy this is a new scenario that I haven't encountered before. Can you send me more details on vagif.abilov at gmail dot com or open an issue at GitHub? If you can attach your schema or a test data, that would be perfect.

# April 10, 2013 8:18 AM
Leave a Comment

(required) 

(required) 

(optional)

(required) 


Please add 6 and 5 and type the answer here: