Nathan J Pledger

Kendo UI: Managing data changes on the client and sending it back to ASP.NET MVC

2011-12-14T15:48:36Z

My few regular readers will know that I’m a big fan of Telerik and their products. Their product range is comprehensive, the support top-notch and their evangelism and engagement is second to none. Whilst usually involved in the .NET space from ASP.NET UI components through to testing platforms, they have recently branched out into an HTML5/JavaScript UI suite, called Kendo UI.

The more time I spend with ASP.NET MVC, the more I learn about everything that is actually going on in a web request, which is no bad thing and highlights a fault of my own professional development; depending on ASP.NET WebForms to hide the guts of the HTTP request away from me. In gaining this “missing knowledge”, I’ve developed my JavaScript skills tremendously and as such am able to think up some sweet user experiences.

My current requirement is a simple master->detail data structure and editing experience. Obviously this data-structure should not require the user’s knowledge or understanding but as anyone knows, effectively creating a seamless master->detail scenario without some degree of complexity or unnecessary user steps can be challenging. In playing around with the ASP.NET MVC grid it quickly became apparent that the guts of the grid control was becoming a barrier to rapid understanding and deployment, again, this considering I’m relatively new to MVC and the powered-by-unicorns “convention over configuration” pattern. What if I could rule out all the complex data-structure parsing, AJAX and the like and have the entire editing experience occur on the client?

The Kendo UI suite does provide integration with the server and therefore actual data, but only insofar as facilitating AJAX requests. This is fine, as I’m fairly well versed in AJAX, JavaScript and the MVC pattern now. However, I wanted to avoid having to return to the server for “mini edits” via a full POST or AJAX. So I did not want:

Two steps: one for editing master data, the other for editing individual child data
AJAX updates: updating child data in lieu of a complete POST and onward persistence (this would require some session state of some form which I am keen to avoid as this project is being developed with future cloud installation in mind)

What I needed was complete client-side editing and a single POST.

And it turns out it’s all very simple.

Given the data structure:

I created the view with the KendoUI grid in, the model containing a series of properties that would hold the modifications made on the client and the controller to tie it all together.

View

The grid is easily created, much like a jQuery UI widget:

<div id="valuesGrid"></div>

I wanted to handle everything on the client, so in in initialising my grid, I needed to consider how I was going to do this. I decided to create a few properties in my Model, which would be rendered as hidden input elements in the form element. This would allow me to manipulate items on the client and post back the individual fields as a single POST, processing them on the server.

My fields are below (in Razor syntax):


@Html.HiddenFor(model=>model.OriginalLookupValuesJson)
@Html.HiddenFor(model => model.NewLookupValuesJson)
@Html.HiddenFor(model => model.EditedLookupValuesJson)
@Html.HiddenFor(model => model.DeletedLookupValuesJson)

On the GET, the controller populates the detail records by setting the OriginalLookupValuesJson field to the JSON representation of the data-structure.

On the client-side, these are manipulated by “overriding” the built-in CRUD methods of the Kendo UI grid:


$(document).ready(function () {
    var json = $('input#OriginalLookupValuesJson').val();
    var data = eval('('+json+')');
    var dataSource = new kendo.data.DataSource({
        transport: {
            read: function (options) { 
                options.success(data);
            },
            create: function (options) {
                $("input#NewLookupValuesJson").val(kendo.stringify(options.data.models));
            },
            update: function (options) {
                $("input#EditedLookupValuesJson").val(kendo.stringify(options.data.models));
            },
            destroy: function (options) {
                $("input#DeletedLookupValuesJson").val(kendo.stringify(options.data.models));
            }
        }, //transport
        batch: true,
        pageSize: 4,
        schema: {
            model: {
                id: "ID",
                fields: {
                    ID: {
                        editable: false,
                        nullable: true
                    },
                    LookupSetID: {
                        editable: false,
                        nullable: true
                    },
                    Value: {
                        editable: true,
                        validation: {
                            required: true
                        }
                    },
                    Description: {
                        editable: true
                    }
                } //fields
            }, //model
            data: "data",
            total: function (result) {
                result = result.data || result;
                return result.length || 0;
            } //total
        } //schema
    });

    $("#valuesGrid").kendoGrid({
        dataSource: dataSource,
        editable: true,
        toolbar: ["create"],
        height: 250,
        scrollable: true,
        sortable: true,
        filterable: false,
        pageable: true,
        columns: [
            {
                field: "Value",
                title: "Value"
            },
            {
                field: "Description",
                title: "Description"
            },
            {
                command: "destroy"
            }
        ]
    });

The script implements the following:

Localises the JSON from the OriginalLookupValuesJson field into a variable, which is then evaluated into a JavaScript object
A dataSource object is then configured that takes the evaluated JavaScript object as a source for the read implementation. The create, update and destroy implementations are overridden by setting their output as the previously mentioned hidden fields. The results of which will be passed back into my Controller within the Model.
Initialises and configures the grid with the dataSource variable and the user-interface aspects I require.

Unfortunately, when you submit the form in a POST, the hidden fields used to monitor modifications are exactly as they were when they were requested: blank. You need to synchronise the internal data structures of the grid with the hidden fields by intercepting the form POST by adding the following BLOCKED SCRIPT


$("form").bind("submit", sync);
});

function sync() {
    alert('sync');
    $("#valuesGrid").data("kendoGrid").dataSource.sync();
}

Model

The Model itself is supremely simple and contains the principle object of the Master object (LookupSet) and the four fields used to store the JSON and allow the manipulation of the LookupValues on the client-side and persistence of the changes on the server-side.


public class LookupModel : ModelBase
{
    public LookupSet LookupSet { get; set; }
    public IEnumerable<SelectListItem> DataTypes { get; set; }
    public string OriginalLookupValuesJson { get; set; }
    public string NewLookupValuesJson { get; set; }
    public string EditedLookupValuesJson { get; set; }
    public string DeletedLookupValuesJson { get; set; }

    public LookupModel()
    {
    }

}

I create my models within a Factory class, ModelFactory, which has a method:


public static LookupModel CreateLookupModel(int id)
    {
        LookupModel model = new LookupModel(); 
        using (CoreEntities coreEntities = CoreEntitiesFactory.Create(configuration))
        {
            model.LookupSet = coreEntities.LookupSets.Include("LookupValues").SingleOrDefault(q => q.ID == id);
            JavaScriptSerializer javaScriptSerializer=new JavaScriptSerializer();
            model.OriginalLookupValuesJson = javaScriptSerializer.Serialize(new { data = GetLookupValueModels(model) });
        }
        return model;
    }

The method creates a new LookupModel and interrogates the data-store for the requested data. This is implemented using Entity Framework but obviously can be replaced with your own framework.

In order to have the detail data (LookupValues) processed at the client, it needs to be serialized into JSON format, using the JavaScriptSerializer class. This is called with the result of the method GetLookupValueModels(), which is shown below:


private static List<LookupValueModel> GetLookupValueModels(LookupModel model)
    {
        List<LookupValueModel> lookupValueModels = new List<LookupValueModel>();
        foreach (LookupValue lookupValue in model.LookupSet.LookupValues)
        {
            LookupValueModel lookupValueModel = new LookupValueModel()
            {
                Description = lookupValue.Description ?? string.Empty,
                ID = lookupValue.ID,
                LookupSetID = lookupValue.LookupSetID
            };
            if (model.LookupSet.DotNetDataType == typeof(string).FullName)
                lookupValueModel.Value = lookupValue.System_String;
            if (model.LookupSet.DotNetDataType == typeof(int).FullName && lookupValue.System_Int32.HasValue)
                lookupValueModel.Value = lookupValue.System_Int32.Value.ToString();
            if (model.LookupSet.DotNetDataType == typeof(decimal).FullName && lookupValue.System_Decimal.HasValue)
                lookupValueModel.Value = lookupValue.System_Decimal.Value.ToString();
            lookupValueModels.Add(lookupValueModel);
        }
        return lookupValueModels;
    }

This takes the Entity Framework objects and creates a LookupValueModel for each, returning the result. This is then wrapped up in an anonymous type, which assigns the result to the “data” property. The result of which would be typically:


{
    "data": [
        {
            "ID": 3,
            "LookupSetID": 2,
            "Value": "XXX",
            "Description": "Description for XXX"
        },
        {
            "ID": 4,
            "LookupSetID": 2,
            "Value": "YYY",
            "Description": "Description for YYY"
        },
        {
            "ID": 5,
            "LookupSetID": 2,
            "Value": "ZZZ",
            "Description": "Description for ZZZ"
        }
    ]
}

As this is placed in the model property OriginalLookupValuesJson, this is sent to the client as part of the form, and correctly rendered by the Kendo UI grid component.

Controller

For completeness, the HTTP GET request is implemented by the controller as follows:


public ActionResult EditLookup(int id)
    {
        ActionResult result;
        LookupModel model = ModelFactory.CreateLookupModel(id);
        if (model.LookupSet == null)
        {
            result = View("InvalidLookup", ModelFactory.Create<GenericPageModel>("Invalid lookup"));
        }
        else
        {
            model.Title = string.Format("Edit lookup '{0}'", model.LookupSet.Name);
            result = View("EditLookup", model);
        }
        return result;
    }

But where we are interested in is the POST, which will take the modified values from the hidden form fields and implement the modifications on the data-store. The modifications arrive within the hidden fields as JSON, so it is a simple matter to rehydrate those to usable data that we can work with by deserializing the LookupValues back into CLR objects. I use the Deserialize<T>() method from this StackOverflow post (external link)" href="http://stackoverflow.com/questions/2246694/how-to-convert-json-object-to-custom-c-sharp-object">StackOverflow post:


private static T Deserialize<T>(string json)
    {
        T obj = Activator.CreateInstance<T>();
        MemoryStream ms = new MemoryStream(Encoding.Unicode.GetBytes(json));
        DataContractJsonSerializer serializer = new DataContractJsonSerializer(obj.GetType());
        obj = (T)serializer.ReadObject(ms);
        ms.Close();
        return obj;
    }

Then I call that for each of the modification types in the HTTP POST handler of my Controller (excluding the wider persistence code):


[HttpPost]
[ValidateAntiForgeryToken]
public ActionResult EditLookup(LookupModel model)
{
    ActionResult result = View(model);
    if (ModelState.IsValid)
    {
                    List<LookupValueModel> modifiedLookupValues = Deserialize<List<LookupValueModel>>(model.EditedLookupValuesJson);
                    List<LookupValueModel> createdLookupValues = Deserialize<List<LookupValueModel>>(model.NewLookupValuesJson);
                    List<LookupValueModel> deletedLookupValues = Deserialize<List<LookupValueModel>>(model.DeletedLookupValuesJson);
    }

    return result;
}

How you handle the individual CUD methods is dependent on your persistence layer.

Silverlight and ASP.NET compatible WCF over SSL using custom username/password authentication

2011-01-20T21:12:00Z

WCF, as you probably know, can be a hugely flexible, but equally complicated beast. The domain knowledge stretches from the .NET stack to identity management, encryption and transport. “Out of the box”, it’s pretty straight forward, but the demos and examples often presented by Microsoft are often far from what is needed in real life. Bloggers and authors present excellent examples, but these often fail at the last hurdle where Silverlight compatibility or specific authentication is required. Hopefully, this solution will be a “one-stop shop”.

We needed to be able to deploy software that was loosely coupled via web services, with multiple client platforms. Therefore, we needed:

Secure communications, even though most our work is intranet based, securing the transport of the data was a must so we can scale up onto the internet as required.
Low maintenance, I approach everything pragmatically. If it is over-complicated for us to produce, it will be over-complicated for the user to maintain. Simplicity is king.
Ability to work over internet as well as internet, according to application and scalability.
ASP.NET and Silverlight compatibility, the latter already being stymied due to lack of support of the altogether simpler wsHttpBinding. (Most of the SSL examples being internet-based using wsHttpBinding)
Allow transmission of credentials in each transaction and have those credentials (which are not Windows-based) verified each time.

What seemed like an impossible combination has at last been been achieved (by myself, anyway). After trawling through countless blogs, questions and white-papers, I’m going to share with you how to create a reliable, secure WCF service that may be redeployed across projects, even with the limited capabilities of Silverlight.

Generation of SSL Certificate

If we’re transmitting sensitive data, it should be secure. Sensitive data includes the username and password and WCF reinforces this by requiring you to use security over transport if using username/password authentication.

To create or assign the certificate, you need to use the IIS Server Certificates administration module within the Server tree:

You can then choose to import an existing certificate, or create a self-signed certificate. If you create a self-signed certificate, it is only of use during the development phase and will introduce problems for you (which we can overcome).

I created a self-signed certificate to show you the difficulties and how to get round them.

Creation of the Service

The service code is straight forward. I like to separate the interface from the implementation.

IClientService

using System;
using System.Collections.Generic;
using System.Linq;
using System.Runtime.Serialization;
using System.ServiceModel;
using System.ServiceModel.Web;
using System.Text;
namespace Services.Api.Wcf
{
    [ServiceContract]
    public interface IClientService
     {
          [OperationContract]
           List<int> GetAllClientIDs();
     }
}

The implementation class includes the requirement for ASP.NET Compatibility mode.

ClientService.svc

<%@ ServiceHost Language="C#" Debug="true" Service="Services.Api.Wcf.ClientService" CodeBehind="ClientService.svc.cs" %>

ClientService.svc.cs

using System;
using System.Collections.Generic;
using System.Linq;
using System.Runtime.Serialization;
using System.ServiceModel;
using System.ServiceModel.Web;
using System.Text;
using System.ServiceModel.Activation;
using System.Web;

namespace Services.Api.Wcf
{
    [AspNetCompatibilityRequirements(RequirementsMode = AspNetCompatibilityRequirementsMode.Allowed)]
    public class ClientService : ServiceBase, IClientService
    {
         #region ~ from IClientService ~
         public List<int> GetAllClientIDs()
         {
                ClientRepository clientRepository = ClientRepositoryFactory.CreateForUser(UserName);
                return new List<int>(clientRepository.Clients.Select(q => q.IndexNo));
         }
          #endregion
     }
}

Web.config

The most challenging part of WCF is getting the configuration right.

<system.serviceModel>
     <behaviors>
         <serviceBehaviors>
              <behavior name="sslBehaviour">
                  <serviceMetadata httpGetEnabled="true" httpsGetEnabled="true" />
                  <serviceDebug includeExceptionDetailInFaults="true" />
                  <serviceCredentials>
                         <userNameAuthentication userNamePasswordValidationMode="Custom" customUserNamePasswordValidatorType="Core.WcfUserNamePasswordValidator, Core"/>    </serviceCredentials>
             </behavior>
         </serviceBehaviors>
    </behaviors>
     <bindings>
         <customBinding>
               <binding name="sslCustomBinding">
                    <security authenticationMode="UserNameOverTransport"/>
                   <binaryMessageEncoding />
                   <httpsTransport />               </binding>
          </customBinding>
      </bindings>
    <services>
         <service name="Services.Api.Wcf.ClientService" behaviorConfiguration="sslBehaviour">
              <host>
                    <baseAddresses>
                             <add baseAddress="https://mymachine.domain.local/"/>
                   </baseAddresses>
             </host>
            <endpoint address="" binding="customBinding" bindingConfiguration="sslCustomBinding" contract="Services.Api.Wcf.IClientService"> </endpoint>
            <endpoint address="mex" binding="mexHttpBinding" contract="IMetadataExchange" />
          </service>
      </services>
       <serviceHostingEnvironment multipleSiteBindingsEnabled="true" aspNetCompatibilityEnabled="true" />
</system.serviceModel>

The first element, behavior, defines the SSL configuration of the services that may be attached to it. The important point is the serviceCredentials/userNameAuthentication, which defines the class that will perform the authentication.

Silverlight can only support basicHttpbinding, so it is necessary to define a customBinding. This is defined in the customBinding/binding element, and is set to use the UserNameOverTransport authenticationMode (which requires secure transport) and defines the transport as secure by using httpsTransport.

The service itself is defined in the service element, which ties together the bindingConfiguration and behaviorConfiguration. Additionally, the baseAddresses element defines the address that the service will respond to. Note that this is HTTPS. This will obviously need to be changed when deployed. We use the Web.config transformation feature to achieve this.

The authentication class is shown below and may be as simple or as advanced as you need. Obviously, this is where you verify the authentication against your custom provider.

WcfUserNamePasswordValidator.cs

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.IdentityModel.Selectors;

namespace Core
{
        public class WcfUserNamePasswordValidator : UserNamePasswordValidator
       {
             public override void Validate(string userName, string password)
              {
                    if (userName!="username" && password!="password") throw new FaultException("Invalid username/password");
              }
          }
}

This service may be deployed on the web server and tested using the HTTPS address. As we created our own self-signed certificate, there will be a Certificate Warning, which we’ll ignore for now.

Creation of the Client

First step when working with the client-side code is to point to the service web-site. Clicking on Add Service Reference and enter the HTTPS address. Again, if you are using a self-signed certificate, you will be warned of this.

Click “Yes”, then “OK” to generate the proxy code. Once generated, you’re ready to use it. This adds a configuration such as the section below:

<system.serviceModel>
       <bindings>
            <basicHttpBinding>
                 <customBinding>
                         <binding name="CustomBinding_IClientService">
                                <security defaultAlgorithmSuite="Default" authenticationMode="UserNameOverTransport" requireDerivedKeys="true" securityHeaderLayout="Strict" includeTimestamp="true" keyEntropyMode="CombinedEntropy" messageSecurityVersion="WSSecurity11WSTrustFebruary2005WSSecureConversationFebruary2005WSSecurityPolicy11BasicSecurityProfile10">
                                        <localClientSettings cacheCookies="true" detectReplays="false" replayCacheSize="900000" maxClockSkew="00:05:00" maxCookieCachingTime="Infinite" replayWindow="00:05:00" sessionKeyRenewalInterval="10:00:00" sessionKeyRolloverInterval="00:05:00" reconnectTransportOnFailure="true" timestampValidityDuration="00:05:00" cookieRenewalThresholdPercentage="60" />
                                        <localServiceSettings detectReplays="false" issuedCookieLifetime="10:00:00" maxStatefulNegotiations="128" replayCacheSize="900000" maxClockSkew="00:05:00" negotiationTimeout="00:01:00" replayWindow="00:05:00" inactivityTimeout="00:02:00" sessionKeyRenewalInterval="15:00:00" sessionKeyRolloverInterval="00:05:00" reconnectTransportOnFailure="true" maxPendingSessions="128" maxCachedCookies="1000" timestampValidityDuration="00:05:00" />
                                        <secureConversationBootstrap />
                                </security>
                                 <binaryMessageEncoding maxReadPoolSize="64" maxWritePoolSize="16" maxSessionSize="2048">
                                        <readerQuotas maxDepth="32" maxStringContentLength="8192" maxArrayLength="16384" maxBytesPerRead="4096" maxNameTableCharCount="16384" />
                                 </binaryMessageEncoding>
                                 <httpsTransport manualAddressing="false" maxBufferPoolSize="524288" maxReceivedMessageSize="65536" allowCookies="false" authenticationScheme="Anonymous" bypassProxyOnLocal="false" decompressionEnabled="true" hostNameComparisonMode="StrongWildcard" keepAliveEnabled="true" maxBufferSize="65536" proxyAuthenticationScheme="Anonymous" realm="" transferMode="Buffered" unsafeConnectionNtlmAuthentication="false" useDefaultWebProxy="true" requireClientCertificate="false" />
                           </binding>
                     </customBinding>
              </bindings>
              <client>
                      <endpoint address="https://api1test.mydomain.local/api/wcf/ClientService.svc" binding="customBinding" bindingConfiguration="CustomBinding_IClientService" contract="ClientEndpoints.Clients.IClientService" name="CustomBinding_IClientService" />
              </client>
         </system.serviceModel>
</configuration>

Before we do, I would recommend you use this segment of code originally developed by Damien McGivern, which will aid in the catching of exceptions. As Damien mentions on his site, the WCF stack doesn’t “break” where you’d expect, making diagnosis difficult.

The ServiceHelper class acts as a Factory for generation of proxy objects.

ServiceHelper.cs

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.ServiceModel;

namespace AspNet
{
       public static class ServiceHelper
      {
           /// <summary>             /// WCF proxys do not clean up properly if they throw an exception. This method ensures that the service proxy is handeled correctly.
            /// Do not call TService.Close() or TService.Abort() within the action lambda.
            /// </summary>
            /// <typeparam name="TService">The type of the service to use</typeparam>
            /// <param name="action">Lambda of the action to performwith the service</param>
            public static void Using<TService>(Action<TService> action) where TService : ICommunicationObject, IDisposable, new()
            {
                   var service = new TService();
                  bool success = false;
                  try
                  {
                      action(service);
                     if (service.State != CommunicationState.Faulted)
                      {
                          service.Close();
                          success = true;
                      }
                  }
                 finally
                  {
                      if (!success)
                       {
                           service.Abort();
                       }
                  }
            }
      }
}

So you can create the reference to the client proxy and use the service using code such as:

Default.aspx.cs

using System;
using System.Collections.Generic;
using System.Linq;
using System.Web;
using System.Web.UI;
using System.Web.UI.WebControls;
using System.Configuration;
using System.Net;
using System.Net.Security;
using System.Web.Security;

namespace Web.Client
{
     public partial class _Default : System.Web.UI.Page
     {
          protected void Page_Load(object sender, EventArgs e)
          {
                 // this delegate may be used to ignore SSL/TLS errors
                 // it disables SSL certificate path verification for the AppDomain
                  // Make sure you remove it before production!
                  ServicePointManager.ServerCertificateValidationCallback = new RemoteCertificateValidationCallback( delegate { return true; });
                 ServiceHelper.Using<ClientServiceClient>(client =>
                        {
                               client.ClientCredentials.UserName.UserName = "username";
                                client.ClientCredentials.UserName.Password = "password";
                               int[] clientIDs=client.GetClientIDs();
                        });
             }
        }
}

Note the delegate that I have highlighted. Without this delegate, a self-signed certificate will cause an exception, preventing communication with the server. This delegate is called during this verification and overrides the default behaviour. As this essentially skips the verification of certificates, it shouldn’t make it to a production deployment. (See The Joy of Code for the source of this information)

This configuration should now work. One piece of the puzzle remains; how to identify which user is accessing the web methods. This may be achieved using the following code segment, which can be added to the service methods:

string userName=OperationContext.Current.IncomingMessageProperties.Security.ServiceSecurityContext.PrimaryIdentity.Name;

There are a few gotchas and tips I learnt during this work which you might find helpful.

Useful bits

Make sure that the server and client times are within the time span specified in the maxClockSkew attribute. If the time difference between the machines is greater than this time span, verification will fail. Frustratingly, Googling/Binging for the exception returned by WCF returns 99.9% related to this issue – you’ll probably need to look deeper to find out what actually causes your issue if your clocks marry up. Of course, if you’re deploying an intranet application, you lose control here.

WCF rarely provides useful exceptions. Remember to add the following configuration segment to your server configuration and use the WcfTraceViewer tool within the Windows SDK to diagnose issues. It is invaluable!

<system.diagnostics>
          <sources>
                  <source name="System.ServiceModel" switchValue="Information, ActivityTracing" propagateActivity="true">
                            <listeners>
                                      <add name="traceListener" type="System.Diagnostics.XmlWriterTraceListener" initializeData= "D:\dev\Services\App_Data\Logging\Wcf\log.svclog" />
                           </listeners>
                  </source>
           </sources>
</system.diagnostics>

Remember that Silverlight won’t communicate across domains to your web service, so you may need a clientaccesspolicy.xml or crossdomain.xml policy on your web site.

Testing Entity Framework 4

2010-06-12T10:30:00Z

Entity Framework has previously been a real pain in the backside to unit test and this slowed my own development of unit testing skills. It’s so much quicker to “just get it done” when deadlines loom. The new Entity Framework 4 promised POCO (Plain-old-CLR-objects) support which allows us to separate our concerns from the underlying database, just what you need to “mock out” the database.

I’ve spent hours trawling through Microsoft’s own recommendations and others’, but I just couldn’t find a model that was actually usable or reasonable. That said, I have drawn heavily from Microsoft’s own solutions:

My complaints for the above solutions were two fold. Firstly, they (and indeed the practicalities of implementing an abstracted Repository pattern) seem to push you towards strongly typing your Repository to a particular class within your domain. So, you would have a CustomerRepository, an InvoiceRepository, etc. These would be either strongly typed or implemented through Generics, such as Repository<TEntity>. Patterns often don’t map well to “real life”, and I’m a great fan of real life. In real life, objects interact with other objects and restricting Repositories by Types isn’t going to help this interaction. Secondly, the UnitOfWork pattern seemed superfluous and it just ended up with added levels of abstraction that I don’t feel were worth the lines of code.

This post will detail how I have implemented a common pattern that is easily unit testable/mockable. It is not a panacea for all, though. Deadlines and commercial constraints enforce a degree of the Duct-tape programmer in my work and I have had to make compromises. While I would have liked to mock the concrete data objects themselves and handle through interfaces (eg. ICustomer), this was not possible due to casting through IObjectSet<T>, so I’ve settled on using the generated concrete data objects instead. Therefore, the references to the Entity Framework DLLs (and therefore database reliant code) is required – though not used for database access.

Here’s how I did it, do let me know if you have any comments on my implementation …

I’ve prepared a diagram to try and illustrate the method:

The Concrete Entity Framework is a standard configuration, and is abstracted away by the IRepositoryObjectContext interface. This allows the development of a side-by-side Testing Context, using Entity Framework-generated POCO classes, but using the same concrete generated classes as the standard Entity Framework. Client applications (including Test harnesses, etc.) access the appropriate context via the Repository, which is injected with the appropriate instance of IRepositoryObjectContext. The Business/ORM objects are implemented as interfaces and presented to the client application as interfaces, not as the original classes. This does pose problems in hydration of objects through WCF, etc. but Entity Framework doesn’t deal well with that anyway, so DTOs are still the order of the day.

First, I created a new Solution. This would represent the end application in reality, but just to test this, it’s just for the purpose of illustrating/testing the methodology. The Solution contains the “business” layer (“PatternsAndPractises.TestableEntityFramework”), the client application (“PatternsAndPractises.TestableEntityFramework4.Web.Client” – in this case an ASP.NET Web Application) and the Test project (“PatternsAndPractises.TestableEntityFramework4.Tests” – using the Test Project template).

The database is stored within the App_Data folder of the web application, which is fairly typical. I’ve created a single Customers table in it.

The Customers table has a primary key of integer set to identity increment and a string (VARCHAR/NVARCHAR) field CustomerName.

The next step is to integrate Entity Framework into the project. I created a Data folder (and therefore namespace for new files) that will contain the persistence-layer objects and contexts. Inside here, I created the Entity Model by using “Add New Item” and selecting “ADO.NET Entity Data Model”, and following the wizard to extract the schema from the database, select the database created earlier and add the tables. I’ve now got a [simplistic] model. Opening the generated “code-behind” within the Model.Designer.cs file reveals the class used to manage the entities, TestDatabaseEntities (which will be derived from the settings you chose while you were configuring the wizard).

This generated class is important as it provides the means by which we can extend the basic generated implementation to abstract away the database-code. To achieve this, we create a new class, make it partial and apply the same class name.

In addition, we make it implement the interface IRepositoryObjectContext. The interface doesn’t exist yet, but this will give us the abstraction layer we need to present different entity contexts to our repository later.

To create the interface, IRepositoryObjectContext, we’ll add a new interface at the project level (ie. above the Data folder/namespace) and define the implementation:

This interface needs to expose the bare minimum required for the Repository to be able to interact with the data-store (physical or otherwise). The CustomersSet is used to abstract away the implementation of the set used by the Entity Data Context class (but not the generated concrete data class). This provides all the services a Repository would expect from an Entity Framework API. The SaveChanges method provides persistence of changes to objects. This interface also implements IDisposable, which allows the use of the using() {} block for automatic resource clean-up even though we’re abstracting away the Entity Framework API. We don’t need to implement the IDisposable interface, the generated code from Entity Framework does this for us.

This abstracts-away the implementation detail of Entity Framework, but it still relies on the concrete generated data classes (for example, Customer) and some awareness that the application is using Entity Framework. This is where the Repository class comes in.

using System;
using System.Collections.Generic;
using System.Linq; 
using System.Text;
using System.Data.Objects;
using PatternsAndPractises.TestableEntityFramework4.Data;

namespace PatternsAndPractises.TestableEntityFramework4 
{
    public class Repository : IDisposable
    {
        protected IRepositoryObjectContext RepositoryObjectContext 
		{ get; private set; }

	#region ~ Constructors ~

	public Repository(
		IRepositoryObjectContext repositoryObjectContext)
        {
            RepositoryObjectContext = repositoryObjectContext;
        }

	#endregion

	public ICustomer CreateCustomer()
        {
            ICustomer newCustomer = new Customer();
            newCustomer.CustomerName = "New Customer";
            return newCustomer;
        }

	public void AddCustomer(ICustomer customer)
        {
            RepositoryObjectContext
		.CustomersSet.AddObject((Customer)customer); 
        }

	public void SaveChanges()
        {
            RepositoryObjectContext.SaveChanges();
        }
	
	public IQueryable<ICustomer> Customers
        {
            get
            {
                return 
			(IQueryable<Customer>)
			RepositoryObjectContext.CustomersSet;
            }
        }

	#region ~ from IDisposable ~

	public void Dispose()
        {
            RepositoryObjectContext.Dispose();
        }

	#endregion
    }
}

This class is the point at which the abstraction of the Entity Framework is implemented, by virtue of injecting the IRepositoryObjectContext implementing class into the constructor. We’ve only got one of these at the moment, being the actual Entity Framework generated class, coupled with our partial extension and implementation of IRepositoryObjectContext.

The Repository uses the IRepositoryObjectContext to implement the standard CRUD functionality of adding, saving and querying data. The CreateCustomer() method simply creates an object, and does not persist it. I like the idea of “default objects” to be creatable then sent to the user interface which can then provide additional logic such as undo-steps, etc. The class provides the Customers of the underlying Entity Framework via the CustomersSet property, but cast to an IQueryable<ICustomer> set. This removes the additional functionality provided by the Entity Framework 4 IObjectSet<T> class and casts it to the ICustomer interface, effectively making the set read-only. The ICustomer interface will abstract away the concrete data class, Customer. Finally, we implement IDisposable, which propagates the disposal upto the repository and finally, the Entity Data Context.

To complete the circle, the ICustomer class means that when the client application works with Customers, it always works with ICustomer and therefore doesn’t need to involve itself with the concrete generated data classes. The ICustomer interface defines the properties and access permissions for individual fields, you wouldn’t want the primary key to be set, for example:

In order that the generated Customer concrete class implements the ICustomer interface, we create a partial class, Customer:

This essentially completes the business object layer, which is immediately usable by a client application. The project would look like this:

Notice the RepositoryFactory class. This simple creates an instance of the Repository class, injecting the appropriate data context as an implementation of the IRepositoryObjectContext interface:

Adding Testability

The testing portions of the solution will be added to the PatternsAndPractises.TestableEntityFramework4.Tests project, which will attempt to mirror the business layer in all respects except for the persistence of data.

In the same way as the business layer, we shall separate our classes into persistence and logical layers by adding the Data folder/namespace.

Within this folder we need to mimic the Entity Framework API, which is made possible by the POCO Template for Entity Framework. Downloading this and installing into Visual Studio allows the creation of T4 templates according to the schema already defined in the Entity Model we created earlier, and it’s easily done.

Going back to the Entity Model defined earlier, right-click on the design surface (not on an object) and select “Add Code Generation Item”.

Once prompted, select ”ADO.NET POCO Entity Generator” and name the item, for example “MockableModel”.

The T4 templates will be created, which in turn auto-generates the concrete classes.

You won’t need the Model templates or classes, so you can remove these from the project (highlighted in red).

Before going any further, the action of generating these POCO classes disables the default code generation of Entity Framework, which needs to be re-enabled. (Opening the generated .designer.cs file illustrates this by a comment.) This may be re-enabled by opening the Model, clicking on the design surface and using the Properties pane to set the Code Generation Strategy back to “Default”. Saving the model will regenerate the underlying classes.

Clearly, the mock templates/code are related to the mocking/testing of the solution so would be more appropriately placed within the PatternsAndPractises.TestableEntityFramework4.Tests project so can be moved to live within the Data folder within the test project.

This will create a problem in that the T4 template needs to locate the source schema provided by the Entity Model, which is now unavailable. This can be corrected by opening the MockableModel.Context.tt file (or whatever you have called yours) and locating the line where the inputFile is set and changing it to the relative location of the .edmx file.

(If your editor doesn’t display with the helpful colouring, download the Tangible T4 Editor – it makes it a lot easier)

Saving the file will cause the MockableModel.Context.cs file to be re-generated with the concrete classes defining ObjectSet accessors for each class/table. These will be referencing the concrete generated classes defined in the data-layer – a weakness in this model.

A further change that needs to be made to the template is to weaken the ObjectSet references to the IObjectSet<T> type to allow abstraction. In order to add the IObjectSet<T> type, the System.Data.EntityClient namespace needs to be added. The changes are highlighted in the screenshot below, in green and indicated by arrows as the template colouring reduces identification.

In the same way that we extended the functionality of the original Model context, we create a similar partial class for the MockableModel class (which actually refers to the local TestDatabaseEntities class). Notice that this is also implementing the IRepositoryObjectContext interface defined earlier. This is the testable/mockable object context. We have to create this within a separate folder, to prevent the generated file from being overwritten. I created a “Partial” folder within Data, making sure that “Partial” did not become part of the namespace – which would prevent the partial compilation from working.

While this essentially completes our obligations in terms of implementing the IRepositoryObjectContext interface, it doesn’t yet work. The CustomersSet is created on demand by the generated code, as shown in the snippet below, from MockabelModel.Context.cs:

However, the ObjectSet doesn’t actually retain any objects added to it, so we need to create an in-memory persistence layer that will essentially perform the same as the database persistence layer of the original Entity Framework. The article Testability and Entity Framework 4 defines the InMemoryObjectSet class:

using System.Linq;
using System.Collections;
using System.Collections.Generic;
using System.Linq.Expressions;
using System;
using System.Data.Objects;

public class InMemoryObjectSet : IObjectSet where T : class
{
	private readonly HashSet _set;
    private readonly IQueryable _queryableSet;

	public string Name { get; set; }

	#region ~ Constructors ~

	public InMemoryObjectSet()
		: this(Enumerable.Empty())
	{
	}

	public InMemoryObjectSet(IEnumerable entities)
	{
		_set = new HashSet();
		foreach (var entity in entities)
		{
			_set.Add(entity);
		}
		_queryableSet = _set.AsQueryable();
	}

	public InMemoryObjectSet(string name)
		: this()
	{
		Name = name;
	}

	public InMemoryObjectSet(string name, IEnumerable entities)
		: this(entities)
	{
		Name = name;
	}

	#endregion


	#region ~ from IObjectSet ~

	public void AddObject(T entity)
	{
		_set.Add(entity);
	}
	public void Attach(T entity)
	{
		_set.Add(entity);
	}
	public void DeleteObject(T entity)
	{
		_set.Remove(entity);
	}
	public void Detach(T entity)
	{
		_set.Remove(entity);
	}
	public Type ElementType
	{
		get { return _queryableSet.ElementType; }
	}
	public Expression Expression
	{
		get { return _queryableSet.Expression; }
	}
	public IQueryProvider Provider
	{
		get { return _queryableSet.Provider; }
	}
	public IEnumerator GetEnumerator()
	{
		return _set.GetEnumerator();
	}
	IEnumerator IEnumerable.GetEnumerator()
	{
		return GetEnumerator();
	}

	#endregion

}

This class uses an instance of a HashSet<T> to persist data, which produces contextless and fast storage without dependency on state or implementation of a database.

In order to implement the addition of items to the various sets, in this case Customers, I overrode the CreateObjectSet<T> method by using “new” within the TestDatabaseEntities class, which now looks like:

This skips the original behaviour and instead creates an InMemoryObjectSet of T to ensure a container is actually created for storage of objects.

As the actual Entity Framework code is called in the mocking of the Entity Framework API, there is a dependency on the connection string – but only for configuration purposes, no connection is made. Therefore, we need to copy the App.config file from the business layer project “PatternsAndPractises.TestableEntityFramework4” to the Test project “PatternsAndPractises.TestableEnityFramework4.Tests”.

The project now looks like:

Now would be a good point at which to confirm everything compiles.

We’re now ready to write our tests. As with the data layer, we need to be able to invoke the Repository class and inject our Entity Context, which implements IRepositoryObjectContext. I find creating a base class (“TestBase”) for tests works just as well as the Factory method used previously:

The CreateRepository method creates the Repository, injecting the [local] instance of TestDatabaseEntities, or IRepositoryObjectContext.

Finally, to write the tests, we’ll create a customer and save it, then check the persistence store to ensure it has been stored. Create a new Unit Test class and add the following tests:

		[TestMethod]
		public void CreateAndAddCustomer()
		{
			string expectedCustomerName = 
				string.Format("New Customer ({0})", Guid.NewGuid());
			using (Repository repository = CreateRepository())
			{
				ICustomer customer = repository.CreateCustomer();
				customer.CustomerName = expectedCustomerName;

				repository.AddCustomer(customer);
				repository.SaveChanges();

				ICustomer retainedCustomer = repository.
					Customers.
					Where(q => q.CustomerName == expectedCustomerName).
					FirstOrDefault();
				Assert.IsNotNull(retainedCustomer);
			}	
		}

This test will create a customer (but not store it), add it to the persistence layer (in this case, an in-memory HashSet), save the changes and use the IQueryable<T> interface to verify the operation worked.

The full Solution (exlcuding the Web UI) would look like:

Like I said, it’s not perfect, nor is it 100% isolated, but I think it strikes a healthy compromise between testability and the ability to concentrate on the job in hand, rather than worrying about minutia of patterns and practises. If you have any comments, do post them.

I’ve added a sample project to download to provide a usable sample. Download the Visual Studio 2010 Solution as a ZIP file.

Silverlight: Changing the synchronous mindset

2009-11-25T17:09:00Z

Working with web sites, web pages, web services and web clients has until recently been very much a synchronous operation, which is despite the very asynchronous nature of the web. Synchronous programming techniques are linear in operation in that process A cannot execute until process B has completed. The very nature of the internet is that not only do you not know when you'll get a response from a server, you may not even get it at all, which can be particularly important when you don't own the server as many mash-up web applications do not. If you adopt a synchronous model, such as basic JavaScript or relying solely on the brower's threading model to load a page, you are likely to get freezes or missing portions of your web page. Asynchronous programming allows Process B to run alongside Process A. The two processes can run one after another, in any order, if at all. As a programmer, you need to work around the many headaches this can produce. Although, the experience for the end user is much improved as a result. Moving from a synchronous model to an asynchronous one can often be a challenge if you've never experienced this model previously.

I'm currently working in Silverlight, which forces users to use an asynchronous programming model, as per the NPAPI specification. This means that although a Silverlight component is very likely to need external data, it cannot rely on its timely execution or even a result. The traditional procedural programming pattern becomes redundant. If you have a complex and large data structure at your server, you certainly do not want to pull it all down to the client when you only need to use 5% of it. Likewise, you don't want to have to rely on lots of "little calls" to get the 5% of data on an "as needed" approach as the user navigates through the application.

The perfect example (which is a real example), is that of a Security Index. If you have a complex Security Index that identifies the permissions a user is has on actions and data, it would be tempting to have some code such as:

   1:  Permissions permissions=Security.GetPermissionsForUserOnItem(int itemID);

In a desktop environment, you could certainly get away with having that call as a function, which can either dig into an in-memory data structure, consult a database or even a web service. You can be reasonably sure that you will get a timely result, particularly for an Intranet application. This is not entirely reliable, and is certainly not recommended for developing over the internet using a platform such as Silverlight or similar client/server applications.

In a client/server environment, the GetPermissionsForUserOnItem() call would need to call back to a web service which may not respond as and when you expect. As Silverlight uses threading internally to generate calls to WCF web services, execution will fall out of the function almost immediately, without a result because the result will still be on its way back from the web server. Take the following code:

(Based on the excellent article by David Betz at http://www.netfxharmonics.com/2008/11/Understanding-WCF-Services-in-Silverlight-2)

   1:  public Permissions RequestPermissionsForItem(int itemId)

   2:  {

   3:       Permissions permissions=0;

   4:       ISecurityBroker securityBroker = UIContext.CreateWcfInterface<ISecurityBroker>("Security/SecurityBroker.svc");

   5:       // Point A: call to the web service

   6:       securityBroker.BeginGetPermittedActionsForDataItemAndUser(clientId, dataTypeId, dataItemId, result =>

   7:            {

   8:                 // Point B: This will only execute on response from the web service

   9:                 permissions = ((ISecurityBroker)result.AsyncState).EndGetPermittedActionsForDataItemAndUser(result);

  10:             }, securityBroker);

  11:        }

  12:        // Point C: Execution will immediately follow Point A, skipping Point B

  13:        return permissions;

  14:  }

This uses the following method, which in the example above in in the UIContext static class, which creates the web service (and represents my own code usage, you may have a more suitable mechanism):

   1:  public static class UIContext

   2:  {

   3:       public static TWcfApiEndPoint CreateWcfInterface<TWcfApiEndPoint>(string serviceUrl)

   4:       {

   5:            // create the binding elements

   6:            BinaryMessageEncodingBindingElement binaryMessageEncoding = new BinaryMessageEncodingBindingElement();

   7:            HttpTransportBindingElement httpTransport = new HttpTransportBindingElement() { MaxBufferSize = int.MaxValue, MaxReceivedMessageSize = int.MaxValue };

   8:            // add the binding elements into a Custom Binding

   9:            CustomBinding customBinding = new CustomBinding(binaryMessageEncoding, httpTransport);

  10:            // create the Endpoint URL

  11:            EndpointAddress endpointAddress = new EndpointAddress(serviceUrl);

  12:            // create an interface for the WCF service

  13:            ChannelFactory<TWcfApiEndPoint> channelFactory=new ChannelFactory<TWcfApiEndPoint>(customBinding, endpointAddress);

  14:            // channelFactory.Faulted += new EventHandler(channelFactory_Faulted); - not implemented in this example

  15:            TWcfApiEndPoint client = channelFactory.CreateChannel();

  16:            return client;

  17:       }

  18:  }

The comments illustrate that the web service call will be executed within its own thread (within the anonymous function), so Point B would not necessarily execute before point C. The result of permissions, therefore, will be 0. This rules out calls to functions as we've come to expect them to work.

There are ways to simulate synchronous behaviour, but as mentioned in Peter Bomberg's post from 2008, they are not at all a solution to an asynchronous problem. Classes such as ManualResetEvent and AutoResetEvent provide locking to prevent execution from a given point until a flag as been set on the object. The above code would become as follows:

   1:  public Permissions RequestPermissionsForItem(int itemId)

   2:  {

   3:       Permissions permissions=0;

   4:       ISecurityBroker securityBroker = UIContext.CreateWcfInterface<ISecurityBroker>("Security/SecurityBroker.svc");

   5:       // Point A: call to the web service

   6:       using (AutoResetEvent autoResetEvent=new AutoResetEvent(false))

   7:       {

   8:            securityBroker.BeginGetPermittedActionsForDataItemAndUser(clientId, dataTypeId, dataItemId, result =>

   9:            {

  10:                 // Point B: This will only execute on response from the web service

  11:                 permissions = ((ISecurityBroker)result.AsyncState).EndGetPermittedActionsForDataItemAndUser(result);

  12:                 // Point C: Sets a flag to signal that the result is ready

  13:                 autoResetEvent.Set();

  14:            }, securityBroker);

  15:            // Point D: Execution waits until the Set() method executes, signalling the result is ready

  16:            autoResetEvent.WaitOne();

  17:      }

  18:      return permissions;

  19:  }

This works - sort of. However, mixing this with user-interface based code, which runs inside its own thread, can soon result in threading deadlocks which will cause your application to stop ... dead. These are always difficult to resolve and fix in an elegant way. I often think, if it can't be fixed elegantly, you're probably doing it wrong.

So the problem remains; how can we request small bits of information from a potentially huge data source held behind a web service from a Silverlight (or similar) application? The challenge in this is not how to simulate a synchronous pattern, but how to match an appropriate callback pattern to your code.

Back to the example, we can provide an event, that is called when the result is ready. This will mean that the function will return void, and it causes added headaches at the consumer side of the function because the caller will need to wire-up an event handler, which synchronous patterns avoid. So we can change the code to:

   1:  public class PermissionsBroker

   2:  {

   3:       public event EventHandler<RequestPermissionsForItemEventArgs> PermissionsForItemReady;

4:

   5:       protected virtual void OnPermissionsForItemReady(RequestPermissionsForItemEventArgs e)

   6:       {

   7:            if (PermissionsForItemReady != null) PermissionsForItemReady(this, e);

   8:       }

9:

  10:       public void RequestPermissionsForItem(int itemId)

  11:       {

  12:            Permissions permissions=0;

  13:            ISecurityBroker securityBroker = UIContext.CreateWcfInterface<ISecurityBroker>("Security/SecurityBroker.svc");

  14:            // Point A: call to the web service

  15:            securityBroker.BeginGetPermittedActionsForDataItemAndUser(clientId, dataTypeId, dataItemId, result =>

  16:            {

  17:                 // Point B: This will only execute on response from the web service

  18:                permissions = ((ISecurityBroker)result.AsyncState).EndGetPermittedActionsForDataItemAndUser(result);

  19:                OnPermissionsForItemReady(new RequestPermissionsForItemEventArgs() { ItemID = itemId, Permissions = permissions });

  20:            }, securityBroker);

  21:      }

Here, I have created a class, RequestPermissionsForItemEventArgs (not shown, but the properties are simple enough) that represents a means of passing a message back to the caller, with the permissions, when ready.

So the caller could:

   1:  PermissionsBroker permissionsBroker=new PermissionsBroker();

   2:  permissionsBroker.PermissionsForItemReady+=new EventHandler<RequestPermissionsForItemEventArgs>(SecurityCache_PermissionsForItemReady);

   3:  permissionsBroker.RequestPermissionsForItem(500);

4:

   5:  void SecurityCache_PermissionsForItemReady(object sender, RequestPermissionsForItemEventArgs e)

   6:  {

   7:       // results, act on permission here

   8:  }

That is a lot of lines just to check permissions. While this works, we need to reduce the headache for the consumer of the functionality by reducing the amount of plumbing they need to create just to get a simple value back from the server. By using an anonymous function, we can simplify this.

First, we use a delegate to allow us to pass a function as a parameter to the function. This will represent the callback operation:

   1:       public delegate void RequestPermissionsForItemDelegate(SecureDataActions secureDataActions);

We add this to our function:

   1:       public void RequestPermissionsForItem(int itemId, RequestPermissionsForItemDelegate requestPermissionsForItemDelegate)

   2:       {

   3:            Permissions permissions=0;

   4:            ISecurityBroker securityBroker = UIContext.CreateWcfInterface<ISecurityBroker>("Security/SecurityBroker.svc");

   5:            // Point A: call to the web service

   6:            securityBroker.BeginGetPermittedActionsForDataItemAndUser(clientId, dataTypeId, dataItemId, result =>

   7:            {

   8:                 // Point B: This will only execute on response from the web service

   9:                permissions = ((ISecurityBroker)result.AsyncState).EndGetPermittedActionsForDataItemAndUser(result);

  10:                requestPermissionsForItemDelegate(permissions);

  11:            }, securityBroker);

  12:      }

And using an anonymous function (using a lambda), we can call it and get the result in just 5 lines of code:

   1:  RequestPermissionsForItem(itemID, permissions =>

   2:       {

   3:               bool canWrite = permissions!=0;

   4:       }

   5:  );

The important point to remember is that the code executing within the anonymous function is a in its own scope, and therefore cannot pass its result out of the anonymous function other than by setting a higher-scoped variable (but again you need to consider asynchronous behaviour) and that it will not be executing on the UI thread so will need to be invoked from the dispatcher accordingly.

The not-so-obvious benefits of Localising your Strings

2009-08-03T10:27:00Z

In developing the next major version of our major product line, I made sure we re-visited every aspect of the application. In a sense, the existing team had to make a valid argument for every feature to be included in the product. One aspect that was not previously considered was internationalisation. Being able to localise your strings to "foreign" languages and format your numbers to an international audience is a key benefit and opens our product up to a massive user-base in the future. Aside from the obvious features that .NET localisation offers, such as multi-lingual and culture-specific formatting, working through your product and using localisation features offers many other benefits which are useful even if you want to stick to "International English".

Reduce language clutter - by focussing on identifying the labels to apply to controls, you minimise your language clutter. When talking about writing files to disk, a Save Filename label could appear as "Save File as:", "Save Filename:", Save as:" or even "Save FileName". This provides an extra level of discipline to ensure terms of reference are consistent, particularly across a team of developers.

DRY (Don't Repeat Yourself) Principle - If a section of code ends up being "duplicated", we extract it into a searate function according to the DRY principle, but this can also apply to text. By re-using your term definitions you make updating your terms simpler. An example is when working around a particular product feature which ma be codenamed while in development, but will be rebranded when Marketing decide how to push the feature to the user-audience. Re-branding across the site in this way becomes much easier than a Search/Replace. This is also very useful for spelling mistakes!

Reduce security threats - Using the asp:Localize control in ASP.NET to display the localised text can provide an additional layer of security protection by using the Mode="Encode" attribute, which reduces the risk of script injection and therefore, cross-site scripting attacks for a site. Even if not referencing a localised term, the asp:Localise control is a must for displaying content from the user or the database. Make it discipline and you can delegate your protection of script injection - to a point.

Lower the technical barrier - by franchising out your strings to what is essentially a text file, you can have your Marketing team or translators modify the file. This is known, but it also allows you to modify the file seperately from the code itself, while potentially keeping the file within source control.

These are all pretty obvious when you think about it, so much so that I'll probably work towards localising all my applications in the future. For the sake of a slight change in how you do things, it's worth the effort in consistency, maintainability and safety.

WCF -> JSON Serialization woes and a solution

2009-03-18T10:13:00Z

I'm working on improving the performance of my current web application project at various points. As I had already planned for a WCF interface for third-party use, I thought I'd utilise that, exposing objects as JSON-serialized strings usable by jQuery/ASP.NET AJAX.

Little did I realise that when I was coming up with my Data Framework (I chose Entity Framework) I should have thought about aspects at the other side of my project such as the User Interface as well as the more obvious aspects of scalability, performance in relation to a pragmatist view of what is needed in the "real world" (therefore, nHibernate is out).

My WCF service had an exposed service that would return a Client object (actually, an interface IClient) and return it as JSON. Simple.

No, I hit a number of issues with this.

1/ Investigations at StackOverflow indicated that Interfaces cannot be exposed through serialization. Makes sense, really, but dents my idealistic view of presenting interfaces and no conncrete objects to third-parties via my API. So I was going to have to either return my concrete object or reduce my return value to a JSON string, thereby bringing serialization "in house" and not relying on WCF to serialize it for me.

From:

[OperationContract]
[WebInvoke(Method="POST",BodyStyle=WebMessageBodyStyle.Wrapped,ResponseFormat=WebMessageFormat.Json)]
IClient GetClientJson(int clientId);

I went to:

[OperationContract]
[WebInvoke(Method="POST",BodyStyle=WebMessageBodyStyle.Wrapped,ResponseFormat=WebMessageFormat.Json)]
string GetClientJson(int clientId);

2/ This actually dodged the issue of the attribute configuration on the service itself, which was becoming a nightmare. Tweak it at the server and it breaks at the client, and vice versa. What was:

From:

[OperationContract]
[WebInvoke(Method="POST",BodyStyle=WebMessageBodyStyle.Wrapped,ResponseFormat=WebMessageFormat.Json)]
string GetClientJson(int clientId);

I went to:

[OperationContract]
[WebInvoke(Method="POST",BodyStyle=WebMessageBodyStyle.Bare,ResponseFormat=WebMessageFormat.Json)]
string GetClientJson(int clientId);

3/ Next was serializing the object into JSON. I was getting the "The type 'xxx' cannot be serialized to JSON because its IsReference setting is 'True'. The JSON format does not support references because there is no standardized format for representing references. To enable serialization, disable the IsReference setting on the type or an appropriate parent class of the type." exception. As I was essentially getting this at the client, it suggested my WCF endpoint configuration was wrong. Looking deeper, it turns out that Entity Framework objects are marked with IsReference=True, meaning the native DataContractJsonSerializer of WCF cannot serialize Entity Framework objects. I proved this by doing a manual serialization:

string jsonClient;
IClient client = GetClient(7);
DataContractJsonSerializer ser = new DataContractJsonSerializer(client.GetType());
using (MemoryStream ms = new MemoryStream())
{
ser.WriteObject(ms, client);
jsonClient = Encoding.Default.GetString(ms.ToArray());
}
return jsonClient;

4/ I needed to serialize using a different serializer, so thought I'd use the ASP.NET AJAX Serializer, which also didn't work, this time falling over the exception "A circular reference was detected while serializing an object of type xxx'." The type it was complaining about wasn't in the object so it was clearly navigating deeper into other objects to find that particular Type anyway.

5/ So now I am left with no other option but to either do it myself or use a third-party library. I'm using Json.NET, which I spotted on Scott Hanselman's blog and seems to be robust enough and simple enough for most purposes. So my code now looks like:

string jsonClient=null;
IClient client=GetClient(1);
JsonSerializer jsonSerializer = new JsonSerializer();
jsonSerializer.Converters.Add(new JavaScriptDateTimeConverter());
jsonSerializer.NullValueHandling = NullValueHandling.Ignore;
jsonSerializer.MissingMemberHandling = MissingMemberHandling.Error;
jsonSerializer.ReferenceLoopHandling = ReferenceLoopHandling.Error;
try
{
   using (StringWriter sw = new StringWriter())
   {
   using (JsonTextWriter jtw = new JsonTextWriter(sw))
   {
   jsonSerializer.Serialize(jtw, client);
   }
   }
}
catch (Exception ex)
{
   ex = ex; // have a breakpoint here so can inspect exception
}
return jsonClient;

Notice I have set ReferenceLoopHandling to ReferenceLoopHandler.Error. This is to try and catch the same Reference Count issue that ASP.NET AJAX JSON Serialization catches. (Actually this was added after realising I had StackOverflows occurring). Sure enough, I have another Reference Count issue as the Exception does get caught and the error is related to possible infinite recursion.

The JSON.NET Framework allows me to disable serializing potentially problematic objects, but this would require applying these changes to elements of code "Bhind the wall" of my API - and essentially add Web-specific functionality into a domain that is supposed to be platform agnostic. This is not an option for me.

So I appear to be stuck. Other than rendering the JSON myself through a StringBuilder, I'm pretty much stuck on this now. Maybe something will hit me in a flash of inspiration. Until then, it's good ol' StringBuilder for me.

Update: The Solution:

Thanks again due to StackOverflow, (John Saunders and Craig Stuntz) I've figured out how I'm going to do it. It's not as pretty as I would have liked, but pragmatism wins out again.

Here it is from start to finish. My UI generates an event that is picked up by some JavaScript. This runs:

var wcfProxy = new serviceProxy("../api/wcf/ClientBroker.svc/");
wcfProxy.invoke("GetClientJson", { clientId: 7 }, updateClient, updateClientError);

serviceProxy is RickStrahl's WCF wrapper, which has two callbacks, success and error (the last two parameters, respectively). The Invoke method invokes the WCF service and obtains the result. The WCF service is exposed via the Interface:

[ServiceContract(Namespace = "xxxWCF")]
public interface IClientBroker
{
   [OperationContract]
   [WebInvoke(Method="POST",BodyStyle=WebMessageBodyStyle.Wrapped,ResponseFormat=WebMessageFormat.Json)]
   string GetClientJson(int clientId);
}

Note that the Body STyle is Wrapped. I couldn't get it working in Bare mode at all. (Despite telling me it had logged messages in the Windows Event Log, no events were to be found).

[AspNetCompatibilityRequirements(RequirementsMode = AspNetCompatibilityRequirementsMode.Allowed)]
public class ClientBroker : IClientBroker
{
   public string GetClientJson(int clientId)
   {
   string jsonClient=null;
   IClient client = GetClient(clientId);
   var j = new { ID=client.ID, BusinessName=client.BusinessName };
   JavaScriptSerializer s = new JavaScriptSerializer();
   jsonClient=s.Serialize(j);
   }
   return jsonClient;
}

Notice I am using Anonymous Types to create a new type which is free of Entity Framework idiosyncracies.

This generates a wrapped JSON string:

{"GetClientJsonResult":"{\"ID\":7,\"BusinessName\":\"XYZ Ltd\"}"}

When returned to the client, the success callback is called:

function updateClient(o) {
eval('var z=' + o');
alert('BusinessName=' + o.BusinessName);
}

And the Business Name of the requested client is displayed.

It just goes to show that Microsoft may put all the features they like into C#, and you might very well know they are there. But you're not going to use them until you need to use them, and then you need to know that you need to use them! Variant types are, to me, an uncomfortable throwback of those bad VB days but sometimes they can prove useful when it comes to the crunch, I'm just keen on restricting my use of them to the absolute minimum so the very premise of a type-strong language is not lost.

Another stone passed.

WCF Exception: "There can be at most one address per scheme in this collection"

2009-03-16T11:39:00Z

I'm beginning to see why people dislike WCF.

I'm working on using WCF to feed an AJAX TreeView, and demos of it are FAST. However, in my debugging on my dev server I bumped into the exception:

"There can be at most one address per scheme in this collection"

The posters at http://social.msdn.microsoft.com/Forums/en-US/wcf/thread/9e248455-1c4d-4c5c-851c-79d9c1631e21/#page:1 seem to be most vocal about the definiciencies of WCF in this respect - and they'd be right. What were Microsoft thinking? Most web applications need to respond to at least two domains to overcome the vagaries of user-inputted URLs, eg:

www.myapp.mysite.com
mapp.mysite.com
myapp

Are all potentially valid patterns for accessing an internal web application. We shouldn't need workarounds to get this to work.

Removing the extra host header record fixes the issue. Surely they weren't expecting people to write their WCF services under a separate, discrete URL, like wcf.myapp.mysite.com?

Telerik: radControls: Resizable Splitter

2009-02-24T13:11:00Z

For my current project, I am working on what is quite a complex user interface. I need to develop an interface that has splitter controls and can load one or more modules within each panel. Sort of like Microsoft Outlook three-panel interface. Sort of like the example below:

The screenshot shows how 3 "modules" have been loaded, one in each Pane.

I am using Telerik's RadSplitter control, which is part of their RadControls for ASP.NET AJAX and provide a great suite of components you can just drop into your project for rapidly prototyping and developing applications for the web.

The Splitter control itself is able to resize itself intelligently, so expanding the left column will automatically resize the content in the two right-hand panes. But there are two things it does not do:

It does not (by default) operate within a "100% browser" environment - you need to encourage it to do so
While resizing content within adjacent panes to a resized pane is pretty good, some intelligence may need to be applied when resizing richer controls such as Grids, where the data itself may be effected.

To solve the first problem, you just need to make sure your RadSplitter is set to use the full browser dimensions as such:

<telerik:RadSplitter ID="MainSplitter" runat="server" Height="100%" Width="100%" HeightOffset="35" Orientation="Horizontal" Skin="Outlook">

</telerik:RadSplitter>

According to the established W3C guidelines, 100% is only 100% of a physically defined height of the calculated height of all containers of the item. The Splitter works exactly the same, so unless you are able to pixel-perfect resize your Splitter, you need to apply some hackery to get your Spliter to behave. Telerik show how to achieve this so I won't repeat them, but if you are having problems, make sure that EVERY container is able to resolve to a pixel-defined height. I have found that across browsers, it isn't always consistent. Therefore, I have the following jQuery to try and force the issue:

function stretchSplitter() {
   var newWindowHeight = $(window).height() - 110;
   $("#formPage").css("height", newWindowHeight);
   $("#<%=radSplitter.ClientID %>").css("height", newWindowHeight); }

$(document).ready(function() {
   $(window).bind("resize", resizeWindow);
   function resizeWindow(e) {
   stretchSplitter(); }
   });
stretchSplitter();

This is using the standard jQuery onLoad trick to resize the splitter once the DOM is ready. I'm taking the 110 pixels off to adjust to the current offsets I need to take account of. This could be used to tweak designs across browsers.

The next problem is how to have the contents of each pane resize according to the pane's new size once resized. Mostly, this isn't an issue. Designing your content around a liquid layout should allow content to resize using standard CSS flowing. If displaying data, or particularly complex designs within one of the pane areas, however, it becomes necassary to finely control positioning.

In the screenshot above, I have a radGrid, which requires resizing to allow its VirtualScrolling to operate correctly. I'm actually trying to create a similar effect illustrated at the older radGrid demo. So resizing the pane will cause the grid to require resizing, which itself needs to go back to the server to rebind its rows. What I needed to do, however, was separate the functionality so that the usercontrol loaded into the pane doesn't know of its position in the Splitter. It may be loaded once (or more) in any Pane/location in the Splitter. So I couldn't "bind" myself to any events exposed by the host Splitter.

Thanks to jQuery (and my understanding of it as opposed to pure JavaScript), this was possible, if a little hacky.

I decided to use an <A> tag to act as an event proxy. This <A> tag would recieve a standard "onclick" event, and repeat it to any subscribers. I created a single <A> tag:

Then, I attached an event to the RadSplitters OnClientResized event:

function radSplitter_OnClientResized(sender, args) {
   var v=$('a#eventProxy').trigger(
   type:'click',
   subType:'splitterPaneResize',
   resizedPanelID:sender.get_id(),
   oldWidth:args.get_oldWidth(),
   oldHeight:args.get_oldHeight(),
   newWidth:sender.get_width(),
   newHeight:sender.get_height() } );

When the RadSplitter pane is resized, this event is called, which creates an 'event' object and Triggers the event with type on the element found by the jQuery 'a#eventProxy'. As type is 'click', it will essentially call any handlers attached to the onclick event of the <A> tag.

Any handlers attaching to the onclick event would be dynamically built, so I couldn't just add them to the onclick="" attribute of the <A> tag. Again, I used jQuery to achieve this. In every Module that was interested or concerned with having to resize itself according to the new size of the content, I added the following jQuery:

$(document).ready(function() {
   $('a#eventProxy').click(
   function(event) {
   switch (event.subType)
   {
   case 'windowClosed':
   var ajaxMgr=$find("<%=AjaxManager.ClientID %>");
   if (ajaxMgr) {
   ajaxMgr.ajaxRequest("RebindUsersGrid");
   }
   break;
   case 'splitterPaneResize':
   if (event.resizedPanelID=='<%=ParentRadPane.ClientID %>') {
   sizeUsersModule(event.newHeight); }
   break;
   }
   });
});

Again, I'm using the jQuery onLoad technique to ensuring my DOM is ready. Note that I have a switch, which shows that I can re-use the same <A> tag for multiple types of event. I have two cases here, one for closing windows and one for performing the resizing of the Users Module - determined by splitterPaneResize. Note that I am checking to ensure the pane I am going to resize is the one I am interested in - otherwise, your panes will interact with each other. The sizeUsersModule is shown below:

function sizeUsersModule(newHeight) {
   var ajxMgr=<%=AjaxManager.ClientID %>;
   if (ajxMgr!=null && ajxMgr.ajaxRequest!=null)
   ajxMgr.ajaxRequest('resizegrid_Height='+newHeight);
   }

This simply takes the new height and calls the ajaxRequest client-side method of the RadAjaxManager. This causes a callback to the server, which runs the server-side C#:

Hook up my event in Page_Load():

AjaxManager.AjaxRequest += new RadAjaxControl.AjaxRequestDelegate(AjaxManager_AjaxRequest);

... and the event:

private void AjaxManager_AjaxRequest(object sender, AjaxRequestEventArgs e)

{

if (e.Argument.StartsWith("resizegrid_Height="))

{

string newHeightAsString=e.Argument.Substring("resizegrid_Height=".Length);

if (newHeightAsString.EndsWith("px")) newHeightAsString = newHeightAsString.Substring(0, newHeightAsString.Length - "px".Length);

int newHeight=int.Parse(newHeightAsString);

radGridUsers.Height = newHeight- 65;

}

In this code, I am parsing out the value (removing the 'px' which browsers such as Firefox include) and again using a known offset to be able to accurately size the grid. The grid is rebound and posted back.

Note that you would need an AjaxManager on the page to be able to achieve the AJAX elements of this, which I define programmatically:

RadCodeBlock radCodeBlock = new RadCodeBlock();
radCodeBlock.ID = "radAjaxManagerCodeBlock";
RadAjaxManager _radAjaxManager = BuildRadAjaxManager();
radCodeBlock.Controls.Add(_radAjaxManager);
Form.Controls.Add(radCodeBlock);

It's in a RadCodeBlock because it modifies the DOM so avoids the Exceptions associated with this.

And in each Module that is interested in Resizing, I have to hook up some AJAX settings (AjaxManager below is the same as _radAjaxManager above):

AjaxManager.AjaxSettings.AddAjaxSetting(AjaxManager, radGridUsers);

AjaxManager.AjaxSettings.AddAjaxSetting(radGridUsers, radGridUsers);

This just sets up the grid (radGridUsers) to use AJAX with the AJAX Manager and itself (as it tells itself to rebind).

I'd be interested to know if I have gone way out on the wrong path on this one, but it seems to be hanging together very well across all the browsers. This is very much thanks to the cross-browser functionality of Telerik's controls and jQuery.

Isle of Man BCS: Test+Behaviour Driven Development

2009-01-14T08:18:00Z

Last night saw an impressive attendance at January's BCS event, on Test Driven and Behaviour Driven Development. This is possibly an indication of what people want from their BCSin terms of subject and engagement with the content.

The talk by Zakir Hoosen, from Fuzzelogic Solutions was very interesting, and proved to introduce the principles behind an increasingly widespread method of development to those who would otherwise not have had the opportunity to see it being used. This included myself. The concept of "Test Driven Development" was alien to my previous job, not because I didn't want to write good code, but because the organisation failed to engender the opportunity to spend time on code improvement processes. After seeing that while most people in the room were developers, only 5 or so actually used these methodologies, it made me feel slightly better that while my career had suffered as a result of the rut I had found myself I certainly wasn't on my own. In the "real world", it isn't always possible to find the time to look into design and test methodologies, let alone have the skills and time to implement them and surrounding processes.

Testing aside, a couple of interesting design principles came out of the discussion (and it was a discussion), which align with my own thoughts around best practice:

Keep inheritance to a minimum, use Interfaces extensively. Whenever I expose any form of object (via an API or otherwise) to a public facing consumer, I always use an Interface. Not only does it provide benefits of abstraction, but it also helps in inversion of control, dependency injection, etc. It's also a great way to clarify in your own mind the true purpose of a particular class. All classes have their helper methods, but those implementing an interface are seen to be the "meat" of the class.
Name your classes around the pattern that it is using. As mentioned above, I use Interfaces extensively, and you can't instantiate an interface (as such), so this is a great opportunity to implement Factory patterns and dependency injection. If you have a class that creates instances of an Interface, call it "Factory". For example, in one project I am working on, I have a ClientFactory, which produces IClients. It does what it says on the tin. Other patterns which naturally lend themselves well to forming part of the class name are Mediation ("ClientSearchMediator" for managing searching for Clients), Wrapping ("OutlookWrapper" for wrapping a component you can't necassarily change, for example, Microsoft Outlook) and Proxy (WCFProxy for when a class simply proxies calls between two objects or services, which may or may not include some form of translation). Unfortunately, some of the principles adopted within the .NET Framework and Visual Studio's automatic code generation don't always lend themselves to this style of naming, in particular, instead of having a ButtonObserver for the Observer pattern, you tend to get a button_Click method within your presentation class (be it a Windows Form or an ASP.NET Web Form).

The content of the talk was clearly welcomed by the group, much of whom were developers. As a developer, on an island, any opportunity I can get to see techniques I might not have been exposed to is welcome. Owen asked if we would like to see not only more sessions of this type, but also small "code camp" sessions where a developers collect and work around a topic, hammering out code together. This is a great idea. As many schemes in the US, whereby programmers congregate from all over the country to attend intensive code-athons, this has many advantages. As an island resident, I get to meet my peers in other businesses, learn about methodologies I haven't necessarily been involved in and be able to implement with guidance of someone who has already traversed the learning curve. While I was particularly pleased that my platform of choice (and language, I would sit there cringing if it was in VB.NET rather than C#!) was being used as the basis of the presentation of test/behaviour driven design techniques, this might not be so easy if we all "jump in" with our own preferences. We all have our own language, platform and coding conventions and standards so this may present a challenge, but also an opportunity to improve our platforms/coding conventions in the workplace as a result of seeing how others do it.

While this technical content was a welcome change, I do not want to see a 100% technical syllabus. I would still like to see "softer" subjects, such as the previous talks we have had on Search Engine Optimisation, social knowledge using Wikis, etc. This presents an opportunity to maybe create an additional technical channel to complement the softer content and encourage more technical professionals to attend, which can only be a good thing for the BCS as not only do members increase, but also professionalism is shared amongst peers which is what the BCS is all about.

Telerik OpenAccess ORM Mapper : Lessons Learnt

2008-12-22T11:22:06Z

When looking at developing new applications I always end up developing my own data layer. Largely because I do not like the dictatorial design principles spouted by the likes of ActiveRecord, nHibernate, et al. I write them according to real life situations where database design isn't as clean as it perhaps should be and there is no opportunity to make key changes, etc. And you know what, rolling your own is so much easier and much less hassle.

For my current project, though, I thought I'd try something "new" ...

I have been looking at Telerik's new OpenAccess ORM application recently. Telerik acquired the software a while back and have introduced it into their excellent value Premium RadControls product suite. The Express version is a good way to investigate if it is for you, so long as you are using it on a "free" database server such as SQLExpress, SQLLite, etc.

I thought I'd jot down some quick lessons I have learnt along the way.

Templates

My coding style does not match Telerik's code style as generated from the OpenAccess Reverse Engineering option. Whereas Telerik's generated form of a private member is in the format:

private string siteName;

My preferred style is:

private string _siteName;

There are few things more frustrating than adopting someone elses coding style, so I set about investigating how to get around this.

There are a number of templates that define how the classes are built when you Reverse Engineer, which are stored in the C:\Program Files\Telerik\OpenAccess ORM\sdk\IDEIntegrations\templates\PCClassGeneration folder (or your equivalent).

To change the private member format, you need to open all the files and look for a reference to the "$fieldName" variable. I changed this to "_$fieldName" and it generated the correct code.

It did not, however, generate the correct mappings, which are stored in the reversemapping.config and App.config files, which I had to tweak manually. I still have a forum post outstanding on this one.

Deferred Execution

If you get an "IObjectScope is closed" exception, you'll probably be doing something intelligent like creating a nice Factory Design Pattern, whereby you get your object within the IObjectScope and return it out for use by your BIZ objects. OpenAccess execution is deferred, however, which means if you access any of the properties outside of the IObjectScope context, you get the exception. So, for my factory method that Gets a Site object:

        public object Get(Guid id)
        {
            Site site = null;
            using (IObjectScope scope = Database.Get(_session.ConnectionId).GetObjectScope())
            {
                scope.Transaction.Begin();
                using (IQueryResult result = scope.GetOqlQuery("SELECT * FROM SiteExtent").Execute())
                {
                    if (result.Count > 0) site = (Site) result[0];
                    string s=site.SiteName;
                }
                scope.Transaction.Commit();
            }
            return site;
        }

Note that I access the site.SiteName property and discard it. This just forces OpenAccess to perform the Get and put the Site object into memory so when I drop out of the method, I don't lose the object.

Using LINQ

Telerik say their Open Access solution is a mid point between LINQ to SQL and Entity Framework. Seems good to me, but there are a few gotchas you need to know about.

As with anything in the OpenAccess ORM (unless you are working with SQL directly), you don't work with tables, you work with Extents. The non-LINQ way of doing things int he code sample above shows the OQL query "SELECT * FROM SectionExtent". The table name is Section, but OpenAccess maps it into an Extent. The same model works when using LINQ. Before you can start using Extents in LINQ, you need to add the Telerik.OpenAccess.Query namespace.

If you're using of the Telerik products, you might want to follow @telerikbuzz on Twitter for useful information and updates.

JavaScript Hall of Shame

2008-12-16T17:00:00Z

Here's another pet hate in web bad-practice: Javascript. Javascript is a client-side scripting language that allows really useful interaction to occur with the page and the browser, thereby improving the end user experience when using a site. It's very easy to insert Javascript in a page, not so easy to do it right.

I despair when I look at many people's sites, blogs and networks due to JavaScript errors. It really spoils the experience for me. If I follow a link to a site with the promise of interesting content, I do not want to be presented with a series of JavaScript errors in order to get to that content. I'll just leave. Over the past couple of weeks, (since 1st December to be exact), I have been collecting some of the sites that have JavaScript errors on them (in Internet Explroer 7):

Rik Talavera's site http://riktalavera.com/
Google Reader http://www.google.com/reader
Chris Brogan's Newsletter Page
Facebook (of course) http://www.facebook.com
TwitPic (on their home page!)
TechCrunch (prepare to be annoyed)
Bizarrely, a page on Javascript Exception Handling (see screenshot below)

JavaScript errors can occur due to a number of problems, which bloggers/webmasters aren't always bothered about. From the most basic blog with a Twitter feed to advanced sites such as Facebook, the result of a Javascript error is the same: unwanted behaviour often accompanied by error messages that mean nothing to the end user.

There are a number of excellent resources for getting JavaScript snippets to add to your own site, many of them contributed to by "real" Javascript programmers more capable of handling Javascript code than me. Unfortunately, however, differing scripting dialects and runtimes mean that sometimes these scripts haven't been thoroughly tested on all browsers.

The increase in social networking, "widgets" and advertising services such as Google Adwords results in external Javascript being injected into the page in order to facilitate a particular requirement. For example, on the home page of my site is a Twitter script that gets my Twitter status from Twitter and injects it into the Document Object Model (DOM) of my home page. It's really neat. But if Twitter fails (as it regularly does) or the JavaScript doesn't work with my browser for whatever reason, I will get a JavaScript error - and it will be beyond my control. If possible, it's a good idea to pull the scripts off the remote servers and host them locally so you can react better in the case of an error.

As JavaScript essentially runs in accordance with the page load cycle, it requires that the programmer has made resources available in the right place and in the right order for scripts to run flawlessly. If the page load is disrupted somehow, then errors may be caused which could remove functionality from the page. For example, if I load invoke a script in the HEAD of a page, which relies on a control in the BODY, the control would not be rendered when the script executes, thereby generating an error. A simple error, but one which is often forgotten about particularly as network speeds get faster. It becomes easier to assume the user's page loads as fast as yours. If a script resource gets "held back" over the network for whatever reason, preceding scripts could easily fail.

There is a definite trend to making the web browser work more like a desktop application with every new web application released. Google is particularly focused on the RIA (Rich Internet Application) which uses techniques such as AJAX to create the impression of a responsive and fully functional application within the web browser. Unfortunately, the script behind these applications is bound to be very complex and errors easily creep in. Take Google Reader, it doesn't work on my copy of IE7. Take GMail, it regularly fails on IE7. The complexity of the applications is no excuse. Windows is a complex product but users will (rightly) criticise it and damn the product when it fails to work. The same should go for web sites that use faulty JavaScript. Users of Sitecore will know the powerful Content Editing environment that runs using extensive Javascript - it works fine. The only time I get JavaScript errors are when I cause them by fiddling with my browsing context (eg. Internet -> Intranet zones), etc.

I can almost hear the cries, "use Firefox 3.1" and "use Google Chrome", due to their JavaScript optimisation. Why should I? I use a browser that is easy to access and is on every machine I touch - that is IE 7, which occupies the majority of the browser market. If you write an web/JavaScript application, it should work on ALL browsers, within reason.

By "All Browsers", I mean:

Internet Explorer 7
Internet Explorer 6
Firefox 2
Firefox 3
Safari
Google Chrome

You'll notice that I have not included beta versions of browsers (although I do test my work on these when I believe it is relevant - ie. close to release) and I have gone back a version for the two main browsers, Internet Explorer and Firefox. If your Javascript doesn't work in all these browsers, or degrade gracefully, you really shouldn't release the code.

The JavaScript error message is a jarring experience and spoils many a site. The most attractive site can be ruined by a JavaScript error or two. Just because your browser places a little yellow triangle on the status bar indicating an error has occured in the page, doesn't mean other users have adopted the same configuration. The average user, who has stumbled on a page with a scrpt error, would not understand why certain aspects of the page are unresponsive as a result of any errors that have occured.

My view on JavaScript best practice is: not to use it. If you have to use it, make sure it is error checked and gracefully degrades functionality if required. Also, make sure that alternative functionality is available if the functionality it provides is important to the use of the site. I would struggle to find any reason for JavaScript to be on any "front-end" site, whether it is a blog, corporate site or e-Commerce outlet. Javascript should be a complement to a well designed site, which should add value to the user experience. Tickers, counters, advertisements, etc. are not essential to the content on a page.

There are few things more satisfying in using a web page than when a subtle visual cue is provided to indicate an action has been actioned. Facebook has a number of scripts that allow you to click on text to edit it and then apply it, for example, when updating your status. It's all happening client-side, with minimal AJAX. It works really well - when JavaScript is available and the script works. If for whatever reason JavaScript isn't available (maybe the browsing platform doesn't support it), there is the alternative of typing it in (in most cases) and clicking 'Post'.

Also, remember that search engines will ignore JavaScript. As I always say, your biggest user is deaf, dumb and blind, and that is the search engine. You have to treat the search engine as if it is a base-level browser as a first step of Search Engine Optimisation (SEO), then adding on additional optimisations both for SEO and user experience, which means JavaScript gets added last and does not prevent loading of the page by search engines. I have seen sites that load the next page in the navigation sequence using JavaScript and while it is all very clever in technique, it is not clever in practice. The search bots are getting cleverer all the time and there are rumours of automated JavaScript execution engines that attempt to mimic a user, but these cannot be gauranteed and therefore you should not wrap content only within JavaScript.

If I can, I always look for tried and tested components, rather then develop my own Javascript. I know my limitations, and Javascript isn't a strong point. But when I need it, I'll make sure it works, or look for a component I know will work and will have support behind it. That is why I use Telerik's RadControls for ASP.NET AJAX for situations where I need complex client-side functionality, possibly to create an RIA experience. I can pay for the components, know that the testing has already been done and assuming I do my work right, should just be able to "drop in" the components and tweak accordingly.

If you are working with JavaScript, good practice is to turn on JavaScript errors in your browser. That way, no matter what rush there is to launch a site, a script error will always be important - because it will directly annoy you everytime it happens. I turn on script debugging and error messages as part of my web work as standard, and it pays off in my site quality.

Update (23 December 2008):

This has been a popular post, and I'd like to thank Jim Connolly for his feedback. After reading this post, he checked the page about Advertising on his site and all seems well now. It's good to see that we share the same views on web site quality, as he was very prompt about addressing the issue and contacting me about it.

I'll keep this list up to date and will name and shame sites that fail to trap JavaScript issues. I encourage you to do the same, if you find a site you're struggling to use, feel free to leave a comment and I'll add it to my list.

All hail Google Chrome?

2008-09-02T22:20:04Z

All hail Google Chrome! Another venture from the megalopolis that is Google Corp.

Google Chrome is a new browser from Google. It has long been rumoured that they were getting closer to the browser space with each new service they offered, particularly with their alliance with Firefox. The thinking was that they'd re-badge Firefox, not bring out their own! But even Google didn't expect to launch it so soon, as the release is actually earlier than they had wanted due to a ***-up in an email by one of their senior staff.

I worry about this browser release. I see it as another way to add to their increasingly threatening hold on the corporate desktop with their below standard, half-functional and eternally-in-beta product line-up published direct from below an as yet to be discovered volcano at Mountain View. Okay, maybe I'm a little too cynical.

I have a number of issues with this browser.

Privacy is key for me, and while they may have a mantra "do no evil", that's not to say no evil is done. While they may or may not be tracking my search behaviour "to optimise and improve" my results (thanks, but I do know how to use Google operators myself to refine my searches), I am not confident that they won't be recording everything I do in their browser and aligning that with my Google Mail, my Google Search and my Google Docs to send me Google Adverts. My colleague quite rightly said that if it was open source, then such data "leaks" would be spotted by the community. But how "open source" is anything Google do? Certainly not their search engine algorithms, search appliances, etc. Android was billed as open source, but it turns out it has proprietary elements which will remain closed.

The Google web experience is increasingly JavaScript-based and, to give them their due, this was largely a reliable system. Now, however, they seem to have either gone beyond where JavaScript was meant to go or are letting their QA standards drop. Their Google Mail service, which I now feel I am a prisoner of - because it is so good, is now slow and buggy in its latest release. Google have announced they will develop their own JavaScript virtual machine that will *compile* what is an interpreted language. This will improve the Optimisation for their own applications, certainly, but at what cost for others vendors' applications? Can we be sure they won't do the old alleged Microsoft trick of putting in a few "undocumented" features? Will the "Google Experience" connected to the Google applications/GMail, etc. become limited to the Google browser, in isolation, or with a limited feature set in more popular browsers, which given their stranglehold of the 'web desktop' or 'cloud', could be anti-competitive? Having tried it on the JavaScript elements of the new www.dukevideo.com site, it FLIES. I'm looking forward to seeing how Sitecore performs.

Chrome also adds to the browser fragmentation which I thought we were getting on top of. If the various browsers didn't quite agree on things, at least the number of hacks and tweaks required for consistent operation across browsers is significantly reduced from those Netscape/IE 4 and 5 days. The key browsers today are Microsoft's Internet Explorer, Mozilla's Firefox and Apple's Safari. Opera, Konquerer and a few others nibble at the edges but even Opera only accounts for less than 1% of browser users on any of the sites I manage. (Opera is very good on mobile phones, though) The top three browsers all sort of agree on things, though there is a lot of improvements that could be made to all three to bring them closer to web standards compliance. IE is probably beyond help as it is coded to deal with poorly written pages, so has now cornered itself to some extent for a good while to come. Firefox, by common consensus, is the closest of all three to web standards. Safari uses WebKit, which Chrome has also started to use. Is there any undertaking to leave the WebKit "as is" without tweaking for their own corporate behaviour, maybe to tweak how their applications work to the further cost of web standards compliance?

Firefox is getting too large, too corporate and too centralised to be truly open source, and is losing open source fans and followers due to their increasingly centrist and controlling attitude. Regular readers will know I'm no fan of open source in a commercial environment, where accountability and contracts are important when developing a reliable and maintainable platform from which to do business. Firefox, however, is a good example of where open source has succeeded. If Google's Chrome is going to be open source, don't they fall in to the same trap as Firefox finds itself with the increasingly centrist Mozilla Foundation?

Will Google respect XHTML in their browser (and finally in their code?) They certainly don't in most of their on-line "applications". On their home page at www.google.com, they still use the <FONT> tag. Companies of any self-respecting web presence have now at least started deprecating their use of these tags in favour of the XHTML standard - it being more accessible and search-engine friendly. If they would have kept their horrid HTML3/4 markup on their own sites it would be okay, but they do insist on pushing it to respectable web sites via their IFRAME Google Ads. Hopefully, the "community" will encourage them to clean up their act.

What I do like about it is that it will partition its tabs from each other. This is ideal for me, because as a web developer, I often end up with multiple IE windows open, with their own set of tabs and sometimes a Firefox or two open. As I run fairly demanding web applications through them (Google Mail, Facebook, Sitecore), they do tend to freeze or crash - which in the supposed on-line world where all my work is in the "cloud", is intensely annoying - as I would lose any work I have been doing in those browsers when I have to forcibly quit. Seems an obvious idea, to me, to be honest.

The dragging of tabs outside of the browser area is very neat. I am disappointed in Microsoft not implementing this, to be honest. You can drag Internet Explorer's tabs around, but not out of the browser to create a new window.

Chrome also comes with a neat little debugger, which every other browser lacks. You always have to install 3rd party add-ons or extensions on to others browsers. It's "in the box" with Chrome and so far, I'm quite impressed with it. It is a lot more responsive than Firebug.

Having just tried my GMail in it, I have to say I like the more passive prompts for remembering passwords, etc. These appear at the top of the window, and don't block your browsing experience as happens with IE and Firefox, which always annoys as I often move between machines and it takes me 5 minutes to configure my browsing profile (never remember passwords, don't remind about redirects to secure sites, etc.).

Funnily enough, I thought this was quite satisfying when I wanted to find out about the Crash Control feature:

Sitecore: Accessing Hierarchical Node structures from SQL

2008-07-29T17:58:41Z

I'm often presented with the need to process nodes in Sitecore, which as they are hierarchical, can prove difficult to do other than via the Sitecore API. While the Sitecore API is excellent, it is a little cumbersome to create the environment when you only want the API for a single purpose. (eg. configuration files, required resources in the VS project, etc.). It's also quite slow when traversing down your node structure.

I've been working on a neat marketing channel to output the best selling products in a particular department within an e-Commerce site. So, in the "Bikes" department, I want to see the best sellers in that department, not any others and to do this I need to work from a particular node in Sitecore (representing the "Bikes" department) and work down on a JOIN from previously purchased items.

SQL is intrinsically not recursive, but we found the following SQL which might be of use:

WITH nodes (id, parentID, name, masterID) AS
(
SELECT id, parentID, name, masterID FROM database_sc53_Web..items WHERE id = '110D559F-DEA5-42EA-9C1C-8A5DF7E70EF9' -- root node
UNION ALL
SELECT r1.ID, r1.parentID, convert(nvarchar(256), r2.Name + '/' + R1.Name), r1.masterID
FROM database_sc53_Web..items r1
INNER JOIN nodes r2 ON R1.ParentID = R2.id
WHERE r1.masterID IN
('E461EFA2-CAC8-4A14-93C2-6966E89AAB5B',
'34E5E086-6BE9-480A-9EE0-A9CD64F52A76'))
select * from Nodes

Basically, you create a SELECT to retrieve the root node (indicated by the comment on the fourth line), and then use the SQL 2005 WITH construct to recurse via a JOIN into itself. In the WHERE, we have also added a limit on what nodes are retrieved, by their Master ID, but this could be their TemplateID or some other field available within the Items table.

This produces output similar to:

This is from the Home node, and runs through the entire site, which is over 10,000 pages. With the filter on the Master IDs, we return 9,057 rows within 1 second. (I know the above says 10 seconds, but the database server is a remote server so this screenshot is a little inaccurate). This runs fast enough and is efficient enough on SQL to allow us to run these queries "live" without a caching layer.

Obviously, I wouldn't recommend putting this in the Sitecore databases themselves, rather a database alongside it and do a cross-database query.

Technorati Tags: Sitecore,SQL Server

Sitecore: Overriding the style of the fields in the Content Editor

2008-06-19T10:45:00Z

This is something I figured out last year how to do, forgot, then struggled to figure out how I did it again! If I had blogged about it, I'd have known! Instead, the SDN Forum was kind enough to remind me.

As a Custom Field Type for this particular aspect of my web page was too cumbersome to develop within my deadline, I opted for an IFRAME Field Type, which when given a URL simply exposes a web page via which you can edit the relavant page data. Unfortunately, the default height is a miniscule 80 pixels. You can override this height, however, by:

Going to Template Manager

Opening your Template and selecting Standard Fields on the View ribbon:

Expanding the individual field tree on the left and selecting the relavant field

FInding the "style" field in the Appearance section and entering your own CSS, in my case "height: 200px".

Then, when you navigate to the Item which implements the Template, you get a nicely increased height:

I should add that you must remember to deselect Standard Fields once you are done, otherwise Saving subsequent content will override any shared fields and therefore break the link between Template > Master > Item.

Flash - My view on best practice

2008-05-24T09:59:54Z

A conversation I had with a designer friend today where he asked me what I thought about Flash web sites got me thinking.

Flash content might be "flash", but it is not necessarily useful, functional or usable. I'm not a great fan of Flash, but I can see it's value to the web. It provides attractive, fluent and dynamic sites and designs that have a level of interaction that is difficult to duplicate in a conventional web-site - even when using techniques such as JavaScript and, increasingly, AJAX.

Users who visit a Flash site may experience a number of effects of the Flash content that can disrupt a positive web experience, which often reduces the benefits of using Flash in the first place. When visiting a Flash site with a system or browser that does not have Flash installed, a pop-up, yellow bar or a simple failed portion of the page will be displayed. This can be frustrating, because before I can enjoy the quick-satisfaction I love about the web, I have to wait for a number of minutes while I download and install the Flash applet. This is, of course, only done once so it's annoying once per browser. I remember I recently had to help someone who was having problems with using a web site that used Flash. His browser was experiencing a HAL-9000 style internal conflict. It wanted to install the applet, but his security suite which had integrated itself into his system didn't like it so much, so caused extra confirmation windows which, quite simple, can be quite scary if you're not sure what you're doing.

Once the Flash applet has been downloaded and installed, we then have to wait for the Flash content to load. Well designed Flash content is, of course, loaded asynchronously often with attractive progress bars. Others, just make you wait. But why should I wait? I often have to go to sites and have to watch a progress bar making it's away across the site, and these are big sites - one key site is Sony Ericsson's product catalogue at http://www.sonyericsson.com/cws/products/mobilephones?cc=gb&lc=en. It even comes up with "There's a lot to load, but it's worth the wait". Well actually, it isn't. I struggle to wait for them to bring out decent and reliable phones - I don't want to add waiting for the most important page on their site to load too. If you haven't got Flash, maybe they think you're not worth the effort? Luckily my Nokia has got Flash, but I won't be browsing their site with it.

When the Flash applet has loaded, we're often presented with a beautifully designed and thought through experience. Immediate feedback is provided on mouse interaction with screen elements. Animation is smooth and can be beautifully subtle. A well designed Flash web site is a joy to use - if you're able to fully utilise and benefit from a web experience. Users who require accessibility features such as alternatives to mouse control, larger text, different colour schemes or spoken content all too often lose their experience at this point. No matter how many times they change their browser settings to affect the stylesheet, the text size or control method, the Flash applet will often sit and ignore their wishes. A real shame, because the effort that goes into a Flash applet is not insignificant and requires a paradigm shift in terms of programming. Additionally, even "able" users can struggle to shift their method of interaction to a Flash applet. For instance, I was at a corporate partner's web-site today and I was navigating their site struggling to find some content. I went on a page that was not relevant and instinctively went for the "Back" button - possibly the biggest button on most browser toolbars. I ended back at my Google search, quite frustrating as I then had to navigate back to where I was (after I had waited for the "web site" to load again).

The support of Flash is subject to a large extent to Adobe's opinion and support of a particular platform. The PC and Mac will enjoy Flash players for the forseeable future - but what about the increasing number of alternative browsers? Television browsing is growing fast - often via a games console connected to the internet. Also growing fast is mobile browsing. I am lucky that my new Nokia N95 phone supports Flash, but other phones may not. Other phones would need more memory, a better CPU and this can have a big impact on battery life. If Adobe does not believe a particular platform will "work" (which may even be down to licensing or political reasons, luckily, not something Adobe seems prone to yet), no matter the technical qualities of the device, they can choose not to implement a Flash player. Would you like to put your web-site into their hands and have them choose who can and cannot access your site?

Finally, there are the inevitable security concerns and the obligations that come with them that are passed on to you, the user. We're now at Flash 9. More features are being added each time and this will only provide greater opportunity for bugs and potential security vulnerabilities. We struggle to make sure that our operating system and browser is "patched up" already, now we have to worry about various applets inside the browser. Granted, this is usually an automatic process, but it is using my time to download and install, my bandwidth allocation which may be capped and places the responsibility on me to make sure it is up to date.

Then there are the servers. I always say Google is the most stupid, deaf, dumb and blind user that will use your site - but you want it to use it regularly. Search engines such as Google cannot use a Flash applet. Even when the "Googleplex" achieves sentient status, it will struggle to effectively index and understand a Flash applet and the content it contains in a structured and useful way suitable for indexing. When a search engine goes to your Flash site, it often sees very little. For heaven's sake, make sure that you at least include a decent <TITLE> tag and effective <META> data.

So how can you use Flash effectively?

I'm not interested in being negative for negativity's sake regarding Flash content. Flash content has it's place, and it can often create a very positive user experience, if implemented with consideration to users and search engines. I have come up with the following guidelines I like to keep in mind when people ask me about Flash.

What is Flash good at?

Recently, we've been playing with streaming high-quality video streams using as little bandwidth as possible for as cheap as possible. There are few products that can truly achieve this and have such a wide user-base as Flash. Windows Media Player is on virtually all PCs thanks to it's integration with Windows (except in Europe, of course!) but this often draws resentment from users, particularly Mac users who often need special software to view the same content. By the same token, Quicktime is available on Macs (although they realistically represent no risk to the true business user market) but PC users may resent (and I do) the repeated attempts by Apple to install iTunes and now Safari along with it. Having a "Go Pro" message everytime you close the application is also frustrating. Finally, there is Real Player, a wild thing that has little respect for your wider use of your PC. It aggressively tries to take control of all your media, constantly wants you to "register" for content and support (read "spam"), places tray icons that have no benefit to users and has effectively turned into bloat-ware. Flash, on the other hand, can handle very efficiently compressed video that can be secured and made available to users no matter what platform they may be using. It is no accident that YouTube use Flash for their video player. While no content on the web is truly secure, I was interested to learn that you can adopt some cute practices to seal up Flash video content - at least to the amateur.

Flash is good at marketing. Marketing channels such as banners were often animated GIF's which were slow to load, animated jerkily and were unattractive due to their 256-colour palette. Now, Flash is used regularly and - if well implemented - can provide a positive user experience. The silly and pointless Flash adverts that you find on sites like Facebook aside, a Flash applet can entice users in by presenting simple games, video's (the IBM ones are very popular, good ol' Gill) and elegantly designed animations. If I see a Flash video ad, I usually play it.

Flash is also great for games. So many hours have been spent playing silly, pointless but highly addictive games built in Flash. These games are great as a viral marketing tool, too. Kids love them, they're safe and they're highly interactive thanks to their ability to utilise the multimedia features of a PC.

Never wrap content up only within a Flash applet

You wouldn't keep your key web site content behind a username/password protected page, preventing your users and customers from accessing it so you shouldn't put such content only within a Flash applet. Users who cannot use Flash must be able to access the same information or functionality. Sure, it may not be as snazzy or beautiful, but some people don't have time, security-privileges to install Flash or just aren't that superficial. I'm a meat and guts person, myself. Flashy graphics has no effect on me, if I can't get at what I want, I'm off.

In providing this alternative content, you can choose to do it in two ways. You can check for Flash on the browser in some JavaScript and fail-over to static HTML if Flash isn't available, or you can present either content and ask the user to choose. But don't do a "Flash or HTML?"-style page, this has no benefit to search engines and requires an extra click for the user to access their content. A perfect solution, for me, is to try and load the Flash version and then load a small portion of the Flash content along with an option to choose the static version. This maintains a quick and attractive user experience, even if the user opts for the static version as they've not been made to wait for the full applet to load.

Flash works really well when it complements your content

Consider my hosting company's site: www.hostinguk.net. The Flash applet along the top is being used for what it is good at: creating an attractive and vibrant promotion mechanism that is central to the user's eye-line. I visit their site regularly, and this applet is used to convey latest marketing and products that are being implemented and gets my attention every time. But after a few seconds absorbing the simple content conveyed through the promotion channel, my eyes fall to the text - I want detail and there's nothing like text for providing it.

So in this case, the Flash content is complementing the site content. It is being used as a vibrant, attractive and sleek channel for conveying a marketing message but if you can't see it, you don't miss out. Additionally, it is small, so it loads fast. Search engines will ignore the Flash and attack the text on the same page, which will drive users to the page and who will see the attractive Flash applet and from that, their initial momentary impression of their browsing experience will decide whether they're likely to stick around and maybe make a purchase.

A client has recently told me that I had lost out on the development of a charity's web site to a Flash web site developer. This infuriates me, as, at the end of the day, they will end up with an attractive site but which would be difficult to find - and certainly next to impossible to find and jump straight in to the context which I want. Flash developers might not have realised it yet, but when you search in search engines, they tend to take you to the content you're after - not the first page and have you find your way through the navigation to the content - which is what a Flash site forces you to do.

In conclusion, we must also visit Silverlight, Microsoft's competitor product. Silverlight offers much the same as Flash does now, or will do in a production capacity when v2 is eventually released. Silverlight offers the developer a lot more features than it offers the user, in all honesty. It's another component that has to be installed in increasingly bloated browsers. From a user's point of view, I struggle to see any benefits. The benefits are entirely for developers, particularly those with a .NET background. The Silverlight platform has one key difference, however. It uses Javascript for it's control mechanism, and holds it's data behind web services and XML files as XAML. While not entirely useful for a search engine, at least it is something. Again, I would never advise implementing a site or part of a site entirely within Silverlight, particularly when it is such a fledgling product. It has a long way to go and while Microsoft may have a link on every page of their site, it's going to take more than that to encourage users to install it - particularly as it's use has actually slowed down key Microsoft sites such as their Download Site.