Erwyn van der Meer

Scott Guthrie did a major announcement on his blog yesterday: Microsoft will be releasing the source code for most .NET Framework libraries with the release of Visual Studio 2008. There will even be integrated support for debugging into framework classes and on-demand dynamic downloading of source files and debug symbols in Visual Studio 2008.

This is great news for .NET developers and a major step forward for Microsoft in my opinion. In and by itself it is enough reason to warrant an upgrade to Visual Studio 2008. In fact, I can think of no reason to keep using Visual Studio 2005 after the release of VS2008.

The source will be released under the Microsoft Reference License which basically means you can view and debug but not change or reuse the source code.

If you want a more liberal license you can look into Rotor aka the Shared Source CLI. Rotor was Microsoft's first effort for open sourcing a .NET CLI implementation. But Microsoft does not guarantee that Rotor has exactly the same codebase as the real .NET Framework.

Check out the full details and screenshots of VS2008 integration on Scott's blog.

Today I encountered a problem with accessing the metadata for a WCF service that was deployed on a Windows Server 2003 machine.

The WSDL part worked just fine for the metadata exchange endpoint (url?wsdl, url?wsdl=wsdl0, etc.). These WSDL files refer to XSD files for the message types. Requesting these files (url?xsd=xsd0, url?xsd=xsd1, etc.) resulted in an empty response from the webserver. Checking the IIS logs indicated a HTTP 200 OK response with 0 bytes transferred. A very weird problem. Checking the config files did not lead anywhere.

Eventually I found a hint in a reply by James Zhang in this MSDN Forum post. The identity that is used for the application pool that hosts the WCF service must have the correct NTFS permissions on the %WINDIR%\temp folder. The identity that I used is a domain account. After setting the right NTFS permissions, the problem disappeared.

The funny thing was that this particular answer wasn't the answer for the original question in this forum post.

James Zhang does not indicate what type of permissions are needed, so I had to experiment a little.

First I added the account to the local Users group. This gives it special access permissions: Traverse folder/execute file, create files/write data, create folders/append data  on this folder and subfolders. This is not enough. Then I realized, the domain account is already implicitly a member of this group because the Users group contains the NT Authority\Authenticated Users group. Next, I duplicated the extra rights that the NETWORK SERVICE account had for the domain account. These are list folder/read data and delete permissions for this folder, subfolders and files. This was enough. But it doesn't seem very secure. Now the service account can access temporary files created by other accounts.

So I experimented a bit more. I tuned back the NTFS permissions for the service account on %WINDIR%\temp to list folder/read data on this folder only. This is just enough. This allows the account to see which files are in the temp folder, but it doesn't allow it to read the data in files that are owned by other accounts.

It is very unfortunate that WCF didn't give any clue about why it couldn't generate metadata in this case. It is also unfortunate that it needs just slightly more permissions that a standard user on the folder for temporary files.

Note that if you run your WCF service in an IIS application pool under the default NETWORK SERVICE account you won't run into this problem, because it has more than enough permissions.

PS: Best practices indicate you shouldn't deploy your services with metadata enabled. We will turn this off eventually. However, of course it should work if you do want to enable this.

After a long day and night of coding, I released version 0.8.0.0 of my Flickr Metadata Synchr tool on CodePlex this morning. I finally solved the long-standing problem I was having with the Windows Imaging Component (WIC) to update metadata. So this is the first fully-functional release of my application.

Functionality of FlickrMetadataSynchr v0.8.0.0

This is what the app does:

• It allows you to select a set of your photos on Flickr and a folder on your hard drive with images.
• It reads the metadata for both Flickr images and the local images. The metadata that is read is:
  
  • Title
  • Description
  • Author
  • Tags
  • Geo-info (GPS coordinates)
  • Date and time taken
  • Last update date and time
• It matches images on Flickr with local images based on the date and time taken.
• It determines on a per picture basis in what direction the metadata should by synced, i.e., which side should be updated, if any. Currently the most recently updated side wins. I am getting help from Timo Proescholdt for a better algorithm that will allow for a merge of metadata, i.e., a two-way synch.
• It updates the metadata on Flickr and in the local images.

This is a screenshot of the app:

Previous posts on this tool

• Here is more info on why and how you should store the metadata in your images.
• Read here how I was previously forced to use C++ to update metadata in images stored on the hard drive (I call these local images).
• Read about how CodePlex lost my source code.

Workaround for WIC problems

During my holiday in France in July I received e-mail from Robert A. Wlodarczyk who works at Microsoft. He pinged me to say that he had released new sample code to update metadata using WIC. Yesterday, I tried to incorporate similar code into my application and ran into the same type of problems as before.

Because his sample was working, I wasn't ready to give up again. I finally tracked the problem down to a threading issue. WIC is throwing strange InvalidOperationException and InvalidFormatException exceptions with messages like "Cannot write to the stream" when it is called from a background thread. My app is multi-threaded so that the UI doesn't hang when it is busy syncing.

After I got confirmation from Robert that WIC indeed suffers from a threading issue, I solved the problem with a work around. I now marshal the call to the code that uses WIC to update metadata to the UI thread using the WPF Dispatcher object. This causes the app to become non-responsive for small amounts of time during the update of local metadata. But that is better than a non-fully-functional app.

Fully functional, give it a try

So all is well that ends well. After finally getting WIC to work, I could do away with the C++ code that was causing me headaches ;) And my app now works on Windows XP again. You just need to have the .NET Framework 3.0 installed.

If you have images on Flickr and you have been busy tagging them, give my app a spin! You can always find the latest release on CodePlex. The source code is also available under a GPL license on CodePlex.

Installing is easy. You just need to unzip the ZIP-file, which contains three files, to a folder. Start the FlickrMetadataSynchr.exe file and you are done. The app remembers the last settings.

If you find any issues, please report them using the Issue Tracker for my app on CodePlex.

The Future

Even though the app is now able to sync metadata in both Flickr images and local images, there is always room for improvement. Here are my ideas, some of which are based of suggestions by people on CodePlex:

• Improve the synchronization to also allow two-way synchronization for a picture pair. I.e., one side doesn't have to win. For example, if the Flickr image has just the title set and the local image the description, the metadata should be merged.
• Add UI to see the match that is made by the tool and how it proposes to sync the metadata.
• Allow you to exclude images if the match isn't good.
• Allow you to overrule the sync proposal and sync the metadata in a different direction (on a per property basis) .
• Add UI to store multiple mappings between Flickr sets and local folders. Currently the app only remembers the last folder and Flickr set that was used.
• Add click-once deployment. That way the app can automatically check for new versions and update itself.

If you have any other ideas please post them at the discussions page for my app on CodePlex.

If you are interested in the history of LINQ to SQL and how it works internally, go watch Charlie Calvert's video with Matt Warren and Luca Bolognese.

The first 30 minutes of this video are primarily about the internals. The last 10 minutes focus on the history of LINQ to SQL and how it relates to its "predecessors" Cω  and ObjectSpaces. Matt and Luca also tell how Erik Meijer and Anders Hejlsberg got involved.

Related posts on this blog:

• .NET Language Integrated Query Framework (July 17, 2005)
• History of LINQ to SQL (May 31, 2007)

Back in January 2006 I wrote a blog post about a way to add input validation to declarative query string parameters in ASP.NET 2.0.

This involved adding code to the code-behind of each ASP.NET page to hook an event handler. This quickly becomes tedious if you have a lot of pages.

Will Shaver has come up with a better approach. He derives a custom class ValidatedQueryStringParameter from the ASP.NET class QueryStringParameter.

Check out the download on his blog post. It contains a more complete ValidatedQueryStringParameter class that supports other types besides Int32 like DateTime and Boolean. The QueryStringParameter already supports a Type property (of type TypeCode) to specify the type for the value of the parameter, so you can use it from markup like so:

<cc:ValidatedQueryStringParameter Name="CategoryId" QueryStringField="CategoryId" DefaultValue="0" Type="Int32" />

PDC07 has been postponed indefinitely. Dennis already warned me just before MIX07 that Microsoft wouldn't have anything to announce at the PDC after the MIX conference.

No new date for a PDC has been given, yet Microsoft already announced the date for MIX08! It will be held March 5-7, 2008 in Las Vegas.

So when will the next PDC be? The official statement by Microsoft is that it will be when they are ready to announce the "next wave of platform technologies".  The .NET technology is still forbidden from entering large parts of the Windows Operating System.

Will Microsoft attempt to pull off a synchronized wave of platform technologies  again? Synchronizing the release of

• Visual Studio .NET 2005, .NET 2.0 and SQL Server 2005
• .NET 3.0 and Windows Vista

proved to be mighty difficult for Microsoft to pull off. They only managed to release Windows Vista by cutting the dependencies between the OS and .NET. .NET 3.0 is only bundled with Windows Vista. Vista doesn't make use of it.

Vista even makes it very difficult to pull some things of using .NET. See my metadata API difficulty requiring me to use C++ and the fact that is is very difficult to create a decent Sidebar gadget using WPF.

The synchronization of .NET 3.5 and the ADO.NET Entity Framework failed. ADO.NET vNext has been delayed and will be released after the LINQ wave.

However, there is hope glimmering on the horizon. The new CoreCLR used by Silverlight 1.1 is able to co-exist with other CLR versions in one process. Silverlight will be a great technology for the Vista Sidebar. ASP.NET Ajax was released out-of-band. Microsoft has said that the next revision of the full CLR (remember .NET 3.5 still uses CLR 2.0) will also have this feature.

Martin Fowler notices a trend: interest in the .NET platform is declining. Ruby is all the rage. He has some recommendations for Microsoft on how to better collaborate with the community instead of going its own way. Martin noticed how Microsoft created its own unit testing variant that is incompatible with NUnit.

Change is visible in some parts of Microsoft, for example Microsoft is embracing dynamic languages, but boats the size of aircraft carries can only slowly change course. There are no clear signs from upper management at Microsoft that they are indeed changing course. The recent threats against the open source community over infringement on software patents seem to indicate that the old school closed-source thinking still has big strongholds within Microsoft.

Google is about to set a standard for offline storage for web applications with Google Gears. Even better, it's open source. Does Microsoft have the guts to embrace this technology instead of creating its own variant? I doubt it.

I am still pretty excited about the .NET platform. Especially about the LINQ wave of technologies coming with .NET Framework 3.5 and Silverlight. But I am also thinking about looking into Ruby on Rails more and more.

Not many Microsoft employees dare to publicly refer to WinFS as a black hole. Matt Warren does in this blog post. He gives us great insight into the origin of LINQ to SQL and how it managed to survive. He ends with:

So, for the sake of LINQ and the customer in general, we took up the OR/M torch officially, announcing our intention internally and starting the political nightmare that became my life for the next three years.

I briefly talked with Matt at PDC05. It was pretty clear already that he had had a big role in the development of LINQ to SQL even though the public presentation was done by Anders Hejlsberg. His blog is pretty quiet, but Matt is much more active on the MSDN Forums answering questions about LINQ to SQL.

WPF comes with great support for animation using XAML without needing to code this in for example C#. With Silverlight (fka "WPF/E") you can also do animations from XAML.

If you want to perform custom animations in code that you can't do using XAML, you need timers. In the full blown WPF you have several options, e.g., System.Threading.Timer, System.Timers.Timer and System.Windows.Forms.Timer.

You normally provide a callback that gets called when the timer elapses from a background thread. Properties on WPF objects can only be set from the foreground thread, so you have to queue a call on the UI thread to perform the actual animation. You can do that by calling the Invoke or BeginInvoke method on the System.Windows.Threading.Dispatcher class. You can access the correct Dispatcher instance to use through the Dispatcher property on the UI element (*).

Another option in WPF is to use the Rendering event of a CompositionTarget instance. In that case you get called when WPF is ready to render a frame. The frame rate depends on CPU speed, GPU performance, graphics complexity and other factors, so it fluctuates. This means that the interval after which you get called also fluctuates. However this is great for some scenarios.

In the current Silverlight 1.1 alpha your options are more limited. The CoreCLR libraries do have a System.Threading.Timer, but there is no Dispatcher class to delegate work to the UI thread. So it is useless for doing custom animation. In the source of the Monotone sample by Lutz Roeder I found there is an HtmlTimer class in Silverlight 1.1. This class is undocumented and marked obsolete. Visual Studio shows a warning after compilation:

'System.Windows.Browser.HtmlTimer' is obsolete: 'This is not a high resolution timer and is not suitable for short-interval animations. A new timer type will be available in a future release.

Lutz shows how to use an HtmlTimer in his sample. HtmlTimer has a Tick event. Any event handler that you wire-up to that event gets called from the UI thread. So that solves the problem for the time being.

When I tried to Google for more info on HtmlTimer, all I found was this blog post by Mike Taulty which mentions this class in passing.

(*) In fact any .NET class that derives from DispatcherObject has a Dispatcher property.

Via Brad Abram's blog: Microsoft has made available a Silverlight 1.1 Developer Reference as a beautiful poster. Go check it out to get an overview of what Silverlight is all about.

The only thing missing is an image of a penguin ;)

If you haven't heard the news yet, you must be living under a rock ;) There is a new CLR in town. Silverlight (fka "WPF/E") 1.1 comes with its own CLR. And the best news is that it runs both on the Mac and the PC.

Go check it out at silverlight.net.

One of the greatest achievements is that the download is a mere 4 MB. If you have a reasonably fast Internet connection you can download and install it in under a minute. Try that with .NET 3.0! That will take at least an order of magnitude longer to install.

Before the announcment at MIX07, I wasn't sure if Microsoft would be able to pull this off. They have certainly gone beyond my expectations.

This is what the Silverlight directory looks like after the install:

It shows you this CLR is completely separate from the standard CLR 2.0 runtime and has no dependency on it. This CLR has 2.1.x.x as version number. It has no Global Assembly Cache (GAC). You can see that support for dynamic languages like Python and JScript is included. Support for Ruby is in the works.

Go listen to the Channel 9 interview with Scott Guthrie if you want to know how Microsoft succeeded in trimming down the CLR 2.0 and the Base Class Libraries to this size.

The Silverlight .NET assemblies have the same format as standard .NET assemblies, so you can view them using Reflector. The type system is the same, so Silverlight supports generics. It will also support C# 3.0, VB 9.0 and LINQ.

These are exciting times for the .NET world. The reach of .NET has been substantially increased. Not just because the few percent of Mac users can run .NET applications now, but because it is such an easy deployment for Firefox users on the PC. And soon for Opera users as well.

Microsoft has announced that the little cross-platform, cross-browser cousin of Windows Presentation Foundation will be called Silverlight. This technology which was first announced at PDC05 was codenamed "WPF/E".

Tim Sneath has the best list so far of the features and power of this "Flash killer" technology. Microsoft doesn't ever call Silverlight a Flash killer, but the overlap in feature set is so large, that it cannot be viewed as anything other than a direct Flash competitor.

However, I do believe that Silverlight leapfrogs Flash in a couple of ways. The programmability and ease of use is better than Flash. You can build Silverlight sites using just Notepad if you want. The direct integration of the Silverlight DOM (Document Object Model) with JavaScript in the browser and the ability to create Silverlight UI elements on the fly with the createFromXaml method is a killer feature.

Tim Sneath mentions a secret number 10 feature on his list:

"Ah... #10. I can't reveal this yet - there's a big surprise up our collective corporate sleeve that will be announced at MIX. I hate to hold back on you, but anticipation is part of the pleasure, as my mother used to tell me as a child when I was waiting impatiently for Christmas to come!"

Could this be the .NET programmability that was previously speculated about? Soma spills some more details in his announcement:

"As I mentioned, this Silverlight announcement at NAB is only part of the story, the rest will be unveiled at MIX including details about how Silverlight is a core component of Microsoft’s broader .NET platform."

I commented previously on Robert McLaws' blog that I didn't think that Microsoft wasn't going to release a lightweight crossplatform CLR for Silverlight programmability. But I also speculated that Microsoft was working on a bigger crossplatform CLR based on the .NET Compact Framework.

What I am pretty sure about, is that Microsoft will announce ASP.NET controls that will allow you to very easily integrate Silverlight on your web pages and to expose dynamic data as XAML to Silverlight controls. I.e., AJAX on steroids UI-wise.

Adding ASP.NET to the mix shows that there is no direct need for a CLR on the client in order to enable C# or VB.NET programmability: coding in C#, compiling to IL and converting that IL to JavaScript on the fly! Prototype efforts by Nikhil Kothari with Script# show that this is quite possible. Check out Nikhil's example.

[Update: The electronic version (in Dutch) is available online now.]

My article on developing interactive TV-websites using ASP.NET 2.0 has been published in the Dutch .NET Magazine #16. I mentioned that I was writing this article in January and I made the deadline.

If you are subscribed to this magazine, you received issue #16 last Friday. The electronic version (in Dutch!) is not online yet, but it should appear in the near future. All previous issues are available here.

The typesetting process has caused a few minor issues that I would like to rectify here:

• The byline of code example 1 reads "WMC.browserbestand om Windows Media Center mee te herkennen". This should be "WMC.browser bestand etc." I.e., it's a file with the .browser extension.
• In the "ASP.NET 2.0 browser sniffing" section, the line "Daarin kun je bestanden plaatsen met de browserextensie." should read "Daarin kun je bestanden plaatsen met de .browser extensie".

If any other issues are discovered, I'll update this post.

*sigh*. It seems you can't strong name a C++/CLI assembly, i.e., sign a managed C++ assembly, with a .pfx file like you can in C# or VB in Visual Studio 2005. You have to revert to using an unprotected .snk file ;(

An MSDN forum post about this issue was never answered. It seems that the Visual C++ team is even unaware of the possibility of using a .pfx file to sign a managed assembly in their sister products. And the writers of this MSDN article were apparently unaware that Visual C++ is also a part of Visual Studio 2005 since they assume it can be done using both a .snk and .pfx file.

The situation

I am currently working on version 0.6.0.0 of my Flickr Metadata Synchr tool. The goals for my open source project on CodePlex are described on the Flickr Metadata Synchr wiki page and you can always find the latest status there.

At the moment the latest public release is version 0.5.5.0. The feature set for v0.5.5.0 is roughly:

• Allow you to select a Flickr photoset and a local directory with images.
• Load metadata from both local and Flickr images into internal metadata structures.
• Compare these metadata structures and synchronize them.
• Update metadata on Flickr after the synchronization.

One of the features planned for v0.6.0.0 is updating the XMP and IPTC metadata in locally stored images. I was planning on doing this through the Windows Imaging Component (WIC) which is part of the .NET Framework 3.0. WIC is also available as a separate download for Windows XP and Windows Server 2003.

Windows Presentation Foundation provides a nice managed API for reading and writing metadata through WIC. It provides the SetQuery and GetQuery methods on the BitmapMetadata class. I was already using the GetQuery method, which works fine. However, I hit a snag when I wanted to use the SetQuery method to update metadata.

Plan A

There is a way to do this through the InPlaceBitmapMetadataWriter class. It just touches the metadata structures in the image file and doesn't have to read or write the entire stream with pixel information. This will give you excellent performance and so you don't run the risk of having to reencode the pixel stream or loosing metadata. The sad thing is that it almost never works. The image file often does not have enough room in its metadata structures to allow metadata fields to be filled or updated. When you try to save the updated metadata, the InPlaceBitmapMetadataWriter fails. That is probably why the save method is called TrySave. By the way, the code sample on that MSDN Page is dead wrong. If you call TrySave before updating metadata, it always succeeds. Probably because there is nothing to save yet. You have to call it after you update the metadata, and then it returns false ;( Which means your metadata was not updated successfully.

Plan B

So I tried plan B: Creating a new image file by writing out a copy of the original image, but now with updated metadata. This means you have to grab the original BitmapFrame from the JpegBitmapDecoder. Clone it, update its metadata and write it out again using the JpegBitmapEncoder.

This is where I hit a major problem. The Save() method on the JpegBitmapEncoder almost always fails with an InvalidOperationException with the error message "Cannot write to the stream". When the encoder is able to write out the image, the JPEG turns out to be reencoded with a different quality than the original. This is noticeable through a significant change in size of the file. This happens even though I specified the BitmapCreateOptions.PreservePixelFormat option when opening the image with the decoder. Googling (or Windows Live Searching if you will) for a solution didn't yield anything useful.

Plan C

I had to come up with a Plan C. The Windows Vista Shell is obviously able to update metadata in images without affecting the JPEG quality and without creating a copy of the image file. This led me to an MSDN article titled "Photo Metadata Policy". This is the introduction:

Metadata (file properties) for photo files can be stored using multiple metadata schemas, in different data formats and in different locations within a file. In Windows Vista™, the Microsoft® Windows® Shell provides a built-in property handler for photo files formats, such as JPEG, TIFF, and PNG, to simplify metadata retrieval.

When a piece of metadata is present in different underlying schemas, the built-in property handler determines which value to return . For instance, the Author property may be stored in the following locations in a TIFF file:

• The Creator tag in the XMP Dublin Core schema:

  /ifd/xmp/purl.org/dc/elements/1.1/dc:creator
  

• The Artist tag in the EXIF schema:

  /ifd/{ushort=315} 
  

• The Artist tag in the EXIF schema embedded in an XMP block:

  /ifd/xmp/ns.adobe.com/tiff/1.0/tiff:artist
  

On read, the property handler determines the value that takes precedence over the others that exist in the file and returns it. On write, the property handler makes sure it leaves each schema in a resolved and consistent state with the others. This may mean either updating or removing the tag in question in a given schema.

This would also help to solve another piece of the metadata puzzle: what to do with the several different options of putting metadata in image files (XMP versus IPTC, multiple possible XMP places, etc.). After updating an image, I want the metadata in the different blocks to be consistent. WIC doesn't help with this. You have to sort it out yourself. The Windows Vista Shell does help with metadata reconciliation.

So all seems to be well. Just use the Shell API to update the metadata. I would love to be able to do this from C#. Yet that doesn't seem to be possible or it is extraordinarily difficult. The "file property" handling is implemented in propsys.dll through a COM based API. But you can't add a reference to this COM library in a C# project. It doesn't have a type library ;( The only option I can find is to use C++ and use the propsys.h and propsys.idl files that are distributed in the Windows SDK. This is horrible. I guess I have to dust off my C++ skills to be able to call a brand-new Windows Vista API. WTF?!

The "Longhorn" promise for managed code

Do you remember the promises Microsoft made back in 2003 for the new Windows Client OS codenamed "Longhorn"? I sure do, since I visited the PDC03 conference where this was all announced. Microsoft promised us a brave new world where all Windows APIs could be accessed easily from managed code. Three and a bit years later we have a new Windows Client OS called Vista that doesn't live up to this promise. Microsoft has implemented new APIs that seem to be inaccessible from managed code other than through C++.

Now I can understand why part of the promise was lost during the infamous "Longhorn Reset" at Microsoft. Microsoft's ambition to completely wrap all existing Win32 APIs in WinFX was too big. But why Microsoft would be creating new APIs without managed code in mind is beyond me...

I found some C++ code on the blog of Ben Karas that is indeed able to update metadata  I hate having to add this C++ code to my project. It would require people to have Visual C++ and the Windows SDK (especially the Windows Vista header and library (*.h, *.idl, *.lib) files) installed to be able to build my code in Visual Studio.

A plan D might be to manually create C# wrappers for the COM interfaces of propsys.dll. This article describes how to do this for COM interfaces in general.