Search results matching tag 'SharePoint'

SharePoint 2013 and Windows 8 apps - better together Part 1: Introduction, background and considerations

Bas — Sun, 17 Mar 2013 18:09:00 GMT

This is the first post as part of a blog series about the integration of using SharePoint 2013 as a datasource for windows 8 apps.

Part 1: Introduction, Background and Considerations (this post)
Part 2: Platform choice, using the right API and data access
Part 3: Search
Part 4: Authentication

Introduction

On 8 march 2013, my colleague Ad Reijngoudt (Windows 8 App developer, follow him on twitter: @Areijngoudt) and I spoke on the dutch Techdays on the subject: "SharePoint 2013 and windows 8 app - better together". After this session, we got a lot of questions on several subjects, so I decided to write some blogposts on these subjects.

Background

My colleague Ad developed a fully functional windows 8 application within 3 days during a workshop with Microsoft in The Netherlands. This application is called "Vergoedingenzoeker" in Dutch (Is it called Compensation finder in correct english?!) and offers functionality to find possible compensations within your health insurance. The downside of this app, was the fact that it contained a static datasource; whenever there is a change within the consumed data, the app needs to be updated and that is, of course, not the most ideal situation. That's why we decided to find out if we could create a nice backoffice for this application.

The "zorgvergoedingen" app

The vergoedingenzoeker app basically exposes the following functionality: On startup, a selection screen is displayed. Users can navigate to information in two ways

Select a compensation level. Based on the selected level (1-4 stars), an overview of all compensations within that level is displayed
Select a character, and all compensations of which the title start with that same character are displayed

Whenever a choice has been made, all item titles that fall within the selection criteria, are displayed:

The last action is to select a subject. After a subject is selected, all details are displayed:

All this data is stored within SharePoint and within the upcoming blogposts, you'll see how we did this.

Considerations for using SharePoint 2013 as the backoffice

As we discussed what kind of backoffice we should use, I suggested SharePoint 2013 immediately:

I love SharePoint! (I can't say if that was the main reason or not? ;)
We already had existing data in our public site, so why not reuse that data? It prevents maintaining data on multiple platforms, which is always a good idea. (altough this was a MOSS 2007 site, we could make an extract of the data and host import that data into SharePoint 2013)
The new API's of SharePoint 2013 are very interesting (in my opionion), so this was a good situation to find out about possible limitations
SharePoint 2013 offers great out of the box functionality, which should easily be integrated with the windows 8 app. Think about Userprofiles, Search, mananged metadata. And all of these service apps are accessible via the new API's

Because of the last three arguments, Ad agreed on using SharePoint 2013 (being an ASP.Net / windows 8 fanatic, he doesn't admit that he loves SharePoint as well ;))

Upcoming blogposts

The upcoming blogposts will cover the following subjects

Using the right API and data access - How did we expose the data to the windows 8 app and what choices did we have to make? What tooling did we use?
Search - How did we implement search functionality in the app. We made some really cool choices :D
Authentication - How to authenticatie to Office 365. There are already some blogposts on the subject, but we improved it a bit and decided to share this with the community.

Content by query webpart not performing due to misconfiguration

Bas — Thu, 03 Jan 2013 20:27:00 GMT

Today we were experiencing some major performance issues on some pages on a new website. The pages that were experiencing these performance issues, all made use of the content by query webpart (we didn't have any performance issues before on these pages). The content by query webpart that we were using on the site, used a pagefield value, which was used to lookup some related pages from another list.

The problem

A quick look at the developer dashboard learned me that something "bad" was happening over there:

At first, critical database errors:

fa42 A large block of literal text was sent to sql. This can result in blocking in sql and excessive memory use on the front end. Verify that no binary parameters are being passed as literals, and consider breaking up batches into smaller components. If this request is for a SharePoint list or list item, you may be able to resolve this by reducing the number of fields.

In addition to that, some very bad performing sql queries took place (which could be related to the critical database errors).

the queries that are marked in the screenshot above, included a query that created a table variable, and inserted data with "some" insert into select statements. #define some 489.

Basically, this meant that from 489 different sources, data was inserted into the table variable. This table was queried on its turn, to get the data out if it, that we needed. As described in the small introduction, we just needed to lookup data from one single list! Altogether, it turned into loading times of 10-20 seconds.Due to all the inserts, I suspected that not the specific pages list was queried, but whole the site collection. With 500 Mb of data this can be quite time consuming ;).

A quick glance at the webpart properties learned me that I was right: the whole site was queried, where a much smaller scope was needed (the scope of a single list). Selecting the pages list as the datasource resulted in a much better, acceptable performance: a steady loading time of about 1s.In addition to the performance gain: the database errors got away, too ;).

Summary

Altough the content by query webpart is a very good and fast mechanism to gather data, it can be misconfigured in many ways, which causes performance issues. In my case, it even caused database locks, which prevented the whole site from performing at all. When configuring the content by query webpart, always make sure to use a datasource that has the smallest scope possible.

SharePoint 2010 signout different behaviours based on the number of selected Authentication Types

Bas — Sat, 29 Dec 2012 11:56:00 GMT

While working on our custom ADFS login component and deployed this version to our DTAP street, we saw different behaviours when signing out of a site, under different circumstances. Wen users tried to logout via the page "/_layouts/signout.aspx" users sometimes where redirected back to the root of the site and in some cases users got the message "please close the browser to signout". As I was curious why this happened, I decided to check a few things out.

In wat cases does what behaviour occur?

After comparing our webapplication settings, we found out that the only difference that we had, was that on our Acceptance environment we had just a single Trusted Identity Provider selected, while on our Test environment we had both windows authentication and a Trusted Identity Provider selected. As this was a nice start, we decided to disable/enable a few identity providers and enable/disable windows authentication. As we couldn't find a real pattern, I decided to fire up reflector and started to do some real digging ;).

As seen in the red section, a few decisions are made. First: ClaimsBasedAuthentication needs to be set. This is always in our case, so that one was easy. The code that is executed afterwards, is more interesting: A decision is made, based on the number of authentication providers OR WindowsIntegratedAuthentication is not turned on. It's easier to see what happens when a table is used. At the bottom of the table the actual action is included: Stay or Redirect

windows authentication	0	0	0	1	1	1	1	1	1	1	1
Integrated	0	0	0	0	1	0	1	0	1	0	1
Trusted Identity Provider 1	1	0	1	0	0	1	1	0	0	1	1
Trusted Identity Provider 2	0	1	1	0	0	0	0	1	1	1	1
num	1	1	2	1	1	2	2	2	2	3	3
use windows integrated	0	0	0	0	1	0	1	0	1	0	1
num!=1 or !windows integrated	1	1	1	1	0	1	1	1	1	1	1
expected action	R	R	R	R	S	R	R	R	R	R	R
actual action: S(tay)/R(edirect)	S	S	R	R	S	R	R	R	R	R	R

This table learns us the following: based on the selected options, we are expecting different behaviour then the actual behaviour: In cases where windows authentication is disabled (and thus, no windows integrated authentication) AND there is just 1 identity provider selected, a redirect to the rootweb of our sitecollection is expected, but this is not the case: users stay on the same page and get forced to close the browser to logout, altough this isn't needed. Manual navigation to the website learns us that the user has been logged out.

Why no redirection?

to find out why this didn't happen, I wrote a small bit of code to checkout the value of iisSettings.UseWindowsIntegratedAuthentication when windows authentication is disabled: This value seems always to be true when windows authentication has been disabled! When filling out these values in the table, the behaviour can be explained:

windows authentication	0	0
Integrated	1	1
Trusted Identity Provider 1	1	1
Trusted Identity Provider 2	0	1
num	1	2
use windows integrated	0	1
num!=1 or !windows integrated	0	1
expected action	S	R
actual action: S(tay)/R(edirect)	S	R

Summary

Signing out within SharePoint 2010 or SharePoint 2013 leads to unexpected/unexplainable behaviour, because it's unclear when users are redirected to the rootWeb of the sitecollection, of are forced to close their windows. This behaviour has to do with the fact that, the value of "IisSettings.UseWindowsIntegratedAuthentication" is set to true when Authentication type "windows authentication" is set to false.I have a gut feeling that this value must explicitly set to false, whenever there is no windows authentication. The combination of multiple identity providers vs a single authentication provider leads to the difference in redirecting the user to the rootweb vs forcing the user to close the browser. I do believe that this is a bug and this will be submitted to Microsoft in the upcoming year (2013).

Customizing ADFS login for SharePoint 2010: how we did it

Bas — Fri, 28 Dec 2012 17:33:00 GMT

In SharePoint 2010 the possibility of claims based authentication was introduced. The out of the box experience of this functionality is often OK, for example in cases of corporate intranets and extranets, but it doesn’t always fulfill the requirements of internet facing websites which require authentication. This blogposts describes why we wanted to implement the active login scenario and learns us what kind of problems we encountered (and nailed ;))

Login requirements

At our company we own a lot of different (insurance) labels and a big part of those labels want users to be able to login via a username (for example their email-address) and password. When a user logs in with his username (e.g. user@outlook.com) on site of label A, he must be able to login with that same email address to site of label B. These accounts attached to the sites may not be the same physical account.
When a user logs in, he may never get redirected to another domain.
Login in when filling out a form, to retrieve his contact details that are already attached to his account, without losing context
Custom styling for the login control to be in line with the labels’ look and feel.
Being able to only supply ADFS login to endusers, but supply ntlm based authentication too, for the search service application. This is needed because our sites are hosted within Host named site collections (HNSC), multiple zones aren’t possible in here.
Because we are using HNSC, and don’t use any form of content deployment, we need to be able to offer authentication for our employees too, to be able to edit content. They may only be able to login from within the corporate network, otherwise, only a customer login must be offered.

Non functional

We want to make use of one single infra-structure for cost-reduction purposes. With tons of sites (and thus url’s), it isn’t very costs-effective to build a separate environment for every site.

Passive login

The out of the box experience that comes with SharePoint 2010 is the so called “passive requestor profile” or “passive login” and works as displayed in the flow below. Please note that users are being redirected to a Resource STS. SharePoint thinks it’s an Identity Provider, but in our infrastructure, this is not the case. Why, is explained further on.

As seen in the diagram, whenever a user needs to login to a SharePoint site, the user is redirected up to 7(!!) times before he gets his page served. As this may be acceptable behavior when accessing a page to check out his customer information, this is not acceptable whenever a customer is in the 3rd step of a flow when he wants to purchase an insurance. In addition to this: the OOTB functionality isn’t able to hide login options based on your source (in- or outside the corporate network)

Claims based authentication infra-architecture for internet sites

To be cost effective, we have a single Resource STS for claims augmentation and a single IP-STS for the identity claims. . At the SharePoint STS we don’t make use of any claims augmentation of transformation, because this can lead to problems in a multi farm (publishing/consuming) setup.

As seen in the diagram, the trusted Identity Provider for the SharePoint STS, is in fact, a resource STS. The real identity providers are federated with this resource-STS. This opens up the possibility to use different identity providers and handle resource claims augmentation on a single place.
The problem with this setup, is that the federated STS-ses are accessible via only url. This is contrary to the requirement REQ2: a user may never leave the domain when logging in.

Our solution

Our solution for this problem, is to create a custom ADFS login control that can be re-used on pages and/or webparts. This is called “Active requestor profile” or “Active Login”. In fact, the solution exists out of two parts
1) Determine user login source and type
2) The login itself

Determine user login source and type

Below is a small flow diagram to determine whether a user is a customer or an employee:

“Determine origin” is executed without any user interaction. Whenever a user tries to login, we need to determine their origin. This can be done in various manners, but we decided to make use of the X-Forwarded-For header. It’s a HTTP-request header, that is inserted from our load balancer. External users get the address of the reverse proxy injected, internal users their own IP address. According to Wikipedia (and everything that’s on Wikipedia, is true ;)), it is the de facto standard for identifying the originating IP-address. Whenever the IP in the XFF HTTP header is not equal to the IP of the reverse proxy, a user selection type screen is show. Otherwise, only the customer login screen will be shown.

Active Login

As seen in the diagram below, this authentication method is much more user friendly, as there are just two redirections: the one to the login page that hosts the user login control and one redirection, back to the source URL, when the user has been authenticated. Please note that no difference between customers and employees has been made here, as the login logic is almost the same.

The downside of this approach is that it requires custom code and needs to handle all the logic that ADFS was handling in the passive scenario:

Get identity claims
- Multiple identity providers – choose which IP to use (for example, based on selected userType)
  - Customers
  - Employees
  - Government
  - Etc.
Augment resource claims
Pass Token to SharePoint

Liam Cleary has done excellent research on this subject and wrote some code for it. I suggest that you read his blogpost. The code is not production ready, but provides a great base to start out on your own active login user control.

Login control

Where liam’s code sample provided an application page with an asp.net login control we decided to create a custom login control, which subclassed System.Web.UI.WebControls.Login. We had a few reasons to subclass this control. One of those reasons is the fact that this Login control is used for forms based authentication. See the highlighted sections in the following, reflected, codesnippet:

Formsbased authentication

As seen in the first highlighted section, this login control places a Forms Authentication cookie, which caused problems when trying to logout. After logging out, the cookie wasn’t deleted, which caused an error. Users had to close the browser and open it again to be able to login again.

Templating

Another reason was more practical. In the existing asp.net login control, templating is already implemented. By subclassing this control, a custom look and feel can easily be implemented, as the templated controls are inherited. The control can even be reused on any sitepage/masterpage/pagelayout: no application page is needed anymore. This makes it possible to give each label its own markup and messages:

Subclassed ADFSLogin control

The subclassed ADFSLogin control is not too complex: the default OnBubbleEvent logic has been overridden, just to be able to implement custom AttemptLogin logic (this can’t be overridden as it is a private function. I didn’t want to use any form of reflection ;)). The AttemptLogin logic doesn’t contain much custom logic: it only got rid of the two highlighted sections.

Issues

Redirecting after SPFederationAuthenticationModule.SetPrincipalAndWriteSessionToken will throw a ThreadAbortedException. When caught, the redirect will still occur and the login behavior is as intended.
applying changes to the web.config isn't as trivial as expected. overwriting sections with SPWebConfigModifications won't work; another approach is needed for this. (MOre on this in a later blogpost)

Summary

This article described what exactly passive and active login is and learned us why we chose for the active login scenario and what challenges we have met. As the standard login methodology doesnt satisfy the business requirements, another solution has been created. this solution has the following advantages:

a custom adfs login control minimizes redirect traffic to a minimum
own authentication logic can be implemented
a custom adfs control provided ultimate flexibility to the business. If build properly, new IP- or Resource-STS'ses can be added on the fly.
the control is ultimately stylable using, for example SharePoint designer.

WSP Deployment scopes and the development lifecycle in large environments

Bas — Sat, 21 Jul 2012 16:13:00 GMT

At our company, we use sharepoint to host all of our websites. As SharePoint is quite scalable, this is of course no problem and the production farm can handle it easily. These websites need to be developed and be tested, before they can be deployed to the production farm. This process is nothing special and doesn't diverge from most popular software development methodologies.

Development Lifecycle

As stated before: Our development process is not something special: we develop, test it, the customer accepts the product and finally we take the product into production.

Most of the developers that I know, develop their products on their own systems and they test it locally before the product is pushed to the development farm. At those companies, there is just one project (2 at most) which makes use of the development farm. The developers are free to deploy whatever and whenever they want, because they don't have to take other projects into account.

But our situation is quite different.We host a large amount of portals and websites (that number is still growing) and part of those existing portals are continously upgraded. As opposed to the situation above (with a single project working at a farm), we do have 10+ projects working simulteaniously at a single farm.

Deployment cycle

When just developing websites in ASP.Net, this isn't much of a problem, as all sites can easily be copied to it's location. As we all know, the deployment scenario in SharePoint, is different. Developers create WSP's, and those WSPs are deployed to the farm. In our case, code is checked into TFS, TFS builds the WSP's and some tooling, ROSS, gets triggered and deploys the generated WSP's to the development farm. That same tool deploys the solutions to Test, Acceptance and Production environments. This tool has been configured in such a way, that a solution only can be deployed to the production farm, when it has passed the Development, Test and Acceptance stages.

This automated deployment is very useful, but causes some problems for us. Due to the fact that we have multiple projects at the same central development farm, which makes it easy for developers to deploy solution pacakges, this causes a lot of retractions and deployments of WSP's. Every single WSP retraction and addition, causes the IIS application pools to recycle. This can (sometimes) take quite some time, can be frustating, and is, sadly enough, inherent to SharePoint Development. These deployments can cause other projects to recycle too. This is mainly due to the scope of the WSP's being deployed: when a WSP is deployed with a global scope, or is retracted, this affects ALL webapplications on the farm. Every single application gets a "free" application pool recycle. With two projects, this doesn't have to cause much problems: even when no deployment planning is made, those projects don't have to suffer much from each other.

But when more projects come in, things get worse, as seen in the diagram below. The availability of all web applications gets lower, which results into less testing time per app, and each deployment of a WSP blocks all other projects from deploying/testing. It's not only annoying, it also cost's a lot of serious money. The next diagram is just an indication with 2 deployments per project a day and assumes that a deployment on a farm with 1 webapplication takes as long as a deployment to a farm with 10+ webapplications. In my experience this is not the case at all: the more applications that co-exist on a farm, the more time a (global) solution deployment takes.

A solution for this problem is easy: split de big central Develoment farm into several small ones, as shown in the picture below:

This is a situation that, in my opinion, has a lot advantages (aside from the availability options), and fits into the cloud strategy of microsft (hosted development images), but doesn't solve all problems. Where the problems will disappear for developers, the problems will stay the same for the Test, Acceptance and Production environments. Not as much as at the development stage, but the deployments are still blocking other projects. The Test environment is heavily used by our testers and needs a lot of uptime, too. When looking from a business perspective: it's not acceptable that a commercial site, which sells products, has downtime because an other site is being deployed to production.

The Solution

The solution to get rid of the blocking deployments, but keep the flexibility of deploying "any time", needs a small hack. The deployment scope of the solution needs to be changed from "Global" to "WebApplication". This is not a setting in the package property window, but is determined "deploy time". In short: whenever there is an artefact or configuration option that needs to be deployed at the webapplication scope, the solution scope is "webapplication". In all other situations, the WSP will be globally deployed. The easiest way to do this, is to manually add a dummy safecontrol to the manifest.xml. Note: this doesn't work in all cases. There are a few situations where a solution only can globally be deployed to a farm. Try to bundle these functionalities into a single package, to keep downtime as low as possible.

As seen in the picture above, in the most ideal situation, projects don't block each other from testing. The only influence that projects have on each other, is the fact that wsp's can't be deployed simultaneously.

Conclusion

SharePoint solution deployments can be quite time consuming, and when not properly planned and configured, it can even block other projects when working simultaneously on a farm. The more projects that are using the farm, the more projects that can suffer from each other. A simple solution for this, is to force the solutions to be deployed at webapplication scope. This can be achieved by manually adding a safecontrol to the project.

GeoSearch with Fast Search for SharePoint 2010 - Part 3: how to use the latitude and longitude in your queries

Bas — Tue, 10 Apr 2012 19:21:00 GMT

This is the third and last part of a series of blogposts about GeoSearch with Fast Search for SharePoint. I recently held some presentations about extending the power of Fast (on DIWUG and the Dutch TechDays 2012) and one of the subjects that I talked about was on how to search for locations, sort on distances etcetera. The recording of my sessions are available on Channel 9 in dutch.

Part 1: why and how

Part 2: the custom pipeline extension to enrich the Fast index with spatial data

Part 3: how to use this metadata in your search queries (this post)

Please make sure that this code (and the code that is included) is not "production ready" at all, doesn't implement the geographic coordinate system correctly, as it has a range of [-180,180][-90,90] and that the computations don't take care of the fact that we live on a globe either.

Creating the queries

After implementing the custom pipeline extension and all the data has been re-indexed, the index isenriched with the latitude and longitude information. This information can be used for some interesting queries and some interesting sorting algorithms. When working with spatial data, there are some different approaches that can be used to retrieve the nearest locations and sort them. There is however one caveat to take care of, when a custom sorting formula is used. For this blogpost, I query directly against the Fast query service application using the code below. I also return a set of 3 managed properties: title, latitude and longitude.

The code gets the proxy that will be used, and instantiates a new keywordQuery object. On this object, I explicitly enable FQL, select my resultsprovider and I set the rowlimit to 20, as this set contains enough results to be displayed

Retrieve all results and sort them by distance to a certain point

This query is easy to execute, as the query "#" will retrieve all items. but when the managed properties are used in a sorting formula, things change. As the managed properties are of type decimal, these properties are handled differently, as described in this blogpost. For the sort formula, different algorithms can be used. Two populair sorting algorithms are the following:

Euclidean distance: the shortest, unique distance between two points.
Taxicab distance: the distance between two points is the absolute difference of their coordinates. The path between the two points doesn't have to be unique.

the next image shows the difference between the two different algorithms. The green line represents the euclidean algorithm and is the unique, shortest path between two points. The red, blue and yellow paths represent variations on the taxicab geometry and are indeed, not unique. The image comes from wikipedia, btw.

For this blogpost, I use the taxicab algorithm, as it is, by far, the most easy to implement ;). Basically, this algorithm is as follows:

distance = abs(queriedLocation.latitude - result.latitude) + abs(queriedLocation.longitude-result.longitude). In code, I implemented this with the following snippet:

this snippet creates the string that is used as sortformula. {0} and {1} are replaced by the latitude and longitude of the location that I need to find the nearest results for. The latitude and longitude in the formula, are the names of the managed properties that are retrieved from the index. For every result that is returned from the query, this formula is executed, and the resultset will be ordered based on the outcome of this formula. This sortformula needs to be added to the Sortlist, and the last step that is needed, is to execute the query. This is shown in the following snippet.

When using this technique of getting a resultset and sort using a sortformula, the rank that is returned is zero. Relevance models can't be used to sort within this resultset, and, for example, operators like xrank can't be used either.

Get all results within a certain area

Another method of getting results, which gives a bit more flexibility, is to retrieve all results within a certain area:

the cool thing about using this technique, is that a sorfformula to get all the nearest results, isn't needed, because that is already managed that in the query itself. Within the returned resultset, relevance models can be applied to rank results based on other properties!

a formula like this looks like the following:

set the latitude and longitude offsets
compute the borders of the bounding box
create the query: return all results with the latitude and the longitude in the respective ranges
execute the query

the fun starts when a xrank operator is thrown in:

the possibilities are almost limitless, when this technique is used! Queries like "return all restaurants in the neighbourhoud, but push all restaurants that pay for advertisements to the top". other possibilities are executing queries to retrieve items within an area, and rank them based on the keywords in the query. Think about using the Entity extractor "Location" that extracts all locations from a document, get all spatial data for that document, and execute a query like "give me all documents on burglaries near my current location".

the query has to be passed to the keywordquery object and thereafter be executed, just like in the other example.

Even more coolness!!

When all the results are returned, we can show the results in a boring list. But there is another option. Because we do have all latitude and longitude information, it's easy to plot them on a map. On a BING-map :D. Documentation on the API can be found here: http://www.microsoft.com/maps/developers/web.aspx. An example can be found as well in the code that is attached to this post.

Basically, all you have to do is the following:

Set the Bing-credentials and the applicationID (a free key that is available via the bing maps developer center)
set the center of the map
for each result to plot, add a pushpin
set the image size
get the map uri from the imageryservice
include the image in your webpart

I dont explain much of the code, because it isn't to hard and self-explanatory

this results in ultimately in the following webpart. Imagine what can be done. Add some nice javascripts to the pins to plot a route to the target or give extra information.

summary

we saw in this blogpost that there are several techniques to query for and sort on geo-locations. using these techniques in combination with some existing services, for example BING-Maps, can result in some really cool webparts. Code will be posted soon!

Upgrading Masterpage and UIVersion from SharePoint 2007 to SharePoint 2010 without v4.master

Bas — Tue, 20 Mar 2012 20:21:00 GMT

When upgrading from SharePoint 2007 to SharePoint 2010, several scenarios are possible. When using the database attach method, it's possible to stay in "V3" mode, or execute a visual upgrade too and make use of all SharePoint 2010 love. Whenever custom masterpages have been created for SharePoint 2007, the visual upgrade itself isn't too much of an option: it upgrades the UIVersion of the current web to V4, but the custom masterpage is replaced by the default v4.master, which (of course) doesn't provide any customizations. the next step is to create a new customized masterpage for SharePoint 2010, add it to your site and make sure to set this masterpage as the new default/custom masterpage.

The scenario I wanted to achieve, was the following:

Deploy my custom solutions with all masterpages already upgraded to SP2010, so only the UIVersion should be upgraded
Attach the content database
Upgrade the site to v4 mode, without the need of a v4.master deployment

The problems of (visual) upgrading to SharePoint 2010

The problems of visual upgrading consist of several issues: it doesn't work right out of the box (but that is expected), and some steps are needed to deploy and undeploy a masterpage, before the real SP2010 masterpage can be set, because the v4.master seems to be required

1 - Upgrade the site doesn't work immediately

As stated above, a visual upgrade using the user interface or upgrade by code/powershell is needed to let the existing site run in v4 mode:

1: $web = Get-SPWeb http://site

2: $web.UIVersion = 4

3: $web.Update()

sadly, this will bug out, because SharePoint requires the v4.master to be able to upgrade to v4-mode. The v4.master has to be made available via a feature (or manually) in the masterpage gallery, before the upgrade can be executed.

2 - Upgrade to a masterpage that isn't needed

Assuming that all customizations that did exist in SharePoint 2007 need to be retained, the above solution isn't the final solution. Next steps are to create a SharePoint 2010 masterpage with the customizations, deploy that masterpage, and set that masterpage as the default/custom masterpage for that site. And because that v4.master wasn't needed at all, that feature should be deactivated, or even better for hygienic reasons, be uninstalled and deleted.

why is this a problem?

As this isn't to much of a hassle for a few sites, it does whenever some big portals/intranets are migrated, with a lot of different masterpages. When working with about 10.000 webs, this needs to be automated, but activating anddeactivating 10000 times a feature can be quite time consuming. A lot of time can be saved, whenever the customized masterpages are already "SP2010-ready", so all is left to do, is to set the UIVersion to 4.

The solution of quickly applying a visual upgrade without a v4.master

As described above. I wanted to achieve the following scenario:

Deploy my custom solutions with all masterpages already upgraded to SP2010, so only the UIVersion should be upgraded
Attach the content database
Upgrade the site.

what happens behind the scenes?

When firing up reflector, we see that the following happens in code:

When the UIVersion is set, an internal function called "SetUIVersion" is executed. This function takes the UIVersion as a parameter and a boolean which decides wether or not to set the masterpage. Unfortunately, this function is an internal function, so it can't be used without using reflection.

The SetUIVersion function contains some logic to determine whether to rollback to UIVersion 3, or to upgrade to UIVersion 4. After the decision, the internal function SwitchMasterPages is called, with 5 parameters:

That function is called with the following parameters

And what happens in this function gets interesting. This function checks what default and custom masterpage are currently set (the V3 masterpages), and stores them in the propertybag of the current web, specified, by the parameters specified in the function. To determine what new masterpage needs to be set, that same propertybag is checked: in this case the properties "v4master" and "v4customMaster". if they are empty, the default v4.master is used:

and in this part of the function, the error is thrown that the default masterpage couldn't be found:

When the propertybag isn't empty, the value of that propertybag is set as masterpage:

And that is our solution!

Simply set add the v4master and v4customMaster to the SPWeb propertybag, with the values of the current masterpage. Update the UIVersion of the web and update it. That should do the trick:

Summary

Upgrading a SP2007 site to SP2010 is easy to do, but it can consume quite some time and operations. To streamline the upgrade process of a SP2007 site to SP2010, it's easier to upgrade the Sp2007 masterpages beforehand and include them, with the same name, in the solution. Deploy the solutions, attach the databases and upgrade every SPWeb using the above powershell. That should do the trick!

GeoSearch with Fast Search for SharePoint 2010 - Part 1

Bas — Fri, 24 Feb 2012 18:51:00 GMT

This is the first part of a series of blogposts about GeoSearch with Fast Search for SharePoint. I recently held some presentations about extending the power of Fast (on DIWUG and the Dutch TechDays 2012) and one of the subjects that I talked about was on how to search for locations, sort on distances etcetera. The recording of my sessions will be made available on Channel 9.

Part 1: why and how

Part 2: the custom pipeline extension to enrich the Fast index with spatial data

Part 3: how to use this metadata in your search queries

Why using Fast search for SharePoint 2010 for this purpose?

It's possible to compute distances between locations in a lot different ways. For example, address can be queried against SQL Server (if the correct data is available), or those addresses can be used in conjunction with the Bing Geocode services. A custom webpart with logic to query one of those services with the users/objects current location and all list items with location information takes little time, but performance issues can pop up in no time. how is the performance when there are 200 items in a list? and 2000? 20000? maybe 200000? Surely, I can imagine that there are some smart solutions to send 200000 locations to the geocode service and receive them back, but remember: how did you extract that information from a sharepoint list? That takes quite some time. It is even getting harder when data comes from several lists, not even thought about data from several site collections, external data (BCS is an option here btw) or, location information that resides inside document.

And this is where Fast comes into play. Fast is a very powerful search engine that can be custimized in various ways. First of all, Fast can index all of the information that lives inside SharePoint, or outside of SharePoint (using, for example, the BCS or a custom connector). Not matter what (besides security ;)), but whenever a query is executed on a certain keyword, all indexed data can be checked against hat keyword. at second comes the ability to enrich the index with extra information. The source for this information can be existing metadata from sitecolumns(address, city), data from inside the document, or data that is already extracted using the entity extractor. These sources can be used to query the geocode service of your choice to retrieve the spatial data for that source and can be added to the fast index.

This metadata can be used to query the index, determine distances to items within the index, or do other, cool things!

Customizing the pipeline

As told before, Fast can be used to index various sources with information. Think about SharePoint, external websites, lotus notes, oracle using jdbc, external systems like SAP using BCS and DUET Enterprise et cetera. When this data is indexed, a custom pipeline extension can be created that retrieves spatial data for that object and is added to the index. In short: all data from various sources can be indexed. If this data contains location information (address, city, country), this data can be used to retrieve the latitude and longitude of the item. This spatial data can be added to the index and thus, be queried.

Query the index

When the data has been indexed, this index can be queried using some easy queries. Some examples of queries that easily can be executed:

Give me all results of items within 5 km of my living area
Give me all information on cars and sort that information on distance, relative to my current location.

I bet that the speed of results showing up, can't be beaten by custom implementations that continiously need to query external services!

Fast Search for SharePoint problems - simplified architecture diagram

Bas — Tue, 09 Aug 2011 21:54:00 GMT

Recently I had a few colleagues (and myself) who tried to install FAST search for SharePoint on their development workstation. Despite the well written installation guidelines on MSDN, they had several problems, and sadly enough, they all faced different problems. These problems varied from not able to crawl the content to not being able to query the indexed content. As most of the colleagues weren't too familiar with Fast (and SharePoint search in general), I decided to create a small architecture diagram of the SharePoint 2010 Search Service Applications, Fast Search for SharePoint and (some of) their junction points. These junction points are displayed in the image below. To simplify the diagram, some items have been removed, the search query proxy is one example.

The diagram helped me and my colleagues to pinpoint the problems and I hope it will help you in the future with pinpointing problems (well, I hope you wont have any problems at all, but hey, it's Fast ;)). This post just covers a small subset of problems, but I think that the majority of most common problems are covered here. Whenever you think that a certain common problem is missing in here, please contact me, I will add it as soon as possible.

Use the following log/event locations to pinpoint problems:

c:\fastsearch\var\log
- contains a lot of folders with logging. Whenever problems arise, I check the folder "syslog" first
the Eventlog - increase the loglevels in the sharepoint diagnostic logging to verbose for extensive logging
- Application log
- Fast Search
- Fast Search Farm
SharePoint diagnostic logging - increase the log level to verbose
The IIS authentication logging
use nctrl status to check the status of your Fast installation

Some basic guidance:

Whenever no log errors appear on the Fast Search farm (event log as well as the \var\log), the problem lies within the SharePoint configuration
Whenever no errors appear on the Fast Search application, but errors do appear on the Fast IIS log, it's most likely a permission problem. My guess: database and/or
Whenever errors appear in Fast, you may not have SharePoint or permission problems, but a wrongly configured Fast environment. I didn't write about these errors below, but hostname configurations appeared to cause problems a lot.

As seen in the diagram, there are several junction points that can cause problems when not configured correctly:

Crawling SharePoint content
Push content into Fast
Communicatin with the Search Query and Settins Service
Accessing the admin service the search query and site settings service
Configure Fast via PowerShell

I have seen problems on all of these levels and I will try to (shortly) describe the possible problems, their symptoms and (a link to) their solution. A small disclaimer is in place: this is by no means a full overview of possible problems, but it should help in troubleshooting whenever problems arise.

1 - Crawling SharePoint content (1)

One of the most common problems are the problems with crawling content. Make sure that your default content access account has read permissions on the sharepoint sites that need to be crawled. Normally, this is handled by SharePoint. Possible error message:

Access is denied. Verify that either the Default Content Access Accouont has access to this repository, or add a crawl rule to crawl this repository. If the repository being crawled is a SharePoint repository, verify that the account you are using has "Full Read" permissions on the SharePoint Web Application being crawled.

2 - Push content into Fast (2)

After installing fast, pushing content into Fast can be a problem too. Several problems are possible here:

- SSL communication between the Fast Content Service and the Content Distributor is not configured correctly.

Visit technet on how to configure SSL communication. Please make sure that the account used in ".\SecureFASTSearchConnector.ps1 –certPath "path of the certificate\certificatename.pfx" –ssaName "name of your content SSA" –username “domain\username” is the account that runs the osearch14. In the Administration Services window this service is called "SharePoint Server Search 14"

- Permissions

After checking the Fast Content Service installation with "docpush -csp c:\testfile.txt" several errors can appear:

sp Reported error with http://cohowinery.com/c:\temp\bob.txt: processing:N/A:ProcessorDeploymentException: For pipeline 'Office14 (webcluster)', creating processor CompanyExtractorInclusionAny failed: ConfigurationError: CompanyExtractorInclusionAny: Failed to access or parse configuration file 'etc/resources/matching/configuration.companyextractor.inclusion.any.xml'.

This error can have several causes. Most likely is that the user doesn't have isn't member of the FastSearchAdministrators group. Another case I have encountered was that the account was member of that group, but for some reason did not have read access to the directory. Changing permissions fixed this issue for me, but this not recommended to do on a production server!! The message about http://cohowinery.com/ is not misconfigured on your system! It's a fictional company, same like the more known virtual companies contoso and wingtip toys

- I see dead crawlers

Another issue that can arise when Fast is installed on the same server as your SharePoint installation (not recommended, only for dev purposes!!), is that the enterprise crawler died for some reason. This can be checked with nctrl status. I described the cause and the solution here

3 - Communication with the Search Query and Settings Service (3)

Accessing Fast from your SiteCollection can cause several problems:

- No search results are returned

When a query is executed and no results are returned, the most likely issue is a permission issue: the account that you logged in with, doesn't have any permissions to view the indexed content. As the returned results are security trimmed by the query service, it's possible that the returned set contains 0 items. This happened to me when I created an anonymous Fast Search Center, whereas all my other indexed site collections weren't anonymously accessible. Another issue can be that (and this happens a lot after a fresh dev-installation!) the test site that is indexed, has content that hasn't been published yet. That content isn't indexed and thus, not returned.

- Internal Server error exception (when querying)

The application pool of the Search Query and settings service is not running. Check your IIS Manager to make sure that your application pool is up and that your website is running.

- Unexpected error occurred while communicating with Administration Service when accessing one of the following site settings menu entries

visual best bets
context information
promotion/demotion (ranking)

When this happens, please check the problems mentioned in paragraph 5 and 6 too. As these functions communicate with the Fast Administration service, it's likely that these problems occur, while the query component works fine. It's most likely a permissions issue, where the Search Query and settings service (mentioned in paragraph 5), doesn't have the right administration account configured.

4 - Accessing the admin service through the search query and site settings service (5)

- Unexpected error occurred while communicating with Administration Service

This error mainly has 3 rootcauses:

The Fast Administration services is not configured correctly (e.g. the wrong location is configured in the Fast Query service application)
The Fast Administration application pool is not running
Permissions ;). Make sure that that the right Fast Admin account is configured in the Fast Query service application.

5 - Fast via PowerShell (7)

Powershell can show some initial "weird" behaviour. Whenever the Fast WCF-services aren't running, all of the powershell commands will bug out with an error message. Get-FastSearchContentCollection returns:

Get-FastSearchContentCollection : Failed to communicate with the WCF service.

Whenever the services are back up, this message will disappear and the right output is returned. But sometimes some Powershell commands do work, and some won't. This is definately a permissions issue, where the executing account isn't member of the FastSearchAdministrators group. It's comparable to the missing/wrong permissions for the fast query service application.

Summary

The majority of problems arise whenever permissions are not set correctly. It's important that the Fast Admin account that is configured in the Fast Query Service, is indeed the right account, is member of the FastSearchAdministrators group and has access the the FastAdmin database. For dev-purposes you can choose the make use of just one main account to run fast, run the app pools and use as the admin account, but this is no recommended approach for production environments. That is why I advise to always use least privilege installations, even on your dev-environments!

SharePoint 2010 - Fast Search Center site template not available (bug?!)

Bas — Wed, 27 Jul 2011 10:08:00 GMT

Recently I found out that, my Fast Search Center was not available from the “New Site” menu. Both the english as well the dutch fast search template was not available.

As I saw this happen before, I checked out the site collection features overview, to enable the “SharePoint Server Enterprise Site Collection features”. I had to scratch my head twice when I saw that that feature was already activated: that feature should add fast search center templates to the available templates. Next thing I did was navigate to the “Page Layout and Site Template settings”, to make sure that the Fast Search Center template was available for creating sub-sites. To my surprise, the template was available! Still, I couldn’t select that template when creating a new site.

I remembered that there is a hidden sitecollection feature (search extensions) that needs to be enabled to make use of the Fast specific features like visual best bets and that enables the administration links to visual best bets, user contexts and site promotion/demotion. After activating that feature, the Fast Search center template became available.

This is strange, because when checking that feature.xml, we learn the following:

<?xml version="1.0" encoding="utf-8" ?>
<!-- Copyright (c) Microsoft Corporation. All rights reserved. -->
<Feature  Id="5EAC763D-FBF5-4d6f-A76B-EDED7DD7B0A5"
          Title="$Resources:SearchExtensionsFeatureTitle;"
          Description="$Resources:SearchExtensionsFeatureDescription;"
          DefaultResourceFile="Microsoft.Office.Server.Search"
          SolutionId="7ED6CD55-B479-4EB7-A529-E99A24C10BD3"
          Version="14.0.0.0"
          Scope="Site"
          Hidden="TRUE"
          xmlns="http://schemas.microsoft.com/sharepoint/">
  <ElementManifests>
    <ElementManifest Location="webPartDWPFiles.xml"/>
    <ElementManifest Location="ExtendedSearchAdminLinks.xml"/>
  </ElementManifests>
</Feature>

There is no feature receiver
There is no feature dependency
The feature only
- Adds a webpart (webpartdwpfiles.xml)
- Adds Search admin links (ExtendedSearchAdminLinks.xml)

For some reason this only happened for existing site collections. That made me think about some recent changes to the platform:

SP1 was installed
SP1 for language packs was installed
CU June was installed.

When deactivating the hidden feature (Search Extensions), the english template was gone again, whereas the dutch search template still existed. I definately this needs some further investigation.

As it only happened for the sites that existed before that change, my guess it has to do with a combination of those updates. For new sites, everything works as intended. My guess is that it has to do with the combination of installed/registred language packs and the activation/deactivation of the search extension feature, which provides a default language resource named "Microsoft.Office.Server.Search".