Search results matching tags 'Sharepoint' and 'Fast'

GeoSearch with Fast Search for SharePoint 2010 - Part 3: how to use the latitude and longitude in your queries

Bas — Tue, 10 Apr 2012 19:21:00 GMT

This is the third and last part of a series of blogposts about GeoSearch with Fast Search for SharePoint. I recently held some presentations about extending the power of Fast (on DIWUG and the Dutch TechDays 2012) and one of the subjects that I talked about was on how to search for locations, sort on distances etcetera. The recording of my sessions are available on Channel 9 in dutch.

Part 1: why and how

Part 2: the custom pipeline extension to enrich the Fast index with spatial data

Part 3: how to use this metadata in your search queries (this post)

Please make sure that this code (and the code that is included) is not "production ready" at all, doesn't implement the geographic coordinate system correctly, as it has a range of [-180,180][-90,90] and that the computations don't take care of the fact that we live on a globe either.

Creating the queries

After implementing the custom pipeline extension and all the data has been re-indexed, the index isenriched with the latitude and longitude information. This information can be used for some interesting queries and some interesting sorting algorithms. When working with spatial data, there are some different approaches that can be used to retrieve the nearest locations and sort them. There is however one caveat to take care of, when a custom sorting formula is used. For this blogpost, I query directly against the Fast query service application using the code below. I also return a set of 3 managed properties: title, latitude and longitude.

The code gets the proxy that will be used, and instantiates a new keywordQuery object. On this object, I explicitly enable FQL, select my resultsprovider and I set the rowlimit to 20, as this set contains enough results to be displayed

Retrieve all results and sort them by distance to a certain point

This query is easy to execute, as the query "#" will retrieve all items. but when the managed properties are used in a sorting formula, things change. As the managed properties are of type decimal, these properties are handled differently, as described in this blogpost. For the sort formula, different algorithms can be used. Two populair sorting algorithms are the following:

Euclidean distance: the shortest, unique distance between two points.
Taxicab distance: the distance between two points is the absolute difference of their coordinates. The path between the two points doesn't have to be unique.

the next image shows the difference between the two different algorithms. The green line represents the euclidean algorithm and is the unique, shortest path between two points. The red, blue and yellow paths represent variations on the taxicab geometry and are indeed, not unique. The image comes from wikipedia, btw.

For this blogpost, I use the taxicab algorithm, as it is, by far, the most easy to implement ;). Basically, this algorithm is as follows:

distance = abs(queriedLocation.latitude - result.latitude) + abs(queriedLocation.longitude-result.longitude). In code, I implemented this with the following snippet:

this snippet creates the string that is used as sortformula. {0} and {1} are replaced by the latitude and longitude of the location that I need to find the nearest results for. The latitude and longitude in the formula, are the names of the managed properties that are retrieved from the index. For every result that is returned from the query, this formula is executed, and the resultset will be ordered based on the outcome of this formula. This sortformula needs to be added to the Sortlist, and the last step that is needed, is to execute the query. This is shown in the following snippet.

When using this technique of getting a resultset and sort using a sortformula, the rank that is returned is zero. Relevance models can't be used to sort within this resultset, and, for example, operators like xrank can't be used either.

Get all results within a certain area

Another method of getting results, which gives a bit more flexibility, is to retrieve all results within a certain area:

the cool thing about using this technique, is that a sorfformula to get all the nearest results, isn't needed, because that is already managed that in the query itself. Within the returned resultset, relevance models can be applied to rank results based on other properties!

a formula like this looks like the following:

set the latitude and longitude offsets
compute the borders of the bounding box
create the query: return all results with the latitude and the longitude in the respective ranges
execute the query

the fun starts when a xrank operator is thrown in:

the possibilities are almost limitless, when this technique is used! Queries like "return all restaurants in the neighbourhoud, but push all restaurants that pay for advertisements to the top". other possibilities are executing queries to retrieve items within an area, and rank them based on the keywords in the query. Think about using the Entity extractor "Location" that extracts all locations from a document, get all spatial data for that document, and execute a query like "give me all documents on burglaries near my current location".

the query has to be passed to the keywordquery object and thereafter be executed, just like in the other example.

Even more coolness!!

When all the results are returned, we can show the results in a boring list. But there is another option. Because we do have all latitude and longitude information, it's easy to plot them on a map. On a BING-map :D. Documentation on the API can be found here: http://www.microsoft.com/maps/developers/web.aspx. An example can be found as well in the code that is attached to this post.

Basically, all you have to do is the following:

Set the Bing-credentials and the applicationID (a free key that is available via the bing maps developer center)
set the center of the map
for each result to plot, add a pushpin
set the image size
get the map uri from the imageryservice
include the image in your webpart

I dont explain much of the code, because it isn't to hard and self-explanatory

this results in ultimately in the following webpart. Imagine what can be done. Add some nice javascripts to the pins to plot a route to the target or give extra information.

summary

we saw in this blogpost that there are several techniques to query for and sort on geo-locations. using these techniques in combination with some existing services, for example BING-Maps, can result in some really cool webparts. Code will be posted soon!

Fast Search for SharePoint problems - simplified architecture diagram

Bas — Tue, 09 Aug 2011 21:54:00 GMT

Recently I had a few colleagues (and myself) who tried to install FAST search for SharePoint on their development workstation. Despite the well written installation guidelines on MSDN, they had several problems, and sadly enough, they all faced different problems. These problems varied from not able to crawl the content to not being able to query the indexed content. As most of the colleagues weren't too familiar with Fast (and SharePoint search in general), I decided to create a small architecture diagram of the SharePoint 2010 Search Service Applications, Fast Search for SharePoint and (some of) their junction points. These junction points are displayed in the image below. To simplify the diagram, some items have been removed, the search query proxy is one example.

The diagram helped me and my colleagues to pinpoint the problems and I hope it will help you in the future with pinpointing problems (well, I hope you wont have any problems at all, but hey, it's Fast ;)). This post just covers a small subset of problems, but I think that the majority of most common problems are covered here. Whenever you think that a certain common problem is missing in here, please contact me, I will add it as soon as possible.

Use the following log/event locations to pinpoint problems:

c:\fastsearch\var\log
- contains a lot of folders with logging. Whenever problems arise, I check the folder "syslog" first
the Eventlog - increase the loglevels in the sharepoint diagnostic logging to verbose for extensive logging
- Application log
- Fast Search
- Fast Search Farm
SharePoint diagnostic logging - increase the log level to verbose
The IIS authentication logging
use nctrl status to check the status of your Fast installation

Some basic guidance:

Whenever no log errors appear on the Fast Search farm (event log as well as the \var\log), the problem lies within the SharePoint configuration
Whenever no errors appear on the Fast Search application, but errors do appear on the Fast IIS log, it's most likely a permission problem. My guess: database and/or
Whenever errors appear in Fast, you may not have SharePoint or permission problems, but a wrongly configured Fast environment. I didn't write about these errors below, but hostname configurations appeared to cause problems a lot.

As seen in the diagram, there are several junction points that can cause problems when not configured correctly:

Crawling SharePoint content
Push content into Fast
Communicatin with the Search Query and Settins Service
Accessing the admin service the search query and site settings service
Configure Fast via PowerShell

I have seen problems on all of these levels and I will try to (shortly) describe the possible problems, their symptoms and (a link to) their solution. A small disclaimer is in place: this is by no means a full overview of possible problems, but it should help in troubleshooting whenever problems arise.

1 - Crawling SharePoint content (1)

One of the most common problems are the problems with crawling content. Make sure that your default content access account has read permissions on the sharepoint sites that need to be crawled. Normally, this is handled by SharePoint. Possible error message:

Access is denied. Verify that either the Default Content Access Accouont has access to this repository, or add a crawl rule to crawl this repository. If the repository being crawled is a SharePoint repository, verify that the account you are using has "Full Read" permissions on the SharePoint Web Application being crawled.

2 - Push content into Fast (2)

After installing fast, pushing content into Fast can be a problem too. Several problems are possible here:

- SSL communication between the Fast Content Service and the Content Distributor is not configured correctly.

Visit technet on how to configure SSL communication. Please make sure that the account used in ".\SecureFASTSearchConnector.ps1 –certPath "path of the certificate\certificatename.pfx" –ssaName "name of your content SSA" –username “domain\username” is the account that runs the osearch14. In the Administration Services window this service is called "SharePoint Server Search 14"

- Permissions

After checking the Fast Content Service installation with "docpush -csp c:\testfile.txt" several errors can appear:

sp Reported error with http://cohowinery.com/c:\temp\bob.txt: processing:N/A:ProcessorDeploymentException: For pipeline 'Office14 (webcluster)', creating processor CompanyExtractorInclusionAny failed: ConfigurationError: CompanyExtractorInclusionAny: Failed to access or parse configuration file 'etc/resources/matching/configuration.companyextractor.inclusion.any.xml'.

This error can have several causes. Most likely is that the user doesn't have isn't member of the FastSearchAdministrators group. Another case I have encountered was that the account was member of that group, but for some reason did not have read access to the directory. Changing permissions fixed this issue for me, but this not recommended to do on a production server!! The message about http://cohowinery.com/ is not misconfigured on your system! It's a fictional company, same like the more known virtual companies contoso and wingtip toys

- I see dead crawlers

Another issue that can arise when Fast is installed on the same server as your SharePoint installation (not recommended, only for dev purposes!!), is that the enterprise crawler died for some reason. This can be checked with nctrl status. I described the cause and the solution here

3 - Communication with the Search Query and Settings Service (3)

Accessing Fast from your SiteCollection can cause several problems:

- No search results are returned

When a query is executed and no results are returned, the most likely issue is a permission issue: the account that you logged in with, doesn't have any permissions to view the indexed content. As the returned results are security trimmed by the query service, it's possible that the returned set contains 0 items. This happened to me when I created an anonymous Fast Search Center, whereas all my other indexed site collections weren't anonymously accessible. Another issue can be that (and this happens a lot after a fresh dev-installation!) the test site that is indexed, has content that hasn't been published yet. That content isn't indexed and thus, not returned.

- Internal Server error exception (when querying)

The application pool of the Search Query and settings service is not running. Check your IIS Manager to make sure that your application pool is up and that your website is running.

- Unexpected error occurred while communicating with Administration Service when accessing one of the following site settings menu entries

visual best bets
context information
promotion/demotion (ranking)

When this happens, please check the problems mentioned in paragraph 5 and 6 too. As these functions communicate with the Fast Administration service, it's likely that these problems occur, while the query component works fine. It's most likely a permissions issue, where the Search Query and settings service (mentioned in paragraph 5), doesn't have the right administration account configured.

4 - Accessing the admin service through the search query and site settings service (5)

- Unexpected error occurred while communicating with Administration Service

This error mainly has 3 rootcauses:

The Fast Administration services is not configured correctly (e.g. the wrong location is configured in the Fast Query service application)
The Fast Administration application pool is not running
Permissions ;). Make sure that that the right Fast Admin account is configured in the Fast Query service application.

5 - Fast via PowerShell (7)

Powershell can show some initial "weird" behaviour. Whenever the Fast WCF-services aren't running, all of the powershell commands will bug out with an error message. Get-FastSearchContentCollection returns:

Get-FastSearchContentCollection : Failed to communicate with the WCF service.

Whenever the services are back up, this message will disappear and the right output is returned. But sometimes some Powershell commands do work, and some won't. This is definately a permissions issue, where the executing account isn't member of the FastSearchAdministrators group. It's comparable to the missing/wrong permissions for the fast query service application.

Summary

The majority of problems arise whenever permissions are not set correctly. It's important that the Fast Admin account that is configured in the Fast Query Service, is indeed the right account, is member of the FastSearchAdministrators group and has access the the FastAdmin database. For dev-purposes you can choose the make use of just one main account to run fast, run the app pools and use as the admin account, but this is no recommended approach for production environments. That is why I advise to always use least privilege installations, even on your dev-environments!

SharePoint 2010 - Fast Search Center site template not available (bug?!)

Bas — Wed, 27 Jul 2011 10:08:00 GMT

Recently I found out that, my Fast Search Center was not available from the “New Site” menu. Both the english as well the dutch fast search template was not available.

As I saw this happen before, I checked out the site collection features overview, to enable the “SharePoint Server Enterprise Site Collection features”. I had to scratch my head twice when I saw that that feature was already activated: that feature should add fast search center templates to the available templates. Next thing I did was navigate to the “Page Layout and Site Template settings”, to make sure that the Fast Search Center template was available for creating sub-sites. To my surprise, the template was available! Still, I couldn’t select that template when creating a new site.

I remembered that there is a hidden sitecollection feature (search extensions) that needs to be enabled to make use of the Fast specific features like visual best bets and that enables the administration links to visual best bets, user contexts and site promotion/demotion. After activating that feature, the Fast Search center template became available.

This is strange, because when checking that feature.xml, we learn the following:

<?xml version="1.0" encoding="utf-8" ?>
<!-- Copyright (c) Microsoft Corporation. All rights reserved. -->
<Feature  Id="5EAC763D-FBF5-4d6f-A76B-EDED7DD7B0A5"
          Title="$Resources:SearchExtensionsFeatureTitle;"
          Description="$Resources:SearchExtensionsFeatureDescription;"
          DefaultResourceFile="Microsoft.Office.Server.Search"
          SolutionId="7ED6CD55-B479-4EB7-A529-E99A24C10BD3"
          Version="14.0.0.0"
          Scope="Site"
          Hidden="TRUE"
          xmlns="http://schemas.microsoft.com/sharepoint/">
  <ElementManifests>
    <ElementManifest Location="webPartDWPFiles.xml"/>
    <ElementManifest Location="ExtendedSearchAdminLinks.xml"/>
  </ElementManifests>
</Feature>

There is no feature receiver
There is no feature dependency
The feature only
- Adds a webpart (webpartdwpfiles.xml)
- Adds Search admin links (ExtendedSearchAdminLinks.xml)

For some reason this only happened for existing site collections. That made me think about some recent changes to the platform:

SP1 was installed
SP1 for language packs was installed
CU June was installed.

When deactivating the hidden feature (Search Extensions), the english template was gone again, whereas the dutch search template still existed. I definately this needs some further investigation.

As it only happened for the sites that existed before that change, my guess it has to do with a combination of those updates. For new sites, everything works as intended. My guess is that it has to do with the combination of installed/registred language packs and the activation/deactivation of the search extension feature, which provides a default language resource named "Microsoft.Office.Server.Search".

Fast: Adding managed metadata properties for the use with scopes in SharePoint 2010

Bas — Fri, 03 Jun 2011 11:17:00 GMT

Lately I was working on a project that required to make use of scopes and some managed metadata properties in those scopes. For some reason, those managed metadata properties didnt show up in my scope selection dropdownlist, but I was sure that this metadata property was available in my Fast Managed Metadata properties list:

as you can see, this property doesnt show up in my properties list:

Whenever the SharePoint Search metadataproperties page is accessed, the option "Use in scopes" for the managed property ContentType is turned off:

When this property is turned on, the property is available to be used in the scopes. According to MSDN, this is working "as intended". The managed properties that are created in the SharePoint search administration page, are just placeholders for the Fast managed metadata properties. The configuration of this managed property is, however, done in the Fast Managed metadata properties page.

So, to be able to use managed properties in Fast Search scopes, the following steps need to be taken:

Add a Managed Metadata property via the Fast Search Administration page and configure it
Add a metadata property via the SharePoint search administration page (found on the Fast Query SSA) and make it available for the use in scopes. No other configurations need to be made.
Now you're ready to use your managed metadata property