Search results matching tag 'Fast'

Fast Search for SharePoint: crawler process blocked due to memory capacity limit

Bas — Wed, 22 Aug 2012 20:42:00 GMT

This week I encountered the problem that the indexation of content didn’t seem to end. After 5 hours of crawling, my 1000 html pages of content still weren't indexed by the Fast crawler. After putting on my Sherlock Holmes outfit, I started to investigate this problem. It turned out to be a real adventure to find a solution for this problem.

Investigating the problem

Firing up the ULS-viewer learned me that my development machine was too low on resources:

08/22/2012 21:22:36.40 mssearch.exe (0x09D4) 0x1448 SharePoint Server Search Content Plugin dtv8 Medium At memory capacity. Load is 82%, configured to block at 80%. have been waiting 01:57 to queue this document [documentmanager.cpp:969] d:\office\source\search\native\gather\plugins\contentpi\documentmanager.cpp

This meant that 6.4Gb of the available 8Gb of RAM was in use. The easy solution was to kill some processes, and after shutting down some applications, I saw my CPU usage increase, which meant that Fast was crawling my content again…. for a minute or two. So I decided to kill some more, somewhat useless, processes: Fast resumed to crawl my content again. But the process of a blocking crawl process repeated itself and it started out to be a race to keep memory usage under the 80%. Every minute I checked out if I could kill some (useless) processes to keep that memory usage below 80%: this killed my productivity. I tried to kill some of the more heavier processes to finish the crawl and even disabled some services and service applications: but for some reason my memory consumption was still too high.

Solving the problem

I decided that, if I couldn't manage my memory to keep it below the magic number, I should change that magic number into something else. Of course today was my lucky day: a smart search query on google for "fast search sharepoint memory capacity block" didn't return any relevant results. Using my brains and a swiss-army knife was now my only option to get a solution for this challenge. I wasn't sure if this was a Fast specific problem, or a SharePoint search problem, and after starting (and finishing) a crawl using the Fast Query Service Application (which, in fact, is a SharePoint Server Search Application), showed me that the problem didn't arise here. I concluded that this was a fast specific problem. I started to check out on configuration files, Fast cmdlets and their parameters, configuration options in the central admin, but as I didn't know what to look for exactly, this turned out to be a hopeless action. Finally, I even decided to manually check out the registry, and, of course, this action turned out to be useless too. This made me starting to believe that this "configurable" setting wasn't so configurable.

As a last hope, I fired up the sysinternals process monitor tool (http://www.sysinternals.com/), which shows all filesystem/registry/network activity for all processes, and configured it to display all registry and filesystem activity for the mssearch process. After restarting this sharepoint search service, I did a search for "memory", and after a few hits, I just "knew" why I couldn't find any configuration setting related to this memory limit: It just wasn't there!

Manually adding this key to the registry was, of course, not the solution. After each restart of the service, the setting got deleted from the registry. But after taking a closer look at those registry keys, I remembered that the FastConnector:Collection and FastConnector:ContentDistributor were Fast specific service application settings and tried to add this key to the Extended Properties of my Fast Service Application

The key was added using the following powershell script:

$ssa = Get-SPServiceApplication | where {$_.DisplayName -eq "FastContentService" }
$property = New-SPEnterpriseSearchExtendedConnectorProperty -SearchApplication $ssa -Name "MaxMemoryLoad" -Value "82"

The MaxMemoryValue of "82" was taken on purpose, because I wanted to make sure that this new value was showing up in the logs when crawling content, to make sure it was really this setting. After restarting the Search service, process monitor told me that the key was loaded!

After checking out the ULS log, I was sure that I found the correct parameter:

08/22/2012 22:27:22.69 mssearch.exe (0x0A78) 0x1648 SharePoint Server Search Content Plugin dtv8 Medium At memory capacity. Load is 90%, configured to block at 82%. have been waiting 02:27 to queue this document [documentmanager.cpp:969] d:\office\source\search\native\gather\plugins\contentpi\documentmanager.cpp

Summary

Changing the maximum memory load isn't hard, it's just undocumented. I strongly advise to not to change this value in production environments, as I don't know if this is supported by Microsoft (it might be for a reason that there is no documentation on this subject). For development environments, this configuration setting is quite useful, as those enviroments, usually, take up a lot of resources. I would suggest that shutting down unnecessary processes should always be the first option prior to adjust this configuration.

Update: limiting the memory consumption of your sql server should be a very interesting option too, to limit your memory usage.

GeoSearch with Fast Search for SharePoint 2010 - Part 3: how to use the latitude and longitude in your queries

Bas — Tue, 10 Apr 2012 19:21:00 GMT

This is the third and last part of a series of blogposts about GeoSearch with Fast Search for SharePoint. I recently held some presentations about extending the power of Fast (on DIWUG and the Dutch TechDays 2012) and one of the subjects that I talked about was on how to search for locations, sort on distances etcetera. The recording of my sessions are available on Channel 9 in dutch.

Part 1: why and how

Part 2: the custom pipeline extension to enrich the Fast index with spatial data

Part 3: how to use this metadata in your search queries (this post)

Please make sure that this code (and the code that is included) is not "production ready" at all, doesn't implement the geographic coordinate system correctly, as it has a range of [-180,180][-90,90] and that the computations don't take care of the fact that we live on a globe either.

Creating the queries

After implementing the custom pipeline extension and all the data has been re-indexed, the index isenriched with the latitude and longitude information. This information can be used for some interesting queries and some interesting sorting algorithms. When working with spatial data, there are some different approaches that can be used to retrieve the nearest locations and sort them. There is however one caveat to take care of, when a custom sorting formula is used. For this blogpost, I query directly against the Fast query service application using the code below. I also return a set of 3 managed properties: title, latitude and longitude.

The code gets the proxy that will be used, and instantiates a new keywordQuery object. On this object, I explicitly enable FQL, select my resultsprovider and I set the rowlimit to 20, as this set contains enough results to be displayed

Retrieve all results and sort them by distance to a certain point

This query is easy to execute, as the query "#" will retrieve all items. but when the managed properties are used in a sorting formula, things change. As the managed properties are of type decimal, these properties are handled differently, as described in this blogpost. For the sort formula, different algorithms can be used. Two populair sorting algorithms are the following:

Euclidean distance: the shortest, unique distance between two points.
Taxicab distance: the distance between two points is the absolute difference of their coordinates. The path between the two points doesn't have to be unique.

the next image shows the difference between the two different algorithms. The green line represents the euclidean algorithm and is the unique, shortest path between two points. The red, blue and yellow paths represent variations on the taxicab geometry and are indeed, not unique. The image comes from wikipedia, btw.

For this blogpost, I use the taxicab algorithm, as it is, by far, the most easy to implement ;). Basically, this algorithm is as follows:

distance = abs(queriedLocation.latitude - result.latitude) + abs(queriedLocation.longitude-result.longitude). In code, I implemented this with the following snippet:

this snippet creates the string that is used as sortformula. {0} and {1} are replaced by the latitude and longitude of the location that I need to find the nearest results for. The latitude and longitude in the formula, are the names of the managed properties that are retrieved from the index. For every result that is returned from the query, this formula is executed, and the resultset will be ordered based on the outcome of this formula. This sortformula needs to be added to the Sortlist, and the last step that is needed, is to execute the query. This is shown in the following snippet.

When using this technique of getting a resultset and sort using a sortformula, the rank that is returned is zero. Relevance models can't be used to sort within this resultset, and, for example, operators like xrank can't be used either.

Get all results within a certain area

Another method of getting results, which gives a bit more flexibility, is to retrieve all results within a certain area:

the cool thing about using this technique, is that a sorfformula to get all the nearest results, isn't needed, because that is already managed that in the query itself. Within the returned resultset, relevance models can be applied to rank results based on other properties!

a formula like this looks like the following:

set the latitude and longitude offsets
compute the borders of the bounding box
create the query: return all results with the latitude and the longitude in the respective ranges
execute the query

the fun starts when a xrank operator is thrown in:

the possibilities are almost limitless, when this technique is used! Queries like "return all restaurants in the neighbourhoud, but push all restaurants that pay for advertisements to the top". other possibilities are executing queries to retrieve items within an area, and rank them based on the keywords in the query. Think about using the Entity extractor "Location" that extracts all locations from a document, get all spatial data for that document, and execute a query like "give me all documents on burglaries near my current location".

the query has to be passed to the keywordquery object and thereafter be executed, just like in the other example.

Even more coolness!!

When all the results are returned, we can show the results in a boring list. But there is another option. Because we do have all latitude and longitude information, it's easy to plot them on a map. On a BING-map :D. Documentation on the API can be found here: http://www.microsoft.com/maps/developers/web.aspx. An example can be found as well in the code that is attached to this post.

Basically, all you have to do is the following:

Set the Bing-credentials and the applicationID (a free key that is available via the bing maps developer center)
set the center of the map
for each result to plot, add a pushpin
set the image size
get the map uri from the imageryservice
include the image in your webpart

I dont explain much of the code, because it isn't to hard and self-explanatory

this results in ultimately in the following webpart. Imagine what can be done. Add some nice javascripts to the pins to plot a route to the target or give extra information.

summary

we saw in this blogpost that there are several techniques to query for and sort on geo-locations. using these techniques in combination with some existing services, for example BING-Maps, can result in some really cool webparts. Code will be posted soon!

GeoSearch with Fast Search for SharePoint 2010 - Part 2: Custom pipeline extensions

Bas — Fri, 24 Feb 2012 19:47:00 GMT

This is the second part of a series of blogposts about GeoSearch with Fast Search for SharePoint. I recently held some presentations about extending the power of Fast (on DIWUG and the Dutch TechDays 2012) and one of the subjects that I talked about was on how to search for locations, sort on distances etcetera. The recording of my sessions will be made available on Channel 9.

Part 1: why and how

Part 2: the custom pipeline extension to enrich the Fast index with spatial data

Part 3: how to use this metadata in your search queries

EDIT: Sourcode for the custom pipeline extension

Indexing the data

As previously blogged, Fast crawles data and puts this data into the index.

One of the processes that happens during the indexation process, is processing the content. During this process, this data is traversed through a "pipeline", which consists of several stages:

The stages as shown in the picture above is just a small subset, but basically, it does the following:

Normalize the document -> the data of each input is normalized, so that every stage doesn't have problems processing this content
Language detection -> determine the language of the document. This metadata is used in other pipelines to, for example, determine what dictionary should be used.
Lemmatizer -> based on the language that is detected, lemma and stemma are determined of words. the definition on wikipedia: canonical form, dictionary form, or citation form of a set of words (headword). Think about the following: bank - banks or good - better - best
Entity Extraction -> extracts entities, based on a dictionary, from the data that is processed. Out of the box Locations, Persons and companies are extracted

The cool thing about this pipeline is: this pipeline can be extended!

Extending the pipeline

As we want to work with spatial data, a custom pipeline needs to be created, which has any location data as input, and which can output spatial data: latitude and longitude. All these properties need to be crawled properties:

What can be used as input...

Before the pipeline can be created, it's needed to know what kind of data is available. It's important to know what data is available, where it resides, and how it can be used to be processed. This data must be available through crawled properties. Some data that is often available is:

locations (Data retrieved using the Location Entity Extractor)
ows_WorkAddress (standard sharepoint sitecolumn)
ows_City (standard sharepoint sitecolumn)

When other data needs to be processed, make sure to check out what it is named inside the crawled properties overview in the Fast Query Service applications. In this example, I am using the address and city as an input.

...and spit it out..

This is an easy one. As we know we want to use latitude and longitude information to do some nice computations with, these ones can be defined as crawled properties. This can be done using the following powershell commands:

$guid = [System.Guid]::NewGuid().toString()
$propset = New-FASTSearchMetadataCategory -Name "spatial" -Propset $guid
New-FASTSearchMetadataCrawledProperty -Name "latitude" -VariantType 5 -Propset $guid
New-FASTSearchMetadataCrawledProperty -Name "longitude" -VariantType 5 -Propset $guid

the command creates a separate category named "spatial" and created two crawled properties of variant type "5", which is, according to Technet, a decimal.

Between Sheets

a custom pipeline extension can be any type of executable: powershell, .bat, or an .exe. In this example a .exe is build using visual studio. All have in common that they have xml as input and xml as output. For the example, the following xml is used as input:

<?xml version="1.0" encoding="utf-8"?>
<Document>
  <CrawledProperty propertySet="00130329-0000-0130-c000-000000131346" varType="31" propertyName="ows_WorkCity">City</CrawledProperty>
  <CrawledProperty propertySet="00130329-0000-0130-c000-000000131346" varType="31" propertyName="ows_WorkAddress">Address</CrawledProperty>
</Document>

and the output that will be created looks the same:

<?xml version="1.0" encoding="utf-8"?>
<Document>
  <CrawledProperty propertySet="00130329-0000-0130-c000-000000131346" propertyName="latitude" varType="5">99.9999999999999</CrawledProperty>
  <CrawledProperty propertySet="00130329-0000-0130-c000-000000131346" propertyName="longitude" varType="5">99.9999999999999</CrawledProperty>
</Document>

Using this information, when creating a custom pipeline, the following pattern is always recognizable:

Read the input:

 XDocument inputDoc = XDocument.Load(args[0]);
 
            //get the Location from the input item:            
            // care: vartype == "31": ...
            var cities = from cp in inputDoc.Descendants("CrawledProperty")
                      where new Guid(cp.Attribute("propertySet").Value).Equals(IN_LOCATION) &&   
                      cp.Attribute("propertyName").Value == "ows_WorkCity" &&                    
                      cp.Attribute("varType").Value == "31"
                      select cp.Value;

and write back the output:

            // write back results
            var outputElement = new XElement("Document");
            if (coordinate!=null)
            {                
                // Add crawled properties
                outputElement.Add(
                    new XElement("CrawledProperty",
                        new XAttribute("propertySet", OUT_GROUP),
                        new XAttribute("propertyName", Propname),
                        new XAttribute("varType", 5),
                            coordinate.Longitude)
                        );
            }
 
            outputElement.Save(args[1]);

But what happens in between, is more interesting! the input data is simply passed through a geocode service, and the results are simply returned as spatial data. Below is a somewhat simplified overview of the code:

            // create geocode request query (comma separated string: location, address)
            geocodeRequest.Query = query;
            //instantiate geocode service
            var geocodeService = new GeocodeServiceClient("BasicHttpBinding_IGeocodeService");
            // and query the service. Do whatever you want with the response!
            var response = geocodeService.Geocode(geocodeRequest);

And that is all that is needed to create the custom pipeline extension.

How to insert it in the pipeline

now the pipeline is ready, all that is needed, is to hook it up to the pipeline. As of service pack 1, this is done as follows:

Go to the bin-directory of your Fast installation (default: C:\FASTSearch\bin)
copy extractor together with it's required files (filename.xmlserializer.dll, filename.app,config)
Open the file "pipelineextensibility.xml" in the etc directory (C:\FASTSearch\etc)
Insert the following xml:

all you need to do is: map the crawled properties to a managed property, execute psctrl reset to restart the processer servers and do a full recrawl. Your data is then ready to be queried..

Summary

In this blogpost, we learned that it is easy to extend the fast pipeline, and how to do it. In the next blogpost I'll show how to query this data.

Fast Search for SharePoint caveat: Apply a sortformula with managed properties of type decimals

Bas — Sat, 05 Nov 2011 08:53:00 GMT

Fast Search for SharePoint provides four great possibilities to influence the sort order of your resultset that was returned:

Sort by Managed property
Sort by Rank
Sort by Formula
Sort in Random order

When I was working on my presentaion and demo for the diwug (Dutch Information Worker User Group) last week, to show off on how you can improve your search experience by using custom ranking models, custom sort orders and extending the pipeline, I ran into an issue that I wasn't aware off. When I used a custom sort formula, I did get the right result set, but not the right result order.

The Scenario

I created a custom pipeline extension, which should enrich the Fast index with the latitude and longitude of a given Location (how this works, will be shown in my next blogpost). Whenever I create custom query, based on a given location, I could create a custom sortformula for Fast, which sorts the resultset for the query based on distance.

The Issue

the issue I came accross, was that the resultset I got returned, wasn't in line with the results I expected at all. The data that was returned from the custom pipeline extensions, were mapped to my mananged properties latitude and longitude, both of type decimal:

The query I executed was the following:

and this query provided me the following dataset. the resultset returned by the query was correct!

Based on the sortformula below, I tried to predict the outcome:

The expected results and the actual results:

As seen in the table, the actual resultorder that was returned, wasn't in line with the expected resultorder at all.

What Happened?

When I digged into this problem, my eye felled on one line of documentation on MSDN:

this line tells us that every managed property of type decimal that is used in a formula, is treated in another way then other numeric types like integers and doubles. Whenever a sortformula is applied to the resultset, all managed properties of type 'decimal', are multiplied by (10^decimal digits). Retrieving the managed property via powerShell, does learn us that "decimal digits" is not a property of that managed property, but it's called "decimal places":

Taken this into account, I made a new computation in excel, and this provided me exactly the results that were returned to me: In the formula I used, fast replaces managed properties based on type decimal by 10^decimalPlaces.

latitude = (latitude * 10^decimalPlaces)

In my case, this becomes:

latitude = latitude * 1000.

Whenever this computation is taken into account with excel, it explains that I didn't get the "right" sort order.

How to Solve this?

Now it is known that Fast handles managed properties of type decimal in another way, we can take this into account into our code. Simply multiply the value that you want to use by 10^decimal places. The results are exactly in line with the results that I couldn't explain before:

While solving the issue, I was thinking about the query I executed. While I made use of the same managed properties, the resultset that was returned by the query, was correct, while the sortorder, wasn't. Analyzing the query executed, learned me that FAST handles the decimalPlaces in the query parameters itself using a query transform:

Conclusion

We learned that Fast does handle managed properties of type decimal in different way than other numeric values when results are sorted using a sortformula. We also learned that Fast does handle the sortFormula in another way then it does when querying Fast: Fast takes the digital places into account when querying the index, but doesn't do this when sorting the results when using a sort formula. Keep in mind, then when those managed properties are used in sortformulas, that the computation you want to execute with that managed property, must take the digital places into account.

Fast Search for SharePoint problems - simplified architecture diagram

Bas — Tue, 09 Aug 2011 21:54:00 GMT

Recently I had a few colleagues (and myself) who tried to install FAST search for SharePoint on their development workstation. Despite the well written installation guidelines on MSDN, they had several problems, and sadly enough, they all faced different problems. These problems varied from not able to crawl the content to not being able to query the indexed content. As most of the colleagues weren't too familiar with Fast (and SharePoint search in general), I decided to create a small architecture diagram of the SharePoint 2010 Search Service Applications, Fast Search for SharePoint and (some of) their junction points. These junction points are displayed in the image below. To simplify the diagram, some items have been removed, the search query proxy is one example.

The diagram helped me and my colleagues to pinpoint the problems and I hope it will help you in the future with pinpointing problems (well, I hope you wont have any problems at all, but hey, it's Fast ;)). This post just covers a small subset of problems, but I think that the majority of most common problems are covered here. Whenever you think that a certain common problem is missing in here, please contact me, I will add it as soon as possible.

Use the following log/event locations to pinpoint problems:

c:\fastsearch\var\log
- contains a lot of folders with logging. Whenever problems arise, I check the folder "syslog" first
the Eventlog - increase the loglevels in the sharepoint diagnostic logging to verbose for extensive logging
- Application log
- Fast Search
- Fast Search Farm
SharePoint diagnostic logging - increase the log level to verbose
The IIS authentication logging
use nctrl status to check the status of your Fast installation

Some basic guidance:

Whenever no log errors appear on the Fast Search farm (event log as well as the \var\log), the problem lies within the SharePoint configuration
Whenever no errors appear on the Fast Search application, but errors do appear on the Fast IIS log, it's most likely a permission problem. My guess: database and/or
Whenever errors appear in Fast, you may not have SharePoint or permission problems, but a wrongly configured Fast environment. I didn't write about these errors below, but hostname configurations appeared to cause problems a lot.

As seen in the diagram, there are several junction points that can cause problems when not configured correctly:

Crawling SharePoint content
Push content into Fast
Communicatin with the Search Query and Settins Service
Accessing the admin service the search query and site settings service
Configure Fast via PowerShell

I have seen problems on all of these levels and I will try to (shortly) describe the possible problems, their symptoms and (a link to) their solution. A small disclaimer is in place: this is by no means a full overview of possible problems, but it should help in troubleshooting whenever problems arise.

1 - Crawling SharePoint content (1)

One of the most common problems are the problems with crawling content. Make sure that your default content access account has read permissions on the sharepoint sites that need to be crawled. Normally, this is handled by SharePoint. Possible error message:

Access is denied. Verify that either the Default Content Access Accouont has access to this repository, or add a crawl rule to crawl this repository. If the repository being crawled is a SharePoint repository, verify that the account you are using has "Full Read" permissions on the SharePoint Web Application being crawled.

2 - Push content into Fast (2)

After installing fast, pushing content into Fast can be a problem too. Several problems are possible here:

- SSL communication between the Fast Content Service and the Content Distributor is not configured correctly.

Visit technet on how to configure SSL communication. Please make sure that the account used in ".\SecureFASTSearchConnector.ps1 –certPath "path of the certificate\certificatename.pfx" –ssaName "name of your content SSA" –username “domain\username” is the account that runs the osearch14. In the Administration Services window this service is called "SharePoint Server Search 14"

- Permissions

After checking the Fast Content Service installation with "docpush -csp c:\testfile.txt" several errors can appear:

sp Reported error with http://cohowinery.com/c:\temp\bob.txt: processing:N/A:ProcessorDeploymentException: For pipeline 'Office14 (webcluster)', creating processor CompanyExtractorInclusionAny failed: ConfigurationError: CompanyExtractorInclusionAny: Failed to access or parse configuration file 'etc/resources/matching/configuration.companyextractor.inclusion.any.xml'.

This error can have several causes. Most likely is that the user doesn't have isn't member of the FastSearchAdministrators group. Another case I have encountered was that the account was member of that group, but for some reason did not have read access to the directory. Changing permissions fixed this issue for me, but this not recommended to do on a production server!! The message about http://cohowinery.com/ is not misconfigured on your system! It's a fictional company, same like the more known virtual companies contoso and wingtip toys

- I see dead crawlers

Another issue that can arise when Fast is installed on the same server as your SharePoint installation (not recommended, only for dev purposes!!), is that the enterprise crawler died for some reason. This can be checked with nctrl status. I described the cause and the solution here

3 - Communication with the Search Query and Settings Service (3)

Accessing Fast from your SiteCollection can cause several problems:

- No search results are returned

When a query is executed and no results are returned, the most likely issue is a permission issue: the account that you logged in with, doesn't have any permissions to view the indexed content. As the returned results are security trimmed by the query service, it's possible that the returned set contains 0 items. This happened to me when I created an anonymous Fast Search Center, whereas all my other indexed site collections weren't anonymously accessible. Another issue can be that (and this happens a lot after a fresh dev-installation!) the test site that is indexed, has content that hasn't been published yet. That content isn't indexed and thus, not returned.

- Internal Server error exception (when querying)

The application pool of the Search Query and settings service is not running. Check your IIS Manager to make sure that your application pool is up and that your website is running.

- Unexpected error occurred while communicating with Administration Service when accessing one of the following site settings menu entries

visual best bets
context information
promotion/demotion (ranking)

When this happens, please check the problems mentioned in paragraph 5 and 6 too. As these functions communicate with the Fast Administration service, it's likely that these problems occur, while the query component works fine. It's most likely a permissions issue, where the Search Query and settings service (mentioned in paragraph 5), doesn't have the right administration account configured.

4 - Accessing the admin service through the search query and site settings service (5)

- Unexpected error occurred while communicating with Administration Service

This error mainly has 3 rootcauses:

The Fast Administration services is not configured correctly (e.g. the wrong location is configured in the Fast Query service application)
The Fast Administration application pool is not running
Permissions ;). Make sure that that the right Fast Admin account is configured in the Fast Query service application.

5 - Fast via PowerShell (7)

Powershell can show some initial "weird" behaviour. Whenever the Fast WCF-services aren't running, all of the powershell commands will bug out with an error message. Get-FastSearchContentCollection returns:

Get-FastSearchContentCollection : Failed to communicate with the WCF service.

Whenever the services are back up, this message will disappear and the right output is returned. But sometimes some Powershell commands do work, and some won't. This is definately a permissions issue, where the executing account isn't member of the FastSearchAdministrators group. It's comparable to the missing/wrong permissions for the fast query service application.

Summary

The majority of problems arise whenever permissions are not set correctly. It's important that the Fast Admin account that is configured in the Fast Query Service, is indeed the right account, is member of the FastSearchAdministrators group and has access the the FastAdmin database. For dev-purposes you can choose the make use of just one main account to run fast, run the app pools and use as the admin account, but this is no recommended approach for production environments. That is why I advise to always use least privilege installations, even on your dev-environments!

SharePoint 2010 - Fast Search Center site template not available (bug?!)

Bas — Wed, 27 Jul 2011 10:08:00 GMT

Recently I found out that, my Fast Search Center was not available from the “New Site” menu. Both the english as well the dutch fast search template was not available.

As I saw this happen before, I checked out the site collection features overview, to enable the “SharePoint Server Enterprise Site Collection features”. I had to scratch my head twice when I saw that that feature was already activated: that feature should add fast search center templates to the available templates. Next thing I did was navigate to the “Page Layout and Site Template settings”, to make sure that the Fast Search Center template was available for creating sub-sites. To my surprise, the template was available! Still, I couldn’t select that template when creating a new site.

I remembered that there is a hidden sitecollection feature (search extensions) that needs to be enabled to make use of the Fast specific features like visual best bets and that enables the administration links to visual best bets, user contexts and site promotion/demotion. After activating that feature, the Fast Search center template became available.

This is strange, because when checking that feature.xml, we learn the following:

<?xml version="1.0" encoding="utf-8" ?>
<!-- Copyright (c) Microsoft Corporation. All rights reserved. -->
<Feature  Id="5EAC763D-FBF5-4d6f-A76B-EDED7DD7B0A5"
          Title="$Resources:SearchExtensionsFeatureTitle;"
          Description="$Resources:SearchExtensionsFeatureDescription;"
          DefaultResourceFile="Microsoft.Office.Server.Search"
          SolutionId="7ED6CD55-B479-4EB7-A529-E99A24C10BD3"
          Version="14.0.0.0"
          Scope="Site"
          Hidden="TRUE"
          xmlns="http://schemas.microsoft.com/sharepoint/">
  <ElementManifests>
    <ElementManifest Location="webPartDWPFiles.xml"/>
    <ElementManifest Location="ExtendedSearchAdminLinks.xml"/>
  </ElementManifests>
</Feature>

There is no feature receiver
There is no feature dependency
The feature only
- Adds a webpart (webpartdwpfiles.xml)
- Adds Search admin links (ExtendedSearchAdminLinks.xml)

For some reason this only happened for existing site collections. That made me think about some recent changes to the platform:

SP1 was installed
SP1 for language packs was installed
CU June was installed.

When deactivating the hidden feature (Search Extensions), the english template was gone again, whereas the dutch search template still existed. I definately this needs some further investigation.

As it only happened for the sites that existed before that change, my guess it has to do with a combination of those updates. For new sites, everything works as intended. My guess is that it has to do with the combination of installed/registred language packs and the activation/deactivation of the search extension feature, which provides a default language resource named "Microsoft.Office.Server.Search".

Fast Search for SharePoint gotcha - query suggestions

Bas — Tue, 19 Jul 2011 10:07:00 GMT

At the company i work for, we do have around 30(?!) different labels and we decided that all websites of those labels, should be hosted on one platform: SharePoint 2010. This consolidation to one platform should really decrease costs, because we now have one generic way of developing, one platform to take care of and we can reuse a lot of components. One of the service applications that we wanted to reuse amongst all applications, is the Fast Query Service Application, as we have a Fast instance running. Our labels are really enthousiast about the Fast Search for SharePoint capabilities due to fact that a lot of interesting features, like visual best bets, can be configured from within the UI. Other things, like new ranking models can easily be configured via powershell. There is just one index, with all public information in it: personal details or things like that, will not be indexed! By configuring scopes in a correct way, we can prevent information from label 1 to appear in the search results on label 2. If, for some reason, those results appeared by accident (wrongly configured scope), that wouldn't be too much of a problem, as that can be fixed real soon.

The gotcha ;)

Query suggestions is one of the features that can be manually configured by registering new resourcephrases from within powershell, but it is self-learing too, based on query terms and click-throughs. Sadly enough, this is service application specific. But what does this mean?

Now the following case: We have two labels "Centraal Beheer" and "Interpolis", who are offering the same, competitive products. Imagine that Interpolis demanded us to add a query suggestion for the word "Interpolis". That could easily be added by executing the following powershell:

$queryapp

= Get-SPEnterpriseSearchServiceApplication 'FastQuery'
New-SPEnterpriseSearchLanguageResourcePhrase -Language en-us -Type QuerySuggestionAlwaysSuggest -SearchApplication $queryapp -Name "Interpolis Verzekeringen"

As this query suggestion is added to the service application itself, and not scoped to a custom scope, webapplication or site collection, every web application that consumes this Search Query Application, will suggest the word interpolis. For competitive labels, this will be a major issue! Features as user contexts (till a certain spot) and keywords can be scoped on site-collection scope, suggestions not :(
Imagine that the suggestion "Interpolis Verzekeringen" pops up when searching for "Intake Insurance" on the competitive site of "Centraal Beheer"? I wouldn't be too happy if I was a business owner of "Centraal Beheer".

Luckily, this can be solved by adding a second search service application, but if that's the way to go: I don't know. A second search service application can be used, but requires 3 extra databases to be created:

Crawlstore_DB
Query_DB
PropertyStore_DB

For smaller organisations, this shouldn't be too much of a hassle, but in a large company, this can cost a lot of extra money, due to (but not limited to) SLA's, extra administring of databases, storage and backup. And maybe are two service applications easy to manage, but what about 30 service applications (one for each label?).I think that the same architecture that is used for keywords and usercontexts, should be used for the query suggestions. Suggestions would even be manageble on site collections, what would be an extra advantage for FAST.

PS: this problem arises for SharePoint enterprise search as well ;)
PS2: Who not using a multi tenant environment? Fast doesn't support that and we don't need it ;)

Fast: Adding managed metadata properties for the use with scopes in SharePoint 2010

Bas — Fri, 03 Jun 2011 11:17:00 GMT

Lately I was working on a project that required to make use of scopes and some managed metadata properties in those scopes. For some reason, those managed metadata properties didnt show up in my scope selection dropdownlist, but I was sure that this metadata property was available in my Fast Managed Metadata properties list:

as you can see, this property doesnt show up in my properties list:

Whenever the SharePoint Search metadataproperties page is accessed, the option "Use in scopes" for the managed property ContentType is turned off:

When this property is turned on, the property is available to be used in the scopes. According to MSDN, this is working "as intended". The managed properties that are created in the SharePoint search administration page, are just placeholders for the Fast managed metadata properties. The configuration of this managed property is, however, done in the Fast Managed metadata properties page.

So, to be able to use managed properties in Fast Search scopes, the following steps need to be taken:

Add a Managed Metadata property via the Fast Search Administration page and configure it
Add a metadata property via the SharePoint search administration page (found on the Fast Query SSA) and make it available for the use in scopes. No other configurations need to be made.
Now you're ready to use your managed metadata property

Fast for SharePoint 2010 Enterprise Crawler is Dead

Bas — Thu, 19 May 2011 08:30:00 GMT

Lately, I installed Fast locally on my machine for development purposes, and it was working as intended. Crawled a lot of local (and external) content, but suddenly, my Enterprise crawler was not working. nctrl status showed me that the Enterprise crawler was Dead

but WHY it was dead? I had no idea at all! Checking the event logs only told me:

M05020-DHEVMU98:13260: Process crawler was not running, restarting it
M05020-DHEVMU98:13260: Process crawler ports are already in use, will not start it

After checking the port 13260 on my machine, I concluded that there was nothing wrong with it. But that port, 13260 is the node controller, the enterprise crawler doesn't use that port. After checking the hostsconfiguration.xml I concluded that I made a mistake: This file contains a property: baseport with value 13000. That day, I created a new SharePoint webapplication on port 13000. I always create new web applications at those ranges (10000, 11000, 12000), it's a automatism of me. After deleting that web application, my Fast Enterprise crawler is up and running again! Too bad that the Fast Logging doesn't include portnumbers in their errorlog when a port is already occupied .

Conclusion: never create a webapplication in the 13000 range when running Fast locally ;)

My SharePoint 2010 & Fast session is online!

Bas — Thu, 19 May 2011 08:23:00 GMT

On the Dutch Microsoft DevDays (or Techdays, as they were called), I had a session about SharePoint 2010 and Fast. It was recorded and is now published on Channel 9. It's pretty weird to see and hear yourself presenting. If you are interested in:

Fast Architecture
Fast OOTB capabilities
Fast configuration capabilities
Ranking & Sorting

you should watch my session. For your information, it's in Dutch.

the presentation can be found at:

http://channel9.msdn.com/Events/DevDays/DevDays-2011-Netherlands/Devdays119