Search results matching tag 'Architecture'

Fast Search for SharePoint problems - simplified architecture diagram

Bas — Tue, 09 Aug 2011 21:54:00 GMT

Recently I had a few colleagues (and myself) who tried to install FAST search for SharePoint on their development workstation. Despite the well written installation guidelines on MSDN, they had several problems, and sadly enough, they all faced different problems. These problems varied from not able to crawl the content to not being able to query the indexed content. As most of the colleagues weren't too familiar with Fast (and SharePoint search in general), I decided to create a small architecture diagram of the SharePoint 2010 Search Service Applications, Fast Search for SharePoint and (some of) their junction points. These junction points are displayed in the image below. To simplify the diagram, some items have been removed, the search query proxy is one example.

The diagram helped me and my colleagues to pinpoint the problems and I hope it will help you in the future with pinpointing problems (well, I hope you wont have any problems at all, but hey, it's Fast ;)). This post just covers a small subset of problems, but I think that the majority of most common problems are covered here. Whenever you think that a certain common problem is missing in here, please contact me, I will add it as soon as possible.

Use the following log/event locations to pinpoint problems:

c:\fastsearch\var\log
- contains a lot of folders with logging. Whenever problems arise, I check the folder "syslog" first
the Eventlog - increase the loglevels in the sharepoint diagnostic logging to verbose for extensive logging
- Application log
- Fast Search
- Fast Search Farm
SharePoint diagnostic logging - increase the log level to verbose
The IIS authentication logging
use nctrl status to check the status of your Fast installation

Some basic guidance:

Whenever no log errors appear on the Fast Search farm (event log as well as the \var\log), the problem lies within the SharePoint configuration
Whenever no errors appear on the Fast Search application, but errors do appear on the Fast IIS log, it's most likely a permission problem. My guess: database and/or
Whenever errors appear in Fast, you may not have SharePoint or permission problems, but a wrongly configured Fast environment. I didn't write about these errors below, but hostname configurations appeared to cause problems a lot.

As seen in the diagram, there are several junction points that can cause problems when not configured correctly:

Crawling SharePoint content
Push content into Fast
Communicatin with the Search Query and Settins Service
Accessing the admin service the search query and site settings service
Configure Fast via PowerShell

I have seen problems on all of these levels and I will try to (shortly) describe the possible problems, their symptoms and (a link to) their solution. A small disclaimer is in place: this is by no means a full overview of possible problems, but it should help in troubleshooting whenever problems arise.

1 - Crawling SharePoint content (1)

One of the most common problems are the problems with crawling content. Make sure that your default content access account has read permissions on the sharepoint sites that need to be crawled. Normally, this is handled by SharePoint. Possible error message:

Access is denied. Verify that either the Default Content Access Accouont has access to this repository, or add a crawl rule to crawl this repository. If the repository being crawled is a SharePoint repository, verify that the account you are using has "Full Read" permissions on the SharePoint Web Application being crawled.

2 - Push content into Fast (2)

After installing fast, pushing content into Fast can be a problem too. Several problems are possible here:

- SSL communication between the Fast Content Service and the Content Distributor is not configured correctly.

Visit technet on how to configure SSL communication. Please make sure that the account used in ".\SecureFASTSearchConnector.ps1 –certPath "path of the certificate\certificatename.pfx" –ssaName "name of your content SSA" –username “domain\username” is the account that runs the osearch14. In the Administration Services window this service is called "SharePoint Server Search 14"

- Permissions

After checking the Fast Content Service installation with "docpush -csp c:\testfile.txt" several errors can appear:

sp Reported error with http://cohowinery.com/c:\temp\bob.txt: processing:N/A:ProcessorDeploymentException: For pipeline 'Office14 (webcluster)', creating processor CompanyExtractorInclusionAny failed: ConfigurationError: CompanyExtractorInclusionAny: Failed to access or parse configuration file 'etc/resources/matching/configuration.companyextractor.inclusion.any.xml'.

This error can have several causes. Most likely is that the user doesn't have isn't member of the FastSearchAdministrators group. Another case I have encountered was that the account was member of that group, but for some reason did not have read access to the directory. Changing permissions fixed this issue for me, but this not recommended to do on a production server!! The message about http://cohowinery.com/ is not misconfigured on your system! It's a fictional company, same like the more known virtual companies contoso and wingtip toys

- I see dead crawlers

Another issue that can arise when Fast is installed on the same server as your SharePoint installation (not recommended, only for dev purposes!!), is that the enterprise crawler died for some reason. This can be checked with nctrl status. I described the cause and the solution here

3 - Communication with the Search Query and Settings Service (3)

Accessing Fast from your SiteCollection can cause several problems:

- No search results are returned

When a query is executed and no results are returned, the most likely issue is a permission issue: the account that you logged in with, doesn't have any permissions to view the indexed content. As the returned results are security trimmed by the query service, it's possible that the returned set contains 0 items. This happened to me when I created an anonymous Fast Search Center, whereas all my other indexed site collections weren't anonymously accessible. Another issue can be that (and this happens a lot after a fresh dev-installation!) the test site that is indexed, has content that hasn't been published yet. That content isn't indexed and thus, not returned.

- Internal Server error exception (when querying)

The application pool of the Search Query and settings service is not running. Check your IIS Manager to make sure that your application pool is up and that your website is running.

- Unexpected error occurred while communicating with Administration Service when accessing one of the following site settings menu entries

visual best bets
context information
promotion/demotion (ranking)

When this happens, please check the problems mentioned in paragraph 5 and 6 too. As these functions communicate with the Fast Administration service, it's likely that these problems occur, while the query component works fine. It's most likely a permissions issue, where the Search Query and settings service (mentioned in paragraph 5), doesn't have the right administration account configured.

4 - Accessing the admin service through the search query and site settings service (5)

- Unexpected error occurred while communicating with Administration Service

This error mainly has 3 rootcauses:

The Fast Administration services is not configured correctly (e.g. the wrong location is configured in the Fast Query service application)
The Fast Administration application pool is not running
Permissions ;). Make sure that that the right Fast Admin account is configured in the Fast Query service application.

5 - Fast via PowerShell (7)

Powershell can show some initial "weird" behaviour. Whenever the Fast WCF-services aren't running, all of the powershell commands will bug out with an error message. Get-FastSearchContentCollection returns:

Get-FastSearchContentCollection : Failed to communicate with the WCF service.

Whenever the services are back up, this message will disappear and the right output is returned. But sometimes some Powershell commands do work, and some won't. This is definately a permissions issue, where the executing account isn't member of the FastSearchAdministrators group. It's comparable to the missing/wrong permissions for the fast query service application.

Summary

The majority of problems arise whenever permissions are not set correctly. It's important that the Fast Admin account that is configured in the Fast Query Service, is indeed the right account, is member of the FastSearchAdministrators group and has access the the FastAdmin database. For dev-purposes you can choose the make use of just one main account to run fast, run the app pools and use as the admin account, but this is no recommended approach for production environments. That is why I advise to always use least privilege installations, even on your dev-environments!

Business Rules Engine: Samples

chilberto — Sat, 21 Nov 2009 07:18:55 GMT

I recently presented at the Auckland Connected Systems User Group. In order to prepare, I created several examples of methods of using the rules engine.

The sample solution is here.

Included in the sample solution are:

Various Vocabulary definitions including using a function in XPath Field (TotalCostOfOrder)
Various Rules definitions including retrieving from a database, using classes, adding XML Nodes to a document, forward chaining and priority
Examples implementations of IFactRetriever
BizTalk Application showing calling rules with both XML Document and objects
BizTalk Application showing an example of using the BRE for routing
A WCF Service and Client including an example tracking interceptor

The WCF Service was my focus of the talk and what I wanted to discuss most of all. I have found using the BRE very beneficial and now that I have spent time understanding it, I believe it helps to make more maintainable solutions by separating the business rules from the application framework. This allows for the business logic to more accessible and readable than embedded in BizTalk expressions, maps or custom classes.

Because of this I wanted to illustrate how a WCF service could be strongly typed but not require costly rebuilds when the facts or rules change. Though I believe strongly in SOA, I am not a fan of the un-typed XML in/XML out design. To achieve this, I took advantage of the KnownTypes attribute to allow WCF to serialize the payloads and to present metadata. In the example service, the specific classes are retrieved from the web.config, but there is not any reason why this could not have been retrieved from a more dynamic resource.

Cheers and feedback appreciated.

The foundations of Service Oriented architecting

Carlo Poli — Mon, 05 Jan 2009 20:31:06 GMT

A lot has been said about the foundations of Service Orientation. But still, way to often I see people deciding to use SOAP/XML and thinking they are doing Service Orientation. You might recognize the following examples I see as getting Service Orientation wrong:

Services are very technically oriented: service-designers know they should do something on business alignment, but say that the business doesn’t get it and using web service standards is the best we can.
Services are point-to-point interfaces: ‘services’ are designed on the fact that some information needs to go from system A to system B and therefore a service needs to be created. They then call the service something like the InterfaceInformationService or the SetSomeDataService. Also recognizable by the fact that they always have one operation.
Services with little responsibility: services are designed around one function and are therefore responsible for only a small bit of data.

I think that, when you talk about architecting Service Oriented solutions, the architecting part is the hardest part. There’s nothing special about that. It’s just decomposing systems into subsystems with clear coherent responsibilities and logical interfaces. But the decomposition is at a higher, abstracter level, which makes it harder. When decomposing applications, you assign responsibilities to concrete components. When decomposing systems, you assign responsibilities to abstract services. But still you need to be concrete and accurate about the responsibility of the service.

Clearly defining the responsibility of the service is the best way to get a service contract right.

Imagine you are designing an Order Management service (OM) in an environment where there is an Customer Management service (CM) available. Now you have to decide whether your OM knows the CM. Whether you want that is based on the way you assign responsibilities. Now let’s say that it’s the CM’s responsibility to manage a customers credit rating. Your options:

If you make the OM responsible for validating the customers credit rating before accepting an order, your OM will need to know the CM.
You could also decide that it’s the service consumers responsibility to perform a credit check (the service consumer could very well be a process manager) and the OM doesn’t need to know the CM.

The first option is more strict. There is now way to create an order if the customer doesn’t have a required credit rating. The second option is more flexible, the OM can even handle orders for customers unknown to the CM (imagine merger scenarios). I think that no one option is better than the other, it depends on where you assign responsibilities and that is driven by architecting requirements.

More generally I think that in any service interaction pattern there are three main parties:

The service consumer. The service consumer knows the service provider. It knows it’s contract, it’s policy, it’s endpoint location. Ideally, the service provider doesn’t know the consumer.
The service provider. The service provider defines the contract and provides the implementation. It’s implementation might (and often will) include a datastore.
Other services. These services are used by the service provider to delegate work that’s necessary to realize the responsibility of the service. The service provider plays the role of service consumer to these services.

Deciding on the interacting pattern between the parties is the key to good service contract design. So far, this looks pretty straightforward, isn’t it? Well, it’s often harder then it looks. It’s easier to directly contact a service to get some work done and if your not careful on that, soon you get a spaghetti of communication services. In following posts I dive deeper into the subject “Deciding on the contract”.

I really value any feedback you have on this topic. I hope it sparks some discussion.

Restarting my blog

Carlo Poli — Thu, 01 Jan 2009 20:10:11 GMT

I haven’t been blogging for over 2,5 years (except for one off topic post). This was partly because I didn’t bother, but the main reason was that I didn’t have a goal, a story I wanted to tell the world. I’ve been doing a lot of stuff around integration and SOA/BPM, ranging from guiding BizTalk development to developing an enterprise integration strategy.

2008 to me was the year I dived into the theory of architecting and architecture on one side and understanding the core foundations of Service Orientation and Business Process Management on the other side.

On the theory of architecting I was greatly inspired by the work done by the Software Engineering Institute. I dived into Software Architecture in Practice (2nd Edition) and was lucky to receive an eight-day training from the writers through Logica, my employer.

On SOA I was greatly inspired by the work done by Thomas Erl.

These sources gave me a lot of background on the complex, and still evolving, playground of the Integration Architect.

For 2009, I’ll take up the next challenge, bringing all this theory into practice. Now, I do think that I already bring a lot of the theory in practice. In my day to day job, I work in an environment where the integration we do is greatly driven (and constrained) by existing systems, but with a vision of having a Service Oriented environment in the long run.

In the past year I’ve learned a lot on what the real key features of integration architectures are and what kind of decisions are important to make steps toward Service Orientation.

My goal for this blog is sharing my view on these topics with the world, sparking some discussion. I already have some stuff to talk about, so I guess in the short run I will sound like a teacher telling his story, In the long run I see this blog more as the diary of a tourist touring an environment that is only roughly explored and documented.

As a side note, I don’t write today as a result of some new year blues with big new intentions. It’s more the holiday season with a lot of days off that allowed me to think about what I want to achieve. But still it is a good day to restart a blog ;-).

Windows Azure

chilberto — Mon, 27 Oct 2008 17:49:32 GMT

At the PDC, Windows Azure was announced as the Microsoft platform for hosting services in the cloud. Similar to Amazon's cloud services, Windows Azure will provide a mechanism for hosting solutions in the cloud. In other words, hosted in Microsoft data centers.

Very interesting. Looks like a SOA has just gained a new perspective: on-premise services and cloud services. Very powerful, and as it affects me, lots of work coming into the integration space.

The Myth of Stored Procedures Preference

Adel Khalil — Sat, 05 Jan 2008 22:16:28 GMT

When looking to the Stored Proscedures debate, there is always those three factors you should measure by.

Productivity

- Span code base over multiple staging environments is a pain and harder to maintain.

- Versioning sp code is way harder than versioning application code.

- Minor change to the design require changing in both the SPs and the DAL code.

- Todays IDEs are more advanced than most of the RDBMS offers, implementing on IDEs is obviously preferable.

- Switching between two seprate world to implement single method is always pain.

- It's impossible to cover every single scenario and write SP for it, which will lead to write these SP as you go, huge consistency problem.

- There is no way to only update single param in the Update method using SPs as there isn't optional parameters, on every update you need to supply full param collection.

- SPs are not portable if you want to develop application that run over multiple DBMS you will be writing SPs for each DBMS, standard SQL is portable.

Security

- Big myth over here, using SPs not guarante best security practice and certinly dosn't mean that your application is SQL Injection proof you can write code like this

string s = "EXEC sp_GetCustomerByEmail '" + txtEmailAddress.Text + "'";

and you will be using SP and still open to all kind of SQL Injection.

- Another myth regarding security is that if you are using Ad-hoc queries you *most likely* grand permissions for CRUD operations for your application user on the database, no you are not, that's why Views are invented.

Performance

often when SPs vs. Ad-hoc queries debate intoduced the performance card played, SPs advocates says SPs are pre-compiled which is not let met quate like Frans did from SQL Server Books Online

SQL Server 2000 and SQL Server version 7.0 incorporate a number of changes to statement processing that extend many of the performance benefits of stored procedures to all SQL statements. SQL Server 2000 and SQL Server 7.0 do not save a partially compiled plan for stored procedures when they are created. A stored procedure is compiled at execution time, like any other Transact-SQL statement. SQL Server 2000 and SQL Server 7.0 retain execution plans for all SQL statements in the procedure cache, not just stored procedure execution plans.

So with no pre-compilation and caching for both SPs and SQL statments there is no advantage for SPs here, in some other databases the SPs compiled into C or C++ but this isn't the case in SQL Server 7.0/2000.

I have introuduced my view on the SP vs. Dynamic SQL i don't see any benfit of SPs over the huge amount of productivity, performance that you will gain with dynamic SQL, the only benfit in peformance you will get it when using Managed SPs (SQL Server 2005) but for 0.7/2000 SPs isn't the right choice for most of the scenarios.

Security Tip

Adel Khalil — Tue, 17 Jul 2007 11:58:00 GMT

Hi folks, When you design your sophisticated "Change Password" feature in your next application make sure to retain the old password so the user can't re-choose them anymore..

You can keep a history of passwords and don't allow to use them again.

[UPDATE]

The reason you may wanted to do this..if your password base compromised you wanted to change the password for your clients to something tempporary and also prevent them from changing this back to the old compromized one, however you also will benfit from this stratigy to force users to never having the same password they already changed - why they change their password on the first place - to be using it again...

Paranoid Programmers

Olaf Conijn — Sat, 16 Jun 2007 13:11:00 GMT

Most people should be familiar with the term “defensive programming”. It’s a way of writing code, in which the default mode is to explicitly check all assumptions you have in mind while writing logic (most of the time throwing exceptions to the caller).Though, when does this make sense?

Assume you are developing a framework, application host or plug-in to an application host. How sure are you all the assumptions you have in mind hold, once having released whatever you are working on “out in the wild”?

What if a “friendly hacker” wants to exploit some unforeseen extensibility point in your framework, making it do cool stuff beyond something you thought was possible… would that be gain or loss in value for your framework?
Think of all the mash-ups people made against google maps, flicker, whatever web 2.0 stuff is out there. Or extensions made to “application hosts” such as the Visual Studio Shell (soon to be released and positioned as a true application host!).

My default mode is different: I won’t check assumptions I have when coding, “unless …”:

- I am exposing some surface of an application to whomever (for instance a public service boundary, user input, whatever “unfriendly hackers” could exploit to do bad things).

- I am shielding from conditions that simply cannot be handled gracefully (and will throw an exception anyways). Most of which can be done through basic parameter guarding.

-Etc

When exposing a public API I’d love to have people do whatever they think makes sense.
In return, whatever I expose meets some basic assumptions, around stuff like thread-safety, security and in general its behaviour (implied through convention or somehow more explicit).
If something is not thread-safe or possibly unsecure it should not be part of the public API.

Example would be a pattern such as:
FooBar foo = bar.GetContextualSomething() as FooBar;
if (foo != null) // whatever ContextualSomething I get back, my logic only applies on a FooBar
{
foo.DoStuff();
}

I agree that having an API that works regardless of how it is called wouldn't make sense either, though the consumers of a framework know the problem they are trying to solve best. So why not give them the benefit of doubt?

UML and the Maverick style

Adel Khalil — Sat, 07 Apr 2007 00:27:00 GMT

I'm very confused about UML i didn't had a chance to deeply work with it but now i guess it's already prisoned inside the boundaries of whiteboards nothing like UML will appear in any place other than the white board in the meeting room. Maybe i don't know I'm not working like you in a fancy software vendor - or it's not a fancy big

sw vendor characteristic - so i don't have all the fancy must well done water flow model so we can have the time to do the work, my style is a Maverick style of coding - don't remember who came up with the name - and potential testing where you just wanted to get the initial specs and run into your favorite kinda of IDE and start getting things done - if you know what i mean - even many of the companies that adopt the Agile methodologies dot relaying on UML in design like the idea of Agile and freedom of developer is liberate them from having everything figured out prior.

I see software methodologies - management, design, developing ..etc.. - are interrelated together it's hard for me to imagine a Ruby developer doing some deep design before start coding on the fly, in the other hand Java folks or other old OOP languages are more imaginable as design duds.

So if this is correct then Agile management will affect other sides of software practise to be agile too as the mentality that choose a style of doing design will choose similar style of management and so on..

Back to UML for instance, i see that converting from the high business level to a DB entities for example is alto harder for developers and starting Architects as we learned and worked with the DB point of view very much that the human entities is now so hard to understand in someway we are thinking like Computer better in a certain way than thinking in normal human way, that why we may spending a lot of time on Visio tying to model the business to create the back-end schema when we - as a developer thinking like computers - can easily created the desired table into the DBMS.

Maybe this is why i hated UML - well not hated it - but at least not prefer working with it outside the whiteboard.

So how you design and what is the pattern of methodologies you are using ? Are you a fan of the Maverick Coders style despite of all it's drawbacks ?

MySpace and ASP.NET

Ramon Smits — Wed, 06 Dec 2006 23:44:00 GMT

I just read Paul Winson's post about the fact that MySpace uses ASP.NET. I know this for months but just creates more questions about how MySpace is build.

Do they use c#, vb?
Do they make use of the standaard system.web.* controls?
What kind of application layers do they have?
How do they manage state? Viewstate, cookies, sql, in memory?
What kind of data caching do they use and in which parts?
How do they manage data persistancy?
How big is there web+sql farm?

I've heard rumours that the total processing power for MySpace exceeds that of Google worldwide!

What I read about persistancy is that they have choosen iBatis.NET. They support three environments: Java, .net and Ruby. I have not yet tested this persistancy O/R framework but I am wondering if it also works with Mono :-).

Update:
I just found the following information about the hardware in use for MySpace:

MySpace's extensive IT architecture currently features 2,682 Web servers, 90 Cache servers with 16GB RAM, 450 Dart Servers, 60 database servers, 150 media processing servers, 1,000 disks in a SAN (storage area network) deployment, three data centers and 17,000MB per second of bandwidth throughput.

...

MySpace currently sets aside about 100 terabytes for MP3s and videos, and another 200TB for dynamic content.

...

MySpace is deploying Isilon Systems' software for MP3 and video streaming, clustering systems together in order to spread files and data across multiple storage nodes. The technology also reduces storage capacity constraints, since new nodes can be added as necessary. Originally starting off with a two-node 3PAR frame, MySpace has since upgraded to an eight-node cluster. Each storage node delivers 600 megahertz per second, while each cluster spits out 10G bits per second.

So this seems like an impressive hardware setup :-)

Well that makes you wonder what will happen to this setup when MySpace will expand to China as is mentioned in the media the last couple of months.