Search engines are dead, long live the search engine!
I have been thinking a lot lately about the relevance of results that search engines like Google, Yahoo or MSN-search return. Although the ‘search experience’ is relatively good when your search criteria have a wide scope, the experience degrades considerably when doing very domain specific searches (try searching for SOA in the Dutch version of Google :-)). I’ll try to explain what I mean with this the preceding statement.
A few days ago I was having a discussion with a co-worker of mine who was looking for some background information on deploying a new technology that hadn’t been used before in the organization. His solution was very nice and well architected; it was simply the fact that the influencers needed to be influenced by providing them with some unique selling points.
So I decided to do a Google search on the use of this particular technology within large enterprises. The results I got back weren’t really interesting for me since most of it were about problems people were having while using this particular technology.
The second thing I tried was doing a search in the data my local RSS reader contains. I got back exactly those things I was looking for in the form of blog posts by several developers.
This is an interesting phenomenon and kept me thinking a while. Although I am not fully aware of the algorithms used by the big search engines to index their sites I somehow get the feeling that the relevance of the search results has degraded over time. It occurred to me that there might just be a solution for this problem in the form of the social tagging phenomenon that has been raging the internet the last couple of months.
Social tagging is in essence a way to describe the internet using meta tags. Delicious (http://del.icio.us) is more or less the most active social tagging community on the internet.
People can post links to delicious and add some meta tags describing the link that was posted. The posted link gets added to your personal bookmarks (there are tools that can synchronize these with your local bookmarks). Nothing new you would say? The power lies within the fact that you can see what tags other people have used to describe the same link you have posted (hence the social aspect), and see which links people have posted under these other tags. This is a completely new way of surfing the web and can be quite rewarding with several gems that you might never have found otherwise.
Now I can hear your thought process going… “a user maintained content database is only as good as the sum of its users, this will never work”. I do agree with that.
However, suppose you take the social tagging approach and combine that with the weighting scenarios search engines like Google apply to their data… you might just have found the search engine that gives you the relevant search results that you have been looking for so eagerly. Human intelligence for relevance and quality, machine intelligence for scoping, relationships and quantity.
What also would need for a successful implementation are focused social tagging groups. Delicious is simply a place for all things interesting to everybody. What you would need is a focused amount of people that are tagging the Internet within a specific niche domain.
My domain is that of a “software developer”. We need a community that tags everything within this domain of “software development”, the big win would be our ability to cluster tags because we can accordingly identify and categorize them since we only exist within this specific domain. When searching for “VSTS” our search engine could identify this term as belonging to the (user maintained) cluster of “Visual Studio Team System” or “Team Foundation Server”, which all belong more or less to the same product. I would expect to see all the search results on a given tag returning everything within the cluster of tags this tag belongs to. You can stretch this out to the extent of clusters being children of a bigger cluster or containing relationships with other clusters… the possibilities are endless.
The third aspect that is required for this kind of solution to work is opening up the API. We need to open up everything. Ideally we would like to have our results be included in the search results of the bigger search engines (Google, Yahoo, MSN search) for a broader audience.
Imagine the power of a big search engine crawling these focused tagging communities on the internet (we need to define a common API for this… SSE?) and making use of their content which is probably a lot more relevant since human-intelligence has been used for kick-starting.
In my opinion Web2.0 is all about this (at least, the non over-hyped aspect of it) openness and integration of technology inside the semantic web.
I am thinking about, and working on, applications for making these kinds of scenarios possible. I do think that this is the way to go in the future for improving our search experience. Expect to see an implementation of this concept in the near future. I am very curious what the effect of this little experiment will be.