Recently we accidentally discovered a bug within the SharePoint Search. To aggregate news from certain locations, we use the SharePoint search to query and display items based on user profile settings. This has always worked the right way: both the incremental crawl and the full crawl indexed all items. Within 10 minutes when a new item was published, it was visible within our aggregation page. This changed after we applied the cumulative update of december (If I am correct). For some reason no items were indexed anymore after an incremental crawl. When starting the full crawl, all items displayed properly. After some investigation, we found out that the settings on the Pages libraries that we try to query, have the setting "Always index" and "Always Visible" set to false. When both settings are set to false, both the incremental and full crawl shouldn't index the items in the library. For some reason, these items have always been indexed, and both the incremental and full crawl didn't behave in a correct way. Since the cumulative update the incremental update does behave correctly, but the Full crawl still doesn't. This matter is under investigation by Microsoft at the moment.
UPDATE:
Microsoft is working on a fix for this, but It can last some time before the fix is rolled out.
What happens exactly:
- When the tag <META NAME="ROBOTS" CONTENT="NOHTMLINDEX"> is in the HTML code, the page is not indexed by the crawler.
- This tag is activated when the search visibility options have the following value:
- Do not index ASPX pages if this Site contains fine-grained permissions: true
- or
- never index any ASPX pages on this site: true
- precondition: Allow this web to appear in search results has to be set.
- Put them both on false, then your results will be crawled again.
- note: that they are gathered by the full search is due to a bug in the search engine!