Have you ever wondered how Google returns a search result in less than a second? It’s all made possible through the magic of indexing. On periodic basis, Google will go out across the Internet and crawl all of the content that it can access. During this process, the crawler pulls each and every document, webpage, or whatever, back to the indexer, were the document is broken down into the list of words it contains. Google creates a database which in the world of search is often called the index. When a user executes a query, the index is what is queried for relevant data resulting is sub-second response time. Think of an index as a data warehouse for unstructured information.

Now all of this sounds great, but there are times, when indexing information is not appropriate or even possible.

There’s an old adage in IT that ‘nothing ever goes away’. Retiring legacy systems is a painful, often expensive process, mostly because every system has some useful information in it, but it’s hard to distinguish jewels from junk within that information. There are times when migrating all your content and cutting over to a shiny new system is the right thing to do, and there are many strong content migration products on the market. Sometimes, however, a total migration is simply not necessary nor useful.

I’m a hopeless book hound, and in my life have spent more on books than on cars. I have thousands of books. The last time we moved, we had 60 boxes of books, and my wife made me a proposition: We would unpack the five boxes that contained the books we knew we wanted, and leave the rest in the basement for two years. Over the next two years, I opened a few more boxes and pulled out what I needed. At the end of the two years we donated the rest to the local library. Needless to say, our bookshelves were much more organized, and had a lot less junk.

A fairly recent trend I’ve noticed in this space is an increasing interest in search connectors. A connector enables a search engine to crawl messaging and line of business systems. SharePoint Search and FAST, provide connectors for File Systems, Exchange Public Folders, Websites, and SharePoint out of the box, but Microsoft leaves it to partners to develop others. This makes sense. There are hundreds of business systems out there and Microsoft is careful about where they devote their resources. We’ve invested heavily in developing I.P. in this area. A list of the connectors that we offer can be found here: http://www.bainsight.com/sharepoint-fast-search-connectors/Pages/default.aspx

So why the sudden uptick in interest in connectors? In my view, it’s because the technology has finally reached maturity. Enterprise Search is about all Enterprise data, not just the data in a file system or SharePoint. For over a decade now, search vendors have been promising a “Unified View” of Enterprise Data; both structured data and unstructured content. It is only recently that we have been able to deliver on this promise. Why did it take so long?

One of the most compelling features in SharePoint 2010 is Managed Metadata Service, or MMS for short. Simply put, metadata is data that describes other data. A good way to conceptualize metadata might be to think of a photograph that you’ve taken in the past. The photo is the actual data, and metadata that describes it might include the size of the file, where the photograph was taken and who is in it. In this post, I’m going to focus on how MMS can help you manage metadata, and how, if leveraged properly, it can significantly improve the quality of Enterprise Search in your organization.

Why do I need metadata? Why can’t I just search like I do on Google?

This is the question that comes up most often when I’m at trade shows, conferences, what have you. From a user’s perspective, all they do on Google is enter two or three keywords, and voila – relevant results! What they are not seeing happens behind the scenes, before their query is ever run. The two or three word query is boosted by metadata.