Enterprise search technology is incredibly useful and powerful, but very few people understand how to apply it well. Perhaps because search looks so simple on the outside, or perhaps because of the universal familiarity with web search, enterprise search is generally poorly understood. There are many misconceptions and misunderstandings about enterprise search, and in many organizations the first comment you hear about an intranet is, “the search is useless.”

We can do search better. There is no silver bullet or secret sauce, but if you follow some basic ground rules and utilize some proven, practical techniques, you can make search something that people not only use but love.

“With great enterprise search, organizations can quickly respond to market changes, innovate and accelerate their time to market. Without good search, people drown in their own information,” writes Jeff Fried, CTO of BA Insight in his new article “The Search Journey”, “When search works well, people use it. Improving search quality will increase its utilization and make people more effective at their jobs.” In the first installment of his regular TechNet column, A View from the Crawl Space, Fried gives practical advice for how to make search the useful, powerful tool it can be. Fried explains that search is not a one-size-fits-all endeavor, and while this means that enterprises are able to create a solution perfectly tailored to their unique needs, it also means that there is no secret sauce or silver bullet strategy for search adoption. There are, however, certain key common denominators.

Fried outlines three ground rules that apply in all cases and should be heeded by any organization, regardless of where they are in the “search journey”. The three rules are simple:

• Start projects with envisioning and out-of-the-box search
• Make search administration someone’s job
• Enlist the users.

Visit TechNet’s Enterprise Search Research Center or read Fried’s full article “The Search Journey” (PDF) to learn more about the three rules and other practical advice for building great search.

Now all of this sounds great, but there are times, when indexing information is not appropriate or even possible.

One of the issues with indexing is that the index is only as up to date as your most recent crawl. If you have data that changes rapidly keeping the index up to date can be a challenge. Booking a reservation on an airline is a good example. When a customer books a flight querying an index doesn’t make sense. The only way to ensure you’re not giving away someone’s seat is to query the reservation system directly. (Then again, they’ll overbook the flight anyway!)

Beyond the “freshness” of the data, there are other reasons why indexing may be impractical. One show stopper is bandwidth. Indexing eats up bandwidth. Every document or piece of data is pulled from the source location back to the Indexer. If you are indexing a terabyte of data, you are moving a terabyte across the network.

Some organizations in countries in the E.U. and Canada, won’t permit parent companies in the U.S. to index their content. Their concern is privacy. The Patriot Act gives the U.S. government the right to seize any data from any organization if there is a perceived security concern. An index essentially copies anything that it crawls; a prime target for a discovery action.

One large organization that BA Insight is currently working with has subsidiaries in various countries across the globe. The subsidiaries have signed agreements with customers that prohibit the data from being transported across country boundaries.

One final example that I can think of is a situation where a subsidiary has a highly specialized search deployment and simply don’t want corporate to centralize search for fear of losing their customizations.

I’m sure there are other scenarios were indexing isn’t practical that I haven’t covered here. So all of this being said, is there a solution to all of this?

Enter Federated Search

Federated Search technology has been around for about a decade now. When a user executes a query, it is intercepted by the Federator. The Federator then passes the query to a number of different search engines, which execute the query. The search result from each search engine is passed back to the Federator, which merges each result into single unified result list.

The value of all of this that all indexing is done locally, by each respective search engine. This solves all of the issues mentioned above including bandwidth and privacy concerns. Since all indexing is done locally, bandwidth is a non-issue. Privacy is not an issue either, because a user can only query for data that he or she has access to in the first place. It can also solve the challenge of “data freshness”. If the relevant data is stored in a transactional database, you are getting data directly from the source system. You’ll always get the most up to date information. Problem solved. Another argument for Federation is it enables access to vast amounts of data that one would not wish to index for sheer lack of resources. Why index the internet, when you can just Federate Google or Bing?

Now, Federation has its own set of challenges which include:

• Performance can be an issue. As an example, as the query is passed to multiple search engines if one is offline the user will have to wait until the request times out.
• Security can be a challenge, if the search engines utilize a different security model, single sign-on must be implemented.
• Data duplication can often clutter the merged search result
• Merging the search result is a challenge in and of itself in the sense that the Federator has to merge results from different engines that probably calculated relevance differently. How does one normalize that?

I am going to be conducting a webinar on the 23rd where I’ll cover Federation in general and present BA Insight’s approach to Federation. Hope to see you there!

Watch On Demand: Tactics for Extending Unified Search to the Four Corners of Your Global Enterprise

I’m a hopeless book hound, and in my life have spent more on books than on cars. I have thousands of books. The last time we moved, we had 60 boxes of books, and my wife made me a proposition: We would unpack the five boxes that contained the books we knew we wanted, and leave the rest in the basement for two years. Over the next two years, I opened a few more boxes and pulled out what I needed. At the end of the two years we donated the rest to the local library. Needless to say, our bookshelves were much more organized, and had a lot less junk.

When it comes to managing and migrating your enterprise information, you can take the same approach. But with the right planning and the proper tools accessing your legacy information it can be performed far more efficiently than my ‘open a box in the basement’ approach. Enterprise Search is a technology that is all about breaking down ‘information silos’. According to recent surveys, the average organization using SharePoint has less than 20% of its enterprise content actually stored in SharePoint. This means that for most organizations, searching for information across multiple systems is simply a fact of life. But by taking a more unified approach to information access – with search that indexes and securely surfaces legacy content from across all of your enterprise systems using a single engine – you can provide accessibility to information without the pain and cost of a comprehensive migration.

Resist the ‘All or Nothing’ Migration

User adoption is the key to a successful SharePoint 2010 deployment, and the fear of upsetting users and management often results in the ‘all or nothing’ migration mentality. IT Pros often feel compelled to migrate all content into SharePoint from legacy systems because they fear they will miss critical content if they attempt to ‘cherry pick’. The problem with the ‘all or nothing’ migration approach is that it is the most complex of all possible options. Not only is IT faced with the often daunting challenge of trying to emulate legacy system functionality within SharePoint, but the legacy data needs to be migrated in its entirety, and tagged with appropriate metadata – which is often complex and not easily translatable from the legacy systems organizational structure.

Take a file system, for example. Content in a file system is often organized and classified by folder names and folder hierarchies. SharePoint was designed to (among other things) make the ‘file system approach’ to categorization obsolete. In SharePoint, each top level folder in a file system would ideally be mapped to a SharePoint Document Library. Each sub-folder would then be represented as a metadata column describing the content residing therein. It’s fair to say that all of us will confess to having gone sub-folder crazy at one point or another in our attempts to classify content using file systems. Attempting to duplicate this antiquated method of classification is SharePoint is problematic, to say the least.

Pondering the varieties of content that we have, the option of migrating from legacy systems into SharePoint, it’s easy to see how, despite best intentions, IT Pros are potentially hurting adoption and productivity by taking a pristine SharePoint deployment and trying to recreate the organizational structure from the legacy system.

Using Search to Avoid Migration Headaches

The term ‘Enterprise Search’ describes technologies that were specifically designed to connect to disparate repositories and securely surface their content. Search technology empowers IT Pros with more discretion and choice when it comes to what to migrate, when to migrate, and how to migrate. What makes this a win-win is that search is embedded in SharePoint already, and most organizations intend on deploying it anyway.

In a phased approach, IT starts with a pristine SharePoint 2010 deployment and moves content that users have identified as being absolutely critical to them. Smaller quantities of data enable IT to really focus on organizing it in SharePoint correctly. IT then deploys search and indexes content stored in legacy systems. This valuable content is readily available to users when they need it. Users are introduced to a “clean” SharePoint deployment and really see SharePoint in the best possible light, but still have immediate access to the legacy data they need. In fact, users can often find the content they are looking for more quickly through search than attempting to browse to it in the other system. This also applies to content stored in your SharePoint 2007 that hasn’t made its way to SharePoint 2010 just yet.

This approach provides several benefits. First of all, SharePoint is deployed much faster than it otherwise would be because IT isn’t in a holding pattern waiting for all of the content to be migrated. Second, users are immediately exposed to virtues of SharePoint at work, as they use and benefit from one of SharePoint’s most powerful features: Search. Third, SharePoint will be much less cluttered with content that will likely never be used.

Microsoft Enterprise Search – Best of Breed Search That You Probably Already Own

According to leading analyst firm, Gartner, Microsoft now provides best-in-class search technology out-of-the-box with both SharePoint 2010 and FAST Search Server for SharePoint. To supplement these search technologies – whether it be to extend their search capabilities or to optimize them for specific industries and verticals – Microsoft relies upon its ecosystem of ISVs and System Integration partners.

One such partner is BA Insight, who offers a collection of connectors that extend the power of FAST and SharePoint Search across a variety of ERP, ECM, CRM, and Messaging enterprise systems – while ensuring adherence to all legacy permissions and security models. With these connectors, organizations can unify and integrate information access across all of their enterprise systems, and approach the migration question in a more deliberate and strategic manner.

Office365 and Azure provide great capabilities from the cloud – but how do they work with the information in your other systems? Enterprise Search technology can bridge information silos, including those you might create by putting some but not all of your content in the cloud. This session covers the scenarios that use search and SharePoint with O365, Azure, and Cloud services, including ways to provide information integration between multiple systems and Office 365. We’ll show what is available OOB for search with Office365, and cover how to enhance it. We provide demos of solutions running with Office365 and with Azure, and provide some ‘secret tips’ for how to use enterprise search in different scenarios. If you want to understand the current capabilities and limitations of search with Office 365 offerings, and how to make them work well with your on-premise systems, come to this session.

Why do I need metadata? Why can’t I just search like I do on Google?

This is the question that comes up most often when I’m at trade shows, conferences, what have you. From a user’s perspective, all they do on Google is enter two or three keywords, and voila – relevant results! What they are not seeing happens behind the scenes, before their query is ever run. The two or three word query is boosted by metadata.

As explained below, you can deliver Google-like search to your users, provided you have the same advantages that Google has:

1. Unfettered access to data repositories
2. Content producers painstakingly tagging content so that it will rank high on Google, Bing, etc.
3. Content consumers unknowingly identifying and tagging the most useful content.
4. Google doing some pretty cool stuff on the backend (Top secret, but much of it is pretty obvious).

So what does this have to do with metadata? Numbers 2 and 3 above are all about metadata. In many ways it is the highest quality, in fact. Starting with 2, human beings are much better at classifying data at this point in time than machines are. On the web, administrators go to great lengths to make sure their content can be indexed and ranked favorably. Google provides a collection of tools to assist admins in complying with their standards.

The point here is that in the Enterprise, most users would never put forth this kind of effort without seeing an immediate benefit. People are too busy. So when you enter that two or three word query, it’s exactly that. The search engine simply does a brute force match of keywords and you end up trolling through hundreds of matching documents; hardly the Google experience.

The 3rd point above is what made Google famous. While in college, Google’s founders recognized that researchers would often reference the work of others if they valued it. Quality research was referenced more often. The “light bulb” moment came when the founders recognized people on the Internet do this as well. They realized that when quality content is published on the Internet, people link to it. This became the foundation of their now famous PageRank algorithm.

Hopefully now that you have a sense of how important metadata is to Google’s success, you’ll take a bit more time to read how MMS can help dramatically improve search for your users.

What is MMS?

Managed Metadata Services is a service provided by SharePoint 2010 that does three things:

1. Provides a central repository for managing an organizations Taxonomy(ies).
2. Provide a central repository for managing an organizations Folksonomy(ies).
3. Provides the ability to syndicate Content Types

Before explaining all of this, let’s get some terminology out of the way:

Taxonomy – A Taxonomy is a collection of related words/phrases that are typically arranged in a hierarchy. The picture below will probably give you a better sense of a taxonomy than me trying to describe it.

Microsoft uses the term Term Set in place of Taxonomy. They use the word Term to describe the words/phrase that constitutes a taxonomy, and they use the phrase Term Store to describe where Terms Sets are maintained and managed. Term Store represents an occurrence of MMS. So think of MMS this way:

TERM STORE (MMS)
     TERM GROUPS
          TERM SETS
               TERM 1
                    TERM 1a.
                    Synonym 1a.

The General Business Taxonomy depicted in the image above represents a Term Set.

Additionally, please note in the hierarchy that I depicted you can associate synonyms with terms which is essential for a good search experience.

Important Point! - Creating/maintaining a taxonomy is not a trivial undertaking. Many people do this for a living. As such, a taxonomy is typically centrally controlled by a Taxonomist, Librarian, or other professionals with deep knowledge in an area of their expertise.

The Internet has shown that social networking is a powerful means of capturing and sharing knowledge. One aspect of social networking is user tagging. Users apply tags to content that they like (or sometimes don’t!). Such tagging is often called a Folksonomy. Microsoft decided to create a new term here as well. In MMS, terms that constitute a Folksonomies are called Enterprise Keywords. Like Taxonomies, these are centrally stored in the Terms Store as users apply them. Unlike a Taxonomy, users can use any tags and are not constrained by the Terms in a Term Set (unless this is desired, but that defeats the whole purpose, right?).

The final capability (labeled 3 above) that MMS brings to the table is around Content Types. Think of a Content Type as a document or list item that has known characteristics (metadata). Let’s say I was a stamp collector and wanted to track my collection in a SharePoint List. I could create a Content Type called ‘Stamp’ with the following required properties (Age, Country of Origin, Condition). When Admins create document libraries, they will often specify a column as being of a particular Content Type, knowing in advance the type of documents that will be stored therein. This enforces consistent property values across the document library.

You might ask what this has to do with MMS? MMS enables the creation of a Content Type Hub, which centralizes the management of Content Types (CT). Site Collections, even other SharePoint Farms, can subscribe to these CT’s enabling consistent metadata across an organization.

Above I described the building blocks of MMS. Now, what can you do with it?

Term Set (Taxonomy) and Term Management

The Term Store enables management of Taxonomy and Terms. Admins can create a taxonomy from scratch, for example. The Term Store gives you the ability to import Taxonomies in a CSV format so you can easily create one in Excel.

An alternative to importing an existing taxonomy is to purchase one. BA Insight has recently partnered with a Taxonomy specialist called WAND, that offers taxonomies across many different industries and domains. As part of this partnership we are offering a FREE Wand taxonomy for General Business needs. Please download it here.

Regardless of what you end up doing, the Term Store enables term management.
The image above depicts some of the operations that can be carried out on terms. Some of the more interesting ones include:

• Merge Terms – Terms that have the same meaning can be merged. One or more of the terms simply becomes synonyms.
• Deprecate Terms – Any term which should no longer be used should be deprecated. This won’t delete the term from documents that have been tagged as such, but it can’t be used going forward.
• Move Terms – Terms can be moved between nodes, or Term Sets.

MMS and Information Architecture

One of the stand-out features of MMS is the ability to distribute the Administration of Term Set globally and locally. Administrating organization’s Taxonomy / Taxonomies is time consuming. They are constantly evolving and require continuous management. To Microsoft’s credit, they recognized that a completely centralized taxonomy is not optimal.

Enter Global and Local Term Stores

As discusses, the Term Store is a central repository where taxonomies, terms, and Enterprise Keywords are managed. This information can then be shared across the organization. Left alone, this would be problematic from an administration perspective. For one group, or coordinating multiple groups to manage a Term Store, would be a challenge.

To lessen the administrative burden on Term Store management, Microsoft introduced the construct of Local Term Stores. A Local Terms Store is a Term Store managed by Site Collection Administrators. What’s fantastic about this is that you can push the administration of taxonomies to the people that have that particular domain expertise.

In the image below, we have the Sales, Marketing, Product, and Legal teams, all subscribing to the Global Term Store (portrayed by the black arrow). Meanwhile, the Product Team (which develops Top Secret products) has their own taxonomy, which is strictly confidential. Both, the Product Team and Legal Team, subscribe to that. The Legal Team also works on confidential M&A activity, therefore must maintain their own Taxonomy as well.

Leveraging Metadata with Search

With MMS in place, and all of this wonderful metadata available, your organization can enjoy many benefits – records retention, metadata driven workflows, etc., but this post is focused specifically on search and findability. Users can expect a dramatic improvement in search by leveraging several features provided out-of-the-box with SharePoint (mentioned below) and optionally, products by third party vendors such as BA Insight’s AutoClassifier.

SharePoint’s Metadata Navigation for Lists and Document Libraries

One of the out-of-the-box features that SharePoint provides is the ability to filter Libraries and Lists. In the image below, the user has the ability to filter on any part of the Term Set. This gives the user the ability to start at the top of a hierarchy and with each selection, see the child relationships of the parent node that was selected. Alternatively, the user can go straight to the child node and filter directly. In either case, a lengthy list can be filtered with a few well pointed clicks.

Another capability that SharePoint provides out of the box is the SharePoint Search refinement capability, depicted below. Metadata is presented on the left of the search result. Users can refine results, what is typically hundreds or thousands of documents, down to a handful by selecting one or more of appropriate refiners.

To Conclude on MMS

Managing metadata is work. Often times, there is a lot of pushback from users because they don’t see an immediate benefit to doing so. But make no mistake though, the organizations that do make this effort will be more competitive as a result. MMS will make it a lot easier for employees to find content and substantially improve search within the enterprise.

The good news is that SharePoint 2010 now supports both Wildcards and Boolean Operators when performing search queries!  What does this mean? 

Wildcards

Let’s start with the Wildcards.  It means that now you can enter a query like this directly into the search box:

share*

This would return results that had keywords that started with “share” – it is very useful when you don’t know the exact spelling of something or you couldn’t remember the exact name of a keyword.  Also, with the new addition of the refinement panel you could start off with a very broad wildcard search and then refine your results to quickly get exactly the results you are looking for. Wildcard searches can be used in property searches as well – the example above is very simplistic but you could combine terms or property searches as needed.  For example this search would search on the Author Metadata property and return all results that started with John:

Author:John*

Boolean Operators

SharePoint 2010 Search now also supports Boolean Operators.  This means that you can now use things like “AND”, “OR”, parenthesis, =, >, <, <=, >= 

Here’s an example of the type of query you could run using the new syntax:

(“SharePoint Search” OR “Bing”) AND (title:”Keyword syntax” OR title:”Query Syntax”)

Boolean Operators have been around for a while but are usually only used by search power users. While wildcard searches allow users to do very broad searches, Boolean Operators allow users to do very specific searches so they can quickly find the results they are looking for that meet their search criteria.

Summary

With SharePoint 2007 the options for performing broad or specific searches were limited – although users could use the Advanced Search box it was confusing to many users.  This new syntax provides users with more options for finding content more quickly and efficiently.

Precision

When a user runs a two or three word query, there will be many documents that contain those terms. Naturally only a few of those documents will be relevant to you. This results in a very lengthy list of hits that the user must troll through. Precision is the measure of how many of these hits are actually relevant. Typically most people use the first page in the search result to measure this since most people wont’ even bother to go past the first page move on to page.

So if on the first page, you have 10 hits and 2 of these are actually relevant, you have a precision of 20%. Obviously the goal is to get this number as high as possible. So what’s the problem? The problem is that the amount of data being indexed by the search engine is huge and getting larger with each passing day. The more data that’s indexed, the more hits you’ll have, and the less likely you’ll find what you are looking for on page one.

Knowing this most companies resort to tagging content prior to it being indexed. The additional metadata provides the search engine with additional information so it’s smarter, and this also provides users with a means to filter content pre and post query. The downside of this is it requires a significant investment in time and money. BA Insight approaches this via a patented technology we call AptivRank.

Recall

Recall is a really simple concept. How much of the content that is in your domain of interest is actually indexed and searchable? If there is content in a system that is useful to you and it’s not indexed, you’ll never get a relevant search result.

So why not index everything? On the Internet it’s because content is published so quickly the search engines can’t even keep up with it. In the Enterprise though it’s largely due to security. Different line of business systems have different security models. Microsoft Search technologies, whether you are using FAST or SharePoint, only recognize Active Directory users and groups. Non Microsoft systems use proprietary security models. The answer to the problem is obvious conceptually but challenging to implement. One must create a map between the Active Directory users and groups, to those in the non AD based system. In terms of measuring relevance, the calculation that combines Precision and Recall can be quite complex. This article on Wikipedia provides a good overview, but will leave you feeling you should paid more attention in match class.

The truth is, you don’t have to calculate relevance to determine how SharePoint or FAST search implementation is performing. You can look at a much more telling KPI. Are users actually finding what they are looking for? BA Insight provides users with a search interface that gives them the ability to act on relevant content when it’s found. This information collected for reporting purposes and can provide insight into how well search is serving their needs.

Content Management and Enterprise Search are kissing cousins, but how can you make them really sing together? This session will provide a deep dive and best practices into how to best use search with ECM with SharePoint 2010. We will show how to optimize findability in different scenarios, including discovery and hold; how to combine search across active and archived content and across ECM as well as other sources; how to improve tagging and determine content migration priorities using search for visibility; and much more. If you are planning a complex ECM initiative (or in the midst of one), this session is for you.

What’s interesting is that Google had this same challenge back in the late 90′s when the amount of content on the internet began to explode. Relevance started to drop like a stone. Google claim to fame was to take a somewhat obvious (in hindsight) approach to improving things. On the Internet users have a tendency to link to quality content. People are more than happy to subscribe to a useful blog, for example. The fact that a user took time out of his or her busy day to do so must mean they value that content. Google began to boost the relevance of this content, as if overnight people began to find what they were looking for.So why not apply this approach to Enterprise Search? The answer is that Enterprise content is different in structure and in how it’s treated by users. Google’s approach doesn’t work well in the majority of organizations, because their content is not web based. Office documents, PDF’s, and the like are not linked. When a valuable document is placed in a file system, or even in SharePoint it’s not rated, reviewed, or linked to (Although SharePoint 2010 has just implemented a rating/review capability). A second difference between web and enterprise is that users on the web want content to be found. They go through great lengths to make it as easy as possible for users to find their content. This is done by making the content search engine friendly. In most cases they’ll submit their content to search engines asking for it to be indexed. Not so in the Enterprise where most people are just happy to hit their deadline.

SharePoint Search 2010 includes a new feature called “Social behavior improves relevance”. This feature monitors what links users click on in the search result and boosts the relevance of documents that are frequently clicked. So this feature like Google’s on the Internet attempts to boost relevance by having users identify relevant content without being so obtrusive that users aren’t turned off. This feature was first introduced on the Internet and is often referred to as click-through analysis. There have been many studies that indicate that click-through can indeed improve relevance. The question is does this naturally translate to the Enterprise and Enterprise content?

On the surface, this appears to be a great feature. In practice, depending on the type of data and metadata one has to work with will determine whether this feature actually helps or hurts relevance. The key to click-through working is relies on the quality of the document surrogates returned in the search result. As search result contains a representation of the documents that were relevant to the search. The surrogate often contains the name of the file, an auto generated summary or abstract of each document, and other metadata. If the surrogate is good, the user can quickly surmise if the document is relevant or not. If the surrogate is poor, users disregard the information, click on the file to download it, and then manually search through the file to assess relevance.

So poor document summaries will force users to click on links in a search result from top to bottom. Click-through will reinforce the original rank setting it in stone.

Administrators should do whatever they can to improve the quality of the information returned in the search result so that it is truly representative of the documents being summarized. This will insure that click-through works as it was intended.

SharePoint 2010, like MOSS 2007, leverages an inverted index and a property store. The drawback to this architecture is the Faceted Search capabilities within SharePoint 2010 will be unable to provide exact hit counts. The reason for this is that the Property Store in SharePoint Search is designed for the quick lookup of meta-data for the top ten documents being returned in the search result. Exact counts for the facet values would require the search engine query for the top ten hits requested by the user, then query the property store for all of the relevant meta-data and aggregate it by value. This is significantly more demanding than querying for a simple search result. Search systems such as FAST ESP are able to provide exact counts because the property store and inverted index are combined into a high-performance OLAP cube. Facet values are pre-aggregated at index time, prior to the user ever running a query. Without this architecture SharePoint 2010 cannot produce exact counts unless the number of documents is small, or for larger document sets, performance will decrease significantly on the server.

There have been prior attempts to provide exact counts on top of SharePoint, for example from the Open Source community on CodePlex, but reports have indicated performance levels dropped unacceptably. Some Administrators have reported query times increasing from sub-second to 10-14 seconds.

There have been several studies that demonstrate users expect a sub-second search result, with longer times resulting in poor user adoption of the search system. In 2005 for example, prior to Web search being standardized as many as 30 hits would be shown on a page (Reiterer et al. 2005). A Google VP reported that despite the fact that users said they wanted more hits per page; an experiment that included 30 hits per page resulted in a 20% reduction in site traffic (Linden 2006). The reason turned out to be that a page with 10 hits took 0.4 seconds to generate while a page with 30 took 0.9 on average. Linden (2006) found similar user sensitivity to half second delays at Amazon.com.

While Refiners with Exact Counts is a desirable feature, the consequences of deploying such a feature could significantly reduce user adoption and the ROI of a search initiative.