Moving Passed the Sizzle of Concept Search
That is usually the reaction when you’re sitting through a demonstration of concept search, also referred to as semantic search. Invariably the Enron data set is trotted out in all it’s glory and the presenter shows you how you simply put in a common word such as ‘revenue’ and the engine will return documents that include that word as well as those that do not, but include related words like ‘income’ or ‘profitability’ or ‘sales’. It truly is impressive how machines can infer so much richness from human language, but invariably the audience walks out of the presentation with nothing more than a good feeling. The question is, how can this actually be put to work that will either make you money, save you money, increase opportunities or decrease risk?
It’s About Finding Things Not Looking For Things
At first the above phrase might seem like double-talk, but to understand the power of semantic search you need to appreciate the difference between ‘looking’ and ‘finding’. Traditional keyword search is great for looking for things of which you already have a sense. If I know I’m trying to find out things about Project XYZ I would look for things by simply searching for ‘Project XYZ’, for team members on Project XYZ staff, and common terms they use in discussing the project. This type of search is what we usually think of when we say ‘search’ and embodies the idea of ‘looking for things’. Classic search is most suited for the late stages of an investigation, audit, or research project. I say late stages, because you have already funneled your area of exploration so that you know specifically what you are looking for.
‘Finding’ things is what semantic search does much better, and it is an analysis that is centered around discovery. These text processing engines can churn through terabytes of seemingly random documents, emails and messages and find what concepts are being discussed. The power here is two fold: first, the text processing can see structure in the data and documents and help reveal patterns that are otherwise difficult to discern, and; second the concept analytics allows for non-literal searching for items related to key words but not textually similar. Both these aspects are incredibly powerful in various parts of the enterprise. For example, an enterprise knowledge management application could use semantic technologies to efficiently categorize and relate disparate work products throughout the organization, highlighting areas of expertise and specialty that may be a competitive advantage. Alternatively, a semantic engine integrated with a records management process could automate much of the tedious classification process that proves to be a barrier to success for so many records management efforts. And where traditional search is powerful at the later stages of investigation processes, the concept-based technologies shine at the early stage of investigations and legal efforts, when the exact details are still unknown and the analyst is looking to ‘find out what happened’.
The point is that concept search is not just a better form of search. Concept search is wholly different in it’s character. It brings all new capabilities to the enterprise and with it the ability to realize new workflows and radical improvements in many knowledge worker processes. In the coming weeks, I’ll be discussing very real world case studies of how these technologies are disrupting traditional business niches.