Skip to main content
Search
Liberating your Data

Liberating your Data with “Google-esque” Enterprise Search

  • 05 June 2020
  • Augmented Intelligence

Search is seen as a mature technology in B2B, B2C, and the enterprise, so why is it given such a bad rap?

The simple answer is that search is easy to deploy but incredibly difficult to make work well in delivering customer outcomes. In fact, 90 percent of enterprise search still runs on legacy technology. The reality today is that from retail through B2B to the many search boxes you may have within your organization, search is not meeting user expectations.

A new wave of AI-powered search, automatically managed in the Cloud and intelligent by design, is upon us. These new platforms from the likes of Google will change search forever.

Watch Karl Hampson, Kin + Carta CTO of AI who has been working with search for over two decades, and Michelle McGuire, Director of Product Strategy, to understand why search is more important than ever, what it means for your business and your customers, and how you can quickly make this transformation to intelligent AI-powered search to make yours “work like Google.”

Return to all on-demand sessions

Go back

Speakers

Karl Hampson, CTO Artificial Intelligence & Data, Kin + Carta
Michelle McGuire, Director of Product Strategy, Kin + Carta

Defining the Search Mechanism

(00:02:53.06)

Karl:
I was at Oracle in the early part of my career, I was lucky enough to meet some UK pioneers of applied neural networks out of Cambridge University. They went on to found a company called Autonomy. They introduced me to the art of the possible with unstructured data. And by that, I mean, human-friendly information, such as text, voice, video, emails, webpages, and so on. And that was in the mid '90s before Google. The web had just taken off here and having spent a decade working with structured data, I was completely hooked on this new opportunity. So I ended up founding an autonomy consulting business with some colleagues during the dot-com boom, here in the UK. And we worked really hard to build a reputation as experts and innovators with unstructured data and search in particular. And the core of that team is actually still here today as part of Kin and Carta.


Michelle:
What a great way to get started! So a lot has changed since you started in the space and search is pretty ubiquitous now. As an expert, how do you define the space today?

Karl:
Well, Google's mission, as we know, is to organize the world's information and make it universally accessible and useful. So the opportunity for organizations is to do the same with their own information for their customers and employees. And key to this is moving on from thinking that search is just matching keywords and seeing it as a way to quickly get some value out of AI and start on the journey towards a more cognitive platform and application set. We're in the golden age of natural language understanding. Search delivered in a Google-like way on top of these new AI-powered platforms is actually a real paradigm shift from what's come before. In terms of the market, it's obviously split into customer-facing and enterprise-facing use cases. So on the customer side, most commonly, site search, in-app search, product search, B2B and B2C e-commerce, and then enterprise facing, enterprise internet search, and knowledge management.

Michelle:
So we know where we are now, but how did we get here? How have you seen search evolve?

Karl:
Well, commercial search technology has changed tremendously, but from an intelligence and a scalability perspective, as you might imagine, millions or billions of documents, not really an issue now. Connectivity to any data source is also not an issue. And search has progressed from just the list of 10 blue links, which leads us to decide if they're useful or not, to a natural language interface for information discovery and decision making. And therein lies the problem, the gap between Google and almost every other search box out there today is huge and growing, because our expectations of searches are extremely high. There's actually no reason why organizations can't modernize to start to address this.

Creating a Relevant Search Experience

(00:04:42.00)

Michelle:
What's changing now and what does it mean for us and our customers to address this issue?

Karl:
Well, the introduction of AI means that we're now able to build our own cognitive search experiences. Gartner and Forrester refer to the search space as insight engines and cognitive search respectively. And that's basically recognition that we've moved on from keywords to AI. It applies in two areas: firstly, enriching content as its index to make it more discoverable using things like image recognition, entity extraction, document understanding, and categorization. But then on the experience side, the ability to understand user intent, allowing us to ask for the information we want in natural language and returning more relevant results and actual answers to questions. We also see things like knowledge graphs to help add context and expand query terms just as Google does.

Michelle:
So you're saying that search experiences vary quite a bit and there's an opportunity to improve? So how do you measure that?

Karl:
So it's a great question, Michelle. The thing about search is that it's a mature technology and most organizations already have experience of a legacy search technology. So recognizing this, we're creating a maturity model to help our customers understand where their search experiences are today and how they can start on their journey to AI. So we identified four maturity phases, each with an organizational and product maturity component. And this is the key reason why you can't just go out and buy the best search product if your organization isn't ready for it. Great search is a combination of technology, expertise, and relentless measurement and optimization.

Managing Growth and Expectations

(00:06:24.03)

Michelle:
How do organizations move from phase one to phase two to phase three?

Karl:
Baseline search is typically a legacy platform or a default implementation of a built-in search engine, like in a CMS, a content management system. There's probably no real stakeholder at baseline maturity. It's usually pretty easy to progress to advanced by addressing the organizational data first. So assuming someone is incentivized to actually fix this, like senior management hearing all about how bad the search is, what I would do is get some advice on the capabilities of the existing platform and modernize it immediately if needed, then set up a search management team and identify a product donor. And their job is to work together to understand, measure, and improve the experience. It's really all about seeing search as never done as we both know, then it really should just be a natural and iterative process driven by analytics and direct feedback. And once you're at advanced and everything's working, well, that might be enough for some to stop there. Phase two to phase three is a bit harder. For this, you'll need to be on a modern search platform, so there could be some technology investment there. And this requires some high level stakeholders on the experience side to really wanna have the best-in-class search experience in your vertical. At this level, you'll definitely be using some of the new AI features such as natural language processing, with data and machine learning to drive relevancy. Someone also needs to be owning an ongoing vision for search and working with either your own development teams or a supplier to deliver that.

Michelle:
You and I have talked through this before, and you've said that organizations can't skip phases. Why is that?

Karl:
Well, as you progress through the maturity levels, you're basically understanding more about what your customers actually want from search, how they use it, and where you can take them to next. So it's a journey and whilst that can be accelerated now with the right technology and guidance, so we can help with that for example, for the most the organizational challenge means you have to work up to the next phase of maturity.

Michelle:
Things get really interesting in phase four. Walk us through that. What's reality there and what is still aspirational?

Karl:
When we first created a version of this model, about four years ago, the right hand side was a vision we'd come up with by connecting the dots across search and conversational UI. We tend to still think of search and chat as different front-end experiences. But on the back-end today, they're both about natural language processing and information retrieval. So why can't a customer today ask a question in the search box on a retail site, for example, do you have this in blue? When is this item back in stock? It doesn't have to be in a separate chat bot. So in 2016, when we did this, it was all a bit blue sky, but today products like Google's Cloud search and Dialogflow can bring together search and chat with the same AI technology. Other vendors, like Lucidworks, have AI-powered search as part of Fusion and have recently added a feature called Smart Answers. So you can basically see where it's all going.

Michelle:
One thing that I hear frequently from organizations is that AI can solve all of their search problems, but AI isn't a silver bullet, right?

Karl:
Absolutely not. Fortunately there are no AI magic ones here. Part of the issue with search for many years has been this idea that you just buy the best and it'll all work. But unfortunately, as we both know, search has never been like that. Whilst it can now be much more intelligent through AI, AI is not gonna help you with some of the more prosaic challenges such as connecting to data sources, data quality, implementing security models, and actually creating a frictionless and engaging user experience.

Search Modernization

(00:10:04.09)

Michelle:
Where are you applying this directly? Can you give us examples of where you're seeing this kind of search modernization taking place and what you can do to get started? Let's start with in-app search and site search.

Karl:
So in-app search, I see a lot of scope to improve. That's on mobile. We've had a nice example in healthcare recently where an insurer is looking to provide better access to care. And this is a good example where a traditional search approach, which is not intelligent, just doesn't work. You've got issues such as members using consumer health terms like knee doctor, when the data's full of medical terms like orthopedic surgeon. And that is then compounded by the typing errors we get from the fat finger effect on a mobile keyboard. As we all know, most of what we type into Google on our phones can be all over the place. But as a solution with search, it can be done, and we should expect more of a typical in-app search feature today. Site search, I'm talking about your typical brand or corporate website search on the CMS, sort of not e-commerce. People are usually looking for information about products, services, company details, and the latest annual report. These kinds of sites have been underserved for years. They're commonly at a baseline level of maturity. We've worked on many of these and there's always an opportunity to create a great customer service channel by adding a modern search experience. And these sites are basically sending a really negative message to customers that we don't care through an existing poor search experience. I'd actually encourage sites to not have a search box at all, rather than a bad one.

Michelle:
Let's talk about retail next. The increase in online demand is seemingly making every day like Black Friday. What are you seeing there?

Karl:
Retail is perhaps the most interesting one right now, given all that’s going on with the pandemic. Most of us are well aware that retail has seen huge increases in demand online, and that also this is likely to continue. But that increased demand has just magnified all of the existing issues with search as well. So more zero results queries, more dissatisfied customers. So this is where there's a real opportunity to embrace a modern search platform and create something truly transformative. Retailers have a ton of behavioral data. So using that along with new AI search technology can really enable this next generation of search experiences in ways that massively reduce friction and improve convenience. The pinch point in search is always single word queries, where there are a gazillion results and you need some intelligent signals to rank them effectively. For example, if I'm a regular customer and I'm searching for something like coffee, I wanna see the kind of coffee that I buy first. In fact, even in the autosuggest dropdown, along with an add to cart button. I don't have to scroll through results to find it. In retail, merchandising is another classic pain point. So merchandising teams commonly have this hairball of hard coded rules to deal with poor search relevancy, but the intelligence is there today to fix a lot of this in the search platform. And that means merchandisers can actually get on with what they should be doing. If I was in charge of search in one of these retail sites today with a scaling or a search quality issue, I'd be seriously planning for the kind of modernization intelligence and elastic scaling that I'm talking about.

Michelle:
And what about B2B?

Karl:
Well, so much of B2B is actually behind retail. I would put this in the same category as retail in terms of the opportunity. But the potential positive impact on customers is huge because the overall maturity for B2B is typically lower than B2C in search.

Michelle:
Having worked alongside B2B organizations, I would definitely agree with you. The B2B space has so many opportunities as customers become more self service. Let's touch on enterprise search.

Karl:
Yeah, great one. So the first opportunity in the enterprise today is to use these new AI search platforms to unlock the value in all of your unstructured data so much more effectively than how things have been done in the past with legacy tech. So it's about unveiling documents, reports, research, finding people in your organization that can help you make better decisions, and just generally being more efficient, and not duplicating effort. Now with this shift to remote working we're all experiencing, employees don't have access to their coping mechanisms, the analog tools in the office. They might use white boards or print outs, asking a colleague. They've all gone. So great search can make a huge difference here.

In terms of the state of search technology in the enterprise, we heard that 90% of enterprise search is still on legacy technology. That was actually researched from Bank of America. Merrill Lynch mentioned on Elastic's first earnings call back in 2018. And I doubt much has changed between now and then. So legacy search tech typically required a huge investment and had a high cost of ownership. A lot of it under delivered, there was a lot of disappointment because the teams were just not there to own and manage the deployments. So one thing I've seen consistently over the years is that few organizations have managed to maintain the maturity required to fully own their enterprise search platform and get value out of it. But with the cloud and AI, it's actually easy to modernize today and a lot of the heavy lifting can be removed. So that means you can have a smaller team looking after it. So whilst on the customer side, AI can create amazing omni-channel experiences in natural language, including search. In the enterprise, the bigger opportunity is to use AI-powered search as a great way to liberate your unstructured data, but to actually use this as a start on your journey towards more cognitive platforms, which inject intelligence and automation into your core business processes, decision making.

The Google Search Experience

(00:16:12.00)

Michelle:
Let's talk a little bit about internet search since Google informs so many of our expectations. I know we both often hear, I want our search to work just like Google. What makes that so difficult?

Karl:
Everyone always asks, why can't our search just work like Google? It remains the single most popular requirement. Usually, people that ask that don't really have a huge appreciation of what the request involves. Google literally has thousands of people working on search. And it's so complex today that having your own search to work, even slightly like Google, is a big deal, but it's actually not as big a deal as it used to be. Also Google isn't just one thing anymore. It's still accessible from a single search box, but it's actually now a highly cohesive ecosystem of specific search applications that are all designed around their individual use cases. So web, news, images, maps, it's a subtle but important point. The experience of search actually has to be experienced and optimized for the use case along with the data. But Google actually does an amazing job of making us think that it's still just one search box. So there's actually a clue there in how we can scale out different kinds of search experiences without them becoming silos in the enterprise.

Michelle:
Modernization of search and the introduction of AI has made things more intelligent, but what about that scale and simple management? How is the cloud impacting this?

Karl:
Well, from a management platform perspective, most of what we're seeing now is cloud first. Every cloud option out there for search is much more flexible than what we used to do with bare metal and VMs. So typical cloud options today range from being able to rapidly provision and manage search clusters for any scale right through to Google's new cloud search approach, which is totally serverless. I mean, you literally get a piece of Google's global search platform complete with the scale and the AI power. So there are actually many options out there depending on your use case and all are completely valid. Really, there should be no excuse going forward for not meeting demand. Many existing search deployments will struggle with a step change in traffic, but they can be modernized to remove that as a future risk.

Michelle:
Since we're coming up on time, Karl, any final thoughts?

Karl:
I was just gonna mention the process of buying new search technology. From what I've seen in recent years, it appears that it's not really kept up with what's going on in the market. So those long form RFP checklists for the features feel like they're well past their sell by date for search. The complexity of the products and the vendor messaging can make everything sound really similar. So if anyone is actually in that position, I'd urge you to think carefully and if possible, get some help to navigate the options. And do that along with an understanding of your target search maturity, where you think you wanna go with that new AI-powered technology.

Michelle:
I think we can all take away three key things. First, the search maturity model is a great guide. Companies should use it to gauge where they are and where they can head next. Second, the next evolution of search should include AI for natural language and cloud platforms for scalability and simpler management. And third and finally, today's search technology is advanced and intelligent. We're finally seeing the promise of AI creates some paradigm shifts in the market and search is a great place to start your journey towards AI. We can now begin to move past traditional search experiences into things like conversational and chat experiences as well.

Want to know more?

Click here to get in touch