A Knowledge Copilot for your Entire Company using Precision RAG
Your knowledge base is more than a repository; it should be a co-pilot for your entire organization as well as your customers. This session will disrupt the way you think about knowledge management, introducing technology and guardrails that ensure the right information is available when and where it’s needed most. We’ll prove that empowering your team with an implicit knowledge engine can lead to transformative business outcomes.
You Might Also Like
>> Hi everyone. I'm next to Corinne for the great presentation. I'm Co-founder
of XFind,
recently acquired by support logic. And now our engine is the basis for support
logic's
latest edition which is resolvasex. What I'd like to do in the coming
presentation is
share first of all the support logic vision for what an enterprise knowledge co
-pilot can
look like or will look like, how that plays into our technology, how it
compares to other
solutions in the market, and how it can really improve on existing knowledge
management processes
and turn many overlooked resources you already have in the enterprise into
valuable real-time
knowledge. And there we go. And as Corinne noted, resolvasex is targeting the
answer
seeker in the enterprise. And this could be really multiple different people in
the workforce.
It can be from a customer wanting validated answers to their issues throughout
many different
people on the team. And when we go specifically into support, really everyone,
customers and
agents, engineers alike, want a great knowledge co-pilot. This is very obvious
in a lot of
research that's been done. So engineers, what they want is secure precise
knowledge co-pilot
embedded into their workflow. And we'll get into what that means exactly. But
what Corinne
and Karthika just showed gives you an idea. And customers also want a similar
ability
to help themselves serve themselves. And this, again, has been really a
recurring theme
in a lot of research that's been done in the past few years and really AIs
bring it to
a new level of expectations from what such a co-pilot can look like. But when
we look
into very specifically B2B support, what defines a good co-pilot is mainly
three things. So
first of all, it has to be in the user's flow. So whether it's customers, when
they're
coming in with an issue or agents, when they're dealing with that issue, it has
to be right
there in front of them and help them as they're working on the issue at hand. B
, it has to
be adapted to really different types of issues and queries. So it's not just
keyword
searches as we've been used to for so many years. Really, you want to be able
to deal
with complex types of issues, to complex types of cases, as queries as well.
And C, and probably
most importantly, is that you want to be able to make use of those internal
sources that
you have that are really very messy a lot of times. So past cases, Slack, Jira,
all those
data sources, you already have them in place. Many times they're overlooked,
they're not
really being used in order to help your customers or your engineers solve the
issues at hand.
And that's what we really want to have and enable in such a co-pilot. So if
everyone
wants this and we already have this data in place, the big question is why isn
't everyone
building it for themselves? AI is out there. AI has been available for two
years now,
we've all seen chat, JPT, and that big revolution. Why isn't everyone going
ahead and building
it for themselves? So the answer really breaks down into two sides. There's the
data side
of things and the tech side. On the data side, and again, when we're talking
specifically
about B2B support, this is really true, it's complex data sources. So as I
noted, a lot
of times, most of the data you want to have in place for such a co-pilot does
not reside
in a nicely curated knowledge base. Usually, a nicely curated knowledge base
covers maybe
20, maybe 50% of the issues you're actually trying to solve with a co-pilot,
while the
rest really is sitting in those past cases, in your Slack or teams in JIRA, and
you really
want to be able to deal with that kind of data. And that data doesn't play
nicely with
generic search tools, with generic AI tools, and it requires quite a lot of
massaging.
And we'll talk about what that looks like as well. The second issue is the
queries. So
as I noted, we're not talking about keyword searches here, we're talking about
actual
issues coming in with logs and descriptions and a lot of different technical
information,
and you want to be able to deal with those types of queries, not just the
simple ones.
So that's on the data side of things. It's on the tech side of things. So you
want to
be able to integrate with these dynamic sources, and that's not a fire and
forget type of integration.
You really need to be able to continuously work on that integration and make
sure you're
pulling the data correctly, doing the work correctly. Another element is
workflow adaptation.
So you want this to be as part of your engineers or customers' workflow. You
want to be building
the correct user experience and UI, and that requires quite a lot of work. It's
not just
something that you can do offhand. Another two important points are cost and
upkeep.
So upkeep, I think, again, these are complex types of workflows, complex types
of data
sources. That's there. Cost. So large language model, the cost there, they are
going down
over time, but still when we're talking about tens of thousands, hundreds of
thousands of
queries, yearly, the cost are there, and you want to be able to deal with that
cost efficiently.
That requires quite a lot of work, quite a lot of technology in place to be
able to do
that. So really, the point is that many legacy search solutions out there, or
do it yourself
rag solutions, where the embeddings are taken off of the internet, just as is
internet
trained embeddings, those really aren't going to be able to deal with the use
cases we're
talking about in B2B support. So if we break it down into the actual use cases,
what common
rag, as we call it, or vanilla rag, as I like to call it, what that can enable
you to do
is work with how to use cases, how to chat bots, right? When your customer
comes in with very
simple issues, and you have a nicely curated knowledge base with maybe a few
hundred articles
explaining how to solve different issues, FAQs and that kind of stuff. Yeah,
that can
do the work. For that, you don't really need us. You don't really need to
resolve a sex.
But what we're talking about, break fix issues, when we're talking about really
the more complex
types of issues that you have in the more complex types of companies and
products, when we're
talking about thousands of barely curated articles or where you want to use
your past
cases in order to help solve those issues, that's when you really need
precision rag.
The generic solutions won't really be helping you in those types of cases. So
let's get
a bit into the technology itself and explain what differentiates us and what we
're doing
from many of those generic solutions. So it starts really with the knowledge
sources.
Knowledge sources, again, many different types of knowledge sources from those
knowledge
bases all the way over to Slack channels, pulling them, really the first part
of making this
work is data cleaning and parsing. So you want to be able to take that data and
have
the models in place to massage that data into something that can really be
worked with in
the models that are going to be working with it. So cleaning it up, removing
irrelevant
information, making sure obviously that no PII, no personal information exists
there.
And then only then when it's really ready, you can take it over to the modeling
and language
processing phase. Now, with another big differentiator is our use of how we use
language models
in our engine. So what we do is instead of using just one approach, which is
many times
a lexical approach from past types of search engines, which is the keyword-
based approach,
or just using a semantic approach which uses internet trained embeddings like
we see in
large language models, what we do is we combine a lot of different approaches.
So lexical,
semantic, local semantic, a lot of different of these models into a stack that
really what
that gives you is a lot of different angles to understanding what is going on
in the data,
both on the query side and in the result side. And that enables us to achieve
very nice accuracy
even when our query, as showed earlier, is an entire case or a summary of a
case. And
summary of a case for us in our engine is a valid query much like a keyword
query is
an apporel. And we want to be able to deal with that quickly. We don't want to
require
months of training in order to achieve good results. We want to be able to do
it, pull
the data and go. So that's another important differentiator. And then comes the
guardrails.
Guardrails are really a critical part. Again, we all know hallucinations is a
big issue with
our language models and with the I in general. And we want to avoid that at all
costs. How
do we do that? So we have two types of guardrails in place. The first is in
domain. So we
want to understand whether the query that we're getting, whether it's from a
customer
or from an engineer as well, whether it's actually in the domain that's
relevant to the
data we have in place. So say for example, a user comes in and asks the
question, how
do I walk my dog? We don't want to be answering that question with data from a
cybersecurity
company because obviously it's just going to hallucinate and provide irrelevant
information.
So we want to detect that first. And that's one proprietary technology that we
've been
developing for quite a while now is how to be able to detect that. The second
after the
end domain element is say even if the query is in the correct domain, do we
have information
in place to actually answer the question? And again, if we do not, we want to
avoid answering
the question. We want to be able to say we don't have any information on this
topic.
Go ahead and maybe create an article or ask experts, but we don't want to just
make up
stuff that doesn't exist. Last and doesn't appear here, but another very
important element,
and this has to do with cost, which I noted earlier, is the whole process of
passes retrieval.
So right before we provide the prompt and the context, if you're all familiar
with the
rag setup, so right, we get the context, which is the relevant items, we have
the query, and
then we provide that as a prompt to a large language model, please answer this
question
with this context. Now the context, if we're just providing it with a three or
five page
long article, that is not precise enough to really be able to find the relevant
information.
That's A, but B, it also has to do a lot with the cost. The more tokens, the
more information
you're feeding to large language model, the higher the cost. And it's really,
it's an
avoidable part. If we're able to find the specific passages that will power the
answer, it's a
much better both in precision and in cost. So that's the whole process that we
've built
in place. It's a process that each data source, each customer that we work with
, this is the
entire process that we're running on their data, and it's built in a way that
really
integrates into many different use cases and workflows. Just to maybe drive
home another
point on the technology side, so this is a benchmark that we ran a few months
ago, taking
X-Find, then X-Find, the engine and comparing it to, again, the vanilla or
generic rag
approach, which is taking embeddings, say, opening eyes, ADA embedding off the
internet,
and comparing the precision between them given a few different sets of data. So
what we did
here is we used four different types of data. One is Ember, they sell coffee m
ugs, Alab,
sell office, some office hardware, eight by eight and waters. What we see very
nicely
is the more complex and domain specific the data that we're dealing with, the
bigger the
better X-Find compares to internet embeddings to ADA in this case. And when we
get to waters
that they sell lab hardware, there the difference is already almost a quarter.
So one in four
questions where X-Find or Resolve us X today will answer correctly versus
generic embedding
off the internet. And one in four is very -- that's a big number when we're
talking about
getting answers correctly in a customer service, customer support type of
scenario, right?
We want to make sure we're getting those answers correctly. 25% that's a big
difference.
So what I'd like to do now is -- so I've talked quite a bit. Now let's see a
few live examples.
And what I'm going to be demoing is two scenarios, two use cases. The first is
support portal,
a scenario, much like you're all familiar with, you want your customer to come
in, ask
questions and get answers. The second is more of a Slack use case. We've
connected the same
engine to a Slack -- in a Slack type of scenario. We want internal workers,
engineers, ask
questions and get relevant answers. In the portal, it's actually demoing on
super-object
data. So in the portal, it's our fresh test knowledge base. And in Slack, it's
using internal
get guru and confluence knowledge bases as well. And what I want to show is a
few edge
cases that will really, I think, bring home the point of the difference between
common
rag approach to what precision rag can really get you. There we go. So I don't
know if you're
not seeing it yet. So right away. There we go. So as I noted, starting with the
portal
side of things. So on purpose, I wrote a query here that's a bit long and B has
quite a
few spelling mistakes. And a lot of the solutions that exist today really are
built as I noted
for keyword searches. They're not really built for this type of length of a
query. And really
the ability here is A, to be able to retrieve the relevant items as you can see
in the center,
those are the relevant knowledge items to the given query. And B, the other
element here
is to be able on the large language model side, which is on the right hand side
, the
answer that's being generated, to be able to take the specific passages from
the top
items and mesh them into a very nice answer to the given questions. Just to
read through
it, right, the customer comes in with the question, which features does core
include?
Does it include voice? And there's a few spelling mistakes as you can see here.
But then by
providing the relevant items, even though the, even given the spelling mistakes
and being
able to retrieve the right passages, we really can get a very nice answer to
the customer's
question. Now another example here is, so what I went ahead and so elevate,
right, is
also of both a verb in English and a support logic solution. And you want to be
able to
be very precise in understanding when it's serving as a verb versus what it's
serving
as a name of a product. So when I ask it how to elevate the solution, which is,
it's a
very general question, but it goes ahead and answers to the point, right,
elevate as a
verb. How do I improve the solution? But then when I ask it how to add elevate
to the solution,
just by adding that word add, it understands that elevate is, we're talking
about the
product here and speaks to that, speaks to how to utilize that, the elevate the
product
in the context here versus, and not just elevate as a verb. So those very
specific, very small
details is what really is really important when you're trying to utilize, rad,
trying
to utilize these types of knowledge solutions in your context, especially again
, given complex
types of data. Now moving over to the Slack use case, what I went ahead and did
is, again,
comparing two seemingly similar queries, but they have a difference in what
they're
trying to achieve. So I'm starting with a query that's more, a question that's
more
process oriented, customer coming in from a customer's perspective, what's
required to
install the iFrame and Salesforce, right, the customer or the engineer is
asking for a
customer here, what do they need in place in order to install the Spore Object
widget?
And the answer here is coming from, mainly from the fresh desk, which makes
sense, right,
we want to, this is an externally relevant question and we want to answer to
that question
what the customer needs to do, what steps they need to take in order to install
in Salesforce.
But then when I move over to the next query here, and by the way, running these
queries
in Slack is very simple, it's a slash command, slash resolve, and you run the
question here,
it looks very similar, but, you know, again, the differences in the details,
the question
is what are the system requirements for installing the Salesforce iFrame? That
's a more technical
question. And here it knows to retrieve the relevant information, and this is
where it's
really critical to be able to deal with not so nicely curated internal
knowledge. Here
it's taking the main information from a Guru, an internal Guru article, and
Confluence,
Confluence and Guru articles, and what it's doing is it's providing the more
technical
information, more internally relevant information of what's required from our
perspective as
Spore Logic in order to make sure we can install the widget, right, the
requirements versus
not what's required, what are the requirements. So again, those little details
are really
important when you're trying to enable such an advanced type of knowledge
retrieval solution
on complex types of data. And of course, the more complex, the more it has to
be precise.
So I hope these cases, these demos gave you an idea, and I'm really, you're all
invited
to ask me later about like more specific examples or more specific questions on
the technology
itself. But maybe moving over back to the presentation. So I showed you a demo
on the
portal and in Slack. The same exact engine can power the CRM assist as Karan
and Krithika
showed earlier as well as the chat bot assist. So really all the different use
cases, all
the different workflows and experiences that you want to have in place can be
powered by
the exact same engine. Now, this is a comparison that Krishna thought of that I
really like,
which is what we see here is a test law and a Hummer, right, both of them are
cars, both
of them get you from A to B. But obviously when you think about it more in
depth, they're
engineered very differently and they're built differently for, in a sense,
different aims
even. So a Hummer, like when you're doing it yourself, when you're doing it
yourself,
a build yourself approach, to Rad, you're probably going to end up with Hummer,
which
is, it's a car, you know, it's a pretty nice car. But it's not engineered for
precision,
it's not engineered for performance and efficiency. What we're offering is a
test law. We're offering
something that's really built for that precision for the performance and the
efficiency and
to be able to work with very complex types of data and scenarios in order to
provide
that. I think, and we'll end with this, I think a good example of that is what
you'll
see in the next slide coming from Vanit, who's a VP of support at Seavent and
his experience
with Xfind. Seavent, one of our first customers and one of our first large
customers as well,
and they compared us, they compared our engine to other solutions out there and
they were
mainly taken away by the level of precision that we were able to provide on
complex types
of queries with their type of data in the use cases they needed, which is
mainly portal
on chatbot. So we'll end with this, his feedback, his thoughts on Xfind now,
Resolve
and the other side of the system. So we're going to start with the results.
>> We evaluated multiple providers to be very honest and fan when we're looking
at Xfind,
we're like still a startup, we're going to try that search engine, but the
results were
really astonishing if I could. We were getting what we needed, we never thought
that's going
to be the case. So we started with that, I think one Xfind did for us really
was three
things I could think of, one unified search. When we're talking about putting
information,
we're putting information from our training content, from our KB articles, from
our community
forums. So all of the knowledge could be designed in any way. It's actually
helping us pull
out the right content there. It's also helped us do really was unified
experience for our
customers. It doesn't matter where you are, you could get the same experience.
>> So as I said earlier, I invite you all to ask more questions and hear some
more information
on this engine. Thank you very much.
[APPLAUSE]
[Silence]