Inside the Intelligence: SupportLogic Machine Learning Architecture
This session will take you through the architectural depths of SupportLogic’s machine learning framework, challenging your preconceived notions and equipping you with the knowledge to harness ML for predictive, actionable insights. We’ll cover the ML engines that drive various parts of the platform, and discuss why technology like gradient boost classifiers and ensemble models benefits support teams.
You Might Also Like
>> Yeah, thank you, Ryan, very much for that introduction.
And thanks to you guys for making the trek down the hall to the gold room for a
solution
deep dives.
Ryan covered this one, but just to give you kind of an outline of what we're
going to
cover today.
So we'll first just have a very late brief overview of AI or support logic
followed by
touching on the data pipeline and ML as a service.
And then after we've kind of chatted about the infrastructure a little bit, we
'll really
like dig down into the different ML engines that we have for support logic.
So those of you who were paying attention to current this morning might notice
that I
actually borrowed the slide from his presentation.
But I do think it's a really good representation of how support logic has been
invested in
ML from the very get-go.
So our signal extraction engine, our kind of first product was backed by what
at the time
was state of the NLP.
The field has moved on a lot since then.
So as the field has expanded it and more and more advanced and powerful
solutions have
become available, support logic has kind of grown with the field.
So incorporating those new products or those new solutions into new products
that we're
able to offer.
So that brings us right up to 2024 with our summarization engine and our answer
engine.
But I think also importantly we're not kind of leaving those legacy products
behind.
We're reinvesting in the products that we already have, the services we already
have.
So if you were to look at our signal extraction engine today, it's as barely
recognizable compared
to our signal extraction engine from 2017.
And just to kind of illustrate how machine learning is kind of embedded in all
over our
products in support logic, what we're going to talk about today touches machine
learning
in all of these different areas highlighted by a yellow arrow here.
So it really is kind of throughout our products and everything you're touching.
So whenever we're developing a new product or thinking about what we want to
build as
support logic, we try and stick to a few kind of general principles around our
AI.
So firstly, we believe that AI should be an invisible force.
I know Krishna likes the electricity metaphor, but I kind of like to think of
it like a
smart home.
Like AI is kind of everywhere.
It's all around you.
It's in everything you're touching, but you're never really thinking about it.
It's just there helping you improve your workflow, second guessing what you're
going
to do next.
And you don't even kind of notice that it's happening.
Secondly explainability.
As Ryan touched on in the intro, explainability is a kind of fundamental
principle with for
us at support logic.
Three or four years ago, explainability was like all the rage in AI and then
that kind
of disappeared.
And it's definitely we're seeing the emergence of a lot more kind of black box
solutions.
And that is kind of a caveat of these very large language models.
But whenever it's possible, we still want to stick to an explainability kind of
paradigm.
We think it's important that our customers know where those predictions are
coming from.
And we think it helps our customers understand their customers and their
organization if they
can kind of really dig into their data and these predictions.
And then thirdly, I have adaptive here.
So when I'm talking about adaptive, I'm covering kind of a few different areas
here.
So firstly, we always want our AI to be improving.
So we wanted to improve with our customers data, improve with their workflows,
how their
organization has changed, improve based on the feedback that they're giving us
as to
how this is working for them and their organization.
And then I also am kind of using adaptive to cover like the being able to
configure
our AI.
So we want, we know that in not all instances, is it kind of a one size fits
all.
So we want to be able to configure our ML models where possible to, you know,
our different
customer needs and requirements and their data.
So I'll touch on it kind of as we go through every model, but a lot of our
models are highly
configurable at, you know, in different places within the kind of training
inference, the
data that's being used and things like that.
To touch very briefly on our data pipeline, this is a very simplified
representation of
our data pipeline.
I hope it doesn't get back to our data platform team.
This is what I simplified it down to.
But so the customer data is obviously coming from the CRM and it comes into Sep
ologic via
the data pipeline.
So for kind of the machine learning aspects of this, what's really important
here is transforming
the data into universal data schema.
So we're able to take data from across a number of different CRMs and, you know
, turn that
into like a uniform data format that then we can pass through the ML pipeline.
And the pipeline can be completely naive to what CRM this data came from.
It doesn't need any information about that.
It's just sees data and will act on it accordingly without kind of caring where
it came from.
And then all of those machine learning, all of the services or the machine
learning in
the data pipeline, sorry, is being serviced now by our ML as a service platform
So ML as a service is something we really invested in, I would say, a couple of
years
ago now.
It's definitely something like a big infrastructure change like this can be a
pretty hard sell
at a company level.
It doesn't result in like a nice new shiny effect in the UI.
But it does like behind the scenes make our ML much better, much more kind of
future-proof
and scalable.
So previously before ML as a service, we were training a lot of our models
locally so the
ML team were training just on their computers locally.
And then these ML models had to be deployed to the customer VPCs.
Some of these models had to be deployed to multiple places within the VPCs.
So that's just from the outset there, you have a risk of like, you know, two
different
models being accidentally deployed to one customer VPC or only one.
It only being updated in one area.
So you're not having like a uniform ML across the customer environment.
Also like training locally is very slow.
It's pretty high risk unless a customer has a staging environment.
We're pulling data directly from production, which is never really anything you
want to
be touching if possible.
So then moving to ML as a service, we're able to move that training into the
cloud, which
means that we can, with the click of a button, we can set off training for a
new model.
And then we're still able to keep all of our customer models siloed and like
segmented
into their own like tenancy if you like, but they're able to be served through
a single
API, a unified API endpoint.
So this makes it really nice for, you know, we're able to have better
monitoring.
Like when we're just, when we are just deploying the machine learning models
like manually to
a VPC monitoring and kind of maintaining of that model is really hard.
But once you have ML as a service, it's much more like available at your
fingertips to
monitor and maintain these models.
It's still less arduous to kind of make sure everything is where it should be
and looks
good.
Yeah, so, and then we, ML as a service basically says this is just an API layer
and it means
that we're able to, with like cloud run endpoints, we're able to seamlessly
kind of update models
without any customer downtime.
We're able to roll back those models if we need to, if like the worst case
happens, it's
very easy to roll back that model.
We can slowly upgrade models across customers.
It's not kind of an all or nothing.
And that's again with like the click of a button.
Just rather than having to go in and manually change models for every single
customer.
Okay, so we've touched on a little bit of the infrastructure supporting the ML.
So yeah, now we can start really getting into the models.
So yeah, as you've heard a lot already today about signals, signals really is
like foundational
to support logic.
I mean, support logic was founded on the belief or the knowledge that there's
so much information
in this text content of support tickets that we're just missing.
So yeah, we are able to, as was discussed earlier, this morning our signals go
much
further beyond kind of the positive and the negative that is kind of your
traditional
sentiment.
So we're able to detect up to 30 different signals and then these fall into a
couple of
different groups.
So we do have like a positive group and a negative group, but they have a lot
of like
finer grained signals within them.
And then we also have things like a commercial group looking for signals
related to commercial
either in the positive or negative.
And then looking specifically at our signal extraction engine pipeline.
So at the beginning of our signal extraction pipeline, text is cleaned and pre
processed.
So this is a lot of kind of removing the junk text that we don't want to
extract signals
from.
It's getting the text in like a good shape to be handed to an LLM to make sure
that LLM
is going to kind of get the best it can out of the data.
The text is then embedded at the sentence level by an LLM.
I do want to point out here that LLM has like become synonymous with JNI now,
but LLMs have
been around kind of long before these like newer generative models have been
kind of
appeared on the scene.
And yeah, so every like generative text AI model is an LLM, but not every LLM
is a generative
model.
We're using an LLM here, but it's not like one of the JNI models.
So yeah, so as I said, the text is embedded at the sentence level, and then it
's sent
to a classifier.
So this is kind of the embedding representation here, and then this is a sigm
oid layer of
classifiers.
So each of these classifiers is able to predict based on the embedded sentence,
sorry, whether
or not the signal that it is designated is present in the text data.
These classifiers are multi-label, which means that you can have multiple
truths predicted
through this layer of 40 plus signals.
And then these classifiers are trained on millions of rows of sentence data so
that we
can get an accurate representation of when a signal is present or not present.
And then the final step in our pipeline is feedback.
So through our UI, we have the ability for customers to provide feedback on the
signals
that they're seeing.
Customers can also add their own signals.
They can highlight text and add, label it with their own signal if they think a
signal
is missing.
And then with this feedback data, we sanitize the feedback.
Unfortunately, customer feedback is not always the cleanest.
So yeah, we see a lot of duplication signals added and then removed and added
again, things
like that.
Like clean it up and make sure that none of that noisy data is going to get
into our classifier
and make sure that the feedback that we have is going to improve the classifier
and not
be detrimental to it.
And then we use various data augmentation techniques.
So I mentioned earlier that classifiers are trained on millions of rows of data
So throwing in one sentence, a feedback is probably not going to shift the
needle very
much.
But we're able to use data augmentation to really kind of grow that feedback
into its
own little data set so that we can see that it is actually having kind of an
effect within
the classifier.
>> Good question.
[INAUDIBLE]
>> Sorry?
>> I want to have a good question for you.
>> Yes, sure, I guess.
>> [INAUDIBLE]
>> Okay, we are creating some custom signal-sized data.
>> [INAUDIBLE]
>> So if we create extra signals, the extra sent human signals, when you do
custom, then you select feedback.
And then we post like a text and write a bit of a negative something else.
>> Mm-hm.
>> So that gets removed from our system, which you think that is not relevant
or it will
just go in trouble?
>> It will affect the classifiers, which will affect your system.
>> Okay, so moving on to the scoring engine.
The scoring engine is kind of the natural progression after the signals.
All of our schools are built on top of very strong foundation of signals.
But they also include a lot of metadata coming from kind of the case or account
level as
well.
So it might be kind of, or like one way to think about how we're building these
signals
is like it might be nice if we could just take the sentiment scores, right?
We have, so say for account health score, we have all these cases.
All these cases have a sentiment score.
And like why can't we just average the sentiment score and then we have an
account health score?
And that would be fantastic.
But unfortunately, the central limit theorem dictates that if you start to
average a large
quantity of numbers, they will converge into a normal distribution, which what
that basically
means for the end user is you're going to see a bunch of scores that sit around
our baseline
for sentiment, which is 70.
And you're not going to see variation across customers.
And if you kind of think of an example of that, say you have three cases with
sentiment
of like 10, 20, and 90, that's going to average to 40.
If you have three cases with sentiment of 35, 40, 45, that's also going to
average to 40.
But there's obviously very different things at play between like one set of
cases and
the other set of cases.
So one of the things we do is use cumulative distribution function.
This is able to kind of recognize the changes, anomalies, like differences that
are happening
in the cases from one time period to the next.
CDF is actually commonly used underlying anomaly detection itself.
So we're able to use cumulative distribution function, look at how these
different inputs
to scores are changing, and then kind of weight them and allow them to
influence the final
score in different ways.
So kind of that example I gave is mostly applicable to account health scores,
but all
of our scoring, so customer effort score, auto QA score, sentiment score,
attention score,
is all powered by kind of a complex set of weighted heuristics.
And so we talked about being able to configure our ML at the beginning.
So these account health scores is one area where we have configuration
available.
So if you as a customer are more interested in how, or you think the escal
ations have
a stronger impact on your account health scores than you think we're giving it
in our
algorithm, we can actually customize that.
So there's various different kind of groups of signals.
Escalations would fall into, I believe, business impact.
So we could adjust at a customer level the weight of the business impact score
and reduce
the weight of some of the other scores so that then the Econ Health scores that
a customer
would see in their system would be more influenced by escalations than
previously.
Okay, so our prediction engine.
So we heard a lot this morning about how we have lots of customers who are
really drastically
reducing their escalations, and that is in no small part to our escalation
model.
So our escalation model is actually a gradient-boosted classifier, so that's
kind of represented
in the diagram here.
And what that means is that it's basically a sequential, like a series of
decision trees
or decision stumps.
So this is a decision tree.
A decision stump would just be the first bit with just like one split.
And all of them individually are very bad at their job.
But when you kind of add them up sequentially, they become greater than the sum
of their parts.
And how that works is you train this first model and it will detect some very
simple
pattern in the data, and it will be able to predict maybe a couple of the cases
or a couple
of the samples of data.
And then you take the prediction residuals, which is just a fancy way of saying
the ones
that got wrong, and you weight those errors so that the next model is able to
kind of
correct the errors of the previous decision tree.
And so there's a few shown here, but this can be hundreds or thousands of
decision trees
running in a sequence.
And at the end of it, you have what actually is a very powerful and very good
model.
One of the kind of features of gradient-boosted classifiers is that they have a
lot of model
capacity, which means they can learn very complex, uncomvoluted patterns in
data, very complex
decision boundaries.
And that's perfect for escalation prediction.
There's so many nuances to data or to a case that can result in escalation.
So having a model like this that can really adapt to all of those nuances is
great.
It does come with a caveat that it needs a lot of data to train on, to not kind
of
overfit and focus too much on some simplistic patterns they're seeing and not
generalized
to the whole environment.
So the data that this has trained on is historical escalation data.
So we look at customers' historical data, cases that have escalated, cases that
haven't
escalated.
We actually use time series analysis to break up those cases into chunks of
time.
So obviously in the lifespan of a case, there's so much that's happening.
So what happens on day one at a case might be nothing to do with why it
escalated on
day 45.
So we want to look at discrete time intervals throughout the case.
So we're able to learn what a case looks like before it escalates, or way
before it
escalates, as well as right up to the boundary of escalation.
We also include our customer feedback here as well if we have customer feedback
So obviously when we're onboarding new customers, we're just looking at
historical data.
But as our customers are using the escalation model and giving their feedback
on it, we
also incorporate those labels into the training data in the same way.
So a new model will incorporate customer feedback as well.
And then so at inference, we're taking a case at that kind of snapshot in time.
And we're looking across like 70 features that this model has learned the
weights for.
And we're applying those weights at inference to see if we think a case is
likely to escalate
in the next time period.
Okay, so our recommendation engine, intelligent case assignments.
So this is also actually a group of different models, a selection of different
models, but
this is not an ensemble model.
Instead, this is allowing us to assign cases based on a number of different
factors based
on the agent and the case at the time.
So in no order of favoritism, we have a skill score.
So our skill score is actually made up of an agent skills model, a case skills
model,
and then a skill match model.
So for a case skills model, we're again training on historical data and we're
able to learn
from how a case opens the skills that it will need at the end of the case.
So if a case opens and it contains certain phrases or it contains certain
skills already,
we're able to learn from historical data the probability that at the end of
that case,
it's going to require like these extra skills.
And so sometimes we see really strong relationships and we're able to stay with
very high probability,
you know, like this case opened with skill A. So we're almost guaranteed that
we're going
to see skill B by the time it closes, but maybe we'll see skill C as well.
And then for our agent skills, we're just extracting the agent skills from the
cases
that they have handled.
So we look at the cases that an agent has held and actually contributed to and
we extract
those skills.
And then we're using kind of a weighted distribution to give a different
expertise level to the
agents. So if an agent has a held or like worked on a lot of cases with a
certain skill,
they would get a higher expertise level and another agent.
But we use like an inverse frequency weight.
So if every agent has handled a lot of cases with a certain skill, they should
all be reduced
because it's like we're kind of having the system know that that's not maybe a
very important
skill.
So if a single agent has the skill, it's not really a defining factor for them.
So a very common skill if even if all agents have it should have a lower
probability.
And then we have the skill match.
So that's basically taking the case skills, both those extracted and the
probability of
the ones we've predicted to appear on a case and the agent skills.
And then we're doing some probabilistic matching to try and find kind of the
best agent for
the case based on the skills that are there, the skills the agents have, the
expertise
level and the probabilities.
We also look at customer history.
So this is, does this agent have a good rapport with this requester, with this
account?
We look at the history again of the agents and the cases they've held for this
account.
And we use the case performance model to basically rate how well the agent
performed on the
case.
We only look at the period of time that there are signs to the case.
You know, if an agent gets handed an escalated case, we don't want that to be a
negative impact
to the score.
But if an agent is holding a case and it gets escalated, then maybe it would be
a negative
impact to the score.
And so once we have the case performance for all of the cases held by an agent
for the
account, we roll that up into a customer history score.
We also have a bandwidth calculation.
So bandwidth is looking at the agent backlog and their availability even take
on a case
at that time.
So obviously not all cases are equal in the amount of time and effort they're
going to
take.
So bandwidth is very customizable based on the status and the priority of the
case.
So if an agent is holding 20 cases but 19 of them are pending, they probably
still have
pretty high capacity still.
But if they're holding 20 cases and they're all open, they're probably pretty
busy.
Agent backlog is also tuned to an individual customer's data.
So we have to know what's like normal for a customer.
You know, we have some customers where just the way they handle their backlogs
and their
system, it's not unusual for an agent to be holding 50 cases.
We have some customers where if an agent is holding 50 cases, we're on fire.
Like something's gone very wrong.
So we need the system to know that so that we don't just have a hard cut
threshold where
we say this is a lot of cases because that's not the case across all of our
different customers.
We're also looking at the incoming case as well.
So we want to make sure that we know the size of the incoming case, the kind of
effort
that that case is going to take to solve.
So we can see from an agent's backlog, you know, they have a little bit of time
but they
don't have enough time for this case.
This is a big case.
You know, maybe this other agent who has a little bit more time has time for a
big case.
And then finally, we have a time overlap.
So this is a time series analysis model.
So we are using time series analysis to predict the times that the request of
the ticket is
going to be online.
And then we're either using time series analysis or a shifts feature to know
when the agent
is going to be online.
And then we're able to match up those two time series and find the kind of
amount of
overlap that the agent and request are going to have in the next 24 hours.
So that's kind of an important thing to note.
Like it's not for the next week, it's just for the next 24 hours.
And then on top of all of these scores, we have a recent assignment penalty.
So our recent assignment penalty is designed to make sure we are dispersing our
assignments.
So we don't want, you know, an agent that has a lot of experience, has a really
high
skill score.
We don't want to penalize them for that by assigning them a lot of cases.
An agent that's very efficient with like turning through their backlog.
We don't want to penalize them by just continuously assigning them cases as
soon as they shift
another case.
So we have a recent assignment penalty.
It acts with a time decay and a customizable weight so that you can kind of
decide for
how long and to what extent you want the agents to be kind of like iced out a
little
bit in the scores before they come back up to the regular score.
And also all of this is configurable.
So if, you know, the most important thing for your assignments is that you just
need kind
of a warm body to send an email, then we would increase the weight of time we
have a lab
and bandwidth and we would just get anyone who is available onto that ticket so
they
can start working on it.
If instead, sorry, if instead your priority is really for like you want the
best agent
to be taking on this case, then we would increase the weight of the skill score
and maybe that
agent isn't like online right now or isn't available but when they come online
they can
pick up the case and that's going to be preferable for the customer.
And then last but not least, we have our generation, generative and answer
engine.
So we've touched a little bit this morning on summarization.
So right now we have summarization at the case, account and knowledge source
level.
And I believe Chris will be talking later about summarization at like a case
cohort level.
And all of this summarization is right now powered by our LLM service.
I did realize putting this together that I said everyone conflates Gennai with
LLM and
we actually have done it here.
We are guilty of that too.
But yeah, so our LLM service is actually really a Gennai service.
The kind of purpose of this service is to take a lot of the heavy lifting away
from the
UI or the products that want to use Gennai.
So this LLM service can take multiple different inputs and it knows how to
format and structure
that input for whichever different service it's using.
So I think this was also covered this morning but these three summarizations
that we have
right now, they use three different endpoints.
So we're using anthropic for one, Mistro for another and OpenAI model for
another one.
So we don't want for that to be the case that would involve at the product
level to have
so many different formats of data and to meet all the different requirements of
all of these
different services.
But we can abstract that away into the LLM service and that means that
basically the product
or the UI can just send its payload and the LLM service decides what that needs
to look
like, which model that's going to, which prompts that's going to use.
So it's powered by a state-of-the-art prompt management system that we built in
-house.
So the really nice thing about this prompt management system is again
customizable.
So we're building a library of prompts.
We're always adding prompts as we have new use cases come through and through
again the
click of a button we can update the prompt for a specific summarization use
case for
a specific customer without being a lot of downtime.
There's obviously it takes time in the prompt engineering phase to develop and
research
that prompt.
But once we have the prompt, we don't need to wait for a release cycle.
It can just be almost instantaneously used within the product.
Okay.
So hopefully I've given you a good kind of demonstration that we have ML and AI
throughout
our products and nearly everything that you're touching and interacting with
when using
support logic.
Good job, Alex.
[applause] [end of transcript]
Good job, Alex.
[applause] [end of transcript]
(applause)
(applause)