How Auto QA Elevates your Post-Sales CX
Quality Assurance has long been the Achilles' heel of support operations, but it doesn’t have to be. This session will confront your assumptions about QA, demonstrating how automation can elevate your entire support experience. You’ll see why relying on manual checks is no longer enough in today’s fast-paced environment.
You Might Also Like
All right, I'm Joe Andrews again and joined by my colleague, Tali.
We're going to talk about AutoQA and by definition we mean quality assurance
and quality monitoring,
just a show of hands.
Who here is doing AutoQA today or any QA, manual QA?
Anyway, okay.
So, what we want to do today is talk about the opportunity for AutoQA, the use
cases,
what it means for the business, the impact, sort of cost considerations, trade-
offs.
I'm going to do just a little bit of TIP around that, but we want to spend most
of the time
with Tali actually showing you the product, support logic, elevate our own Auto
QA solution.
So with that, I'd start with the thinking around that a lot of companies are
faced with challenges
around agent retention.
It's harder than ever today, you're all aware of supporting organizations where
firefighting
leads to stress, things like escalations, things like upset customers, just are
part
of support engineers and support agents every day.
And so the opportunity to coach and to help enable your agents and your support
teams
is tremendous.
But the reality today is that coaching takes a lot of time.
It often gets left to spend less time than we want.
And the majority of cases happen when it's not a good time.
The feedback is inabsorbed.
Reviews take a lot of time.
When you're in a manual situation, you're only looking at a fraction of the
data.
We'll talk about that in a second.
And then there's no real consistent coaching practice.
So typically quality assurance or quality monitoring organizations are set up
teams to
accomplish a couple things.
One is around coaching and another is around compliance, like ensuring that
certain process
steps are done with your customers.
So just curious, like for the ones who are looking at AutoQA solutions, are you
most
interested in the coaching side of things?
Or is it more compliance or something else?
Coaching.
Yep.
So it's all focused on coaching.
What about over there?
There's interest or looking at AutoQA?
A little bit.
A little bit of both.
Okay.
These are -- so agent retention is harder.
Coaching can have a tremendous impact on agent retention.
But this is the reality, right, for those of you who are using manual QA
solutions.
Typically it's a sampling process.
It's similar to surveying customers.
You're picking one to two percent, maybe, of cases.
And typically you're -- it's either random or it's manual selection, which is
even worse
because there's more bias introduced.
And often it's delayed.
It's after the fact.
Maybe it's once a week.
Maybe it's at the end of a month.
And it's subjective.
So it basically barely scratches the surface of where organizations could be
and should
be in terms of their agent coaching practice.
So this is what support logic set up -- set out to solve with our AutoQA
solution using
AI.
We monitor -- we ingest 100 percent of all of the cases, all of the
conversations, the
interactions that happen with cases.
It's close to real time.
So it's timely, it's contextual, and it's consistent and objective because the
machine
learning is actually powering the scoring.
So that you can adjust the rubric, the algorithm, right, the factors that are
important to coach
on, but the bottom line is everyone is viewed through an equal lens.
And this will help to transition your QA organization from laborious scoring to
more
of a value-add coaching organization.
And that's where I think everyone wants to be.
I'm seeing some head nods here.
And these are the primary use cases that we're going to talk about.
Manual and automated QA -- there are uses for manual QA, but at a baseline you
want to
start with automated.
You want to ingest everything and use that as your starting point.
We want to talk about coaching feedback loops and how do you build this into
your business
process and ultimately about how do you do it more efficiently?
That's the big challenge because large organizations that have a QA process,
let's say for a thousand
agents, you typically have 20, 25 QA people on a team that are doing that.
And then how do you ultimately measure things like customer effort score, how
do you ensure
compliance, zero tolerance policies around behaviors that are not meeting the
mark?
So these are all the things that we're going to talk about in terms of the
solution just
really quickly.
Oh, sorry.
>> You might have asked questions.
>> Yeah, absolutely.
>> On the use case of the career, if you go back to life, it makes a ton of
sense to
do this on send and analysis of something talking and then in our A.M.
around the common cause.
>> Do you want to address that?
>> Yeah.
>> Yeah, I think that, oh my, sorry.
Yes, there is some solutions around it as well.
We have some configurations that you can address particular QS cases.
So if you need particular keywords, you will see when I'm in demo for those
assignment
Qs, when we decide what cases we're going to QA, both for automation and the
manual,
you can restrict to particular keywords, for example, or particular text
phrases that you
more care about them.
So you definitely have a control over it.
And the good thing is that you will see the Qs that you kind of querying, it's
not a singular
one.
So you can create as many as you want.
So you can create one is more technical, one is more coverage, like white
coverage,
et cetera.
You will see my demo and then you can ask more technical questions.
>> It was very strong.
>> Yeah, so the question similar is, is the coaching feedback around the
communication
skills or the technical feedback?
The answer is both and we'll get into that.
>> Yeah, I'm sorry.
It's very similar.
>> So, as we said, there is a matter of fact, if you are in a situation where
you are in a
situation, you can try what you need.
You obviously probably need to make it updated based on your timeline, but you
do have to
make sure that it's flexible enough for you.
>> These are great questions and we're going to get into the product so you can
actually
see it.
I just want to touch on a couple more points.
We work across all major systems of record on the left.
Multilingual and multimodal, we're ingesting different data streams from
different channels.
And you can see general outputs, escalations, behaviors, brand trust safety,
performance,
performance is where you get into the technical aspects.
We will show more details of that.
Really quickly wrap up a couple slides and then we'll get into the product.
You can see comparing traditional or manual QA against auto, typically tickets
reviewed,
low single digit percentage versus auto QA 100.
You can see what that means for the human resources, the team that's doing it,
typically
one analyst for 25 agents if you're doing manual, where it's one for 100.
If you're doing auto, you move from coaching feedback that's sporadic to near
real time,
scalability, rigid and expensive versus flexible, rigid because it's largely
people resources.
I'm not going to go into this in depth.
We're happy to do an ROI analysis.
We actually have a pretty in depth calculator on this, but again, assuming for
about a thousand
agents, you have an annual cost from right sizing the QA team.
There is an opportunity to shrink the number of full-time QA analysts and then
you have
improvements in operating expenses, OPEX, that come from lower handle time,
fewer repeat
calls, etc., improved quality.
We can run through this calculation for your situation if you're interested in
that.
Let's quickly get over to the demo because I think showing is more powerful
than talking
about it.
I can try to hold it down.
Can you hear me?
Yeah.
I'm usually a loud person, so I don't want to be too loud.
Closer?
Good.
All right.
I'm going to show you some use cases that we do with our auto and manual QA.
I'm going to demo from three percent a kind of point of view.
The first one is the regular support manager or support leader that all what he
needs to
know or wants to know is in any second of his life what is the quality of his
support.
So it's the question that is always hanging there but we not always have answer
for it.
So now with the tool we can easily come from the core product, click on the
elevate and
launch the elevate solution.
And once we're coming to the elevate you immediately can focus on the score
cards and
pick auto QA.
So auto QA is where we do auto QA on top of all the cases all the time.
This is something that is unbelievable.
It's a full coverage.
No need any manual interaction.
So theoretically any case that comes in with any interactions within the case I
will go
to the case level just to showcase you the coverage.
We are doing the full QA based on the score card that you decide to score.
So you have a flexibility of what behaviors we scan, what skills we want to
score and
based on all this setup we running on every single case and doing the
automation.
So you as a manager come to this tool and immediately can see that in the last
30 days
my QA score of my organization is whatever number it is.
The good news is also that you have another information on the screen such as
trends.
So if you have enough data and QA is running you can see over time you can do
it weekly
or monthly and you can see several trends running and see how the QA is
evolving.
You can also create different groups by the interest.
I have a trick examples here that I can drill down into them and actually
perform the analysis
of this QA, Auto-QA in this example on those particular groups.
Groups can be groups of agents, it can be areas, it can be whatever you want to
structure
into the group.
Lower on the screen just to get you familiar with the flow of the screen you
have those
fields and behaviors.
As I said every scorecard will have a variation of those.
So in the skills you will have for example here I have the opening, the
efficiency, the
empathy etc.
And the behaviors will take us one level down.
When you can see for example for the opening we will have metrics like greet
ings and assistants
and introduction.
So if the agent will not do one of them we will grade him the ways that it
defined for
example negative.
Below you can see exactly the same breakdown for those behaviors just on the
group level.
So let's just drill down to show you how impactful it is.
If you go down to the team this is the group that I picked you will see the
exact statistics
on the team.
Again the same trends, the breakdown of the agents within the team and the
metrics of
the skills and behaviors.
And I can go all the way down to the individual agent.
So again as a manager if I want to start from the top and just get an overall
understanding
what's going on in my organization I can do it from the top, I can drill down
to the
group, I can go to the individual agent.
From the individual agent obviously you can see all the statistics but you can
also drill
down into individual case if you identify some trend that is kind of for
example negative.
Let's say I picked up now a critical one of the agents here in the system and I
can see
that on the greetings she has all the time red score.
Everything is not going well there.
I can write from here to say what is the negative and it will bring me to this
screen
where it will filter for me the agent, the 30 days of experience and the
behavior I
was on and show me the exact cases where the behavior was caught.
So if I open one of the cases again I'm going through the route that kind of
investigation
when I want to build up and kind of bring some value when I have a conversation
with
the agent it will lend me right into the case and will show me all my scorecard
on the right
side.
So as we said the scorecard is flexible this is what we decide for the auto
assignment
and this is what you see.
On the opening you see there are trick criteria and one of them is negative on
the introduction.
If I will look at the behavior here I can see hello person name is our
reduction so we
redact data we have a service of reduction if we want.
Thank you for reaching out to me and etc.
So we already see that the agent did not introduce himself or herself.
This is something that we point out as a negative and this is what the system
thought and this
is why the score is negative for the introduction.
The other parameters here are positive and some of them are not scored so I can
as an
auditor if I'm the auditor start my manual QA right on top of this automation
and edit
it further in order to complete fully the QA on the case.
I will show it in a second.
And eventually once the full QA is complete the email sent to the agent to
notify him that
the manual QA was complete on his case.
The agent will get an email and will need to acknowledge the email.
We can define the time frame of the acknowledgment it doesn't need to be right
away we can decide
like in the next three or five days.
And once he acknowledged the QA that done on his case he can review obviously
the results
and raise something that we call dispute.
Dispute is if there is anything in the score that he was given is not as he was
expecting
and he disagree he can resume and explain what is the item that he is disagree
with and it
will go back to the editor and I will show you the flow of the dispute.
Just to point out that this case has all kind of variation of the data.
For example we have also voice file on the case.
So the QA done not just on the comments but also on the chats that link to the
comment
to the case as well as the voice.
On the voice you can see it is highlighted in different colors those are the
signals.
Those are the elements that we are listening to and tagging as a QA and those
are also
converted into the scorecard.
Plus you can see I just went through the case and I saw some foreign language
on the case
here.
So I was curious I checked on the translation.
This case is already translated and the QA is done on the translated version.
We can see the original version if we flip it to the original.
So the text was given as an original we did the translation and we applied the
auto QA
on the translation.
So just to show you the powerful that you will not just take the comments as
they are.
If needed translation we translate them.
If there is a voice file or chat file we will break it down and work on it as
well.
So now just to show you as an auditor what else can I do.
As you already saw the auto QA is like time save right?
Everything is done for you.
You just see the stuff and you can drill down to the points that interest for
you.
But there is also always a question if I want to manual QA stuff and I want I
have auditors
that doing what manual QA.
What should I do?
Like what cases I need to focus on.
This is what we have on our assignment board.
So theoretically we can create an assignment which is the list of cases for
auditor, assignment
for the auditor and let him to work on the cases in the assignment.
Now in the create assignments area this is when we decide what is the
assignment list
going to be.
It can be a single list like I literally cherry pick the cases or I defined the
filter
for the cases and I send it as an assignment auditor do the assignment and the
list is over.
I also can create a list that is re-occuring and this list will kind of have a
filter like
a query behind it and will feel itself with the cases based on the filtration
that we
provide and to radically let the auditor revisit this list over and over and
the cases will
be there.
So just get some examples.
Somebody asked me here how do we control what we QA.
So here you have all the flexibility to do some filtration.
Ffiltration can be based for example on the signals.
We can find out only the cases that has individual signals or individual LTE
for example predictions.
We can filter out cases that have customer efforts score high or low.
Another example.
We have different keywords.
We asked what about technical terms.
So we can put here some technical terms and then we will target those cases
that have
those technical terms into this assignment queue, assign it to the appropriate
auditor
and he will take care of those cases for the manual QA.
Obviously you have trivial stuff like how many ticket going to be in the queue
or what
is the due date of the queue.
Those are the basic default.
So all those assignment lines, assignment queues will be in this list and you
can see
him, see those assignments.
You can see how many cases assigned to each one of them and if you can start
playing it
as an auditor you come to the page which will tell you now you have five cases
in this assignment
line that you need to manual co-accurate.
You just go through the manual QA, you start your QA, you pick up the responder
and you
just go in with your scorecard and obviously every selection can have a comment
, etc.
One more thing to mention on the assignment.
Since most of the organizations have more than one editor and a lot of time
editor have
more experience or editor is new, they are not always aligned with how they do
the QA.
So we want to make them trained and aligned and kind of consistent across the
board.
So we have a tool that we call calibrations.
This is something that we can create.
It's very similar to assignment when we create a queue with several cases in
and we assign
it to several auditors.
It doesn't really do QA as an agency but the report is go to the manager.
So I can see each individual editor how he performed the manual QA.
I get the report and then we can review it as a group and kind of align us and
bring
our knowledge base to the similar baseline.
So dispute as we said once the manual QA done the case is sent to the agent,
agent get
the case, he might be happy or not, he will acknowledge.
And then he might raise the dispute.
The disputes will be visible for me here as an auditor.
I will see my disputes and then I can go in and see what was the dispute about.
For example, let's see.
So this is the dispute I got.
I can see what was the dispute, who raised the dispute, what was the comment
about it
and then I can resolve the dispute.
Again, when I resolve the dispute it will open the editing mode and I can say,
yep,
you write, I missed and correct the score or you can say you are not right and
then explain
again how.
And once you resolve the dispute it will be cleaned from your page and agent
will get
a notification about it.
Scorecards just to touch it very briefly, as I said, we can create several
scorecards.
The manual one is the base one which we want to have perfectly set up because
we want a
great auto QA.
But you can create also a lot of manual variations here.
The creation is through the button.
So you can see here all the information, it will have the skill, it will have a
behavior,
you can design which one you tag, you can decide the weighting of them, you
have a flexibility
and obviously the grade itself if it's negative, positive or any other
combination.
And one more thing to say, we love to welcome everybody, all the users into the
platform.
So it's not only designed for the managers, it's designed also for agents.
If agents will come here, they will see the disputes that they raised.
They also can visit the tickets of interest.
This is kind of a dashboard.
We have different widgets with different stuff.
Joe mentioned previously, for example, ZTP tickets.
If we have tickets that are cases that have some not polite language, I don't
want to
be rude.
Or cases that we have some confident information shared both from the customer
or from the
agent, we want to be very cherry-picking on those and make sure that we're not
losing
them.
So those will appear on this gadget.
We also have a feature in both Elevate site and the core product when we can
open the case,
even if it's not closed, like through the lifecycle of the case and flag it to
the manual QA because
we see some potential there and we want to not forget about this.
This will also appear in one of those gadgets and you will see all manually
tagged cases
here so you don't need to kind of figure out how I find the case or whatever.
I think I covered most of the stuff.
There is more functionality here, like in the feedback area, so you can record
the video
through Loom explaining some information about the case, about the dispute,
about whatever
you see in the application and you can share it with your team.
Very good for training for coaching as well as getting feedback.
So more is here.
Yes I can get back to the open QA which is the just summarization of all.
If I go back to the dashboard, I started with the Auto QA just to focus on and
then explain
a little bit about the manual.
But theoretically on the dashboard you can see both of them.
So the summarization of both gives you a perspective both on the general score
as well as the
customer effort score.
So just to make sure that we're not just doing QA on the cases in order to
score our
agents.
We're also analyzing all the account, all the customer kind of data and giving
a C set,
I don't want to call it C set because it's not a survey.
It's just something we're doing automatically across all the cases.
You can hear the power of it, right?
We don't need to send a survey anyway.
We don't need to wait and beg the customer to reply the survey.
We just do it automatically based on the data and believe me, it's good quality
without
any additional effort.
So you see it right away here on the dashboard.
It covers all the kind of functionality over and manual automation side by side
split here
on the dashboard.
You can see them set beside.
See, cover, touch.
Yeah