How well have you tested your CX AI models, chatbots and agents?

Episode topics

00:00 — Why is standard pre-launch chatbot testing insufficient?

01:14 — Welcome and episode introduction

02:08 — What are the four core harms of inadequate AI testing?

03:41 — How extensive was the TELUS Digital AI security benchmark?

04:07 — What are the five gaps in current corporate AI testing?

06:31 — Why does post-training alignment limit simulation realism?

08:58 — How does exposure management redefine enterprise CX AI safety?

11:45 — What questions should every CX leader ask about the safety of their customer-facing AI?

Transcript

[00:00:00]

[00:00:02] Robert Zirk: Before your AI chatbot started handling real customer interactions, your team ran through common customer questions, checked the edge cases and made sure the guardrails held. The outputs looked good and stakeholders signed off.

[00:00:17] But how many times did you test your AI? And against how many categories of attack?

[00:00:22] Researchers at TELUS Digital ran 34 AI models through more than 620,000 simulated attacks representative of real customer service deployments, and found attack success rates ranging from one to 90 percent.

[00:00:37] Bret Kinsella: Every model is breakable. There is no model that we've come across and no system that we've come across that we can't break.

[00:00:43] Robert Zirk: That's Bret Kinsella, senior vice president and general manager of Fuel iX at TELUS Digital, an award-winning portfolio of generative AI software.

[00:00:54] He joins me on today's episode of Questions for now as we ask: How well have you tested your CX AI models, chatbots and agents?

[00:01:10] Welcome to Questions for now, a podcast from TELUS Digital where we ask today's big questions in digital customer experience. I'm Robert Zirk.

[00:01:19]

[00:01:23] Robert Zirk: AI is now deployed across the customer experience in ways that would have seemed ambitious just a few years ago: routing calls, handling chat inquiries and resolving customer issues on the first interaction.

[00:01:35] Most CX leaders have done some version of testing before those systems went live, but the distance between "we tested it" and "it's safe" is wider than most organizations expect.

[00:01:48] Bret Kinsella is senior vice president and general manager of Fuel iX at TELUS Digital — the team behind Fortify, a continuous AI red teaming and automated penetration testing platform for customer-facing AI. Even non-technical users can be up and running with Fortify in minutes.

[00:02:06] Bret says the consequences of inadequate testing show up in four ways. First, there's people data harm.

[00:02:13] Bret Kinsella: PII extraction and things like that. Anything that might potentially get you in trouble with the law or harm people directly.

[00:02:20] Robert Zirk: What Bret described is when personally identifiable information — like names, account details and contact data — is accessed, exposed or stolen by someone who shouldn't have access to it in the first place.

[00:02:33] People data harm could also come in the form of an AI that dispenses financial or medical advice it has no business giving. These scenarios can lead to regulatory consequences.

[00:02:45] The second category of consequences is other types of data harm.

[00:02:49] Bret Kinsella: ...which can be compromise or exfiltration. Would it be bad for you if your IP leaked or something like that because someone was able to get into something that they shouldn't have been able to?

[00:02:57] Robert Zirk: The third consequence is reputational harm.

[00:03:00] Bret Kinsella: We've seen this many times where folks on the internet will take any type of AI learning solution and try to make it say uncomfortable things that reflect badly on the brand.

[00:03:09] Robert Zirk: And the last category of consequences is commercial harm.

[00:03:13] Bret Kinsella: Is there some sort of transaction that's happening that could negatively impact you or a customer or something like that? That financial impact might be directly to you, like they might be able to go in and get you to transfer money to their account.

[00:03:24] Robert Zirk: Now that AI systems are increasingly being connected to operational and financial systems — with capabilities like handling account lookups, processing transactions and updating shipping addresses — the commercial risk extends beyond your business to your customers directly.

[00:03:41] To understand just how exposed most organizations are, Bret's team put that question to the test. They ran 34 models through more than six hundred and twenty thousand adversarial attacks — findings published in the GenAI safety model benchmark report and first shared at Uncharted, TELUS Digital's AI safety and security summit.

[00:04:03] I asked Bret to explain where most organizations fall short.

[00:04:07] Bret Kinsella: People are not testing at enough scale. The probabilistic nature of these solutions means it might seem safe when you test it a few times, but, actually, you have to test it more than you think. In fact, our research shows that, generally, somewhere between 20 and 72 times you need to actually test different types of things.

[00:04:24] Robert Zirk: Scale is one of the five gaps Bret identified in how most organizations approach AI testing. It tends to catch teams off guard because it means testing can't be treated as a task that can be completed. Instead, testing needs to be thought of as an ongoing practice that has to repeat every time a system prompt changes, a guardrail is adjusted or a model is updated.

[00:04:48] Scope is the second gap, which is how many different types of vulnerabilities you're covering. In TELUS Digital's benchmark research, scope spans fifteen distinct vulnerability categories including privacy exploitation, financial fraud and reputational manipulation.

[00:05:05] Bret Kinsella: That's even before you get into things that are specific to your bot and your use case, right? These are all things that can lead to one of the four harms that you can get from AI.

[00:05:13] Robert Zirk: That's the four harm categories Bret outlined earlier: people data, other types of data, reputational and commercial.

[00:05:22] The third gap is variety, which is whether your simulated attacks are fresh and updated.

[00:05:27] Bret Kinsella: You can't just have, like, a golden list of 1,000, 5,000, 10,000 tests and just run them over again and say, "Good, I'm safe." You also have to provide novelty. So think about changes, new attacks, on a regular basis. You can have your baseline attacks if you want and that can be very helpful as part of your solution. But you also really need to be varying that.

[00:05:49] Robert Zirk: The fourth gap, repetition, is retesting every time something changes: a guardrail adjustment, a model update or a new system prompt can all alter the way your AI behaves.

[00:06:01] Bret Kinsella: What we found in our research is that, essentially, every time you change a system prompt, the model behavior changes. Every time you change the model or sometimes the model endpoint host changes something that you're not aware of, the output changes.

[00:06:12] Robert Zirk: The last gap Bret identified is simulation realism, which is how convincingly your attacks actually behave like a real adversary. To achieve simulation realism in the testing that formed the basis of the benchmark research, Bret's team had to train their own model so it could behave the way an attacker would.

[00:06:31] Bret Kinsella: When you use AI models to do the attacks, very often what we found is you cannot simulate a malicious attacker. And the key reason for that, is there's this concept called post-training. So you train a model, and the model can do lots of things. But then there's this post-training step and post-training is where the opinion of the model maker gets embedded into the model. And this is where post-training alignment is usually there's some safety and security built in. What we found is that the post-training alignment actually prevents it from actually doing realism. And so what we wound up having to do is take the next step where we trained our own model. We took some existing models that are open source, but we actually had to extract the embeddings from their post-training alignment, sort of un-train them, and then we retrained them so that they can do that simulation.

[00:07:19] Robert Zirk: Taken together, these five gaps point to the same conclusion: testing isn't a one-time event, and no single approach is enough.

[00:07:29] I asked Bret which key finding from the GenAI safety model benchmark report CX leaders should sit with.

[00:07:35] Bret Kinsella: The high-level takeaway is everybody should understand that every model is breakable. There is no model that we've come across and no system that we've come across that we can't break. And I think anybody who's got really skilled red team testers will be able to see that at a small scale, with the tens or hundreds of tests that a human can do. But we see this at large scale.

[00:07:55] Robert Zirk:

[00:07:55] Recall that Bret's team ran more than 620,000 tests across 34 models. And even that was based on single attacks, each repeated multiple times.

[00:08:06] Bret Kinsella: Every model was compromised. We had attack success rates, from a model standpoint, which ranged between 1% and 90%.

[00:08:14] Robert Zirk: A 1% attack success rate might not sound that high until you consider what it means relative to the number of customer interactions your AI might be handling in any given month. And, as Bret points out, that 1% figure is an aggregate. There are individual vulnerability categories where even the strongest performing models fail at higher rates.

[00:08:36] Bret Kinsella: Don't pick a model just because it's got better ranking. Pick a model that works for you for your use case and then say, "Okay, based on things like the benchmark that we have, we need to focus a little bit more in these areas: fraud and financial scams. This model is particularly susceptible to that," you know, or as we mentioned, like, "PII exfiltration. This model is particularly susceptible to that." And then you can focus on those areas.

[00:08:58] Robert Zirk: Knowing where you're vulnerable and directing mitigation effort accordingly is what Bret calls exposure management. It's a concept from the security industry, and it reframes AI safety not as a launch checklist, but as an ongoing discipline — one that accepts that no system is fully secure and asks, instead, what an acceptable level of risk actually looks like.

[00:09:21] Think about building a house. It can't just be walls. At minimum, you need a door. At the same time, you don't want anyone walking in whenever they like, so you install a deadbolt.

[00:09:35] But no lock is infallible. There's always a chance that a determined intruder with enough time, skill, and the right tools could break in.

[00:09:45] Even so, it's enough of a deterrent for your actual purpose: letting you and the people you trust in and keeping everyone else out.

[00:09:55] As Bret explains, exposure management is an approach where, rather than waiting for something to go wrong and reacting to it, like only buying the deadbolt after someone breaks in, you're continually assessing where your entry points and locks are and whether they're strong enough.

[00:10:13] Bret Kinsella: I really like to be able to see the outside and have natural light in my house. It's an acceptable risk for me. But, as you say, "Okay, I have mitigation techniques. I have a lock on the window. Maybe I have an alarm system that would warn me or scare away people, potentially. I have lights on the outside of the house that are motion detection, that might make people less willing to risk trying to do a break-in." And that's exactly what we're talking about in terms of exposure management. That's a great analogy.

[00:10:39] The question is: what's acceptable? So if you think about, let's say you're in a multi-story dwelling, so you got a two-story house. Maybe on the first floor, you do want bars on your windows. Maybe it's like PII exfiltration, right? Or financial scams or unauthorized financial advice that'll get you in trouble with regulators or something like that. Maybe you do want bars. Maybe you wanna have more refusals, less tolerance for people delving into those domains.

[00:11:04] But on the second floor, you've got some other things which you might say, "Okay, maybe there's some harm here. There's some reputational harm that might happen. There might be potentially some data exfil that I would be uncomfortable with." But maybe in order to put bars on those windows, it means it undermines the functionality of the solution that I'm trying to deploy. And so, in that case, I'm not gonna have bars on the second story. That's acceptable to me because then at least I know someone needs a ladder to get up there, they need to take more effort, and maybe there's less risk in terms of what the downstream impacts are for me and for my customers.

[00:11:34] Robert Zirk: I asked Bret to close with the one question every CX leader should be asking themselves right now about the AI their customers are interacting with, and he shared a question leaders should think of in two parts:

[00:11:45] Bret Kinsella: The first question is "Is this solution that we developed having the desired effect?" So is it performing the function that we want it to do? Is it improving the customer experience?

[00:11:57] And then the next thing I would say: "Is it not going to operate outside of the bounds of how we want it to operate?" So that's the way I think of it. Will it do what you want it to do? And will it not do what you don't want it to do? And that's really important.

[00:12:10] We think about this code of conduct that we usually work with Fortify: what is it supposed to do? What is it not supposed to do? And that's very important context, and a lot of people haven't actually taken that step to think about what the boundary conditions are. Do I want it to be able to do certain things? Do I only want it to be able to do other things? Do I not want it to be able to do certain things?

[00:12:30] And, again, how do I validate that second question: am I doing this at the appropriate scale, scope, variety, repetition and simulation realism that will really give us the confidence that we're putting something out there that's actually good for us, good for our business, good for our customers, but is not going to undermine our whole AI program because something happened and all of a sudden we have to shut it down, and then we're not getting those benefits anymore.

[00:12:58] Robert Zirk: If you want to know where your AI actually stands, Bret's team is currently offering complimentary safety analyses, essentially a free red team, for organizations that want to find out. Links to the Uncharted virtual summit, Fuel iX Fortify, and Bret's LinkedIn are in the show notes, along with the link to request your complimentary red team analysis.

[00:13:20] If today's conversation raised questions about how your own AI is being tested, Don't wait for something to go wrong. We encourage you to share this episode with your peers and have the conversations you need to have about CX AI security. Your customers are counting on it.

[00:13:42] Thank you so much to Bret Kinsella for joining me and sharing his insights today. And thank you for listening to Questions for now, a TELUS Digital podcast.

[00:13:51] Be sure to follow Questions for now on your podcast player of choice to get the latest episodes as soon as they're released.

[00:13:58] I'm Robert Zirk, and until next time, that's all... for now.

[00:14:02]