Travelers deploys AI-powered claims countrywide with OpenAI

Well, hi everybody. I am thrilled to be joined today by Eric Rowan, who's the senior vice president and chief information officer at Travelers Insurance. Eric, great to see you. >> Great to see you, Denise. >> So, let's start out at the highest level. So, maybe just tell us a little bit about your role at Travelers Insurance, what you look after, and then how AI plays a role in that. >> Sure. So, I've been at Travelers 28 years. Uh, yeah, a long time. Uh, last 15 years within the claim organization. uh in the last 7 years uh also within our technology organization. So I have overall responsibility for technology analytics and business delivery uh within claim uh which includes all things related to AI which is why I'm here today. So Eric, Travelers has been moving really quickly to deploy AI at the insurance company and very specifically in first notice of loss, which what I was going to ask you to do is just explain for those who maybe don't know what is first notice of loss and it and it is a critical workflow. >> Yeah. So first notice loss really sets the tone for the entire claim process. So you think about it from a customer perspective. Uh they may have just had been in an accident, maybe their car was damaged in a storm. So they may be stressed, they may have a lot of questions um and they're calling um really to find out what they should do next, whether they should file the claim and then how to go about doing it. So it's really important that we get that right right at the beginning. >> Uh secondly, for us from a traveler's perspective, operationally, it's so critical. Um, if we get that information accurate and timely right up front, it really sets up the claim process to go much more smoothly and to kick off a lot of different uh services. Me, whether it's triaging, whether it's assigning it to a claim professional, maybe it it's setting up self-service capabilities to schedule a body shop or anything like that. The reason we started with first of loss was it's high volume. Um, we get about 1.5 million claims a year. Um, and it crosses all of our lines of business. So, we knew it was something that we'd be able to scale to other lines even though we're starting with autophysical damage uh right now. >> Um, uh, it's also it's structured in the sense of what we need to start the claim, but it also varies a lot in terms of the questions and um, uh, explanations that the customer may need when they're deciding whether they want to file the claim. So it's really a place where non-deterministic agents can really add a lot of value. Right. >> Right. They can adapt to different questions and responses. It can provide clarity. It can really help navigate them through that claim filing process. Yeah. >> Um in a variety of ways, which is really critical. >> And I'm just curious, what gave you the confidence that you could bring AI into that process? Well, so we we started very early in our in our uh journey on this with OpenAI uh even before some of the models were GA. Um and so these models and the real-time APIs are really powerful and we knew that um even in our early evaluations and benchmarking and testing, it was going to give us a very different chatbot like experience than what everybody is used to experiencing over the past few years. And so, uh, that gave us a lot of, uh, a lot of excitement and a lot of confidence. Confidence that we could kind of pull it off. >> Yeah. Can you just explain a little bit about what what processes exactly were covered in this agent and then what the customer experience feels like? >> Sure. So, maybe to start, I'll just describe a little bit of what it looked like before we put this capability in place. So, um, customers had a lot of different options to file a claim with us. Uh, they could use our digital channels, whether that's on our website or our mobile app. Uh, and a lot of customers did that. Uh, but we also had a lot of customers that were calling in to file a claim in one of our our contact centers. Uh, and depending on um u the type of event, if we have a hurricane, we have a lot of calls that come in a very short period of time. And so that can be hard for us to staff to that those infrequent peak demands that can sometimes lead to longer hold times to get hold of somebody to actually help with your claim process. right when you need it most. >> Exactly. So, we really wanted to tackle that last opportunity there so that now when a customer calls in uh to file a claim, they're giving given the option to actually interact with our AI claim assistant to walk through that claim process um and make help them make that determination as to whether they want to file. >> Give them a choice. >> Exactly. And so uh when you opt in uh the way it works is we have multiple agents that are working seamlessly totally undetectable to the customer. Uh walking them through the process. Um the the system is listening. It's understanding intent. It's providing clarity. Uh it's providing them explanations to the questions that may have. Uh and eventually it's actually setting up the claim in our legacy system and kicking off a lot of other activities uh such as scheduling a body shop or uh scheduling a rental car um or things like that. Uh one of the things that is really differentiating about um the this AI claim assistant is our loss consultation agent. So a lot of times people call in, >> they're not sure whether they even should file a claim. They don't know what coverage applies. They may not know what their deductible is. They may not be sure uh what the impact may be on their pricing if they file a claim. Um they may not even be sure if they're at fault whether they should file with travelers or maybe go through the other uh insurance company. And that loss consultation agent is able to handle all of those different situations and many more. Uh and again at any time uh we give the opportunity for the customer to actually speak to one of our contact center specialists if they want to. >> One thing that stands out to me I think when you think about all the success and your ability to have the confidence to deploy in such critical processes like first notice of loss >> is that you sort of embrace the philosophy that we see with some of the most successful companies which is AI is not just a technology or an application layer. it's an operating layer and you've really leaned into that and the way that you operationalize AI, the governance that you put around it. Can you share a bit about how you've approached that and your philosophy in doing that? >> Sure. That that was huge. So, uh, operating from an operating model perspective, very early on, uh, we felt we needed to bring everybody together that was going to be working with the OpenAI team. So the data engineers, the software engineers, the data scientists, our legal folks, our architects, our subject matter experts, we thought it was absolutely critical that they were involved very early on uh because how we were going to go about building this and testing it and deploying it was going to be very different. If you think about our traditional software application development, 80% of the work was mainly on the tech side, 20% was on the business. You kind of get requirements and then you ask >> you had a finite beginning and roughly an end with the go live and then kind of just start >> exactly and then you would have some UAT testing on the back end and you to deploy it. This was completely different. And so we knew that ratio was going to be really 50/50 in terms of how much resource we would need from our business folks part of the evals building the LLM judges iterating through with the team. >> So that was absolutely critical. So getting that operating model right right up front was um was number one. Two, we did a lot of change management. So going back to my comment earlier about everybody's had a chatbot experience that was probably not ideal. Uh we brought everybody from our senior leadership all the way down to the contact center into into the room to really understand and look under the hood as to how these things were actually working. Yeah. How the agents worked, how we would do observability, uh and allow them to actually experience what this would look like. and that went a long way as well of getting them comfortable with going in this direction. Um, from a governance perspective, um, we've been doing AI and machine learning within claim for the last 15 years. So, we have a a really good responsible AI framework that we've followed. And so, that along with our tech governance process was already in place to make sure we did this responsibly. But we also wanted to make sure it was aligned to what we call our claim three laws. uh anything we build and deploy, we want to make sure one, we always pay what we owe on every single claim. Two, we want to provide a great experience to that customer or agent as long as it doesn't violate the first law. And three, we want to do it efficiently and effectively um internally as long as it doesn't violate those other two laws. And so those three things, the laws, the AI responsible framework, and the tech governance were kind of the three legs of the stool that really allowed us to do and innovate quickly but also responsibly. >> And the thing that we often see, and I it sounds like you had the same experience. It's an iterative process where you have to test and retest and help the agent really perform. You give it context, it learns more. You give it feedback, it learns. And I think what we've seen with the most successful deployments is when you bring all of the cross functional stakeholders into the into the process along the way, you get great feedback, but also everybody understands what to expect as opposed to in traditional software. You have the kickoff, you do the design, you do the conference room pilot, you do the UAT, you go live and and tada. It's just not that way anymore. No, it it again that transparency of bringing everybody along. Everybody got really familiar with the agents. Everybody got really familiar with the prompts. They were in there. They were very familiar with the evals and it allowed us to really iterate through in that test and learn process. We're putting changes in place daily where in the past we may have it may have gotten the business folks in a room maybe once a week to kind of check. Eric, you mentioned this term LLM judges and can you tell us a little bit about how that how that concept applied in going from deploying in eight states to truly doing it nationwide? >> Yeah. Um, and we did it in a pretty short period of time. And so we went from piloting in the eight states to countrywide within two months, which is pretty incredible. >> So we had this idea called mission control. So mission control was something that we wanted to have in place um for all of our testing and then for pilot and then obviously for production that streamed all of our data in near real time. So 15minute increments we were able to see business outcomes, system performance, model performance. Wow. >> Um customer experience and intervention monitoring. And so all of those things were in place and then we had a series of eval. So one eval that we did was we actually created a synthetic caller with AI >> that could call into the IVR itself and run through thousands of different scenarios of of claim calls. Um, and then we had LLM judges on the back end that were assessing tone, accuracy, was it getting the information right, all of those things just to help us iterate through the testing very quickly. And then again, part of the mission control, we had LLM judges that were really there for fail safes. Yeah. So once we went to pilot, we wanted to make sure >> were any of the models or the agents hallucinating? Mh. >> Were they creating um uh inaccurate information? Were they making promisory statements they shouldn't have? >> And we actually created some of those LLM judges that if any of those things were triggered, the team would actually be alerted. Yeah. >> And we could actually turn the agent off within 10 minutes if we ever needed to do so. And so having that observability and that mission control and all of that data at our fingertips really allowed us to continue to move with confidence with responsibility. >> I mean I think that's such a great learning for anyone listening is when you think about the changes of moving from just having AI as a technology layer to AI as an operating layer and then what that actually means in practice. I think that's a great tangible example for people to think about how they do this because it really is different. So I think the stat now is you have 80 n 80 to 90% of customers completing their claim through the assistant now. So tell us a little bit about how you got to that level. That's remarkable how you got to that level of adoption and what people can learn from that process. >> Yeah. Um so so that adoption was remarkably right from the pilot. Um, so we didn't have to ramp too much off of that. And I think again it goes back to >> the amount of eval going into that. Again, having that that that AI create that synthetic caller to be able to go through that to tell us what maybe the backlog should be in terms of making adjustments. >> Um, that really I think led to a really good experience right out of the gate there. almost obsessing over the user experience is what it sounded like on all angles. Human oversight, the LLM judge. >> Yeah. And we're having really great um feedback from customers that can't believe how good the experience is. We actually have calls that people call in and say, "I've never experienced something like that before." Now, we still have 35% of folks that when they're given the option >> to use the AI claim assistant, still want to talk to a person. Um, and I think that is really just a behavioral piece that they haven't experienced with something new. And I think as more and more companies start to roll out and you you're starting to see these uh in maybe even in your personal life, >> um, >> as you start to experience a better >> Yes. >> aentic experience with voice, people will be a little bit more open to trying it. >> So Eric, you you defined this process. It was crossf functional. It was iterative. You have mission control. you have the LLM judges, but we also worked together in partnership and you worked very closely with our team and so I two questions actually. >> Um, how do you feel the partnership between OpenAI and travelers ultimately worked well to deliver this outcome and what more can we be doing to help our customers as we work with so many large enterprises in the all around the world and helping them to really meet this moment? Yeah. Again, when we when we chose OpenAI, it was through a pretty pretty extensive benchmarking and testing that we we did there and and OpenAI did a great job there. But we were really looking for a partner that would be in the trenches with us with the team. Um that would roll up their sleeves and that as we were doing the evals, as we were building the judges, as we were actually in there building and testing it, we were looking for a partner that was really hungry for the feedback. And the OpenAI researchers, they loved that feedback because they were able to take a lot of the things that we were seeing >> and roll it right into the next version of the model, which was super exciting and it allowed, I think, both of us to go faster. um when it was that real concrete feedback. And so I think to your second question around um where can people lean in, I think dedicating the time, I think having dedicated folks that um can really pick a problem and work with their open AI partners uh to solve that problem and and and be comfortable with a test and learn approach to that. >> Exactly. We talked about how different >> and lean on those evals and lean on those LLM judges and lean on that observability to give you the confidence to go faster. Um because you're never going to have perfect information and you know you don't want these things to kind of be hanging out there for too long before you get them into production. >> The other part of the AI conversation, we're talking about all of the incredible work that you've done and changing the customer experience and the employee experience. there's often uh discussion about the concern around AI taking jobs away, but you actually see it very differently. Can you share your philosophy and what you've experienced there? >> Yeah. Um again, similar to how much time we put into the operating model um and the evals, we put a lot of time into the change management. Um not just sort of within the team, but also down to our contact center employees. And I give a lot of credit to our contact center leadership >> for really leading their organization through this change. Um they were very thoughtful in taking the time to make sure the local leadership understood what this change was going to mean. But >> we also were taking a very thoughtful approach to how we could upskill and reskill and redeploy talent. Yes. Um into other parts of the claim organization as well. Um, we think AI will be um, part of what all of our functions are at Travelers and so we're all leaning into that upskilling and reskilling. >> Eric, so just to wrap us up, uh, you've accomplished so much. It's an incredible story. It's so inspiring. I'm sure everybody listening is going to feel like they gained some sort of knowledge that they can apply and be inspired by what you've done. So, where do you go from here? Where do you see the potential of AI within the industry itself and at Travelers, of course. >> Yeah. So, we're kind of continuing on this journey within claim with what we did with the AI claim assistant and we're rolling it out to other lines of business as well. Uh, so we can handle all different claims within our within our first notice of loss in this way. Um, but we're also leaning in across the entire claim life cycle. So, we have about 20 other initiatives underway right now um just within claim and even more across travelers. Wow. um around looking for where AI can really help us to um again further make sure we pay what we owe, provide that great experience and do it in an efficient and effective way. And I couldn't be more excited about I think the level of enthusiasm, the thought leadership, the technical leadership that we have uh within Travelers, within the teams uh they're all leaning into that and I think the the partnership with OpenAI has been great and we're looking forward to continuing it and expanding it. We're really grateful for the partnership. Thank you for the trust. We're very excited for all the work that you've done and we're just so grateful uh that you were willing to share your story with us. Thank you so much.