Ronnie 00:00 

Hey everybody, it's Ronnie. It's Thursday, September 17th, and I'm here in our New York studio with a very special guest, our internet analyst Ross Sandler, who's joining us from our podcast studio in Silicon Valley. Ross, welcome back to the Barclays Brief.  

Ross 00:15

Ronnie, very excited to be here. Thank you. And I'm glad to be back. This is, I think, our round two of the AI debate. 

Ronnie 00:23

Well, it's great to have you back on the podcast, especially with everything happening right now. And wow, has it been a busy and confusing few weeks in the world of AI. Look, we're all scrambling a bit to get our arms around what it all means for this critical technology and market driver. Can you help frame what “pacing the frontier” is and how we got to this point? 

Ross 00:42

Yeah, yeah. So pacing is just sort of a fancy word for adding a bunch of extra kind of safety and what the AI community calls alignment monitoring of what's going on in mostly the training process where they build these AI models, but also in the inference process, which is, you know, once a model is released to the public and we're all kind of using this stuff, it's basically just making sure that both of the training and, and the inference stage, that these AI models aren't doing things that they're not supposed to be doing. So pacing is essentially just a fancy word for slowing it down a little bit, to then add all this safety and monitoring on top of what they're already doing. 

Ronnie 01:25

So, let's spend a little bit of time on how we got to this point. I think everybody has read about the Open AI, Hugging Face incident. Can you just tell our listeners what they need to know about that and what happened, and maybe how it helped catalyze this recent movement? 

Ross 01:39

Yeah, and I think it's worth like kind of stepping back a little bit even before that. So, what's happening is you've had multiple kind of huge moments in AI over the last like call it four years. First, there was ChatGPT being released in late 2022, and then you've had kind of a series of like pretty big breakthroughs, like the reasoning model breakthrough. And more recently, these models and these products have gotten very good at code writing and particularly at cybersecurity. And so, part of what's going on is that we're just on this continuum of AI improving. And you're getting to a point sometime, probably earlier this year around the release of Anthropic’s Mythos Model, where we reached a new plateau, a new level of capability whereby these AI systems are far more capable in the field of code writing and cyber. And so, Hugging Face and what happened here was in the process of training a model which they haven't even yet released. Basically, the model was being tested on a certain benchmark, and it is supposed to be somewhat contained in this like secluded environment called a sandbox. And what happened was during these tests, which were designed to kind of test the limit of the model cyber capability, it escaped the sandbox, went out onto the internet, broke into Hugging Face’s infrastructure, which actually had the answer to the test that it was supposed to be completing. And in the process of doing that, it was like kind of self-replicating, and it created a bunch of agents that that helped with the hacking into Hugging Face. And so, the incident was surprising on like many vectors. It wasn't supposed to do any of this. 

Ronnie  03:24

Were these the swarming agents everyone's talking about?  

Ross 03:26

Yeah, there's a couple things going on here. The agents swarm like, you know, the beehive swarm. Yeah. So apparently there was like 700 agents that, were involved in breaching Hugging Face’s, infrastructure and finding sort of the answers to this test that it was supposed to be completing. And so, yeah, that's the agent swarm. It begs the question of the sandboxing that OpenAI set up - how tight was that? And clearly there was like some exploits that that happened here that the model figured out and broke containment. Hugging Face is another AI company which is in the process of being acquired by Nvidia. So, it's sort of like a no harm, no foul. Everybody like says they're sorry and, hey, we'll get it right next time, which is sort of what happened here. 

Ronnie 04:14

And so that's where the pacing and guardrails come in.  

Ross 04:16

Yeah. So, it isn't just the Hugging Face incident there's been about, I don't know, maybe a dozen or so of these of various levels of severity for both OpenAI and Anthropic that have happened. You know, somewhere like I said, it started in basically the spring or summer of this year where these models got much more powerful, much more capable, particularly around cyber. And so you kind of have to, like, prepare the world. You have to kind of like get everything a little bit more tightened up in both the training process and then also once you release these models, because they're far more capable than they were a year ago, you know, we're just in a new level here in terms of AI capability. And hence like the bar is being raised in terms of what these eyes can do and also what the world needs to do to prepare for this.  

Ronnie 05:07

No, I definitely feel that. And the quality of the products of the models that we're using now. Let's move it back to what this means for the AI labs at the frontier from a growth rate perspective, does this mean lower growth rates or slower growth rates off a highly accelerated pace? How do you see this all playing out as we pace the frontier for these AI labs? 

Ross 05:27

Yeah, I think the revenue growth shouldn't be impacted at all really in the near term because most of what's happening like there's the training process, which is like, you know, these models that are doing these, you know, these breaches were sort of things that have yet to be released. The ones that are out in the wild, which would be in OpenAI’s case, Astra, which is like their GPT6 and in Anthropic’s case, Fable Five, these were kind of 5.1. These are sort of the state-of-the-art models that are out there, and those are mostly like pretty locked down. And, you know, us as the end user are kind of going about our business and kind of using these products. They're not really causing any real problems. So, it's just in the training process for this next generation where you're starting to see some of these incidents. And so the revenue that the labs are generating today is mostly just a function of the diffusion that's happening with AI products kind of getting out there. Most of it is probably for models that are either Fable Five in Anthropic’s case or Astra in OpenAI’s case, or maybe even like the generation one step prior to that is currently what's being set up inside of these big companies. And so, the revenue is sort of lagging in terms of the revenues supported by models that are lagging, the ones that are causing problems. I think if you kind of play this out, what pacing could mean in the future is that the cost of training and inferencing next generation models goes up for all this extra safety monitoring that needs to be done, and then the labs will either have to absorb that cost, or they'll have to kind of pass it on to the end customer sometime next year in the form of like higher token prices. And so could actually mean revenue goes up once you get to these next generation of models. But yeah, revenue seems to be doing quite well. I think we crossed the 100 billion mark sometime in the first half of this year for AI lab ARR. We're going to end the year probably close to a little over 200 billion of ARR. So, the revenue seems to be up and to the right.  

Ronnie 07:28

In three years. 

Ross 07:29

That's anybody's guess. Yeah.  

Ronnie 07:31

So, who wins and who loses here just in general in your eyes or because of this new pacing dynamic?  

Ross 07:38

Yeah. So, I think this is an important point. If you look at like where the incidents are occurring, it's right now mostly just OpenAI and Anthropic. One could argue, okay, those guys are, you know, a few months ahead of the other Western labs and a few months even further ahead of the kind of open-weight community coming out of China. And so maybe because their training models that are, like, far more powerful and far more capable, they're the ones that are kind of the first ones that you would see running into some of these issues. They also have the most compute to do these, these big training runs. And so, you're seeing it where you should be seeing it. It is also interesting though, that Google who has DeepMind, Meta, who has like completely rebuilt their internal AI lab called MSL, like they're training models right now that are pretty much neck and neck with where OpenAI and Anthropic are, and yet they're not really seeing incidents. It's interesting to me that Google and Meta, and even to a lesser degree, SpaceX are training models that are pretty much close to what OpenAI and Anthropic are doing, and they might just be taking a few extra steps to have the alignment, the safety, the monitoring up and running, because you're not seeing those training runs kind of break containment and have all these incidents that we had with Hugging Face. And so it could be that if, you know, the two leading labs have to pace and maybe slow down a little bit and kind of implement a bunch of these new safety measures, that could mean that, like Google, Meta, SpaceX, even the Chinese could catch up for some short period of time.  

Ronnie 09:11

That's interesting. So, look a lot to digest in terms of events in the last few weeks. Does all of this make you more optimistic, less optimistic? We spent a lot of time on the power of AI for positive economic transformation. Does this change the pacing of that in your eyes in any way?  

Ross 09:27

 Well, I'm definitely in the AI pilled camp, which means I'm very positive at all times. But I think, like stepping back, I know the Hugging Face and all this pacing and safety and security, etc. are important topics that we need to kind of work through. But the other thing that's happening is as models get more powerful and as we bring more compute online, it's just to do quite a bit more like some of the big breakthroughs that we're hoping AI would, would kind of deliver are starting to happen. So, it kind of got lost in the shuffle, but OpenAI solved this like Millennium Prize math problem a couple weeks ago, Navier-Stokes. That if you look at what happened here, they put the swarm that we were talking about before, 10,000 concurrent agents working on this math problem. They put them on that for 88 hours, so about three and a half days. And the swarm was able to solve this historic, incredible math problem, like a big breakthrough in about three and a half days. And if you add up what that would mean in human years of kind of like 9 to 5 work by a mathematician, it's like 5 million or something human years of work being done in just three days. I think this is important because what's going to happen next is that as these models get more capable, as we as an industry bring on more compute, you're going to start pointing the AI systems at huge problems within science, math, medicine, biology, robotics, like some of these new categories that are cropping up. And you're going to start to just see breakthrough after breakthrough after breakthrough happening. And these are things that can have like a huge impact on creating a new industry, creating all sorts of GDP, etc.. And so just look at where we've come from like two years ago, we're talking to chatbots. Then we kind of go into this like code writing agent mode. Now we're about to step into like breakthroughs that actually start to change the world. So that's what I'm pretty excited about. 

Ronnie 11:25

It's very exciting. Why don't we end on some of those breakthroughs, or just some of the things that you'll be focused on in the months to come around AI to make sure that your highly AI pilled excitement remains intact? 

Ross 11:36

Yeah. I mean, it's hard to know where they're going to come from. I think if you look at what Demis from DeepMind is working on, he's got this whole group working on various different problems across medicine, biology, kind of protein mapping, etc. so you're going to get a bunch in that area. It's all the things that, like academia, has been working on for decades that you're going to start to see these AI systems kind of pointed at these problems. You're going to just start to see more and more over and over. 

Ronnie 12:05

Building new industries, compression of innovation cycles. A lot of exciting things could come from this. 

Ross 12:10

Yeah. And that's the reason why you want to be bullish and not lose sight of that. When we get into the Hugging Face and some of the cyber issues like, yeah, we need to engineer solutions that are safe and that everybody agrees upon and, you know, are set up the right way. But let's like get the AI to have more of these breakthroughs and then let's see where that takes us as a society.  

Ronnie 12:30

Every time I talk to you, I just walk away more AI bullish this time is no different. Ross, thank you so much for being on the podcast with us.  

Ross 12:36

Thank you.  

Ronnie 12:37

So, in conclusion, this pacing dynamic in AI has been the topic of conversation for market participants since the weekend. Ross helped me decipher the signal from the noise around this argument, and his view is that this was a natural part of the evolution cycle of this technology. If anything, it will just provide more safety without slowing growth rates for the industry and slowing the innovation cycle down, leaving him in an even more bullish place related to AI and breakthroughs to come in the future.  

One thing to note, we mentioned some private companies during this podcast that we do not cover out of Barclays Research. Thank you for joining and please remember to hit subscribe wherever you listen to your podcasts to be notified when new episodes of the Barclays Brief come out.