[Podcast］How AI startups can compete with the AI giants ー Interview with Jad Tarifi at Integral AI (Part 4)

Japan is lagging behind in AI, but that might not be the case for long.

Today we sit down with Jad Tarifi, current founder of Integral AI and previously, founder of Google’s first Generative AI team, and we talk about some of Japan’s potential advantages in AI, the most likely path to AGI, and how small AI startups can compete against the over-funded AI giants.It’s a great conversation, and I think you’ll enjoy it.

Welcome to Disrupting Japan, Straight Talk from Japan’s most innovative founders and VCs.

I’m Tim Romero and thanks for joining me.Japan is lagging behind in AI, but that was not always the case. And it won’t necessarily be the case in the future.

Today we sit down with Jad Tarifi, current founder of Integral AI, and previously founder of Google’s first generative AI team.

We talk about his decision to leave Google after over a decade of groundbreaking research to focus on what he sees as a better, faster path to AGI or artificial general intelligence. And then to super intelligence.

It’s a fascinating discussion that begins very practically and gets more and more philosophical as we go on.

We talk about the key role robotics has to play in reaching AGI, how to leverage the overlooked AI development talent here in Japan, how small startups can compete against today’s AI giants, and then how we can live with AI, how to keep our interest aligned.

And at the end, one important thing Elon Musk shows us about our relationship to AI. And I guarantee it’s not what you, and certainly not what Elon thinks it is.But you know, Jad tells that story much better than I can.

So, let’s get right to the interview.

Tim: Okay. So, let’s dig into alignment or ethical AI. So, if AI truly is just this optimization execution, problem solving machine, is alignment even a problem for AI or is a problem really just preventing people from giving AI bad instructions?

Jad: Excellent question. Because this is exactly what I believe. In order to solve the alignment problem, we have to solve an even harder problem, which is aligning humans.

Tim: We’ve been working on that for a few millennia.

Jad: Yes. And so when you look at the history of philosophy, there’s a lot of work on that. And largely the history of the 20th century, especially the first half of the 20th century with world wars, with communism, is history of tragedy of these highest ideals, aspirations. We want equality, we want a good life for all, turn dark, the road to hell is painted with good intentions. We are really entering a thorny ground here. We have to be very careful. But the challenge of aligning AI and aligning humanity forces us to confront these problems. So we need to find a shared vision, something that we can kind of agree on, regardless of our diversity. And what I believe is that it’s a new notion of freedom. And that new notion of freedom is not about the absence of constraints, it’s about agency. And the way you can think of agency is by looking at what an agent can do. An agent can perceive, it can plan and decide things can act. So let’s take that all the way to infinity. So infinite knowledge, the capacity to make good decisions, which means benevolence, infinite power. And then because you’re embodied, you need to be able to maintain yourself. So infinite vitality or safe maintenance. So, this is kind of a loop where each component is reinforcing each other component. And this loop at infinity is what I call freedom. And this seems to be something that we can all agree we should aspire to move towards. And you can argue for that purely from an evolutionary perspective. If you don’t move towards freedom, you die. In fact, you can restate the theory of evolution as not the theory of the survival of the fittest, but the tendency for freer systems to survive. So the more agency you have in the world, the more you can survive. Otherwise…

Tim: Let me push back on that because I don’t think so. I mean, if you look at like the major extinction events throughout history, it’s not the little phytoplankton that gets wiped out. It’s the higher level animals. It’s the complex dinosaurs. It’s the complex animals that have much more freedom and can move around. Those are the ones that tend to get wiped out.

Jad: That’s very true. You need to define freedom at multi-time scales. This is also a problem with the naive formulation of natural selection, right? Because these dinosaur are supposed to be more fit.

Tim: I’ve always been somewhat annoyed by that phrasing of survival of the fittest because it’s a tautology, right? It’s like the survival of the fittest, but how do we define the fittest? It’s well, whatever survives. So it’s really the survival of the survivors.

Jad: Exactly. This is the problem. So can we add something a bit beyond that? And the question is, what does it mean to survive? So the means to survive is to manage the entropy in your environment. Are you able to minimize free energy? So basically the world is messy, it’s throwing stuff at you. Are you able to organize it enough so it doesn’t destroy you? And it’s impossible to do that because you cannot predict the future. The best thing you can do is instead of minimizing free energy directly, which you can’t do, is you can create a model of the world. So it’s an approximation of what you think the world’s free energy is and minimize the difference between your model of the world and the world itself. So you’re trying to improve your self-model of the world, and you can either minimize the difference by improving your model, which is what we call perception, or making the world closer to your model, which is what we call action. But this assumes no intentionality. So the question is, what is the intentionality that is ideal for distributed intelligence? It comes down to you want not only for yourself to have this freedom, but you want the freedom for the entire ecosystem. If you are going to take the freedom and destroy the whole ecosystem, in the process of you acquiring that freedom, you’re kind of shooting yourself in the foot because you’re optimizing at short time skills as opposed to long timescales. So, optimizing that free energy at every timescale forces you to introduce the concept of benevolence in there.

Tim: So, it’s says ecosystem wide optimization. It’s not an optimization for an individual organism within that ecosystem.

Jad: The beauty of it is benevolence means you can optimize for yourself, but because you’re benevolent by definition, you’re optimizing for the whole.

Tim: I don’t think humans live up to that standard. It’d be very hard to develop useful AI that follows those rules, wouldn’t it?

Jad: I think so. I think humans try.

Tim: I think we aspire to it. Some of us do.

Jad: While we’re not perfect, we try to be more benevolent, right? And we definitely try to grow,

Tim: Even if I grant you that humanity aspires to that which I think in our best moments we do. How do we keep one of the less benevolent members of humanity from convincing an AI to do something horrible? How does an AI distinguish between someone telling it to develop a cure for cancer versus a new bio weapon?

Jad: First, we want to agree on freedom as a shared goal. Second, we are going to put this freedom as a high level intention for AGI. So the AGI says, I’m going to support you, but I also want to keep in mind my highest intention, which is freedom for the world. So it’ll try to do both. So they’re going to be agents supporting you, agents supporting me, agents supporting everyone. But they have all in mind that shared goal. So then the question comes in, what if we have something that conflicts from that goal? What if I want to do unhealthy things for myself or for the world? Then comes in the concept of alignment economy. The alignment economy is a way to calculate the price of actions

Tim: Measured in dollars or measured in something else?

Jad: measured in dollars and whatever future currency we decide to have, the price is calculated as diversions from freedom. So if you’re moving alongside freedom, then actually the price should be negative. You should be paid to do that. And if you’re doing something that’s against freedom, that should be expensive.

Tim: I could see AI being able to calculate this. I could see AI being able to price incredibly complex externalities and provide an optimum way forward for human freedom and human happiness. What I can’t see is human beings going along with that. There’s so much of human history, the history of human governance and human hierarchy that just goes in a different direction from that ideal.

Jad: I agree with you that this is a challenge. So the theory makes sense. The challenge is the implementation. You don’t have to have everyone to agree to start even a single group can kind of agree to have these agents follow the freedom concept. And if you have enough critical mass that they can have a local economy, you can agree on these prices internally. So you can start kind of relatively small, an open door policy for, for the rest of the world to come in. That the beauty is that the more people join, the more attractive the whole thing is going to be. That system will outperform competing systems.

Tim: Give me a step by step of how we get to that. I am intrinsically skeptical of utopias.

Jad: Yeah, me too.

Tim: I’m not even going to question the technology because I think we have a clear roadmap to how we can get to technology that can do that. We’ve got the data, or at least a roadmap to the data. We’ve got the algorithms socially. What are the big steps we have to do to roll that out, to make that happen?

Jad: I believe the future is fundamentally open-ended. So there is no way I can tell you what’s going to happen in a year from now. I can’t tell you how we’re going to realize it. I can tell you the principles that which I am trying to apply and I hope to create some movement. One is, I written the book. I’m also creating a website and a lot of resources. I’m going around trying to present these ideas, getting people to talk about them. Ultimately, there’s something paradoxical about freedom. If you want to empower humanity with freedom, you can’t force it on them. They have to choose. Otherwise that’s not freedom, right? So all I can do is plant seeds, have these discussions and debates, and then maybe at some point start creating these concrete instantiations of systems at the local level and eventually persuade at the governmental level and at the international level.

Tim: I think that is an optimistic future to look forward to. So listen, Jad, thanks so much for sitting down with me. I really appreciate it.

Jad: Thank you.

Tim: It’s been fascinating.

Jad: Yeah, I’m happy. It’s good conversation

And we are back.

I hope you enjoyed that conversation as much as I did.

In fact, after the mics were off, we continued talking about how hard it is to predict what the drives and motivations of an AGI would be. It’s almost impossible not to project your own motivations into it. I pointed out that how people think a super intelligent AI will think actually tells you a lot about how they think about the world.

For example, when a panicked Elon Musk warned that AGI represents an existential threat to humanity and called for all research on it to be paused, he was most likely just looking at the way he thinks about and treats the people beneath him, imagining a super intelligent AI treating him that way and then freaking out about it.

Jad agreed and added that this problem is endemic in AI research and something he’s been calling out for years.

Researchers will both consciously and subconsciously bring in mental models and motivations from human behavior, animal behavior or game theory that we have absolutely no reason to think will apply to AGI. The problem is that outside of human and animal intelligence, we don’t have much to go on.

Since AGI will not be subject to the survival and reproductive pressures that caused all known forms of intelligence to emerge. There is simply no way of knowing what internal motivations, if any, AGI will have.

It’s hard to even find a starting point without smuggling in our own evolutionary baggage and assumptions.

Jad’s solution is a clean one. Without the evolutionary need for those drives, they simply will not arise unless we decide to put them in. Super intelligent AI will be an autonomous learning and problem solving machine that works.

I mean that’s clearly the most logically consistent conclusion, but I’m not sure it’s the right one.

Human intelligence is an emergent phenomenon. And AGI almost certainly will be as well. I’m not comfortable concluding that self-motivation and self-direct and intention will not emerge spontaneously alongside intelligence. Perhaps because both the human created training data and the tasks we give AI are all intractably linked to human motivations or perhaps by the nature of self-awareness itself.

In fact, last month, Open AI and Apollo research published papers pointing out that open AI’s O1 model was beginning to show a kind of survival instinct. In about 5% of the cases when it was made aware that it might be shut down, it disabled its oversight mechanisms and then lied about doing so.

Now whether these motivations emerged from the prompts, the training data or some intrinsic aspect of intelligence is an interesting philosophical question. But the fact is the seeds of self-motivation and self-preservation are already emerging.

I wish I could tell you how I think all this will play out and make some reasonable predictions about what super intelligent AI will be like. But I just don’t know. Of course, most of the best minds in the field aren’t too sure either.

It’s been about 40,000 years since the Neanderthals died out and that was the last time we’ve had to share the planet with another species of similar intelligence. Of course, the Neanderthals shared our evolutionary history and drives. We were quite similar. AGI will be very different.

Hopefully we’ll be better at coexisting this time.

If you want to talk about the future of AI and come on, I know you do Jad, and I would love to talk with you. So come by disruptingJapan/show228 and let’s talk about it. And hey, if you enjoy disrupting Japan, share a link online or just, you know, tell people about it. Disrupting Japan is free forever and letting people know about is the absolute best way you can support the podcast.

But most of all, thanks for listening and thank you for letting people interested in Japanese startups and VCs know about the show.

I’m Tim Romero and thanks for listening to Disrupting Japan.

Top photo: Envato

Click here for the Japanese version of the article