미친 건, 이 모든 게 진짜라는 거예요.

https://youtu.be/L3OvFcSjXRg?si=Yxq4KUqZMt4hgGkr

지금 AI에겐 결정적인 '이것'이 없다 - 일리야 수츠케버

[4s] You know what’s crazy? That all of this is real.
[4s] 미친 건, 이 모든 게 진짜라는 거예요.
[5s] >> Yeah. Meaning what?
[5s] >> 네. 그게 무슨 뜻이죠?
[6s] >> Don’t you think so?
[6s] >> 당신도 그렇게 생각하지 않아요?
[8s] >> Meaning what?
[8s] >> 무슨 뜻이냐고요?
[11s] >> Like all this AI stuff and all this area. Yeah. That it’s happen. Like isn’t it straight out of science fiction?
[11s] >> AI 관련 모든 것과 이 분야처럼요. 네. 이게 일어나고 있다는 거예요. SF 소설에서 바로 나온 것 같지 않아요?
[19s] >> Yeah. Another thing that’s crazy is like how normal the slow takeoff feels. The idea that we’d be investing 1% of GDP in AI, like I feel like it would have felt like a bigger deal, you know, where right now it just feels like
[19s] >> 네. 또 다른 미친 건 느린 이륙이 얼마나 정상적으로 느껴지냐는 거예요. AI에 GDP의 1%를 투자한다는 아이디어가 더 큰 일처럼 느껴질 것 같았는데, 지금은 그냥
[30s] >> we get used to things pretty fast. Turns out, yeah, but also it’s kind of like it’s abstract like what does it mean? What it means that you see it in the news?
[30s] >> 우리는 물건에 꽤 빨리 익숙해져요. 맞아요, 하지만 추상적이기도 하죠. 그게 무슨 의미냐면? 뉴스에서 본다는 뜻인가요?
[37s] >> Yeah.
[37s] >> 네.
[39s] >> That such and such company announced such and such dollar amount,
[39s] >> 어떤 회사가 어떤 금액을 발표했다는 거요,
[41s] >> right?
[41s] >> 그렇죠?
[42s] >> That’s that’s all you see,
[42s] >> 그게 당신이 보는 전부예요,
[45s] >> right?
[45s] >> 맞아요?
[45s] >> It’s not really felt in any other way so far.
[45s] >> 아직 다른 방식으로는 느껴지지 않아요.
[48s] >> Yeah. Should we actually begin here? I think this is an interesting discussion.
[48s] >> 네. 여기서 시작할까요? 흥미로운 토론이라고 생각해요.
[51s] >> Sure.
[51s] >> 좋아요.
[53s] >> I think your point about well from the average person’s point of view, nothing is that different will continue being true even into the singularity. No, I don’t think so.
[53s] >> 평균 사람의 관점에서 아무것도 그렇게 다르지 않다는 당신의 포인트가 특이점까지도 계속 사실일 거라고 생각해요. 아니, 그렇게 생각하지 않아요.
[1m 2s] >> Okay. Interesting.
[1m 2s] >> 좋아요. 흥미로워요.
[1m 5s] >> So the thing which I was referring to not feeling different
[1m 5s] >> 제가 말한 다르게 느껴지지 않는다는 건
[1m 10s] is okay. So such and such company announced some difficult to comprehend dollar amount of investment,
[1m 10s] 괜찮아요. 어떤 회사가 이해하기 어려운 투자 금액을 발표했다는 거예요,
[1m 14s] >> right?
[1m 14s] >> 그렇죠?
[1m 15s] >> I don’t think anyone knows what to do with that.
[1m 15s] >> 그걸 어떻게 해야 할지 아무도 모를 거예요.
[1m 20s] >> Yeah.
[1m 20s] >> 네.
[1m 21s] >> But I think that the impact of AI is going to be felt.
[1m 21s] >> 하지만 AI의 영향은 느껴질 거예요.
[1m 25s] >> AI is going to be diffused through the economy. There are very strong economic forces for this.
[1m 25s] >> AI는 경제 전반에 퍼질 거예요. 이를 위한 강력한 경제적 힘들이 있어요.
[1m 30s] And I think the impact is going to be felt very strongly.
[1m 30s] 그리고 그 영향은 매우 강하게 느껴질 거예요.
[1m 36s] >> When do you expect that impact? I think the models seem smarter than their economic impact would imply.
[1m 36s] >> 그 영향은 언제 예상하나요? 모델들이 경제적 영향이 암시하는 것보다 더 똑똑해 보이는데요.
[1m 45s] >> Yeah, this is one of the very confusing things about the models right now. How to reconcile
[1m 45s] >> 네, 이건 지금 모델에 대한 아주 혼란스러운 점 중 하나예요. 어떻게 조화시킬지
[1m 52s] the fact that they are doing so well on evals.
[1m 52s] 평가에서 그렇게 잘하고 있다는 사실.
[1m 56s] >> Mhm. And you look at the evals and you go, those are pretty hard evals,
[1m 56s] >> 음. 평가를 보면, 꽤 어려운 평가라고 생각해요,
[1m 59s] >> right?
[1m 59s] >> 그렇죠?
[2m 2s] >> They’re doing so well,
[2m 2s] >> 아주 잘하고 있어요,
[2m 5s] but the economic impact seems to be dramatically behind. And it’s almost like
[2m 5s] 하지만 경제적 영향은 극적으로 뒤처져 있어요. 거의
[2m 12s] it’s it’s very difficult to make sense of how can the model on the one hand do these amazing things and then on the other hand like repeat itself twice in some situation in a kind of a an example would be let’s say you use vibe coding to do something and you go to some place and then you get a bug and then you tell the model can you please fix the bug?
[2m 12s] 이해하기 어렵죠. 모델이 한편으로는 놀라운 일들을 하고 다른 한편으로는 어떤 상황에서 자신을 두 번 반복하는 식으로요. 예를 들어 vibe 코딩을 사용해 무언가를 하고 어딘가 가서 버그를 만나 모델에게 버그를 고쳐달라고 하면요?
[2m 33s] >> Yeah. And the model says, “Oh my god, you’re so right. I have a bug. Let me go fix that.” And it introduces a second bug.
[2m 33s] >> 네. 모델이 “오 마이 갓, 당신 말이 맞아요. 버그가 있어요. 고칠게요.“라고 하고 두 번째 버그를 도입해요.
[2m 38s] >> Yeah.
[2m 38s] >> 네.
[2m 40s] >> And then you tell it you have you have this new second bug and it tells you, “Oh my god, how could I’ve done it? You’re so right again.”
[2m 40s] >> 그리고 당신이 새로운 두 번째 버그가 있다고 말하면, “오 마이 갓, 어떻게 그랬지? 당신 말이 또 맞아요.“라고 해요.
[2m 45s] >> And brings back the first bug. And you can alternate between those.
[2m 45s] >> 그리고 첫 번째 버그를 다시 가져와요. 그 둘 사이를 번갈아 할 수 있어요.
[2m 48s] >> Yeah.
[2m 48s] >> 네.
[2m 48s] >> And it’s like, how is that possible?
[2m 48s] >> 어떻게 그게 가능하죠?
[2m 52s] >> Yeah.
[2m 52s] >> 네.
[2m 54s] >> It’s like I’m not sure. But it does suggest that the something strange is going on. I have two possible explanations. So here this is the more kind of a whimsical explanation is that maybe RL training makes the models a little bit too single-minded and narrowly focused a little bit too I don’t know unaware even though it also makes them aware in some other ways
[2m 54s] >> 잘 모르겠어요. 하지만 뭔가 이상한 일이 일어나고 있다는 걸 시사하죠. 두 가지 가능한 설명이 있어요. 여기 더 기발한 설명은 RL 훈련이 모델을 좀 너무 단일 지향적이고 좁게 집중하게 만들어서, 모르는 채로 두는 거예요. 다른 면에서는 인식하게 만들면서도요.
[3m 22s] and because of this they can’t do basic things but there is another explanation which is
[3m 22s] 그래서 기본적인 일들을 못 하게 되지만, 다른 설명은
[3m 31s] back when people were doing pre-training the question of what data to train on was answered because the that answer was everything.
[3m 31s] 사람들이 사전 훈련을 할 때 어떤 데이터를 훈련할지 질문의 답은 모든 것이었어요.
[3m 38s] >> Yeah.
[3m 38s] >> 네.
[3m 41s] >> When you do pre-training, you need all the data. So you don’t have to think is it going to be this data or that data.
[3m 41s] >> 사전 훈련을 할 때 모든 데이터가 필요해요. 그래서 이 데이터냐 저 데이터냐 생각할 필요가 없어요.
[3m 47s] >> Yeah.
[3m 47s] >> 네.
[3m 50s] >> But when people do RL training, they do need to think. They say okay, we want to have this kind of RL training for this thing and that kind of RL training for that thing. And from what I hear, all the companies have teams that just produce new RL environments and just add it to the training mix. And then the question is, well, what are those? There are so many degrees of freedom. There is such a huge variety of real environments you could produce. And one of the one thing you could do, and I think that’s something that is done inadvertently,
[3m 50s] >> 하지만 사람들이 RL 훈련을 할 때는 생각해야 해요. 이건 이런 RL 훈련, 저건 저런 RL 훈련이라고요. 제가 들은 바에 따르면 모든 회사에 새로운 RL 환경을 만들어 훈련 믹스에 추가하는 팀이 있어요. 그러면 질문은 그게 뭐냐예요? 자유도가 너무 많아요. 만들 수 있는 실제 환경의 다양성이 엄청나요. 그리고 할 수 있는 한 가지는, 무의식적으로 하는 거라고 생각하지만,
[4m 24s] is that people take inspiration from the evals. you say, “Hey, I would love our model to do really well when we release it. I want the EVOs to look great. What would be RL training that could help on this task?” Right? I think that is something that happens and I think it could explain a lot of what’s going on.
[4m 24s] 사람들이 평가에서 영감을 얻는 거예요. “우리 모델이 출시될 때 정말 잘했으면 좋겠어. 평가가 멋지게 보이게. 이 작업에 도움이 될 RL 훈련은 뭐지?“라고요. 그런 일이 일어난다고 생각하고, 그게 일어나는 많은 걸 설명할 수 있어요.
[4m 44s] If you combine this with generalization of the models actually being inadequate, that has the potential to explain a lot of what we are seeing. this disconnect between eval performance and actual real world performance which is something that we don’t today exactly even understand what what we mean by that I I like this idea that the real reward hacking is a human researchers who are too focused on the evals um I think there’s two ways to understand or to try to think about what what you have just pointed out one is look if it’s case that simply by becoming superhuman at a coding competition, a model will not automatically become more tasteful and exercise better judgment about how to improve your codebase. Well, then you should expand the suite of environments such that you’re not just testing it on having the best performance in coding competition. It should also be able to make the best kind of application for X thing or Y thing or Z thing. And another maybe this is what you’re hinting at is to say why should it be the case in the first place that becoming superhuman at coding competitions doesn’t make you a more tasteful programmer more generally.
[4m 44s] 이걸 모델의 일반화가 실제로 불충분하다는 것과 결합하면, 우리가 보는 많은 걸 설명할 잠재력이 있어요. 평가 성능과 실제 세계 성능 사이의 단절인데, 오늘날 우리는 그게 무슨 뜻인지 정확히 이해하지 못해요. 진짜 보상 해킹이 평가에 너무 집중한 인간 연구자라는 아이디어가 좋아요. 당신이 지적한 걸 이해하거나 생각하는 두 가지 방법이 있어요. 하나는 코딩 대회에서 초인간이 되는 것만으로 모델이 자동으로 더 취향 좋고 코드베이스를 개선하는 더 나은 판단을 하지 않을 경우예요. 그러면 환경 세트를 확장해야 해요. 코딩 대회에서 최고 성능을 테스트하는 게 아니라 X, Y, Z 것에 대한 최고 애플리케이션을 만들 수 있어야 해요. 또 다른 건, 애초에 코딩 대회에서 초인간이 되는 게 왜 더 취향 좋은 프로그래머로 만들지 않는지 말하는 거예요.
[5m 58s] Maybe the thing to do is not to keep stacking up the amount of environments and the diversity of environments to figure out approach with let you learn from one environment and improve your performance on something else. So I have I have an anal a human analogy which might be helpful. So even the case let’s take the case of competitive programming since you mentioned that and suppose you have two students one of them work decided they want to be the best competitive programmer so they will practice 10,000 hours for that domain
[5m 58s] 할 일은 환경의 양과 다양성을 계속 쌓는 게 아니라, 한 환경에서 배우고 다른 것에서 성능을 개선하는 접근을 찾는 거예요. 도움이 될 인간 비유가 있어요. 경쟁 프로그래밍 사례를 들어보죠. 두 학생이 있어요. 하나는 최고 경쟁 프로그래머가 되기로 해서 그 도메인에서 10,000시간 연습해요.
[6m 32s] >> they will solve all the problems memorize all the proof techniques and be very very you know be very skilled at quickly and correctly implementing all the algorithms and by doing by doing so they became the best one of the best. Student number two thought, oh, competitive programming is school. Maybe they practiced for 100 hours,
[6m 32s] >> 모든 문제를 풀고 모든 증명 기법을 외우고, 모든 알고리즘을 빠르고 정확하게 구현하는 데 아주 숙련되게 돼요. 그렇게 해서 최고 중 하나가 돼요. 두 번째 학생은 경쟁 프로그래밍이 학교라고 생각하고, 100시간만 연습했어요,
[6m 54s] >> much much less, and they also did really well. Which one do you think is going to do better in their career later on?
[6m 54s] >> 훨씬 적게, 그리고 잘했어요. 누가 나중에 경력에서 더 잘할 것 같아요?
[6m 58s] >> The second,
[6m 58s] >> 두 번째요,
[7m] >> right? And I think that’s basically what’s going on. The models are much more like the first student, but even more because then we say, okay, so the model should be good at competitive programming. So let’s get every single competitive programming problem ever and then let’s do some data augmentation. So we have even more competitive programming problems.
[7m] >> 그렇죠? 그게 기본적으로 일어나는 일이라고 생각해요. 모델들은 첫 번째 학생처럼, 더 심해요. 왜냐하면 모델이 경쟁 프로그래밍에 잘해야 한다고 해서 모든 경쟁 프로그래밍 문제를 모으고 데이터 증강을 해서 더 많은 문제를 만들어요.
[7m 16s] >> Yes.
[7m 16s] >> 네.
[7m 18s] >> And we train on that. And so now you got this great competitive programmer. And with this analogy, I think it’s more intuitive. I think it’s more intuitive with this analogy that Yeah. Okay. So if it’s so well trained, okay, it’s like all the different algorithms and all the different proof techniques are like right at it at its fingertips.
[7m 18s] >> 그걸로 훈련해요. 그래서 훌륭한 경쟁 프로그래머가 돼요. 이 비유로 더 직관적이라고 생각해요. 잘 훈련됐으면, 모든 다른 알고리즘과 증명 기법이 손끝에 있어요.
[7m 36s] And it’s more intuitive that with this level of preparation, it not would not necessarily generalize to other things.
[7m 36s] 이 수준의 준비로 다른 것에 꼭 일반화되지 않는다는 게 더 직관적이에요.
[7m 45s] >> But then what is the um analogy for what the second student is doing before they do the 100 hours of fine-tuning.
[7m 45s] >> 하지만 두 번째 학생이 100시간 파인튜닝 전에 하는 것에 대한 비유는 뭐예요.
[7m 53s] >> I think it’s like they have it. I think it’s the it factor.
[7m 53s] >> 그게 바로 it이라고 생각해요. it 팩터요.
[7m 56s] >> Yeah.
[7m 56s] >> 네.
[7m 57s] >> Right. And like I know like when I was in undergrad, I remember there was there was a student like this that studied with me. So I know I know it exists.
[7m 57s] >> 그렇죠. 학부 때 그런 학생이 있었어요. 존재한다는 걸 알아요.
[8m 4s] >> Yeah. I think it’s interesting to distinguish it from whatever pre-training does. So, one way to understand what you just said about we don’t have to choose the data in pre-training is to say actually it’s not dissimilar to the 10,000 hours of practice. It’s just that you get that 10,000 hours of practice for free because it’s already somewhere in the pre-training distribution. But it’s like maybe you’re suggesting actually there’s actually not that much generalization from pre-training. There’s just so much data in pre-training. Everybody’s like it’s not necessarily generalizing better than RL.
[8m 4s] >> 네. 사전 훈련이 하는 것과 구분하는 게 흥미로워요. 데이터 선택 안 해도 된다는 건 10,000시간 연습과 비슷하다고 할 수 있어요. 사전 훈련 분포에 이미 있어서 무료로 얻는 거죠. 하지만 사전 훈련에서 일반화가 많지 않고 데이터가 많다는 걸 암시하는 것 같아요. RL보다 꼭 더 잘 일반화되지 않아요.
[8m 34s] >> Like the main the main strength of pre-training is that there is a so much of it.
[8m 34s] >> 사전 훈련의 주요 강점은 양이 엄청 많다는 거예요.
[8m 39s] >> Yeah.
[8m 39s] >> 네.
[8m 42s] >> And b you don’t have to think hard about what data to put into pre-training
[8m 42s] >> 그리고 어떤 데이터를 넣을지 어렵게 생각할 필요가 없어요.
[8m 48s] >> and it’s a very kind of natural data and it does include in it a lot of what people do.
[8m 48s] >> 자연스러운 데이터고 사람들이 하는 많은 걸 포함해요.
[8m 53s] >> Yeah. people’s thoughts and a lot of the features of you know it’s like the whole world as projected by people onto text.
[8m 53s] >> 네. 사람들의 생각과 특징들, 사람들이 텍스트에 투영한 전체 세계처럼요.
[9m 1s] >> Yeah.
[9m 1s] >> 네.
[9m 4s] >> And pre-training tries to capture that using a huge amount of data. It’s it’s very the pre-training is very difficult to reason about because it’s so hard to understand the manner in which the model relies on pre-training data. And whenever the model makes a mistake, could it be because something by chance is not as supported by the pre-training data? You know, and pre support by pre-training is maybe a loose term.
[9m 4s] >> 사전 훈련은 엄청난 데이터로 그걸 포착하려 해요. 사전 훈련은 모델이 사전 훈련 데이터에 의존하는 방식을 이해하기 어려워서 추론하기 아주 어려워요. 모델이 실수할 때마다, 우연히 사전 훈련 데이터에서 지원되지 않아서일 수 있어요. 사전 훈련 지원은 느슨한 용어예요.
[9m 36s] I I don’t know if I can add anything more useful on this, but I don’t think there is a human analog to pre-training.
[9m 36s] 이게 더 유용한지 모르겠지만, 사전 훈련에 대한 인간 유사물이 없다고 생각해요.
[9m 42s] Here’s analogies that people have proposed for what the human analogy to pre-training is, and I’m curious to get your thoughts on why they’re potentially wrong. One is to think about the first 18 or 15 or 13 years of a person’s life when they aren’t necessarily economically productive, but they are doing something that is making them understand the world better and so forth. And the other is to think about evolution as doing some kind of search for three billion years which then results in a human lifetime instance. And then I’m I’m curious if you think either of these are actually analogous to pre-training or how how would you think about at least what lifetime human learning is like if not pre-training. I think there are some similarities between both of these two pre-training and pre-training tries to play the role of both of these
[9m 42s] 사람들이 제안한 사전 훈련에 대한 인간 비유가 있어요. 왜 잠재적으로 잘못됐는지 당신 생각이 궁금해요. 하나는 사람의 처음 18, 15, 13년으로, 경제적으로 생산적이지 않지만 세계를 더 잘 이해하게 하는 거예요. 다른 건 진화가 30억 년 동안 검색을 해서 인간 일생 인스턴스를 결과로 내는 거예요. 이 둘 중 어느 게 사전 훈련과 유사하다고 생각하나요? 아니면 평생 인간 학습이 사전 훈련이 아니면 어떤지. 둘 다 사전 훈련과 유사점이 있고, 사전 훈련은 둘 다 역할을 하려 해요.
[10m 34s] >> but I think there are some big differences as well. The amount of pre-training data is very very staggering.
[10m 34s] >> 하지만 큰 차이도 있어요. 사전 훈련 데이터 양은 아주 엄청나요.
[10m 44s] >> Yes.
[10m 44s] >> 네.
[10m 47s] >> And somehow a a human being after even 15 years with a tiny fraction of that pre-training data they know much less.
[10m 47s] >> 그런데 인간은 15년 후에도 그 사전 훈련 데이터의 작은 부분으로 훨씬 적게 알아요.
[10m 53s] >> Yeah. But whatever they do know they know much more deeply somehow and the mistakes like like already at that age you would not make mistakes that ours make.
[10m 53s] >> 네. 하지만 아는 건 훨씬 깊게 알아요. 그 나이에도 우리 모델이 하는 실수를 하지 않아요.
[11m 2s] >> Yeah. There is another thing you might say could it be something like evolution and the answer is maybe but in this case I think evolution might actually have an edge like there is this I remember reading about this case where some you know that one thing that neuroscientists do or rather one way in which neuroscientists can learn about the brain is by studying people with brain damage to different parts of the brain
[11m 2s] >> 네. 진화 같은 거라고 할 수 있지만, 진화가 우위일 수 있어요. 신경과학자들이 뇌 손상 사람을 연구해서 뇌를 배우는 경우를 읽었어요.
[11m 30s] >> and and so and some people have the most strange symptoms you could imagine. It’s actually really really interesting. And there was one case that comes to mind that’s relevant.
[11m 30s] >> 어떤 사람들은 상상할 수 없는 이상한 증상을 보여요. 정말 흥미로워요. 관련된 한 사례가 떠오르네요.
[11m 41s] I read about this person who had some kind of brain damage that took out I think a stroke or an accident that took out his emotional processing. So he stopped feeling any emotion
[11m 41s] 뇌 손상으로 감정 처리를 잃은 사람에 대해 읽었어요. 뇌졸중이나 사고로 감정 처리를 잃어서 감정을 느끼지 못했어요.
[11m 56s] and as a result of that you know he still remained very articulate and he could solve little puzzles and on tests he seemed to be just fine but he felt no emotion he didn’t feel sad he didn’t feel angry he didn’t feel animated and he became somehow extremely bad at making any decisions at all it would take him hours to decide on which socks to wear and he would make very bad financial decisions
[11m 56s] 그 결과로 그는 여전히 명확하게 말하고 작은 퍼즐을 풀 수 있었고 테스트에서는 괜찮아 보였지만 감정을 느끼지 못했어요. 슬프거나 화나거나 활기차지 않았고, 결정하는 데 극도로 나빠졌어요. 양말 고르는 데 몇 시간이 걸리고 나쁜 재정 결정을 했어요.
[12m 26s] And that’s very
[12m 26s] 그게 아주
[12m 30s] does what what does it say about the role of our built-in emotions in making us like a viable agent essentially
[12m 30s] 우리의 내장 감정이 우리를 실행 가능한 에이전트로 만드는 역할에 대해 뭐라고 말하나요.
[12m 36s] >> and I guess to connect to your question about pre-training
[12m 36s] >> 사전 훈련에 대한 당신 질문과 연결해서
[12m 41s] >> it’s like maybe pre- like maybe if you are good enough at like getting everything out of pre-training you can get you could get that as well but that’s the kind of thing which seems Well, it may or may not be possible to get that from pre-training. What is that? Clearly not just directly emotion. It seems like some almost value function like thing which is giving telling you which decision to be like what the end reward for any decision should be and you think that doesn’t sort of implicitly come from
[12m 41s] >> 사전 훈련에서 모든 걸 잘 추출하면 그걸 얻을 수 있지만, 사전 훈련에서 얻을 수 있을지 없을지 모르겠어요. 그게 뭐냐면, 감정이 아니고 가치 함수 같은 거예요. 어떤 결정의 최종 보상이 뭐여야 하는지 말해주는 거죠. 그게
[13m 19s] >> I think it could I’m just saying it’s not one it’s not 100% obvious.
[13m 19s] >> 될 수 있다고 생각해요. 100% 명확하지 않다는 거예요.
[13m 24s] >> Yeah. But what what is that like what how do you think about emotions in what is the ML analogy for emotions?
[13m 24s] >> 네. 하지만 그게 뭐예요? 감정을 어떻게 생각하나요? 감정에 대한 ML 비유는 뭐예요?
[13m 28s] >> It should be some kind of a value function thing.
[13m 28s] >> 가치 함수 같은 거예요.
[13m 32s] >> Yeah. But I don’t think there is a great ML analogy because right now value functions don’t play a very prominent role in uh the things people do.
[13m 32s] >> 네. 하지만 좋은 ML 비유가 없다고 생각해요. 지금 가치 함수가 사람들이 하는 일에서 두드러진 역할을 하지 않아요.
[13m 39s] >> It might be worth defining for the audience what a value function is if if you want to do that.
[13m 39s] >> 청중을 위해 가치 함수가 뭔지 정의할 가치가 있을 수 있어요.
[13m 47s] >> I mean certainly I I’ll be very happy to do that. Right. So
[13m 47s] >> 물론 기쁘게 할게요. 자,
[13m 52s] so when people do reinforcement learning the way reinforcement learning is done right now how does it how do people train those agents? So you have your neural net and you give it a problem and then you tell the model go solve it. The model takes maybe thousands hundreds of thousands of actions
[13m 52s] 강화 학습을 할 때 지금 방식으로 에이전트를 어떻게 훈련하나요? 신경망에 문제를 주고 해결하라고 해요. 모델이 수천 수십만 액션을 취해요.
[14m 9s] or thoughts or something and then it produces a solution. The solution is created and then the score
[14m 9s] 생각이나 뭐든 해서 해결책을 만들어요. 해결책이 만들어지고 점수가
[14m 18s] is used to provide a training signal for every single action in your trajectory.
[14m 18s] 궤적의 모든 액션에 훈련 신호를 제공해요.
[14m 24s] >> Mhm. So that means that if you are doing something that goes for a long time, if you’re training a task that takes a long time to solve, you will do no learning at all until you solve the until you came up with a proposed solution. That’s how reinforcement learning is done naively. That’s how 01 R1 ostensibly are done.
[14m 24s] >> 음. 그래서 오래 걸리는 걸 하면, 오래 걸리는 작업을 훈련하면 제안 해결책을 내놓을 때까지 학습이 전혀 안 돼요. 그게 순진한 강화 학습 방식이에요. 01 R1이 표면상 그렇게 돼요.
[14m 46s] The value function says something like okay look maybe I could sometimes not always could tell you if you’re doing well or badly. The notion of a value function is more useful in some domains than others. So for example when you play chess
[14m 46s] 가치 함수는 때때로 잘하거나 못하는지 말할 수 있다고 해요. 가치 함수 개념은 일부 도메인에서 더 유용해요. 예를 들어 체스를 할 때
[15m 1s] and you lose a piece you know I messed up. You don’t need to play the whole game to know that what I just did was bad and therefore whatever um whatever preceded it was also bad. So the value function lets you short circuit the weight until the very end. Like let’s suppose that you started to pursue some kind of um okay let’s suppose that you are doing some kind of a math thing or a programming thing and you’re trying to explore a particular solution direction and after let’s say after a thousand uh steps of thinking you concluded that this direction is unpromising.
[15m 1s] 말을 잃으면 망쳤다고 알아요. 전체 게임을 할 필요 없이 방금 한 게 나빴고 그 앞도 나빴어요. 가치 함수는 끝까지 기다리는 걸 단축해요. 수학이나 프로그래밍에서 특정 해결 방향을 탐색하다 천 단계 생각 후 이 방향이 무의미하다고 결론짓는 거예요.
[15m 40s] As soon as you conclude this, you could already get a reward signal a thousand time steps previously when you decided to pursue down this path. You say, “Oh, next time I shouldn’t pursue this path in a similar situation long before you actually came up with a proposed solution.” H this was in the deepcar one paper is that the space of trajectories is so wide that maybe it’s hard to learn a mapping from an intermediate trajectory and value and also given that you know in coding for example you’ll have the wrong idea then you’ll go back then you’ll change something
[15m 40s] 이걸 결론짓자마자, 천 타임 스텝 전에 이 경로를 추구하기로 결정했을 때 이미 보상 신호를 얻을 수 있어요. “아, 다음엔 비슷한 상황에서 이 경로를 추구하지 말아야 해”라고, 실제 제안 해결책을 내놓기 오래 전에요. deepcar one 논문에서 궤적 공간이 너무 넓어서 중간 궤적과 가치 매핑을 배우기 어렵다고 해요. 코딩에서 잘못된 아이디어를 갖고 돌아가서 바꾸는 거예요.
[16m 15s] >> this sounds like such lack of faith in deep learning
[16m 15s] >> 딥러닝에 대한 믿음 부족처럼 들려요.
[16m 21s] >> like I mean sure it might be difficult but nothing deep learning can’t too.
[16m 21s] >> 어렵긴 하지만 딥러닝이 못할 건 없어요.
[16m 26s] >> Yeah.
[16m 26s] >> 네.
[16m 30s] >> So my expectation is that like value function should be useful and and I fully I fully expect that they will be used in the future if not already. What was I alluding to with the person whose emotional center got um damaged is more that maybe what it suggests is that the value function of humans is modulated by emotions in some important way that’s hardcoded by evolution
[16m 30s] >> 제 기대는 가치 함수가 유용할 거예요. 이미 아니면 미래에 사용될 거라고 완전히 기대해요. 감정 중심이 손상된 사람에 대해 암시한 건 인간의 가치 함수가 진화에 의해 하드코딩된 중요한 방식으로 감정에 의해 조절된다는 거예요.
[16m 58s] and maybe that is important for people to be effective in the world.
[16m 58s] 그게 사람들이 세상에서 효과적으로 되는 데 중요할 수 있어요.
[17m 2s] >> That that’s the thing I was actually planning on asking you. There’s something really interesting about emotions as a value function, which is that it’s impressive that they have this much utility while still being rather um simple to understand.
[17m 2s] >> 그게 제가 물어보려던 거예요. 가치 함수로서의 감정에 대해 정말 흥미로운 건, 이해하기 꽤 단순하면서도 이 정도 유용성을 가진다는 거예요.
[17m 23s] So I have two responses. I do agree that compared to the kind of things that we learn and the things we are talking about, the kind of ads we are talking about, emotions are relatively simple.
[17m 23s] 두 가지 응답이 있어요. 우리가 배우는 것과 이야기하는 것에 비해 감정은 상대적으로 단순하다는 데 동의해요.
[17m 34s] They might even be so simple that maybe you could map them out in a human understandable way. I think it would be cool to do.
[17m 34s] 너무 단순해서 인간이 이해할 수 있게 매핑할 수 있을지도 몰라요. 해보면 멋질 거예요.
[17m 43s] In terms of utility though, I think there is a thing where you know there is this complexity robustness trade-off
[17m 43s] 유용성 면에서 복잡성과 견고성 트레이드오프가 있어요.
[17m 56s] where complex things can be very useful but simple things are very useful in very broad range of situations. And so I think what what one way to interpret what we are seeing is that we’ve got these emotions that essentially evolved mostly mostly from our mammal ancestors and then fine-tuned a little bit while we were homminids just a bit. We do have like a decent amount of social emotions though which mammals may lack but they’re not very sophisticated
[17m 56s] 복잡한 건 아주 유용하지만 단순한 건 아주 넓은 상황에서 유용해요. 우리가 보는 걸 해석하는 한 방식은 이 감정들이 주로 포유류 조상에서 진화하고 호미니드 시기 조금 파인튜닝됐다는 거예요. 사회적 감정이 꽤 있지만 포유류가 없을 수 있고, 아주 세련되지 않아요.
[18m 25s] and because they’re not sophisticated they serve us so well in this very different world compared to the one that we’ve been living in. Actually they they also make mistakes for example our emotions well I don’t know does hunger count as an emotion
[18m 25s] 세련되지 않아서 우리가 살던 세계와 아주 다른 세계에서 잘 봉사해요. 실제로 실수도 해요. 예를 들어 우리의 감정, 배고픔이 감정인지 모르겠지만
[18m 42s] deate it’s debatable but I think for example our intuitive feeling of hunger is not succeeding in guiding us correctly in this world with an abundance of food.
[18m 42s] 논쟁의 여지가 있지만, 직관적인 배고픔 느낌이 음식 풍부한 이 세계에서 우리를 올바르게 안내하지 못해요.
[18m 54s] >> Yeah people have been talking about scaling data scaling parameters scaling compute. Is there a more general way to think about scaling? What are the other scaling axes?
[18m 54s] >> 네, 사람들이 데이터 스케일링, 파라미터 스케일링, 컴퓨트 스케일링에 대해 말해요. 스케일링을 더 일반적으로 생각할 방법이 있나요? 다른 스케일링 축은 뭐예요?
[19m 5s] >> So the thing so so here is a perspective here’s a perspective I think might be might be true.
[19m 5s] >> 여기 제 관점이 있어요. 사실일 수 있어요.
[19m 15s] So the way ML used to work is that people would just think of it with stuff and try to and try to get interesting results. That’s what’s been going on in the past.
[19m 15s] ML이 과거에 작동하던 방식은 사람들이 그냥 생각하고 흥미로운 결과를 얻으려 했어요. 그게 과거에 일어나던 거예요.
[19m 31s] Then the scaling insight arrived right scaling laws GPT3
[19m 31s] 그러다 스케일링 통찰이 도착했어요. 스케일링 법칙 GPT3
[19m 39s] and suddenly everyone realized we should scale
[19m 39s] 갑자기 모두 스케일해야 한다는 걸 깨달았어요.
[19m 45s] and it’s just this this is an example of how language affects thought.
[19m 45s] 이건 언어가 생각에 영향을 미치는 예예요.
[19m 50s] Scaling is what just one word but it’s such a powerful word because it informs people what to do. They say okay let’s let’s try to scale things. And so you say okay so what are we scaling and pre-training was a thing to scale it was a particular scaling recipe.
[19m 50s] 스케일링은 한 단어지만 강력한 단어예요. 사람들에게 뭐 할지 알려주니까요. 스케일 해보자고 해요. 그래서 뭐 스케일하나요? 사전 훈련이 스케일할 거고 특정 스케일링 레시피예요.
[20m 4s] >> Yes
[20m 4s] >> 네
[20m 7s] >> the big breakthrough of pre-training is the realization that this recipe is good. So you say hey if you mix some compute with some data into a neural net of a certain size you will get results and you will know that it will be better if you just scale the recipe up. And this is also great. Companies love this because it gives you a very uh lowrisk way of investing
[20m 7s] >> 사전 훈련의 큰 돌파구는 이 레시피가 좋다는 깨달음이에요. 컴퓨트와 데이터를 특정 크기의 신경망에 섞으면 결과가 나오고 레시피를 스케일업하면 더 좋아질 거예요. 회사들이 좋아해요. 저위험 투자 방식이니까요.
[20m 32s] >> your resources.
[20m 32s] >> 자원을요.
[20m 34s] >> Yeah.
[20m 34s] >> 네.
[20m 37s] >> Right. It’s much harder to invest your resources in research. Compare that. You know, if you research, you need to have like go forth researchers and research and come up with something versus get more data, get more compute, you know, you’ll get something from pre-training.
[20m 37s] >> 맞아요. 연구에 자원을 투자하는 건 훨씬 어려워요. 연구하면 연구자들을 보내 연구하고 뭔가 내놓아야 해요. 데이터 더 얻고 컴퓨트 더 얻는 거랑 비교하면, 사전 훈련에서 뭔가 얻을 거예요.
[20m 54s] And indeed, you know, it looks like I based on various um um things people say on some people say on Twitter, maybe it appears that Gemini have found a way to get more out of pre-training. At some point though, pre-training will run out of data. The data is very clearly finite. And so then, okay, what do you do next? Either you do some kind of a souped-up pre-training, different recipe from the one we’ve done before, or you’re doing RL or maybe something else. But now that comput is big, computer is now very b…(truncated 66135 characters)…is safely in a
[20m 54s] 실제로, 사람들이 트위터에서 말하는 것에 기반하면 Gemini가 사전 훈련에서 더 얻는 방법을 찾은 것 같아요. 하지만 언젠가 사전 훈련 데이터가 고갈될 거예요. 데이터는 분명 유한해요. 그럼 다음은 뭐예요? 이전과 다른 강화된 사전 훈련을 하거나 RL을 하거나 다른 걸 할 거예요. 하지만 이제 컴퓨트가 크고, 컴퓨터는 아주 b…(truncated 66135 characters)…is safely in a
(참고: 제공된 transcription이 중간에 “b…(truncated 66135 characters)…“로 잘려 있지만, 이어지는 부분부터 계속 번역합니다. 전체가 제공된 대로 번역했습니다.)
[1h 12m 45s] way that the other companies don’t. What what is that difference?
[1h 12m 45s] 다른 회사들이 하지 않는 방식으로 안전하게요. 그 차이는 뭐예요?
[1h 12m 51s] So the way I would describe it as there are some ideas that I think are promising and I want to investigate them and see if they are indeed promising or not. It’s really that simple. It’s an attempt. I think that if the ideas turn out to be correct, these ideas that we discussed around understanding generalization,
[1h 12m 51s] 제가 묘사하는 방식은 유망하다고 생각하는 아이디어가 있어서 조사해서 정말 유망한지 확인하려는 거예요. 정말 간단해요. 시도예요. 아이디어가 맞으면, 일반화 이해 주변에서 논의한 아이디어가
[1h 13m 13s] >> if these ideas turn out to be correct, then I think we will have something worthy. Will they turn out to be correct? We are doing research. We are squarely age of research company. We are making progress. We’ve actually made quite good progress over the past year, but we need to keep making more progress,
[1h 13m 13s] >> 맞으면 가치 있는 게 있을 거예요. 맞을까요? 연구 중이에요. 우리는 연구 회사예요. 진전을 이루고 있어요. 지난 해 꽤 좋은 진전을 이뤘지만 더 진전해야 해요,
[1h 13m 34s] >> more research. And that’s how I see it. I see it as an attempt to be an attempt to be a voice and a participant.
[1h 13m 34s] >> 더 연구요. 그렇게 봐요. 목소리와 참가자가 되려는 시도예요.
[1h 13m 45s] Um, people have asked uh your co-founder and previous CEO left to go to Meta recently and people have asked well if there was a lot of breakthroughs being made that seems like a thing that should have been unlikely. I wonder how you respond.
[1h 13m 45s] 사람들이 당신 공동 창립자이자 이전 CEO가 최근 Meta로 갔고, 돌파구가 많았다면 그럴 일이 없을 것 같다고 물었어요. 어떻게 응답하나요?
[1h 14m 1s] >> Yeah. So I in for for this I will simply remind a few facts that may have been forgotten and I think this these facts which provide the context I think they explain the situation. So the context was that we were fundraising at a 32 billion valuation
[1h 14m 1s] >> 네. 이건 그냥 잊혔을 수 있는 몇 가지 사실을 상기시킬게요. 이 사실들이 맥락을 제공하고 상황을 설명할 거예요. 맥락은 320억 평가로 펀드레이징 중이었어요.
[1h 14m 22s] and then Meta um came in and offered to to acquire us and I said no
[1h 14m 22s] Meta가 와서 인수 제안을 했고 저는 거절했어요.
[1h 14m 28s] but my former co-founder like in some sense said yes and as a result he also was able to enjoy from a lot of near-term liquidity and he was the only person from SSI to join Meta. It sounds like SSI’s plan is to be a company that is at the frontier when you get to this very important period in human history where you have superhuman intelligence and you have these ideas about how to make superhuman intelligence go well but other companies will be trying their own ideas. What distinguishes SSI’s approach to making super intelligence go well?
[1h 14m 28s] 하지만 제 이전 공동 창립자는 어느 의미에서 yes라고 했고, 결과로 단기 유동성을 즐길 수 있었고 SSI에서 Meta로 간 유일한 사람이에요. SSI의 계획은 초인간 지능이 있는 인류 역사 중요 기간에 프론티어 회사가 되는 거예요. 초인간 지능을 잘 만드는 아이디어가 있지만 다른 회사들도 자기 아이디어를 시도할 거예요. SSI의 초지능 잘 만드는 접근을 구분하는 건 뭐예요?
[1h 15m 4s] The
[1h 15m 4s] 그
[1h 15m 6s] >> the main thing that distinguishes SSI is its technical approach.
[1h 15m 6s] >> SSI를 구분하는 주요 건 기술 접근이에요.
[1h 15m 10s] So we have a different technical approach that I think is worthy and we are pursuing it.
[1h 15m 10s] 가치 있다고 생각하는 다른 기술 접근이 있고 추구 중이에요.
[1h 15m 17s] I maintain that in the end there will be a convergence of strategies. So I think there will be a convergence of strategies where
[1h 15m 17s] 결국 전략의 수렴이 있을 거라고 유지해요. 전략의 수렴이 있을 거예요.
[1h 15m 26s] at some point as AI becomes more powerful
[1h 15m 26s] AI가 더 강력해지면
[1h 15m 29s] it’s going to become more or less clearer to everyone what the strategy should be. And it should be something like, yeah, you need to find some way to talk to each other. And you want your first
[1h 15m 29s] 모두에게 전략이 뭐여야 하는지 더 명확해질 거예요. 서로 대화할 방법을 찾아야 하고, 첫 번째
[1h 15m 46s] actual like real super intelligent AI to be aligned and somehow be you know care for sentient life, care for people, democratic, one of those, some combination of thereof. And I think this is the condition
[1h 15m 46s] 진짜 초지능 AI가 정렬되고, 지각 있는 생명, 사람을 돌보고 민주적이고 그런 조합이 돼야 해요. 이게 조건이라고 생각해요.
[1h 16m 4s] that everyone should strive for and that’s what SSI is striving for and I think that with time if not already all the other companies will realizing they’re striving towards the same thing and we’ll see. I think that the world will truly change as AI becomes more powerful.
[1h 16m 4s] 모두가 노력해야 하고 SSI가 노력 중이에요. 시간이 지나면 다른 회사들도 같은 걸 향해 노력한다는 걸 깨달을 거예요. AI가 강력해지면 세계가 정말 변할 거예요.
[1h 16m 20s] >> Yeah. And I think a lot of these forecasts will like I think things will be really different and people will be acting really differently.
[1h 16m 20s] >> 네. 많은 예측이 달라질 거예요. 것들이 정말 달라지고 사람들이 정말 다르게 행동할 거예요.
[1h 16m 27s] >> What speaking of forecast what are your forecast to this system you’re describing which can learn as well as a human and subsequently as a result becomes superhuman.
[1h 16m 27s] >> 예측 얘기하니, 당신이 묘사한 인간만큼 잘 배우고 결과로 초인간이 되는 시스템에 대한 예측은 뭐예요.
[1h 16m 39s] >> I think like uh 5 to 20
[1h 16m 39s] >> 5에서 20년쯤요.
[1h 16m 39s] >> 5 to 20 years.
[1h 16m 39s] >> 5에서 20년.
[1h 16m 43s] >> Mhm.
[1h 16m 43s] >> 음.
[1h 16m 45s] >> So I just want to unroll your how you might see the world coming. It’s like we have a couple more years where these other companies are continuing the current approach and it stalls out and stalls out here meaning they earn no more than low hundreds of billions in revenue or how do you think about what stalling out means?
[1h 16m 45s] >> 당신이 세계를 어떻게 보는지 풀어보고 싶어요. 다른 회사들이 현재 접근을 계속하는 몇 년 더 있고, 정체되는데, 수익이 수천억 이하라는 뜻인가요? 정체가 뭔지 어떻게 생각하나요?
[1h 17m 2s] >> Yeah,
[1h 17m 2s] >> 네,
[1h 17m 5s] I think the re I think it could I think it could stall out and I think stalling out will look like it will all look very similar.
[1h 17m 5s] 정체될 수 있다고 생각해요. 정체는 아주 비슷하게 보일 거예요.
[1h 17m 12s] >> Yeah.
[1h 17m 12s] >> 네.
[1h 17m 13s] >> Among all the different companies something like this. I’m not sure because I think I think I think even with I think even I think even with stolen out I think these companies could make a stupendous stupendous revenue maybe not profits because they will be it will be they will need to work hard to differentiate each other from themselves but revenue definitely
[1h 17m 13s] >> 다른 회사들 사이에서 비슷할 거예요. 확실하지 않지만, 정체되더라도 이 회사들이 엄청난 수익을 낼 수 있다고 생각해요. 이익은 아닐 수 있지만, 서로 차별화하기 위해 열심히 해야 하니까요. 수익은 확실해요.
[1h 17m 34s] >> but there’s something in your model implies that the when the correct solution does emerge there will be convergence between all the companies and I’m curious why you think that’s the case
[1h 17m 34s] >> 하지만 당신 모델에서 올바른 해결책이 나오면 모든 회사 간 수렴이 있을 거라는 암시가 있어요. 왜 그렇게 생각하나요?
[1h 17m 44s] >> well I was talking more about converg convergence on their larger strategies.
[1h 17m 44s] >> 더 큰 전략에 대한 수렴을 말했어요.
[1h 17m 47s] >> I think eventual convergence on the technical approach is probably going to happen as well, but I I was alluding to convergence to the larger strategies. What what what exactly is the thing that should be done?
[1h 17m 47s] >> 기술 접근에 대한 최종 수렴도 일어날 거지만, 더 큰 전략에 대한 수렴을 암시했어요. 정확히 뭐 해야 하나요?
[1h 17m 58s] >> I I just want to better understand how you see the future on rolling. So currently we have these different companies and you expect their approach to continue generating revenue. Yes.
[1h 17m 58s] >> 미래가 어떻게 펼쳐지는지 더 잘 이해하고 싶어요. 현재 다른 회사들이 있고 그 접근이 수익을 계속 낼 거라고 기대하나요. 네.
[1h 18m 5s] >> But not get to this humanlike learner.
[1h 18m 5s] >> 하지만 인간 같은 학습자에는 도달하지 않아요.
[1h 18m 7s] >> Yes.
[1h 18m 7s] >> 네.
[1h 18m 9s] >> So now we have these different forks of companies. We have you, we have thinking machines, there’s a bunch of other labs.
[1h 18m 9s] >> 이제 다른 회사 분기들이 있어요. 당신, thinking machines, 다른 랩들.
[1h 18m 14s] >> Yes. and maybe one of them figures out the correct approach
[1h 18m 14s] >> 네. 그중 하나가 올바른 접근을 알아낼 수 있어요.
[1h 18m 18s] >> but then the release of their product makes it clear to other people how to do this thing.
[1h 18m 18s] >> 하지만 제품 출시가 다른 사람들에게 어떻게 하는지 명확히 해줘요.
[1h 18m 22s] >> I think it won’t be clear how to do it thing but it will be clear that something different is possible
[1h 18m 22s] >> 어떻게 하는지는 명확하지 않지만 다른 게 가능하다는 건 명확할 거예요.
[1h 18m 26s] >> right
[1h 18m 26s] >> 맞아요.
[1h 18m 29s] >> and that is information and I think people will will then be trying to figure out how how that’s how that works. I do think though that one of the things that’s that I think you know not addressed here not discussed is that with each increase in the AI’s capabilities I think there will be some kind of changes but I don’t know exactly which ones in how things are being done.
[1h 18m 29s] >> 그게 정보예요. 사람들이 어떻게 작동하는지 알아내려 할 거예요. 여기서 논의되지 않은 건 AI 능력이 증가할 때마다 변화가 있을 거지만 정확히 어떤 건지 모르겠어요.
[1h 18m 57s] So like I think it’s going to be important yet I can’t spell out what that is exactly.
[1h 18m 57s] 중요할 거지만 정확히 뭐라고 말할 수 없어요.
[1h 19m 3s] >> And how how are the by default you would expect the company that has the model company that has that model to be getting all these gains because they have the model that is learning how to do all has the skills and knowledge that it’s building up in the world. What is the reason to think that the benefits of that would be widely distributed and not just end up at whatever model company gets this continuous learning loop going first?
[1h 19m 3s] >> 기본적으로 그 모델을 가진 회사가 모든 이득을 얻을 거라고 기대하나요? 세상에서 쌓는 기술과 지식을 배우는 모델이 있으니까요. 그 이득이 널리 분배되고 첫 번째 연속 학습 루프를 가진 모델 회사에만 가지 않을 이유는 뭐예요?
[1h 19m 27s] >> Like I think that empirically what happen so here here is what I think is going to happen. Number one, I think empirically
[1h 19m 27s] >> 경험적으로 일어날 거라고 생각해요. 첫째, 경험적으로
[1h 19m 34s] when let’s let’s look at let’s look at how things have gone so far with um the AIS of the past. So one company produced an advance and the other company scrambled and produced some competi some some similar things after some amount of time and they started to compete in the market and push their push the prices down
[1h 19m 34s] 과거 AI로 어떻게 됐는지 봐요. 한 회사가 발전을 내고 다른 회사가 서둘러 비슷한 걸 만들고 시장에서 경쟁하고 가격을 낮춰요.
[1h 20m] >> and so I think from the market perspective I think something similar will happen there as well even if someone okay we talking about the good world by the way where
[1h 20m] >> 시장 관점에서 비슷한 일이 일어날 거예요. 좋은 세계에서요.
[1h 20m 12s] what’s the good world. What’s the good world?
[1h 20m 12s] 좋은 세계는 뭐예요?
[1h 20m 19s] Where we have these powerful humanlike learners that are also like and by the way maybe there’s another thing we haven’t discussed on the on the the spec of the super intelligent AI that I think is worth considering is that you make it narrow
[1h 20m 19s] 강력한 인간 같은 학습자가 있고, 논의하지 않은 초지능 AI 스펙은 좁게 만드는 거예요.
[1h 20m 34s] can be useful and narrow at the same time. So you can have lots of narrow super intelligent AIs. But suppose you have many of them and you have some and you have some company that’s producing a lot of um profits from it and then you have another company that comes in and starts to compete and the way the competition is going to work is through specialization.
[1h 20m 34s] 유용하고 좁을 수 있어요. 많은 좁은 초지능 AI를 가질 수 있어요. 많은 게 있고 이익을 내는 회사가 있고 다른 회사가 경쟁을 시작하면 전문화를 통해 작동할 거예요.
[1h 20m 59s] I think what’s going to happen is that the way competition like competition loves specialization and you see it in the market, you see it in evolution as well. So you’re going to have lots of different niches and you’re going to have lots of different companies who are occupying different niches in in this kind of world where you might say yeah like one AI company is really quite a bit better at some area of really complicated economic activity and a different company is better at another area and the third company is really good at litigation and
[1h 20m 59s] 경쟁은 전문화를 사랑하고 시장과 진화에서 봐요. 다른 틈새가 많고 다른 틈새를 차지하는 다른 회사들이 있을 거예요. 한 AI 회사가 복잡한 경제 활동 영역에서 훨씬 낫고 다른 회사는 다른 영역, 세 번째는 소송에 좋아요.
[1h 21m 30s] >> is this contradicted by what human like learning implies is that like it can learn
[1h 21m 30s] >> 인간 같은 학습이 암시하는 것과 모순되나요? 배울 수 있다는 거요.
[1h 21m 36s] >> it can but but you have accumulated learning you have a big investment. You spent a lot of compute to become really really really good really phenomenal at this thing and someone else spent a huge amount of comput and a huge amount of experience to get really really good at some other thing
[1h 21m 36s] >> 할 수 있지만 누적 학습이 있고 큰 투자예요. 이 것에 정말 환상적으로 되기 위해 많은 컴퓨트를 썼고 다른 사람은 다른 것에 정말 좋게 되기 위해 많은 컴퓨트와 경험을 썼어요.
[1h 21m 49s] >> right
[1h 21m 49s] >> 맞아요.
[1h 21m 52s] >> you apply a lot of human learning to get there but now like you you are at this high point where someone else would say look like I don’t want to start learning what you’ve learned to go
[1h 21m 52s] >> 거기 도달하기 위해 많은 인간 학습을 적용하지만 이제 높은 지점에서 다른 사람이 당신이 배운 걸 배우기 시작하고 싶지 않아요.
[1h 22m] >> I guess that would require many different companies to begin at the human like continual learning agent at the same time so that they can start their different research in different branches. But if one company, you know, gets that agent first or gets that learner first, it does then seem like well, you know, they could like if you just think about every single job in the economy, you just have uh instance learning each one seems tractable for a company.
[1h 22m] >> 그건 많은 다른 회사들이 동시에 인간 같은 연속 학습 에이전트를 시작해야 다른 브랜치에서 다른 연구를 시작할 수 있어요. 하지만 한 회사가 먼저 그 에이전트를 얻으면, 경제의 모든 직업을 인스턴스 학습으로 하는 게 회사에게 실현 가능해 보이네요.
[1h 22m 33s] >> Yeah, that’s that’s that’s a valid argument. My my strong intuition is that it’s not how it’s going to go.
[1h 22m 33s] >> 네, 유효한 주장이에요. 제 강한 직관은 그렇게 가지 않을 거예요.
[1h 22m 39s] My strong intuition is that yeah like the argument says it will go this way.
[1h 22m 39s] 제 강한 직관은 주장이 이 방식으로 간다고 하지만.
[1h 22m 41s] >> Yeah.
[1h 22m 41s] >> 네.
[1h 22m 43s] >> But my strong intuition is that it will not go this way.
[1h 22m 43s] >> 하지만 제 강한 직관은 이 방식으로 가지 않을 거예요.
[1h 22m 47s] >> That this is the you know in in theory there is no difference between theory and practice. In practice there is and I think that’s going to be one of those
[1h 22m 47s] >> 이론과 실천 사이에 차이가 없다는 이론이에요. 실천에는 차이가 있고 그게 그중 하나일 거예요.
[1h 22m 55s] >> a lot of people’s models of recursive self-improvement literally explicitly state we will have a million Ilias in a server that are coming in with different ideas and this will lead to a super intelligence emerging very fast. Do you have some intuition about how parallelizable the thing you are doing is? How how what are the gains from making copies of Ilia? I don’t know. I think
[1h 22m 55s] >> 많은 사람들의 재귀적 자가 개선 모델이 서버에 백만 Ilia가 다른 아이디어로 들어와 초지능이 아주 빠르게 나올 거라고 명시적으로 말해요. 당신이 하는 게 얼마나 병렬화 가능하나요? Ilia 복사에서 어떤 이득이 있나요? 모르겠어요. 생각해요.
[1h 23m 19s] I think there’ll definitely be a there’ll be diminishing returns because you want you want people who think differently rather than the same. I think that if they were literal copies of me, I’m not sure how much more incremental value you’d get. I think that
[1h 23m 19s] 확실히 감소 수익이 있을 거예요. 같은 게 아니라 다르게 생각하는 사람을 원하니까요. 제 리터럴 복사라면 추가 가치가 얼마나 될지 모르겠어요. 그게
[1h 23m 34s] but people who think differently that’s what you want.
[1h 23m 34s] 다르게 생각하는 사람들이 원하는 거예요.
[1h 23m 38s] >> Why is it that it’s been if you look at different models even released by totally different companies trained on potentially non-over overlapping data sets. It’s actually crazy how similar LLMs are to each other.
[1h 23m 38s] >> 다른 회사에서 출시한 다른 모델을 보면, 잠재적으로 겹치지 않는 데이터셋으로 훈련됐는데 LLM들이 얼마나 비슷한지 미쳤어요.
[1h 23m 52s] >> Maybe the data sets are not as non-over overlapping as it seems. But there’s there’s some sense there’s like even if an individual human might be less productive than the future AI, maybe there’s something to the fact that human teams have more diversity than teams of AIs might have. But how do we elicit meaningful diversity among AI? So I think just raising the temperature just results in gibberish. I think you want something more like
[1h 23m 52s] >> 데이터셋이 생각만큼 겹치지 않을 수 있어요. 하지만 개인 인간이 미래 AI보다 덜 생산적이라도 인간 팀이 AI 팀보다 다양성이 더 있다는 게 있어요. 하지만 AI 중 의미 있는 다양성을 어떻게 이끌어내죠? 온도 올리는 건 그냥 횡설수설이에요. 더 뭔가
[1h 24m 12s] >> different scientists have different different prejudices or different ideas.
[1h 24m 12s] >> 다른 과학자들이 다른 편견이나 아이디어를 가지는 거예요.
[1h 24m 17s] How do you get that kind of diversity among AI agents? So the reason there has been no diversity I believe is because of pre-training.
[1h 24m 17s] AI 에이전트 중 그런 다양성을 어떻게 얻나요? 다양성이 없던 이유는 사전 훈련 때문이라고 믿어요.
[1h 24m 27s] All the pre-trained models are the same pretty much because the pre-train on the same data. Now RL and postraining is where some differentiation starts to emerge because different people come up with different RL training.
[1h 24m 27s] 모든 사전 훈련 모델은 같은 데이터로 사전 훈련해서 거의 같아요. RL과 포스트 훈련에서 차별화가 시작돼요. 다른 사람들이 다른 RL 훈련을 생각하니까요.
[1h 24m 42s] >> Yeah. And then I’ve heard you hint in the past about selfplay as a way to either get data or match agents to other agents of equivalent intelligence to kick off learning. How should we think about why there’s no public um proposals of this kind of thing working with LLM?
[1h 24m 42s] >> 네. 과거에 셀프플레이를 데이터 얻거나 동등 지능 에이전트 매칭으로 학습 시작하는 방식으로 암시한 걸 들었어요. LLM에서 이런 게 작동하는 공개 제안이 없는 이유를 어떻게 생각하나요?
[1h 25m 2s] >> I would say there are two things to say. I would say that the reason why I thought selfplayful was interesting is because it offered a way to create models using compute only without data.
[1h 25m 2s] >> 두 가지 말할 게 있어요. 셀프플레이가 흥미로웠던 이유는 데이터 없이 컴퓨트만으로 모델을 만드는 방식을 제공했어요.
[1h 25m 15s] Right? And if you think that data is the ultimate bottleneck, then using compute only is very interesting. So that’s what makes it interesting. Now the the thing is
[1h 25m 15s] 맞아요? 데이터가 궁극적 병목이라면 컴퓨트만 사용하는 게 아주 흥미로워요. 그게 흥미롭게 만드는 거예요. 이제
[1h 25m 30s] that selfplay at least the way it was done in the past when you have agents which are somehow compete with each other it’s only good for developing a certain set of skills. It is too narrow.
[1h 25m 30s] 과거 방식의 셀프플레이는 에이전트가 서로 경쟁할 때 특정 스킬 세트 개발에만 좋았어요. 너무 좁아요.
[1h 25m 42s] It’s only good for like negotiation uh conflict certain social skills strategizing that kind of stuff. And so if you care about those skills then selfplay will be useful. Now actually I think that selfplay
[1h 25m 42s] 협상, 갈등, 특정 사회 스킬, 전략화 같은 데만 좋아요. 그 스킬에 신경 쓰면 셀프플레이가 유용해요. 실제로 셀프플레이가
[1h 25m 58s] did find a home but just in a different form in a different form. So things like debate prove a verifier. You have some kind of an LLM as a judge which is also incentivized to find mistakes in your work. You could say this is not exactly selfplay but this is you know a related adversarial setup that people are doing. I believe
[1h 25m 58s] 다른 형태로 자리를 잡았어요. 토론, 증명 검증자 같은 거요. LLM 판사가 작업에서 실수를 찾도록 인센티브를 줘요. 정확히 셀프플레이는 아니지만 관련된 적대적 설정이에요.
[1h 26m 18s] >> and really selfplay is an example of um is a special case of more general like um competition between between agents right the response the natural response to competition is to try to be different and so if you were to put multiple agents and you tell them you know you all need to work on some problem and you’re an agent and you’re inspecting what everyone else is working you’re going to say well if they already taken this approach it’s not clear I should pursue it I should pursue something differentiated
[1h 26m 18s] >> 셀프플레이는 에이전트 간 경쟁의 특별 사례예요. 경쟁에 대한 자연 반응은 다르게 되려는 거예요. 여러 에이전트를 두고 문제를 작업하라고 하면, 다른 사람들이 하는 걸 검사하고 이 접근이 이미 취해졌으면 추구하지 말고 차별화된 걸 추구할 거예요.
[1h 26m 49s] And so I think that something like this could also create an incentive for um a diversity of approaches.
[1h 26m 49s] 이런 게 접근의 다양성에 대한 인센티브를 만들 수 있어요.
[1h 26m 58s] >> Yeah. Um final question,
[1h 26m 58s] >> 네. 마지막 질문,
[1h 27m 1s] what is research taste? You’re obviously the person in the world who is considered to have the best taste in doing research in AI. you were uh the co-author on many of the biggest the biggest things that have happened in the history of deep learning from Alex net to GPT3 to so on what is it that how do you characterize how you come up with these ideas
[1h 27m 1s] 연구 취향은 뭐예요? 당신은 AI 연구에서 최고 취향을 가진 사람으로 여겨져요. AlexNet부터 GPT3까지 딥러닝 역사에서 가장 큰 것들의 공동 저자예요. 어떻게 이런 아이디어를 생각하나요?
[1h 27m 26s] >> I can answer so I can comment on this for myself
[1h 27m 26s] >> 제 자신에 대해 답할 수 있어요.
[1h 27m 29s] >> I think different people do it differently
[1h 27m 29s] >> 다른 사람들이 다르게 한다고 생각해요.
[1h 27m 34s] >> but one thing that um guides me personally is an aesthetic of how AI should be
[1h 27m 34s] >> 하지만 저를 개인적으로 안내하는 건 AI가 어때야 하는지에 대한 미학이에요.
[1h 27m 44s] >> by thinking about how people are but thinking correctly
[1h 27m 44s] >> 사람들이 어때야 하는지 올바르게 생각함으로써요.
[1h 27m 48s] >> like it’s very easy to think about how people are incorrectly but what does it mean to think about people correctly
[1h 27m 48s] >> 사람들이 잘못된 건 생각하기 쉽지만 올바르게 생각한다는 건 뭐예요.
[1h 27m 57s] >> so I’ll give you some examples the idea of the artificial neuron is directly inspired by the brain and it’s a great idea why because you say sure the brain has all these different organs has the faults but the faults probably don’t matter M
[1h 27m 57s] >> 예를 들어 인공 뉴런 아이디어는 뇌에서 직접 영감을 얻었고 훌륭한 아이디어예요. 왜냐하면 뇌에 다른 기관과 결함이 있지만 결함은 아마 중요하지 않아요.
[1h 28m 10s] >> why do we think that the neurons matter? Because there’s many of them. It kind of feels right. So you want the neuron.
[1h 28m 10s] >> 뉴런이 중요하다고 왜 생각하나요? 많으니까요. 느낌이 맞아요. 그래서 뉴런을 원해요.
[1h 28m 14s] >> Yeah.
[1h 28m 14s] >> 네.
[1h 28m 15s] >> You want some kind of local learning rule that will change the connections. You want some local learning rule rule that will change the connections between the neurons,
[1h 28m 15s] >> 연결을 바꾸는 로컬 학습 규칙을 원해요. 뉴런 간 연결을 바꾸는 로컬 학습 규칙을요,
[1h 28m 24s] right? It feels plausible that the brain does it. The idea of the distributed representation,
[1h 28m 24s] 맞아요? 뇌가那样 한다는 게 그럴듯해요. 분산 표현 아이디어,
[1h 28m 30s] the idea that the brain, you know, the brain responds to experience or neural network should learn from experience, not response. The brain learns from experience.
[1h 28m 30s] 뇌가 경험에 반응하거나 신경망이 경험에서 배워야 한다는 아이디어, 반응이 아니라요. 뇌는 경험에서 배워요.
[1h 28m 40s] the neural natural level of experience and you kind of ask yourself is some is something fundamental or not fundamental how things should be
[1h 28m 40s] 신경 자연적 경험 수준이고, 뭔가 근본적인지 아닌지 어떻게 돼야 하는지 물어봐요.
[1h 28m 48s] >> and I think that’s been guiding me a fair bit kind of thinking from multiple angles and looking for almost beauty beauty simplicity ugliness there’s no room for ugliness it’s just beauty simplicity elegance correct inspiration from the brain and all of those things need to be present at the same time and the more they are present the more confident you can be in a top- down belief. And then the top down belief is the thing that sustains you when the experiments contradict you. Because if you just trust the data all the time, well, sometimes you can be doing a correct thing, but there’s a bug. But you don’t know that there is a bug. How can you tell that there is a bug?
[1h 28m 48s] >> 그게 저를 꽤 안내했어요. 여러 각도에서 생각하고 아름다움, 단순함, 추함은 없고 아름다움 단순함 우아함 올바른 뇌 영감이 동시에 있어야 해요. 더 많을수록 top-down 믿음에 더 자신 있어요. top-down 믿음은 실험이 모순될 때 당신을 지탱해요. 데이터를 항상 믿으면 올바른 걸 하다가 버그가 있지만 버그를 모르죠. 버그를 어떻게 알죠?
[1h 29m 25s] >> How do you know if you should keep debugging or you conclude it’s the wrong direction? Well, it’s the top down. Well, how should you can say the things have to be this way? Something like this has to work. Therefore, we got to keep going. That’s the top down. And it’s based on this like multifaceted beauty and inspiration by the brain.
[1h 29m 25s] >> 디버깅을 계속할지 잘못된 방향이라고 결론지을지 어떻게 알아요? top-down이에요. 것들이 이렇게 돼야 한다고 말할 수 있어요? 이런 게 작동해야 해요. 그래서 계속 가야 해요. 그게 top-down이에요. 다면적 아름다움과 뇌 영감에 기반해요.
[1h 29m 43s] >> All right, we’ll leave it there.
[1h 29m 43s] >> 좋아요, 여기서 마칠게요.
[1h 29m 46s] >> Thank you so much.
[1h 29m 46s] >> 정말 감사해요.
[1h 29m 47s] >> Thank you so much.
[1h 29m 47s] >> 정말 감사해요.
[1h 29m 48s] >> All right. Appreciate it. That was great.
[1h 29m 48s] >> 좋아요. 감사해요. 훌륭했어요.
[1h 29m 52s] >> Yeah, I enjoyed it. Yes, me too.
[1h 29m 52s] >> 네, 즐거웠어요. 저도요.