Submitted by googoobah t3_11cku2n in Futurology
Cryptizard t1_ja4y67r wrote
Reply to comment by Mason-B in So what should we do? by googoobah
You are really not following what is going on, or else you have closed your mind so much you can't process it. 90 years for general intelligence? Buddy, 30 years ago we didn't even have the internet. Or cell phones. Computers were .001% as fast as they are now. And technological progress speeds up, not slows down.
I don't think it is coming tomorrow or anything, but look at current AI models and tell me it will take 3 more internets worth of advancement to make them as smart as a human. Humans aren't even that smart!
>Skipping the obvious answer of "programmers will be the last people to be programmed out of a job."
This is a really terrible take. Programmers are going to be among the first to be replaced, or at least most of them. We already have AI models doing a big chunk of programming. Programming is just a language problem, and LLMs have proven to be really good at language.
Background_Agent551 t1_ja5hzot wrote
I’m sorry, but I’ve seen your comments through this thread and the only close-minded person I’ve seen is you.
Cryptizard t1_ja5ik9t wrote
Cool comment. Excellent details to back up your assertion lol
[deleted] t1_ja5j1yk wrote
[removed]
Cryptizard t1_ja5jkba wrote
>it’s at least 60 years into the future.
With no argument, cool cool.
>We’re not in a courtroom, I don’t need to cite evidence
And I don't need anything to call you a dumb piece of shit with his head stuck up his ass. Miss me with your bullshit please.
Background_Agent551 t1_ja5js54 wrote
I’m the one with no argument, are you trolling or is someone in real life really this fucking dense?
[deleted] t1_ja5md9m wrote
[removed]
Background_Agent551 t1_ja5mjj8 wrote
Your argument is that the A.I revolution is around the corner, do you have any sources or evidence about this opinion or are you just talking out your own ass like you’re leading me to believe?
Le_Corporal t1_ja5pybj wrote
I dont think anyone can be certain of how long it will take for something to develop, if theres anything to learn from the past, its that its very difficult to predict the future
Mason-B t1_ja5xgvt wrote
> You are really not following what is going on, or else you have closed your mind so much you can't process it. 90 years for general intelligence? Buddy, 30 years ago we didn't even have the internet. Or cell phones. Computers were .001% as fast as they are now. And technological progress speeds up, not slows down.
Early internet was 40 years ago if not longer. Which is about the same time as we had wireless phones. Further, we've had technologies like global instant communications, or hyperlinked knowledge bases, for at least a hundred years. I don't think technology moves quite as rapidly as you are imagining here. But even if they do...
Computer speeds can keep doubling but at a certain point we hit atomic limits for silicon transistors. And fundamentally we have a simple problem that AI today is at least seven orders of magnitude away from the needed efficiency to even reach parity with biological intelligence. Which, by the way is beyond that atomic limit. But even if it wasn't, it would take 70 years at a minimum if the computer industry managed to double at it's current speeds (which has been slowing down for decades, take that into account and it's closer to 90). Modern AI models are effectively cockroach brain, or thin slices of larger brains, trained on very specific problems.
And fundamentally, we are at the end of this boom in AI tech. We are at the end of the sigmoid curve of adoption. Things like deep fakes and early drafts of mid journey and GPT were being developed 15 years ago when the latest DNN breakthroughs were made. In that time the technology has matured, it's been put into easily accessible libraries, and engineering work has gone into getting the technology to efficiently run on cutting edge hardware (re: GPUs). Google even got specialized chips with the latest fab cycle made for it to push the envelope of what is possible. And now we are here, at the end of the curve, where it's being put into people's hands and being made generally accessible.
But there is no follow up. There are no new theory breakthroughs in the last decade, there is no more easy hardware performance gains to be grabbed. All the things that were imagined being possible with the tech 15 years ago is now today here. With nothing much new on the horizon. And from an algorithmic complexity standpoint, every doubling of performance you put into these models will give you a sub-linear improvement in output. So as computers get twice as fast, and the companies spend a year training them instead of 6 months, and they buy twice as many servers with their capital to train on, we'll get maybe a 30% improvement. I cannot emphasize enough how in the last 10 years we have gone from researchers running python code on their one computer for a week to teams of engineers running hand tuned assembly on clusters of bespoke hardware for months to get here. That's easily 20 or 30 doubling of performance increases that we cannot repeat, meanwhile hardware doubles every 18 months (more like 2 years)?
But we are at the end of this curve, not the beginning. Just because you haven't been reading AI theory papers for the last 20+ years, just because you have not been paying attention does not make this technology novel or surprising or somehow going to hit big strides. Speaking of, if you had paid attention, you would see that we are coming up on an AI winter scenario again. Probably around 2025/2026.
> We already have AI models doing a big chunk of programming. Programming is just a language problem, and LLMs have proven to be really good at language.
Tell me you don't do programming or computer science without telling me. Sure programming is just a language problem, like rocket science is just a math problem, or like a cow is actually an idealized point in space. What this skips is the HUGE gap in the details that actually allows anything to actually work. Yes it can write code to do a thing, but actually designing a good solution, actually problem solving edge cases, or debugging a specific error? None of that (and tons more) is covered by these models yet. Not even close.
So much of the complexity in non-trivial code bases is in very large language inputs. Like "crash the AI" sized inputs. So yes, we could train the AI on the language model and the code base, but how does that get us to the next step, that just makes it a domain expert in what already exists.
Using these extant models to program something that is even "two steps" complex usually just fails to get anywhere. Yes it can download a website, and yes it can put data in a data base (besides, these are the rote tasks that most programmers have already abstracted into libraries anyway). But it can't put the two together, even if the specification for the data usually wasn't so large that it couldn't even comprehend it anyway. This isn't just something that can be overcome incrementally, it's the whole goddamned ball game.
Going back to the earlier point of history of technology. We are at the end of this technology, we don't have a way forward to solve these fundamental issues with the technology without going back to the drawing board. Perhaps by integrating more classic techniques we will find a path forward. But like last time, that will take an AI winter to reset the space.
Cryptizard t1_ja5yk73 wrote
It’s astonishing how you make like a dozen points and almost every single one of them is flat wrong. I don’t want to argue with you since it seems like you are not open to new information, but I will say that Moore’s law has not been slowing down for decades, transformer/attention models are explicitly a new theory that has made the current wave of AI possible and was not like anything that was done before, and I am a computer science professor and I program all the time an am well-versed in what AI can and can’t do at the moment.
Mason-B t1_ja5zzwq wrote
> It’s astonishing how you make like a dozen points and almost every single one of them is flat wrong.
Hmm, I think I'll just quote you: "Cool comment. Excellent details to back up your assertion lol"
> but I will say that Moore’s law has not been slowing down for decades
Links story showing Moore's law is two years instead of the original assertion of 18 months. Cool story bro.
> transformer/attention models are explicitly a new theory that has made the current wave of AI possible
Which is over 6 years old from publication now, with some lead time before that. I may have rounded up to decade, but still, no new theory on the horizon.
> and I am a computer science professor and I program all the time an am well-versed in what AI can and can’t do at the moment.
After getting my graduate degree I got distracted by my better paying side gig in the industry. But more or less, same.
Cryptizard t1_ja618dl wrote
You said moores law has been slowing for decades and would be the main bottleneck for the future, I show you actual evidence that it has only very slightly started to slow since 2010 and somehow now that was your argument the whole time lol.
You say that current AI is the same as it was 15 years ago (I am using your exact language here), I point out that transformers are very new and different, you say oh but those are 5 years old.
This is the definition of moving the goalposts. Like I said, you are not interested in an actual discussion, you want to stroke your ego. Well, you aren’t as smart as you think friend. Bye bye.
Mason-B t1_ja61gpk wrote
Getting this in before you try and block me again to snipe your responses in.
> You say that current AI is the same as it was 15 years ago (I am using your exact language here)
No, my exact language was:
> > All the things that were imagined being possible with the tech 15 years ago is now today here.
Also (edit),
> You said moores law has been slowing for decades and would be the main bottleneck for the future, I show you actual evidence that it has only very slightly started to slow since 2010 and somehow now that was your argument the whole time lol.
I said Moore's law in either incarnation would be a bottle neck, but it is also slowing (in a parenthetical no less!). Over long periods of time the slowing trend becomes obvious.
> > it would take 70 years at a minimum if the computer industry managed to double at it's current speeds (which has been slowing down for decades, take that into account and it's closer to 90)
It's like you are trying to find enough nitpicks to justify stopping arguing over a technicality.
Cryptizard t1_ja61rh1 wrote
You said “there are no new theory breakthroughs in the last decade.”
Mason-B t1_ja624u2 wrote
I already admitted I rounded up the 6 year old by publication date paper there. I should have said half a decade.
Do you have a substantive counter point instead of nitpicks? (In something I wrote from memory in 30 minutes?)
Because I would very much enjoy being wrong on this.
Cryptizard t1_ja648ox wrote
Yes like I said everything you wrote is wrong. Moore’s law still has a lot of time left on it. There are a lot of new advances in ML/AI. You ignore the fact that we have seen a repeated pattern where a gigantic model comes out that can do thing X and then in the next 6-12 months someone else comes out with a compact model 20-50x smaller that can do the same thing. It happened with DALLE/Stable Diffusion, it happened with GPT/Chinchilla it happened with LLaMa. This is an additional scaling factor that provides another source of advancement.
You ignore the fact that there are plenty of models that are not LLMs making progress on different tasks. Some, like Gato, are generalist AIs that can do hundreds of different complex tasks.
I can’t find any reference that we are 7 orders of magnitude away from the complexity of a brain. We have neural networks with more parameters than there are neurons in a brain. A lot more. Biological neurons encode more than an artificial neuron, but not a million times more.
The rate of published AI research is rising literally exponentially. Another factor that accelerates progress.
I don’t care what you have written about programming, the statistics say that it can write more than 50% of code that people write TODAY. It will only get better.
Mason-B t1_ja67eyi wrote
> You ignore the fact that we have seen a repeated pattern where a gigantic model comes out that can do thing X and then in the next 6-12 months someone else comes out with a compact model 20-50x smaller that can do the same thing. It happened with DALLE/Stable Diffusion
DALLE/Stable Diffusion is 3.5 Billion to 900 million. Which is x4. But the cost of that is the training size. Millions of source images versus billions. Again, we are pushing the boundaries of what is possible in ways that cannot be repeated. With a 3 orders of magnitude more training data we got a 4x reduction in efficiency (assuming no other improvements played a role in that). I don't think we'll be able to find 5 trillion worthwhile images to train on anytime soon.
But it is a good point that I missed it, I'll be sure to include it in my rant about "reasons we are hitting the limits of easy gains"
> You ignore the fact that there are plenty of models that are not LLMs making progress on different tasks. Some, like Gato, are generalist AIs that can do hundreds of different complex tasks.
If you read the paper they discuss the glaring limitation I mentioned above. Limited attention span, limited context length, with a single image being significant fraction (~40%) of the entire model's context. That's the whole ball game. They also point out the fundamental limit of their design here is the known one: quadratic scaling to increase context. Same issues of fundamental design here.
I don't see your point here. I never claimed we can't make generalist AIs with these techniques.
> I can’t find any reference that we are 7 orders of magnitude away from the complexity of a brain. We have neural networks with more parameters than there are neurons in a brain. A lot more. Biological neurons encode more than an artificial neuron, but not a million times more.
Depends how you interpret it. Mostly I am basing these numbers on super computer efficiency (for the silicon side) and the lower bound of estimates made by CMU about what human brains operate at. Which takes into account things like hormones and other brain chemicals acting as part of a neuron's behavior. Which yes, does get us to a million times more on the lower bound.
If you want to get into it there are other issues like network density and the delay of transmission between neurons that we also aren't anywhere close to at similar magnitudes. And there is the raw physics angle about how much waste heat the different computations generate at a similar magnitude difference.
To say nothing of the mutability problem.
> The rate of published AI research is rising literally exponentially. Another factor that accelerates progress.
Exact same thing happened with the boom right before AI winter in the 80s. And also stock market booms. In both cases, right before the hype crashes and burns.
> I don’t care what you have written about programming, the statistics say that it can write more than 50% of code that people write TODAY. It will only get better.
The github statistics being put out by the for-profit company that made and is trying to sell the model? I'm sure they are very reliable and reproducible (/s).
Also can write the code is far different than would. My quantum computer can solve problems first try doesn't mean that it will. While I'm sure it can predict a lot of what people write (I am even willing to agree to 50%) the actual problem is choosing which prediction to actually write. Again to say nothing of the other 50% which is likely where the broader context is.
And that lack of context is the fundamental problem. There is a limit to how much better it can get without a radical change to our methodology or decades more of hardware advancements.
Cryptizard t1_ja69a0b wrote
It seems to come down to the fact that you think AI researchers are clowns and won’t be able to fix any of these extremely obvious problems in the near future. For example, there are already methods to break the quadratic bottleneck of attention.
Just two weeks ago there was a paper that compresses GPT-3 to1/4 the size. That’s two orders of magnitude in one paper, let alone 10 years. Your pessimism just makes no sense in light of what we have seen.
Mason-B t1_ja6cwsg wrote
> It seems to come down to the fact that you think AI researchers are clowns and won’t be able to fix any of these extremely obvious problems in the near future.
No, I think they have forgotten the lessons of the last AI winter. That despite their best intentions to fix obvious problems, many of them will turn out to be intractable for decades.
Fundamentally what DNNs are is a very useful mechanism of optimization algorithm approximation over large domains. We know how that class of algorithms responds to exponential increases in computational power (and re, efficiency), more accurate approximations at a sub linear rate.
> For example, there are already methods to break the quadratic bottleneck of attention.
The paper itself says it's unclear if it works for larger datasets. But this group of techniques is fun because it's a trade off of accuracy for efficiency. Which yea, that's also an option. I'd even bet if you graphed the efficiency gain against the loss of accuracy across enough models and sizes it would match up.
> That’s two orders of magnitude in one paper, let alone 10 years.
Uh what now? Two doublings is not even half of one order of magnitude. Yes they may have compressed them by two orders of magnitude but having to decode them eats up most of those gains. Compression is not going to get enough gains on it's own, even if you get specialized hardware to remove a part of the decompression cost.
And left unanalyzed is how much of that comes from getting the entire model on a single device.
Fundamentally I think you are overlooking the fact that research into this topics has been making 2x, 4x gains all the time but a lot of those gains are being done in ways we can't repeat. We can't further compress already well compressed stuff for example. At some point soon (2-3 years) we are going to hit a wall where all we have is hardware gains.
Viewing a single comment thread. View all comments