Bitter Lesson: Morgan Stanley estimates GPT-5 is currently being trained on $225 million dollars worth of NVIDIA GPUs
Morgan Stanley Note
Morgan Stanley Note
🤯14❤2🔥1😱1
Bitter Lesson: Increase in training costs for large AI models has been consistently OUTPACING Moore’s Law ever since 2011
Cost of training foundation models has been steadily, exponentially increasing ever since 2011, with shows no signs of slowing, possibly ever.
Let's spell it out:
MOORE'S LAW CANNOT BRING FOUNDATION MODEL TRAINING COSTS DOWN
MOORE’S LAW CAN’T CATCH UP
INCREASED EFFICIENCY DUE TO MOORE’S LAW IS BRINGING DOWN COSTS AT A RATE MUCH SLOWER THAN MODEL TRAINING NEEDS ARE DRIVING COSTS UP
MOORE’S LAW < MODEL SIZE GROWTH
Clear now?
Article
Cost of training foundation models has been steadily, exponentially increasing ever since 2011, with shows no signs of slowing, possibly ever.
Let's spell it out:
MOORE'S LAW CANNOT BRING FOUNDATION MODEL TRAINING COSTS DOWN
MOORE’S LAW CAN’T CATCH UP
INCREASED EFFICIENCY DUE TO MOORE’S LAW IS BRINGING DOWN COSTS AT A RATE MUCH SLOWER THAN MODEL TRAINING NEEDS ARE DRIVING COSTS UP
MOORE’S LAW < MODEL SIZE GROWTH
Clear now?
Article
👍9🤯3❤2🔥1💯1
Is the growth of LLM model sizes on: (A) an S-CURVE growth path, i.e. LLM model sizes will soon plateau and stop growing exponentially? — Or, (B) on a PERPETUAL exponential growth path = LLM model sizes will keep growing at an exponential rate, forever?
Anonymous Poll
54%
S-CURVE GROWTH: the pace in growth of LLM model sizes will soon slow down, no longer be exponential
46%
PERPETUAL GROWTH: the pace in growth of LLM model sizes will never slow down, always be exponential
👏6👍4❤2🗿2🔥1
In the real world, every
exponentially-growing process eventually saturates.
(Eventually = Within 1000 years.) (Exponential = with a non-trivial growth rate.)
exponentially-growing process eventually saturates.
(Eventually = Within 1000 years.) (Exponential = with a non-trivial growth rate.)
Anonymous Poll
63%
TRUE: Every exponential process eventually SATURATES. IMPOSSIBLE not to, even in principle.
37%
FALSE: In principle POSSIBLE for exponential process to continue without bound, WITHOUT SATURATING.
❤1
Forwarded from Chat GPT
Moore’s Law is not Dead, but Gordon Moore, RIP (aged 94, January 3, 1929 – March 24, 2023)
Moore’s Law’s has been rewritten by journalists throughout the decades. Few today realize that Moore’s Law’s original stated formulation had nothing to do with clock speeds or circuit sizes, but rather:
“complexity for minimum component costs”
Now, “minimum component” has changed many times throughout the decades, as indicted by the vague wording in this original formulation, so the best general translation of this concept, in a way that ignores the particular component type and further solidifies the meaning of complexity, is then:
“calulations / second @ $1 spent”
In 1965, when Moore’s Law was coined, it had already been holding strong for ~65 years, across many technology regimes,
(1) Rooms full of human “computers”
(2) Mechanical like Babbage’s
(3) Relays
(4) Vacuum Tubes
(5) Discrete Transistors
— Moore’s Law Coined —
(6) Integrated Circuits
(7) Quantum?
Since then, in the 58 years since Moore’s Law was coined, it has continued to hold, near-perfectly.
In the future, Moore’s law will continue to enter new regimes, as it already has many times in the past, perhaps quantum next, but the law will never die.
It has been said that "The number of people predicting the death of Moore’s Law doubles every two years."
I say "The number of people trying to redefine Moore's Law from its original intent doubles every two years."
Moore's Law slowdown was always a willful lie, told by journalists and attention-seekers fully-aware of their lies.
Few willing to grasp the consequences of unending exponential growth.
Moore’s Law’s has been rewritten by journalists throughout the decades. Few today realize that Moore’s Law’s original stated formulation had nothing to do with clock speeds or circuit sizes, but rather:
“complexity for minimum component costs”
Now, “minimum component” has changed many times throughout the decades, as indicted by the vague wording in this original formulation, so the best general translation of this concept, in a way that ignores the particular component type and further solidifies the meaning of complexity, is then:
“calulations / second @ $1 spent”
In 1965, when Moore’s Law was coined, it had already been holding strong for ~65 years, across many technology regimes,
(1) Rooms full of human “computers”
(2) Mechanical like Babbage’s
(3) Relays
(4) Vacuum Tubes
(5) Discrete Transistors
— Moore’s Law Coined —
(6) Integrated Circuits
(7) Quantum?
Since then, in the 58 years since Moore’s Law was coined, it has continued to hold, near-perfectly.
In the future, Moore’s law will continue to enter new regimes, as it already has many times in the past, perhaps quantum next, but the law will never die.
It has been said that "The number of people predicting the death of Moore’s Law doubles every two years."
I say "The number of people trying to redefine Moore's Law from its original intent doubles every two years."
Moore's Law slowdown was always a willful lie, told by journalists and attention-seekers fully-aware of their lies.
Few willing to grasp the consequences of unending exponential growth.
👍4❤1
Forwarded from Chat GPT
First 3 words of article that coined Moore’s Law — “With unit cost falling”
Concluding sentence of the introduction - “Machines similar to those in existence today will be built at lower costs and with faster turnaround.”
Moore’s Law was always, first and foremost, about
i.e.
All else is journalist lies.
Gordon Moore’s 1965 Article Coining Moore’s Law
Concluding sentence of the introduction - “Machines similar to those in existence today will be built at lower costs and with faster turnaround.”
Moore’s Law was always, first and foremost, about
Min Cost / Computei.e.
Max Compute / CostAll else is journalist lies.
Gordon Moore’s 1965 Article Coining Moore’s Law
👍6❤1
False Impossibilities: ChatGPT weighs in on unbounded growth in reality
ChatGPT, Is it impossible, even in principle, in the real world, for an exponentially-growing process to continue growing without bound and without saturating, for at least 1000 years? Answer yes or no.
If the answer is yes, that it is impossible, then state which principle of nature, specifically, is the principle that prevents this? Do not say vague concepts. Only say the formal name(s) for formal principles, if any.
If the answer is no, that exponential growth without saturation is in fact possible, then why is it that people might wrongly assume that exponential growth is impossible, even in principle?
Do not answer anything other than the questions asked here.
ChatGPT, Is it impossible, even in principle, in the real world, for an exponentially-growing process to continue growing without bound and without saturating, for at least 1000 years? Answer yes or no.
If the answer is yes, that it is impossible, then state which principle of nature, specifically, is the principle that prevents this? Do not say vague concepts. Only say the formal name(s) for formal principles, if any.
If the answer is no, that exponential growth without saturation is in fact possible, then why is it that people might wrongly assume that exponential growth is impossible, even in principle?
Do not answer anything other than the questions asked here.
💯6❤1👏1
False Impossibilities: 1969, Minsky published a book which strongly implied that neural networks as a whole were dead-end, implying it was “impossible” for neural networks to solve even the simple XOR problem.
So many morons believed these obviously-wrong false-impossibility “proofs”, that this is widely cited being the event that crashed the entire AI field, plunging it into decades of AI winter.
Same then, as now, morons continue to fall for false impossibility claims.
Article
So many morons believed these obviously-wrong false-impossibility “proofs”, that this is widely cited being the event that crashed the entire AI field, plunging it into decades of AI winter.
Same then, as now, morons continue to fall for false impossibility claims.
Article
👏5👍2❤1
Media is too big
VIEW IN TELEGRAM
False Impossibilities: Lighthill Debate, 1973 — Lighthill implied that “combinatoral explosion” problem makes AI effectively impossible, even in principle, and even with future advancements fo Moore's Law
Likewise, he claimed that methods bypassing these “combinatorial explosions”, what he called “heuristics”, would “depend critically
on data derived through the use of human intelligence” and therefore end up useless on most problems, due to the size of the combinatorial explosion.
This Lighthill Debate is widely seen as having collapsed funding for AI in the UK.
Problem?
His impossibility claims were lies, false impossibilities.
Was obviously wrong to anyone with a brain at the time, later solidly emperically proven wrong, on particular examples he cited, via e.g. MuZero.
False impossibility claims, wrecking progress in countless fields for centuries.
Could it happen again?
Already starting, right here.
Clarke: If an elderly but distinguished scientist says that something is possible, he is almost certainly right; but if he says that it is impossible, he is very probably wrong.
Beware the false impossibility proof, wrecker of fields.
Lighthill Report Transcript
Likewise, he claimed that methods bypassing these “combinatorial explosions”, what he called “heuristics”, would “depend critically
on data derived through the use of human intelligence” and therefore end up useless on most problems, due to the size of the combinatorial explosion.
This Lighthill Debate is widely seen as having collapsed funding for AI in the UK.
Problem?
His impossibility claims were lies, false impossibilities.
Was obviously wrong to anyone with a brain at the time, later solidly emperically proven wrong, on particular examples he cited, via e.g. MuZero.
False impossibility claims, wrecking progress in countless fields for centuries.
Could it happen again?
Already starting, right here.
Clarke: If an elderly but distinguished scientist says that something is possible, he is almost certainly right; but if he says that it is impossible, he is very probably wrong.
Beware the false impossibility proof, wrecker of fields.
Lighthill Report Transcript
👍8❤1👏1
False impossibilities: Guess what else the 1972 Lighthill Report mentioned ~6 times — as the the reason that AI would never be able to do the things… that now can do today?
Machines lacking “emotion.”
“Most robots are designed from the outset to operate in a world as like as possible to the conventional child s world as seen by a man they play games they do puzzles they build towers of bricks they recognise pictures in drawing books (“ bear on rug with ball” ) although the rich emotional character of the child s world is totally absent”
“This is partly because chess is a complic ated enough game so that in a contest between a computer and a human player the computer s advantages of being able to calculate reliably at a speed several orders of magnitude faster need by no means be decisive the number of possible positions being incomparably greater and so there is real interest in whether or not they are outweighed by the human player s pattern- recognition ability exibility of approach learning capacity and emotional drive to win.”
“It is a truism that human beings who are very strong intellectually but weak in emotional drives and emotional relationships
are singularly ine ffective in the world at large. Valuable results flow from the integration of intellectual ability with the capacity to feel and to relate to other people until this integra tion happens problem solving is no good because there is no way of seeing which are the right problems.”
Sadder part is how many people still fall for this nonsense.
Part I Articial Intelligence - A general survey by Sir James Lighthill
Machines lacking “emotion.”
“Most robots are designed from the outset to operate in a world as like as possible to the conventional child s world as seen by a man they play games they do puzzles they build towers of bricks they recognise pictures in drawing books (“ bear on rug with ball” ) although the rich emotional character of the child s world is totally absent”
“This is partly because chess is a complic ated enough game so that in a contest between a computer and a human player the computer s advantages of being able to calculate reliably at a speed several orders of magnitude faster need by no means be decisive the number of possible positions being incomparably greater and so there is real interest in whether or not they are outweighed by the human player s pattern- recognition ability exibility of approach learning capacity and emotional drive to win.”
“It is a truism that human beings who are very strong intellectually but weak in emotional drives and emotional relationships
are singularly ine ffective in the world at large. Valuable results flow from the integration of intellectual ability with the capacity to feel and to relate to other people until this integra tion happens problem solving is no good because there is no way of seeing which are the right problems.”
Sadder part is how many people still fall for this nonsense.
Part I Articial Intelligence - A general survey by Sir James Lighthill
😁3❤2👏1🤣1