Harvard’s new computer science teacher is a chatbot
“Our own hope is that, through AI, we can eventually approximate a 1:1 teacher:student ratio for every student in CS50, as by providing them with software-based tools that, 24/7, can support their learning at a pace and in a style that works best for them individually,”
“Our own hope is that, through AI, we can eventually approximate a 1:1 teacher:student ratio for every student in CS50, as by providing them with software-based tools that, 24/7, can support their learning at a pace and in a style that works best for them individually,”
👍26😈6🤯4❤1😁1
Once again trying to deny the 1st Bitter Lesson: “The bigger-is-better approach to AI is running out of road”
“This gigantism is becoming a problem. If Epoch ai’s ten-monthly doubling figure is right, then training costs could exceed a billion dollars by 2026—assuming, that is, models do not run out of data first.”
= Combustion engines won’t overtake horses, because that would mean that the car industry might be investing over a billion dollars in creating cars soon! Obviously no way that can happen!
Nonsense, not even a real argument.
”An analysis published in October 2022 forecast that the stock of high-quality text for training may well be exhausted around the same time.”
= Training will hit a brick wall because we’re running out of text! I.e. It’s impossible to train LLMs without human-made training data.
Wrong. Already thorougly disproven since long before LLMs even existed, previously with MuZero & EfficientZero, and more recently with LLMs showing great success in learning from their own syntheticly generated training data. Self-supervised training data creation is not only theoretically possible but already widely done.
“And even once the training is complete, actually using the resulting model can be expensive as well. The bigger the model, the more it costs to run. Earlier this year Morgan Stanley, a bank, guessed that, were half of Google’s searches to be handled by a current gpt-style program, it could cost the firm an additional $6bn a year.”
= We can’t create huge models that, because they’re expensive to run.
No, the opposite, surprisingly, and for reasons that are not yet fully understood. Emperically, and despite great effort trying to get around this, turns out the only way to get cheap-to-run powerful models is to first train a gigantic, extremely over-parameterized model, and then after dramatically prune that down into a smaller cheaper model.
Economist article trying to deny the 1st Bitter Lesson
“This gigantism is becoming a problem. If Epoch ai’s ten-monthly doubling figure is right, then training costs could exceed a billion dollars by 2026—assuming, that is, models do not run out of data first.”
= Combustion engines won’t overtake horses, because that would mean that the car industry might be investing over a billion dollars in creating cars soon! Obviously no way that can happen!
Nonsense, not even a real argument.
”An analysis published in October 2022 forecast that the stock of high-quality text for training may well be exhausted around the same time.”
= Training will hit a brick wall because we’re running out of text! I.e. It’s impossible to train LLMs without human-made training data.
Wrong. Already thorougly disproven since long before LLMs even existed, previously with MuZero & EfficientZero, and more recently with LLMs showing great success in learning from their own syntheticly generated training data. Self-supervised training data creation is not only theoretically possible but already widely done.
“And even once the training is complete, actually using the resulting model can be expensive as well. The bigger the model, the more it costs to run. Earlier this year Morgan Stanley, a bank, guessed that, were half of Google’s searches to be handled by a current gpt-style program, it could cost the firm an additional $6bn a year.”
= We can’t create huge models that, because they’re expensive to run.
No, the opposite, surprisingly, and for reasons that are not yet fully understood. Emperically, and despite great effort trying to get around this, turns out the only way to get cheap-to-run powerful models is to first train a gigantic, extremely over-parameterized model, and then after dramatically prune that down into a smaller cheaper model.
Economist article trying to deny the 1st Bitter Lesson
🔥5👍4😱2❤1💯1
Misleading chart used by The Economist to try to deny the 1st Bitter Lesson
Looks like it’s hitting a wall, and couldn’t possibly go much higher, right?
No.
ML training entered a new era.
Why?
Because, like relays and vacuum tubes and transistors at their start, LLMs suddenly reached minimum economic viability. They reached the point where their marginal productivity surpassed their marginal cost.
New era.
2018 OpenAI article explaining the new era
Looks like it’s hitting a wall, and couldn’t possibly go much higher, right?
No.
ML training entered a new era.
Why?
Because, like relays and vacuum tubes and transistors at their start, LLMs suddenly reached minimum economic viability. They reached the point where their marginal productivity surpassed their marginal cost.
New era.
2018 OpenAI article explaining the new era
👍5🤯2❤1🔥1👏1
Demolishing the “We’re hitting a brick wall because we’re running out of human training data” theory - LARGE LANGUAGE MODELS CAN SELF-IMPROVE, Oct 2022
“We show that it is possible for the LLM to self-improve even on its own generated questions and few-shot Chain-of-Thought prompts.”
(Numerous subsequent papers further strongly confirming this.)
Paper
“We show that it is possible for the LLM to self-improve even on its own generated questions and few-shot Chain-of-Thought prompts.”
(Numerous subsequent papers further strongly confirming this.)
Paper
👍12🤬2❤1💯1
Using GPT4 to Make an AI Bartender App
“Last time I made an app it took almost 6 months and nearly $10K in art costs. This time we built the app in about a week and then took a couple of weeks to test and refine it. We used GPT4 to help build the app, which is based on the OpenAI’s GPT API. We also used AI tools for all of the graphics, from the icon to the bartender animations, bar backgrounds, and voices. In this post we’ll touch on the tools used to create the app. The cost of creating the app has been super-low.”
Article
“Last time I made an app it took almost 6 months and nearly $10K in art costs. This time we built the app in about a week and then took a couple of weeks to test and refine it. We used GPT4 to help build the app, which is based on the OpenAI’s GPT API. We also used AI tools for all of the graphics, from the icon to the bartender animations, bar backgrounds, and voices. In this post we’ll touch on the tools used to create the app. The cost of creating the app has been super-low.”
Article
👍24❤4👨💻2
retroactively applying generational labels backward through time
Founding Fathers are Generation L&M
Founding Fathers are Generation L&M
❤8👍2