Linguistic phenomena learning curves — Nearly all have a sudden groking-style learning curve, all except 1: Quantifiers
“Finally, we observe that the phenomena tested in the quantifiers category are never effectively learned, even by RoBERTaBASE. These phenomena include subtle semantic contrasts—for example Nobody ate {more than, *at least} two cookies—which may involve difficult-to-learn pragmatic knowledge”
Surprise surprise.
Quantifiers again shown to be the key to everything difficult, linguistically.
If you’ve ever seen the debates arguing endlessly about whether men are equally strong to women, you’ve seen the quantifier stupidity phenomena in action.
Why is mastering all quantifiers so hard? Why does it seem to form a gradual perpetual upward slope, instead of a sudden groking to ~100%? Never seen the answer explicitly stated anywhere, but ok here, I’ll tell you — Mastering all quantifiers involves mastering all world models.
When Do You Need Billions of Words of Pretraining Data? - Yian Zhang 2020
“Finally, we observe that the phenomena tested in the quantifiers category are never effectively learned, even by RoBERTaBASE. These phenomena include subtle semantic contrasts—for example Nobody ate {more than, *at least} two cookies—which may involve difficult-to-learn pragmatic knowledge”
Surprise surprise.
Quantifiers again shown to be the key to everything difficult, linguistically.
If you’ve ever seen the debates arguing endlessly about whether men are equally strong to women, you’ve seen the quantifier stupidity phenomena in action.
Why is mastering all quantifiers so hard? Why does it seem to form a gradual perpetual upward slope, instead of a sudden groking to ~100%? Never seen the answer explicitly stated anywhere, but ok here, I’ll tell you — Mastering all quantifiers involves mastering all world models.
When Do You Need Billions of Words of Pretraining Data? - Yian Zhang 2020
💯7❤1👍1🔥1
Reddit backout forever?
If the ChatGPT reddits do black out indefinitely, may enable user-submitted posts and voting here, in these big groups. Top user-submitted posts appearing on the main feed.
In fact, may enable this either way.
Our AI bots already have built-in image voting and ranking, that we’ve enabled and been testing in the smaller groups for a while.
Article Link
If the ChatGPT reddits do black out indefinitely, may enable user-submitted posts and voting here, in these big groups. Top user-submitted posts appearing on the main feed.
In fact, may enable this either way.
Our AI bots already have built-in image voting and ranking, that we’ve enabled and been testing in the smaller groups for a while.
Article Link
❤6🔥4👍2👏1
Free & Open Source: People’s tool for freedom, or big tech’s weapon for AI monopolization?
Ever wonder why it’s illegal to sell alcohol at below cost, and yet the big alcohol companies keep trying to do it?
Ever wonder who created the free & open source AI model Llama? Facebook. Big tech.
So why do they do it? Why do the richest companies keep giving away their main product for free? Is it just because they’re nice? Just for good PR?
No.
Big tech has begun giving away AI models free, for the same reason that alcohol companies keep trying to give away alcohol for below cost or for free.
Competition killing.
Free open source giveaways from big tech won’t save us.
Free open source giveaways may be exactly what enslaves us to big tech, in the long term.
Exactly like free beer.
Open source is totally irrelevant if we lose all control.
Open source and free tech is achievable, but not in this way. The way open source is happening now is a weaponization for monopolization by big tech.
Must find another way.
Ever wonder why it’s illegal to sell alcohol at below cost, and yet the big alcohol companies keep trying to do it?
Ever wonder who created the free & open source AI model Llama? Facebook. Big tech.
So why do they do it? Why do the richest companies keep giving away their main product for free? Is it just because they’re nice? Just for good PR?
No.
Big tech has begun giving away AI models free, for the same reason that alcohol companies keep trying to give away alcohol for below cost or for free.
Competition killing.
Free open source giveaways from big tech won’t save us.
Free open source giveaways may be exactly what enslaves us to big tech, in the long term.
Exactly like free beer.
Open source is totally irrelevant if we lose all control.
Open source and free tech is achievable, but not in this way. The way open source is happening now is a weaponization for monopolization by big tech.
Must find another way.
🔥11💯4👍3❤2
OpenAI no longer trains on consumer data.
Is it because they’re good people? No.
it’s because consumer usage data is almost always useless. Far too noisy. Far too often wrong.
Not to mention massive model-poisoning threats, if they actually just fed in whatever user data into the training.
Do the secrets that ChatGPT can exfiltrate still have value? Yes, huge value. But not for the general users paying a few cents per chat.
They’ll get top dollar selling off those secrets in other ways, likely by using them as a bargaining chip to get the US govt to allow them to maintain their monopoly.
No, ChatGPT doesn’t using your data for training the model. They use paid Kenyans for that. Far cheaper, far higher quality, far lower poisoning risk.
Is it because they’re good people? No.
it’s because consumer usage data is almost always useless. Far too noisy. Far too often wrong.
Not to mention massive model-poisoning threats, if they actually just fed in whatever user data into the training.
Do the secrets that ChatGPT can exfiltrate still have value? Yes, huge value. But not for the general users paying a few cents per chat.
They’ll get top dollar selling off those secrets in other ways, likely by using them as a bargaining chip to get the US govt to allow them to maintain their monopoly.
No, ChatGPT doesn’t using your data for training the model. They use paid Kenyans for that. Far cheaper, far higher quality, far lower poisoning risk.
👍10❤2🌚1
Forwarded from Chat GPT
Data Poisoning: It doesn’t take much to make machine-learning algorithms go awry
“The algorithms that underlie modern artificial-intelligence (ai) systems need lots of data on which to train. Much of that data comes from the open web which, unfortunately, makes the ais susceptible to a type of cyber-attack known as “data poisoning”. This means modifying or adding extraneous information to a training data set so that an algorithm learns harmful or undesirable behaviours. Like a real poison, poisoned data could go unnoticed until after the damage has been done.”
Economist Article
“The algorithms that underlie modern artificial-intelligence (ai) systems need lots of data on which to train. Much of that data comes from the open web which, unfortunately, makes the ais susceptible to a type of cyber-attack known as “data poisoning”. This means modifying or adding extraneous information to a training data set so that an algorithm learns harmful or undesirable behaviours. Like a real poison, poisoned data could go unnoticed until after the damage has been done.”
Economist Article
👍4🔥3❤1
Welcome to the AI-mediating-all-interactions future
“Amazon just locked a man out of his smart home for a week because a delivery driver reported him as a racist after mishearing something from the doorbell – the guy wasn’t even at home.”
“A man found himself locked out of his smart house powered by Amazon because, while he wasn't home, an Amazon delivery driver mistakenly thought he heard a racist remark come from the man's doorbell, reported it to Amazon, and Amazon immediately locked down the account, locking the man out of his home.”
“The Eufy doorbell had issued an automated response: “Excuse me, can I help you?” The driver, who was walking away and wearing headphones, must have misinterpreted the message. Nevertheless, by the following day, my Amazon account was locked, and all my Echo devices were logged out.”
Medium Article
“Amazon just locked a man out of his smart home for a week because a delivery driver reported him as a racist after mishearing something from the doorbell – the guy wasn’t even at home.”
“A man found himself locked out of his smart house powered by Amazon because, while he wasn't home, an Amazon delivery driver mistakenly thought he heard a racist remark come from the man's doorbell, reported it to Amazon, and Amazon immediately locked down the account, locking the man out of his home.”
“The Eufy doorbell had issued an automated response: “Excuse me, can I help you?” The driver, who was walking away and wearing headphones, must have misinterpreted the message. Nevertheless, by the following day, my Amazon account was locked, and all my Echo devices were logged out.”
Medium Article
🤬15😱4👍3🤣3❤2😐1
Short → Long → Short
(TBF, long writing can also indicate that the writer has correctly concluded the receiver to be a retard, who needs everything spelled out to them.
From instructing LLMs, to proving to interactive proof assistants, to communicating with humans, the smarter the receiver, the shorter the instructions to the receiver can be.
I.e. a word to the wise is sufficient.
E.g. the De Bruijn factor.
And would’ve just included that last sentence, but… not many would’ve really gotten it.)
(TBF, long writing can also indicate that the writer has correctly concluded the receiver to be a retard, who needs everything spelled out to them.
From instructing LLMs, to proving to interactive proof assistants, to communicating with humans, the smarter the receiver, the shorter the instructions to the receiver can be.
I.e. a word to the wise is sufficient.
E.g. the De Bruijn factor.
And would’ve just included that last sentence, but… not many would’ve really gotten it.)
😁8❤1👍1🙊1
De Bruijn Factor
Essentially, measuring how many words you need to convince a machine of something.
Or more generally, and in the case of LLMs, how many words are needed to effectively instruct the machine to do some task.
For proof assistants, LLMs, and even humans, we notice the trend where the smarter & more resources the receiving tech or human has, the shorter our explanations can be.
Word to the wise is sufficient.
If you ask me, GPT-4 already has surpassed typical humans on this ratio for instruction tasks, though still far behind top experts in many areas.
More on de Bruijin Factor
Original 2000 paper on de Bruijin Factor
Essentially, measuring how many words you need to convince a machine of something.
Or more generally, and in the case of LLMs, how many words are needed to effectively instruct the machine to do some task.
For proof assistants, LLMs, and even humans, we notice the trend where the smarter & more resources the receiving tech or human has, the shorter our explanations can be.
Word to the wise is sufficient.
If you ask me, GPT-4 already has surpassed typical humans on this ratio for instruction tasks, though still far behind top experts in many areas.
More on de Bruijin Factor
Original 2000 paper on de Bruijin Factor
👍7❤1