ChatGPT Outperforms Crowd-Workers for Text-Annotation Tasks
Using a sample of 2,382 tweets, we demonstrate that ChatGPT outperforms crowd-workers for several annotation tasks, including relevance, stance, topics, and frames detection. Specifically, the zero-shot accuracy of ChatGPT exceeds that of crowd-workers for four out of five tasks, while ChatGPT's intercoder agreement exceeds that of both crowd-workers and trained annotators for all tasks. Moreover, the per-annotation cost of ChatGPT is less than $0.003 -- about twenty times cheaper than MTurk.
ChatGPT outperforms humans on 4/5 tasks, while being 20x cheaper.
Paper
Using a sample of 2,382 tweets, we demonstrate that ChatGPT outperforms crowd-workers for several annotation tasks, including relevance, stance, topics, and frames detection. Specifically, the zero-shot accuracy of ChatGPT exceeds that of crowd-workers for four out of five tasks, while ChatGPT's intercoder agreement exceeds that of both crowd-workers and trained annotators for all tasks. Moreover, the per-annotation cost of ChatGPT is less than $0.003 -- about twenty times cheaper than MTurk.
ChatGPT outperforms humans on 4/5 tasks, while being 20x cheaper.
Paper
👍8👀4😱2❤1🔥1🗿1
Words of input -vs- Age, for humans and GPT-3
GPT-3 receives 10,000x to 100,000x more words than humans do, but in many areas humans still crush it.
Something is missing. Something that could enable AI to achieve higher intelligence with 10,000x to 100,000x less external training input.
Upper bound from: Cognitive science in the era of artificial intelligence: A roadmap for reverse-engineering the infant language-learner
Lower bound from: What Do North American Babies Hear? A large-scale cross-corpus analysis
GPT-3 receives 10,000x to 100,000x more words than humans do, but in many areas humans still crush it.
Something is missing. Something that could enable AI to achieve higher intelligence with 10,000x to 100,000x less external training input.
Upper bound from: Cognitive science in the era of artificial intelligence: A roadmap for reverse-engineering the infant language-learner
Lower bound from: What Do North American Babies Hear? A large-scale cross-corpus analysis
👍7👏2❤1
Verbal analogies edition, and then feeding the question to GPT-4 in a different session, and then feeding the answers back into the first.
Of course GPT-4 rates itself as brilliant.
Ask me 5 different multiple-choice verbal analogy questions and analyze how smart you think I am according to my answers. Ask all the questions in one go and I will reply.
Of course GPT-4 rates itself as brilliant.
Ask me 5 different multiple-choice verbal analogy questions and analyze how smart you think I am according to my answers. Ask all the questions in one go and I will reply.
👍2😁2❤1🎉1
Verbal Analogies Test: GPT-3.5 edition.
GPT-4 makes questions,
GPT-3.5 answers questions,
GPT-4 grades them.
Result:
GPT-4 says GPT-3.5 is a retard.
GPT-4 makes questions,
GPT-3.5 answers questions,
GPT-4 grades them.
Result:
GPT-4 says GPT-3.5 is a retard.
😁20👏5❤2
Times Magazine: Eliezer Yudkowsky, chief proponent of AI-Slowdown camp, calls for Shutting down AI, tracking of all AI hardware, elimination of all competition, giving government absolute control
“Shut down all the large GPU clusters (the large computer farms where the most powerful AIs are refined). Shut down all the large training runs. Put a ceiling on how much computing power anyone is allowed to use in training an AI system, and move it downward over the coming years to compensate for more efficient training algorithms. No exceptions for governments and militaries. Make immediate multinational agreements to prevent the prohibited activities from moving elsewhere. Track all GPUs sold. If intelligence says that a country outside the agreement is building a GPU cluster, be less scared of a shooting conflict between nations than of the moratorium being violated; be willing to destroy a rogue datacenter by airstrike.”
“Frame nothing as a conflict between national interests, have it clear that anyone talking of arms races is a fool. That we all live or die as one, in this, is not a policy but a fact of nature. Make it explicit in international diplomacy that preventing AI extinction scenarios is considered a priority above preventing a full nuclear exchange, and that allied nuclear countries are willing to run some risk of nuclear exchange if that’s what it takes to reduce the risk of large AI training runs.”
“That’s the kind of policy change that would cause my partner and I to hold each other, and say to each other that a miracle happened, and now there’s a chance that maybe Nina will live. The sane people hearing about this for the first time and sensibly saying “maybe we should not” deserve to hear, honestly, what it would take to have that happen. And when your policy ask is that large, the only way it goes through is if policymakers realize that if they conduct business as usual, and do what’s politically easy, that means their own kids are going to die too.”
“Shut it all down.”
“Shut down all the large GPU clusters (the large computer farms where the most powerful AIs are refined). Shut down all the large training runs. Put a ceiling on how much computing power anyone is allowed to use in training an AI system, and move it downward over the coming years to compensate for more efficient training algorithms. No exceptions for governments and militaries. Make immediate multinational agreements to prevent the prohibited activities from moving elsewhere. Track all GPUs sold. If intelligence says that a country outside the agreement is building a GPU cluster, be less scared of a shooting conflict between nations than of the moratorium being violated; be willing to destroy a rogue datacenter by airstrike.”
“Frame nothing as a conflict between national interests, have it clear that anyone talking of arms races is a fool. That we all live or die as one, in this, is not a policy but a fact of nature. Make it explicit in international diplomacy that preventing AI extinction scenarios is considered a priority above preventing a full nuclear exchange, and that allied nuclear countries are willing to run some risk of nuclear exchange if that’s what it takes to reduce the risk of large AI training runs.”
“That’s the kind of policy change that would cause my partner and I to hold each other, and say to each other that a miracle happened, and now there’s a chance that maybe Nina will live. The sane people hearing about this for the first time and sensibly saying “maybe we should not” deserve to hear, honestly, what it would take to have that happen. And when your policy ask is that large, the only way it goes through is if policymakers realize that if they conduct business as usual, and do what’s politically easy, that means their own kids are going to die too.”
“Shut it all down.”
🤬26👍9❤5😁3🤯3👀2
Must “be willing to destroy a rogue datacenter by airstrike” to enforce immediate AI shutdown - Eliezer Yudkowsky
Times Article
Times Article
🤬19💯7😱4🤯2❤1