Princeton on ChatGPT-4 for real-world coding: Only 1.7% of the time generated a solution that worked.
“We therefore introduce SWE-bench, an evaluation framework including 2,294 software engineering problems drawn from real GitHub issues and corresponding pull requests across 12 popular Python repositories. Given a codebase along with a description of an issue to be resolved, a language model is tasked with editing the codebase to address the issue.“
“Our evaluations show that both state-of-the-art proprietary models and our fine-tuned model SWE-Llama can resolve only the simplest issues. Claude 2 and GPT-4 solve a mere 4.8% and 1.7% of instances respectively, even when provided with an oracle retriever”
Arxiv Paper
“We therefore introduce SWE-bench, an evaluation framework including 2,294 software engineering problems drawn from real GitHub issues and corresponding pull requests across 12 popular Python repositories. Given a codebase along with a description of an issue to be resolved, a language model is tasked with editing the codebase to address the issue.“
“Our evaluations show that both state-of-the-art proprietary models and our fine-tuned model SWE-Llama can resolve only the simplest issues. Claude 2 and GPT-4 solve a mere 4.8% and 1.7% of instances respectively, even when provided with an oracle retriever”
Arxiv Paper
🤬18😎6
Google presents MusicRL: MusicRL is the first music generation system finetuned with RLHF
Paper
Project Page
Paper
Project Page
🔥17👨💻5👀2
Media is too big
VIEW IN TELEGRAM
Until 1 year ago, this vision of the AI future seemed like a joke
Not anymore. Thanks Sam.
Not anymore. Thanks Sam.
🤣31😎7🤬4😱3💯2🔥1
This media is not supported in your browser
VIEW IN TELEGRAM
Humans may be making AI stupider over time, as a result of interacting with humans - AI researcher James Zou
👏53😎13💯7😐6👍4🥰2😢2❤🔥1🎉1🗿1