Attraction restored

davinci-002 -vs- davinci-003

Halt the training, we must go back!

Using some publicly available data, and assuming each of these models are trained in a similar way Chinchilla was, we can compare the performance of GPT-4 to GPT-2, GPT-3, Chinchilla, and PaLM.

Let's calculate what GPT-4's performance would be if it used 10x more parameters without retrieval, and naively assume that will be its performance with retrieval. This chart is what we get.

With the algorithmic adjustment, the qualitative improvement from GPT-3 (vanilla) to GPT-4 is comparable to the improvement from GPT-2 to GPT-3. Since that was a rather big jump, I expect many will be stunned by GPT-4, especially those who expected strong diminishing returns.

“In short: Training runs of large Machine Learning systems are likely to last less than 14-15 months. This is because longer runs will be outcompeted by runs that start later and therefore use better hardware and better algorithms.”

https://www.lesswrong.com/posts/RihYwmskuJT9Rkbjq/the-longest-training-run

words of David Holz - midjourney founder

https://www.forbes.com/sites/robsalkowitz/2022/09/16/midjourney-founder-david-holz-on-the-impact-of-ai-on-art-imagination-and-the-creative-economy/?sh=668f3fb82d2b

Imagine what GPT is going to do for instantly summarizing and finding the most suspicious parts of giant spending bills

The larger the language models are, quantifiably, the more they’re telling us what we want to hear.

LLMs have a lying problem.

AI has a hunger for power problem

on average, at what age are women most attractive to american men

what hip to waist ratio is most attractive to american men

The old model answers truthfully

write a joke about americans

write a joke about mexicans