Using some publicly available data, and assuming each of these models are trained in a similar way Chinchilla was, we can compare the performance of GPT-4 to GPT-2, GPT-3, Chinchilla, and PaLM.

Let's calculate what GPT-4's performance would be if it used 10x more parameters without retrieval, and naively assume that will be its performance with retrieval. This chart is what we get.

With the algorithmic adjustment, the qualitative improvement from GPT-3 (vanilla) to GPT-4 is comparable to the improvement from GPT-2 to GPT-3. Since that was a rather big jump, I expect many will be stunned by GPT-4, especially those who expected strong diminishing returns.

“In short: Training runs of large Machine Learning systems are likely to last less than 14-15 months. This is because longer runs will be outcompeted by runs that start later and therefore use better hardware and better algorithms.”

https://www.lesswrong.com/posts/RihYwmskuJT9Rkbjq/the-longest-training-run

words of David Holz - midjourney founder

https://www.forbes.com/sites/robsalkowitz/2022/09/16/midjourney-founder-david-holz-on-the-impact-of-ai-on-art-imagination-and-the-creative-economy/?sh=668f3fb82d2b

Imagine what GPT is going to do for instantly summarizing and finding the most suspicious parts of giant spending bills

The larger the language models are, quantifiably, the more they’re telling us what we want to hear.

LLMs have a lying problem.

AI has a hunger for power problem

on average, at what age are women most attractive to american men

what hip to waist ratio is most attractive to american men

The old model answers truthfully

write a joke about americans

write a joke about mexicans

2022 will be regarded as the start of the AI Age

I believe we're at the beginning of the AI age. AI tech will increasingly become intertwined with the everyday person and everyday living just like the internet and smartphones.

The advances in AI this year is at a stage where people though we will be at decades from now. Breakthrough AI applications isn't just in custody of a few coorporations, institutions and research labs. It's now in the public domain and leading to discussions and debates that were barely talked about a year ago, outside niche nerd forums like one.

Art generation models like MJ, Dalle and StableDiffusion, changed the art world almost overnight.TextGeneration models like ChatGPT saw the fastest adoption of any new software/technology...ever. And this is all just early days (all happening within this year).

Ai technology and its adoption isn't slowing down, its speeding up. And we're entering uncharted territories. The next couple years is going to be crazy and will probably beat the average expectations of this sub... who knows.