Axis of Ordinary
3.73K subscribers
4.34K photos
1.23K videos
6 files
5.34K links
Memetic and cognitive hazards.

Substack: https://axisofordinary.substack.com/
Download Telegram
R1 refers to its own thought process in the response. First time I have seen this.
😨11🥴4👍3🤡1
Look closely and realize that the X axis is a log scale with exponentially cheaper being on the right.
🔥10🤯3👍1
A quick note on R1 and today's extremely irrational market behavior:

1. There is no such thing as enough energy, because there is no such thing as enough computing power, because there is no such thing as enough intelligence.

More efficient algorithms do not eliminate the need for expensive development of better processors. No one has stopped building more powerful supercomputers because of hardware optimizations and algorithmic advances. Doing more with less means being able to do even more with more.

The idea that more efficient use of resources makes further growth unnecessary is fundamentally wrong. It is to be expected that such arguments come from green degrowth nutters. But people who trade stocks for a living should really know better.

2. We are now in an arms race with China for dominance of the universe until the end of time. Whoever has more computing power wins. If R1 is as revolutionary as some claim, it will inevitably accelerate innovation and drive substantially higher demand for companies like Nvidia, as computing power becomes an even greater competitive advantage.

3. Although China’s semiconductor industry has made significant strides, it remains years behind Western companies. Just because they can catch up to the latest Western AI model in a few months does not mean they will be able to continue to do so as Western companies increase their computing power exponentially.

4. That R1 is more efficient is no surprise, since it is several months younger than o1. A better comparison would be between R1 and o3-mini.

5. Today’s market crash represents a strategic victory for the CCP. Faced with the realization that they might not win a prolonged arms race, China opted to open-source R1. This move creates the illusion that Western companies’ competitive edge in closed models is diminished, discouraging investment and confidence.

However, this perception is fundamentally flawed for the reasons outlined above. Nonetheless, the strategy worked—at least in the short term—by rattling markets and sowing doubt.
👍14👎6💯5🥰1🎉1
Links for 2025-01-27

AI:

1. “For weakly superhuman intelligence, I think the path to that is increasingly quite clear. Doesn't have too many missing pieces left and I think it's very credible what the leaders are saying that we're looking at one to three years, where AI is better than any human in every task in 2027.” https://www.youtube.com/watch?v=MbX9J1Tt_I0&t=5237s

2. Yann Lecun says within the next 3-5 years we will see the emergence of a new paradigm of AI architectures that don't have the limitations of current systems, leading to a revolution in AI capabilities https://youtu.be/MohMBV3cTbg?si=aClBtK2mfdYE-1PD&t=272

3. Salesforce CEO Marc Benioff says the $500 billion Stargate Project is "the beginning of trillions" and "the very beginning of what will be one the biggest investment levels in the history of the world" that will "completely transform how we do everything" https://youtu.be/DasG1N68Wog?si=_qiMvfp-BTnnFX7Z&t=43

4. Reliance plans world’s biggest AI data center in India, report says https://techcrunch.com/2025/01/23/reliance-plans-world-biggest-ai-data-centre-in-india-report-says/

5. Chain-of-Retrieval Augmented Generation https://arxiv.org/abs/2501.14342

6. Standard LLMs can do protein sequence analysis - no modification required https://arxiv.org/abs/2501.09274

7. Anomalous Tokens in DeepSeek-V3 and r1 https://www.lesswrong.com/posts/xtpcJjfWhn3Xn8Pu5/anomalous-tokens-in-deepseek-v3-and-r1

8. Deployment of Aerial Multi-agent Systems for Automated Task Execution in Large Scale Environments https://www.youtube.com/watch?v=4eyRCCRAEYg

9. Six Thoughts On AI Safety https://windowsontheory.org/2025/01/24/six-thoughts-on-ai-safety/

10. Asked about risks from artificial superintelligence, President Trump says the first question he asks is how to absolve yourself from mistake and make sure it is not "the rabbit that gets away" https://youtu.be/jIE8lFeHpag?si=lTCIbTqOwNlhkA2X&t=761

Miscellaneous:

1. How to Prove False Statements: Practical Attacks on Fiat-Shamir — “We break soundness of a standard protocol (essentially commit to witness and run GKR) by constructing a circuit for which we can prove a false statement.” https://eprint.iacr.org/2025/118

2. “Useful quantum computing is inevitable—and increasingly imminent” https://www.technologyreview.com/2025/01/27/1110540/useful-quantum-computing-is-inevitable-and-increasingly-imminent/ [no paywall: https://archive.is/OofYe]

3. Regions of our brains making multiple predictions of others' actions. https://mindblog.dericbownds.net/2025/01/regions-of-our-brains-making-multiple.html

Short SF:

1. CIV: a story https://www.lesswrong.com/posts/SSNfgL49Bx2uATPv8/civ-a-story

2. The Rising Sea https://www.lesswrong.com/posts/XvyAeymaRi95MSLZD/the-rising-sea
👍1
This media is not supported in your browser
VIEW IN TELEGRAM
Dario Amodei says 2026-2027 is the critical window in AI and if you're ahead then, the models start getting better than humans at everything including AI design and using AI to make better AI, so export controls to prevent DeepSeek keeping up with US companies are worth continuing with
🤡5🤔4😁2
😁40🤯2🤩1🤡1
Is the increased efficiency of DeepSeek R1’s training and inference an unexpected breakthrough that calls into question massive spending on computing power? No. The cost of training and running large language models (LLMs) has been declining for years, largely due to rapid improvements in algorithms.

For instance, recent experiments show that in 2024, GPT-2 could be trained in about 24 hours for only $672 on an 8×H100 GPU node—significantly less than the roughly $50,000 it cost to train in 2019. The cost of running LLMs has also dropped dramatically: in 2021, GPT-3 (then the only model achieving an MMLU score of 42) cost $60 per million tokens, whereas today, Llama 3.2 3B achieves the same score at $0.06 per million tokens, a 1,000-fold reduction in just three years. Even more striking, the cost of GPT-4-level intelligence has fallen 1,000× in the last 1.5 years.

Such algorithmic progress has been underway for decades. Take, for example, one family of algorithms designed to solve the maximum subarray problem: between 1970 and today, the time required to solve this problem for n = 10^6 inputs has declined by a factor of about 10^12—a reduction of 99.9999999999%. With today’s algorithms, an average 1994 desktop computer would have beaten the world chess champion.

Owing to advances like these, the level of compute required to achieve a given AI performance level has been estimated to halve roughly every eight months. In this context, DeepSeek’s R1 delivering a 27× improvement is a natural progression rather than a paradigm shift.

Despite this, the demand for computing power has grown at a phenomenal rate. Experts estimate that the computational power required for AI doubles every 100 days—an increase of roughly 1,220% per year.

In summary, anyone closely following AI developments has already factored in algorithmic and hardware optimizations. Thus, it was expected that a model like R1, emerging a few months after o1, would be cheaper to train and run. A more telling comparison would be between R1 and the forthcoming o3-mini.

P.S. While DeepSeek R1’s training cost is remarkably low, it’s important to consider the broader research and development context. The combined R&D budget for DeepSeek V3 and R1 is estimated to be around $100 million, underscoring the substantial resources behind this achievement.
👍9🥱3👎1
Links for 2025-01-28

AI:

1. Towards General-Purpose Model-Free Reinforcement Learning https://arxiv.org/abs/2501.16142

2. Eagle 2: Building Post-Training Data Strategies from Scratch for Frontier Vision-Language Models https://arxiv.org/abs/2501.14818

3. Qwen2.5-Max: Exploring the Intelligence of Large-scale MoE Model https://qwenlm.github.io/blog/qwen2.5-max/

4. Mixture-of-Mamba: Enhancing Multi-Modal State-Space Models with Modality-Aware Sparsity — Mixture-of-Mamba reaches similar loss with just half of the FLOPs https://arxiv.org/abs/2501.16295

5. Demis Hassabis on how AI can revolutionize scientific discovery, and why we could be 5-10 years away from AGI. https://www.youtube.com/live/ICv03VysLaE?si=r44hKGCPDZKlVWAK&t=1097

AI politics:

1. President Trump says China's DeepSeek AI model is a "wake-up call" for American companies but it is a good thing for faster and cheaper methods of AI to be developed https://www.youtube.com/live/AitXub2TE5s?si=IWjs-xpm6xAocfaE&t=3218

2. Trump To Tariff Chips Made In Taiwan, Targeting TSMC https://uk.pcmag.com/computers-electronics/156458/trump-to-tariff-chips-made-in-taiwan-targeting-tsmc

3. Former NSA chief revolves through OpenAI's door https://responsiblestatecraft.org/former-nsa-chief-revolves-through-openai-s-door/

4. ChatGPT Gov is designed to streamline government agencies’ access to OpenAI’s frontier models. https://openai.com/global-affairs/introducing-chatgpt-gov/

Miscellaneous:

1. Images show China building huge fusion research facility, analysts say https://www.reuters.com/world/china/images-show-china-building-huge-fusion-research-facility-analysts-say-2025-01-28/ [no paywall: https://archive.is/pKovf]
1🤔1
DeepSeek R1 2X speed boost was apparently coded by R1 itself:

https://simonwillison.net/2025/Jan/27/llamacpp-pr/
🔥14🤯131
😁22💩3👍1😐1
New essay by Anthropic CEO Dario Amodei: https://darioamodei.com/on-deepseek-and-export-controls

"If China can't get millions of chips...there's at least the possibility that, because AI systems can eventually help make even smarter AI systems, a temporary lead could be parlayed into a durable advantage."
🥴9🤔1💯1
Links for 2025-01-29

AI:

1. Why reasoning models will generalize. DeepSeek R1 is just the tip of the ice berg of rapid progress. People underestimate the long-term potential of “reasoning.” https://www.interconnects.ai/p/why-reasoning-models-will-generalize

2. “...it's plausible that LLMs can discover better ways of thinking, of solving problems...” https://x.com/karpathy/status/1884336943321997800

3. SFT Memorizes, RL Generalizes: A Comparative Study of Foundation Model Post-training https://tianzhechu.com/SFTvsRL/

4. CodeMonkeys: a system for solving SWE-bench issues specifically designed to leverage test-time compute! https://scalingintelligence.stanford.edu/blogs/codemonkeys/

5. “DeepSeek-R1 has been making waves recently by rivaling OpenAI's O1 reasoning model while being fully open-source. We explored how to enable more local users to run it & managed to quantize DeepSeek’s R1 671B parameter model to 131GB in size, a 80% reduction in size from the original 720GB, whilst being very functional.” https://unsloth.ai/blog/deepseekr1-dynamic

6. Optimizing Large Language Model Training Using FP4 Quantization https://arxiv.org/abs/2501.17116

AI politics:

1. First-ever International AI Safety Report, backed by 30 countries and the OECD, UN, and EU. It summarises the state of the science on AI capabilities and risks, and how to mitigate those risks. Experts agree loss of control isn't a big concern *right now* — but if these key capabilities improve rapidly, it could quickly become a huge risk. [PDF] https://assets.publishing.service.gov.uk/media/679a0c48a77d250007d313ee/International_AI_Safety_Report_2025_accessible_f.pdf

2. The Game Board has been Flipped: Now is a good time to rethink what you’re doing https://www.lesswrong.com/posts/ynsjJWTAMhTogLHm6/the-game-board-has-been-flipped-now-is-a-good-time-to

3. Planning for Extreme AI Risks https://www.lesswrong.com/posts/8vgi3fBWPFDLBBcAx/planning-for-extreme-ai-risks

4. It's time to come to grips with AI https://www.natesilver.net/p/its-time-to-come-to-grips-with-ai

5. A defence of slowness at the end of the world https://longerramblings.substack.com/p/a-defence-of-slowness-at-the-end

6. “The only logical move is to invest in AI. In fact, you should invest pretty much everything you can spare, in order to maximize your chances…” https://x.com/fchollet/status/1884350877995282663

7. AGI Cannot Be Predicted From Real Interest Rates https://nicholasdecker.substack.com/p/will-transformative-ai-really-raise

8. DeepSeek Panic at the App Store https://www.lesswrong.com/posts/hRxGrJJq6ifL4jRGa/deepseek-panic-at-the-app-store

9. Groq CEO Jonathan Ross says DeepSeek is Sputnik 2.0: "You that story about how NASA spent a million dollars designing a pen that could write in space and the Russians brought a pencil? That just happened again" https://youtu.be/4P6CUIPrrtU?si=YiBpZWwSxzYJsiKd&t=125

10. Palmer Luckey says while DeepSeek is impressive, there is also CCP propaganda about the model's purported $5 million training cost that is designed to crash American stocks and "useful idiots" in the media are parroting these talking points without regards to their veracity https://www.youtube.com/watch?v=O53--zQYOpA

11. Chinese Quant Whiz Built DeepSeek In The Shadow Of a Hedge Fund Rout https://www.bloomberg.com/news/articles/2025-01-28/chinese-quant-whiz-built-deepseek-in-the-shadow-of-a-hedge-fund-rout [no paywall: https://archive.is/Ve3CS]

Science:

1. Jeremy Avigad's excellent talk, "You Want Proof? I'll Give You Proof! ...Euclid to Lean," that bridges classical mathematics with modern tools like Lean is now available on YouTube! https://www.youtube.com/watch?v=HT0qJTB_f0s

2. NASA 3D-Printed Antenna Takes Additive Manufacturing to New Heights https://www.nasa.gov/technology/nasa-3d-printed-antenna-takes-additive-manufacturing-to-new-heights/

3. Researchers confirm the existence of an exoplanet in the habitable zone https://www.ox.ac.uk/news/2025-01-28-researchers-confirm-existence-exoplanet-habitable-zone
Reasoning systems can generate high-quality data during inference, which can then be used to further train and improve the model: https://arcprize.org/blog/r1-zero-r1-results-analysis

"Ultimately, R1-Zero demonstrates the prototype of a potential scaling regime with zero human bottlenecks – even in the training data acquisition itself."
👏10
"On IQ questions that only exist offline, Deepseek R1 beats all free models, but is behind OpenAI’s paid models"

https://www.maximumtruth.org/p/chinas-deepseek-is-not-as-smart-as
4🤬1
Media is too big
VIEW IN TELEGRAM
OpenAI CPO Kevin Weil on DeepSeek: The 'first big salvo' in U.S.-China AI arms race

- [They] "have a new model coming soon...that is head and shoulders above anything that's out there"

- [They] "are about to launch some models that are meaningfully ahead of state of the art"

- OpenAI partners with U.S. National Laboratories on scientific research, nuclear weapons security
🥴8👍1😁1
This media is not supported in your browser
VIEW IN TELEGRAM
AI has reached the Russian Orthodoxy. Patriarch Kirill in a speech in the State Duma:

It is important that artificial intelligence serves the benefit of people, and that people can control it. According to some experts, a generation of more advanced machine models, called General Artificial Intelligence, may soon appear that will be able to think and learn — that is, improve — like a person. And if such artificial intelligence is put next to ordinary human intelligence, who will win? Artificial intelligence, of course!


Maybe it would be a good idea for Russia to put more resources into building data centers and attracting smart people to work on AI safety than conquering rubble and fields? Just kidding, Putin is smart. Controlling Ukraine is much more important for Russia than controlling superintelligence.
🤡23😁7💯4🔥3👍1
Zuckerberg predicts a mid-level coding engineer AI agent for 2025 calling it "potentially one of the most important innovations in history".

He also says that their reasoning models and larger model are looking good too.

Source: https://www.facebook.com/share/p/15n83eiPNF/
🥴6🤓31👎1👏1
[2012] Google/DeepMind: We're going to build artificial general intelligence before 2030

Skeptics: It's just hype to attract investors

[2012] Google's driverless car passes the test for Nevada self-driving vehicles

[2016] AlphaGo defeats the world champion in Go

[2019] AlphaStar defeats professional StarCraft II players demonstrating advanced strategic planning, real-time decision-making, and adaptability in a complex, dynamic environment

[2020] MuZero achieves superhuman performance in complex board games and learns to play Atari games without prior knowledge of their rules

[2020] AlphaChip accelerates and optimizes chip design and creates superhuman chip layouts that are now used in hardware around the world

[2020] AlphaFold accurately predicts the three-dimensional structures of proteins

[2022] Flamingo, a generalist visual language model, rapidly adapts its behaviour given just a handful of examples

[2022] AlphaTensor found a way to speed up a calculation at the heart of many different kinds of code, beating a 50-year record

[2023] AlphaDev discovers and optimize fundamental computer science algorithms

[2023] FunSearch cracks a famous unsolved problem in pure mathematics

[2023] AlphaCode 2 reaches the 85th percentile on the Codeforces platform

[2023] RoboCat, a self-improving robotic agent, learns to solve new tasks on different robotic arms with as few as 100 demonstrations - and improves skills from self-generated training data

[2024] AlphaGeometry solves Olympiad geometry problems at a level approaching a human gold-medalist

[2024] AlphaProof achieves silver-medal standard solving International Mathematical Olympiad problems

[2024] SIMA is the first generalist AI agent to follow natural-language instructions in a broad range of 3D virtual environments and video games

[2024] Genie 2, a large-scale foundation world model, creates a path to unlimited environments for training and evaluating embodied agents

[2024] DeepMind scientists create a computerized insect that can walk and fly just like the real thing

[2024] DeepMind achieves human level competitive robot table tennis

[2024] After 25.3 million autonomous miles driven, Google's Waymo vehicles have an 88% reduction in property damage claims and a 92% reduction in bodily injury claims compared to human drivers per mile driven

[2024] Veo 2 generative AI video model demonstrates a superior understanding of real-world physics, human motion and facial expressions

[2024] Google releases Flash Thinking, a new reasoning model with a one-million token context window

[2025] Google/DeepMind: We're on track to build artificial general intelligence before 2030.

Skeptics: It's just hype to attract investors.
🥱17👍16🤡32💯2🤔1