Ali's Notes
1.06K subscribers
9 photos
2 videos
61 links
من علی نجفی ام.
اینجا برای خودم نوت برداری میکنم
شاید به درد شما هم بخوره :‌)


Website: www.najafi-ali.com
BlueSky: https://bsky.app/profile/najafialiai.bsky.social
Linkedin: www.linkedin.com/in/najafi-ali1998
Download Telegram
تویت جالب اندرو کارپاسی درمورد

LLMs and code generation


https://x.com/karpathy/status/1930305209747812559




You could see it as there being two modes in creation. Borrowing GAN terminology:
1) generation and
2) discrimination.
e.g. painting - you make a brush stroke (1) and then you look for a while to see if you improved the painting (2). these two stages are interspersed in pretty much all creative work.

Second point. Discrimination can be computationally very hard.
- images are by far the easiest. e.g. image generator teams can create giant grids of results to decide if one image is better than the other. thank you to the giant GPU in your brain built for processing images very fast.
- text is much harder. it is skimmable, but you have to read, it is semantic, discrete and precise so you also have to reason (esp in e.g. code).
- audio is maybe even harder still imo, because it force a time axis so it's not even skimmable. you're forced to spend serial compute and can't parallelize it at all.

You could say that in coding LLMs have collapsed (1) to ~instant, but have done very little to address (2). A person still has to stare at the results and discriminate if they are good. This is my major criticism of LLM coding in that they casually spit out *way* too much code per query at arbitrary complexity, pretending there is no stage 2. Getting that much code is bad and scary. Instead, the LLM has to actively work with you to break down problems into little incremental steps, each more easily verifiable. It has to anticipate the computational work of (2) and reduce it as much as possible. It has to really care.

This leads me to probably the biggest misunderstanding non-coders have about coding. They think that coding is about writing the code (1). It's not. It's about staring at the code (2). Loading it all into your working memory. Pacing back and forth. Thinking through all the edge cases. If you catch me at a random point while I'm "programming", I'm probably just staring at the screen and, if interrupted, really mad because it is so computationally strenuous. If we only get much faster 1, but we don't also reduce 2 (which is most of the time!), then clearly the overall speed of coding won't improve (see Amdahl's law).

@css_nlp
👍43🔥1
🔹🔹🔹🔹🔹🔹🔹🔹🔹
Open Source LLM Engineering Platform
🔹🔹🔹🔹🔹🔹🔹🔹🔹

🔗 https://langfuse.com/

@css_nlp
Please open Telegram to view this post
VIEW IN TELEGRAM
👍41
🚨 Paper Alert

🔹🔹🔹🔹🔹🔹🔹🔹🔹
Critique Fine-Tuning: Learning to Critique is More Effective than Learning to Imitate
🔹🔹🔹🔹🔹🔹🔹🔹🔹

🔗 https://arxiv.org/pdf/2501.17703


@css_nlp
Please open Telegram to view this post
VIEW IN TELEGRAM
👍31🆒1
Forwarded from Memes
👍7👏3🔥21🆒1
🔹🔹🔹
Diagram Generator
🔹🔹🔹

A tool for generating diagrams using LLM, with a Python backend API and React frontend.

🙂🙂🙂Disclaimer
This project was purely an experiment, it is entirely generated with guidance from myself.
Otherwise known as "vibe coding".
Would I recommend it? No, in fact don't even credit me if you use this code.
I do however believe that as engineers we need to explore options we disagree with in order to widen our perspective. I did in fact learn a lot about how LLMs "think" during this project and am glad to have done it regardless of how frustrating it was.


🔗 https://github.com/minimalefforttech/diagram_generator

@css_nlp
Please open Telegram to view this post
VIEW IN TELEGRAM
👍3