GraphML News (Oct 26th) - LOG meetups, Orbital round, ESANN 2025

🍻 The Learning of Graphs conference continues to update the list of local meetups - the networks already includes 13 places from well-known graph learning places like Stanford, NYC, Paris, Oxford, Aachen, Amsterdam, Tel Aviv down to Tromsø, Uppsala, Siena, New Delhi, Suzhou, and Vancouver (Late November in Tromsø, talking graphs with a cup of glühwein and snow outside must be a quite a cozy venue). The call for meetups is still open!

On this note, the European Symposium on Artificial Neural Networks, Computational Intelligence and Machine Learning (ESANN 2025) will host 3 special sessions on graph learning: Foundation and Generative Models for Graphs, Graph Generation for Life Sciences, and Network Science Meets AI. Submission deadline is November 20, 6 pages tops. Thanks to Manuel Dileo for the pointer! ESANN 2025 will take place on April 23-25 (2025) in Bruges (jokes about the movie and Tottenham are welcome).

💸 Orbital Materials secured a new funding round led by NVIDIA Ventures (financial details undisclosed) – timed nicely coinciding with the recent release of the ML potential GNN Orb-v2. A new unicorn from AI 4 Science is coming? 🤔

Weekend reading:

Learning Graph Quantized Tokenizers for Transformers by Limei Wang, Kaveh Hassani et al and Meta - an unorthodox approach for graph tokenization via vector quantization and codebook learning, conceptually similar to VQ-GNNs (NeurIPS 2021), strange to not see this older paper cited

Relaxed Equivariance via Multitask Learning by Ahmed Elhag et al feat Michael Bronstein - instead of baking equivariances right into models, let’s add it as a loss component and allow a model to learn and use as much equivariance as necessary, brings 10x inference speedups.

Homomorphism Counts as Structural Encodings for Graph Learning by Linus Bao, Emily Jin, et al - introduces motif structural encoding (MoSE) for graph transformers. Paired with GraphGPS, brings MAE on ZINC from 0.07 down to 0.062 and to 0.056 with GRIT.

GraphML News (Nov 2nd) - The Debate on Equivariance, MoML and Stanford Graph workshop

🎃 Writing ICLR reviews and LOG rebuttals might have delivered you enough of the Halloween spirit with spooky papers and (semi)undead reviewers - it’s almost over though!

🥊 The debate on equivariance, namely, is it worth to bake symmetries right into the model or learn from data, remains to be a hot topic in the community with new evidence appearing every week supporting both sides. Is torch.nn.TransformerEncoder all you need?

In the blue corner, the work Does equivariance matter at scale? by Johann Brehmer et al compares a vanilla transformer with the E(3)-equivariant Geometric Algebra Transformer (GATr) on the rigid-body modelling task with a wide range of sizes to derive scaling laws (akin to Kaplan and Chinchilla laws) and finds that the equivariant transformer scales better overall.

In the red corner, we have The Importance of Being Scalable: Improving the Speed and Accuracy of Neural Network Interatomic Potentials Across Chemical Domains by Eric Qu et al who modify a vanilla transformer and outperform hefty Equiformer, GemNet, and MACE on ML potential benchmarks on molecules and crystals. Another wrestler in the red corner is the tech report on ORB v2 by Mark Neumann and Orbital Materials - ORB v2 is a vanilla MPNN potential trained with a denoising objective and delivers SOTA (or close to) performance while trained on only 8 A100s (compared to 64+ GPUs needed for Equiformer V2 but subject to different training datasets).

🏆 Overall, “no equivariance” wins this week 2-1 (2.5 - 1 if including a recent work on relaxed equivariance).

🎤 Next Tuesday, Nov 5th, is not just the election day in the US, but also the day of two graph learning events: MoML 2024 at MIT and the Graph Learning Workshop 2024 at Stanford. Programs of both events are now visible and there might be livestreams as well, keep an eye on the announcements.

Weekend reading:

Generator Matching: Generative modeling with arbitrary Markov processes by Peter Holderrieth feat. Ricky Chen and Yaron Lipman - a generalization of diffusion, flow matching (both continuous and discrete), and jump processes (outstanding paper award at ICLR’24). Expect a new generation of generative models for images / proteins / molecules / SBDD / RNAs / crystals to adopt this next year.

Long-context Protein Language Model by Yingheng Wang and (surprisingly) Amazon team - introduces a Mamba-based bidirectional protein LM that outperforms ESM-2 on a variety of tasks while being much smaller and faster.

Iambic announced NeuralPLexer 3 competitive with AlphaFold 3. While we are waiting for the tech report and more experiments, it seems that NP3 features Triton kernels for efficient triangular attention akin to FlashAttention but on triples of nodes.

GraphML News (Nov 9th) - ELLIS PhD Applications, Protenix AF3, New papers

The next week is going to be busy with ICLR rebuttals, so we still have a bit of time to check the news and read new papers.

🎓 The call for PhD applications within ELLIS, the European network of ML and DL labs, is ending soon (November 15th) - this is a great opportunity to start (or continue) your academic journey in a top machine learning lab!

🧬 ByteDance released Protenix, a trainable PyTorch reproduction of AlphaFold 3, with model checkpoints (so you can run it locally) and with the inference server. The tech report is coming, would be interesting to compare with Chai-1 and other open source reproductions.

Weekend reading:

A Cosmic-Scale Benchmark for Symmetry-Preserving Data Processing by Julia Balla et al feat Tess Smidt - introduces cool new graph datasets - given a point cloud of 5000 galaxies, predict their cosmological properties on the graph level and node level (eg, galaxy velocity).

FlowLLM: Flow Matching for Material Generation with Large Language Models as Base Distributions by Anuroop Sriram and FAIR (NeurIPS 24): extension of FlowMM (ICML 2024), a flow matching model for crystal structure generation, but now instead of sampling from a Gaussian, the authors fine-tuned LLaMa 2 on the Materials Project to sample 10000 candidates (using just a small academic budget of 300 A100 gpus) - this prior yields 3x more stable structures.

Flow Matching for Accelerated Simulation of Atomic Transport in Materials by Juno Nam and MIT team feat. Rafael Gómez-Bombarelli - introduces LiFlow, a flow matching model for MD simulations of crystalline materials: where ab-initio methods would take 340 days to simulate 1 ns of a 200-atoms structure, LiFlow takes only 48 seconds 🏎️

GraphML News (Nov 16th) - ICLR 2025 Stats, Official AlphaFold 3 and RFam, Faster MD

📊 PaperCopilot comprised the basic statistics of ICLR 2025 initial reviews and best scored papers - a few GNN papers are in top-100 and we’ll keep an eye on them! Meanwhile, Azmine Toushik Wasi compiled a list of accepted graph papers at NeurIPS 2024 grouped by categories - from GNN theory to generative models to transformers to OOD generalization and many more.

🧬 Google DeepMind finally released the official code for AlphaFold 3 featuring a monstrous 3k lines of code Structure class and some kernels implemented in Triton (supporting the fact that Triton is the future and you should implement your most expensive neural net ops as efficient kernelized ops). Somewhat orthogonally to AF3, the Baker Lab presented RFam, the improved version of RFdiffusion, in a paper about metallohydrolases. RFam now uses flow matching (welcome on board), allows for scaffolding arbitrary atom-level motifs and sequence-position-agnostic scaffolding. Waiting for the code soon!

🏎️ Microsoft Research announced AI2BMD - a freshly accepted to Nature method for accelerating ab-initio molecular dynamics of proteins with equivariant GNNs (based on VisNet, already in PyG) scaling it up to impressive 10k atoms in a structure (far beyond the capabilities of standard MD tools). Besides, the authors collected a new dataset of 20M DFT-computed snapshots which would be of great help to the MD community.

🌊 Continuing the simulation note, NXAI and JKU Linz presented NeuralDEM, a neural approach to replace Discrete Element Method (DEM) in complex physical simulations (like fluid dynamics) with transformers and neural operators. NeuralDEM is as accurate and stable as vanilla DEM while being much faster and allowing for longer sim times.

Weekend reading:

Generalization, Expressivity, and Universality of Graph Neural Networks on Attributed Graphs by Levi Rauchwerger, Stefanie Jegelka, and Ron Levie - it is known that the vanilla WL test assumes no node features. This is one of the first works to study GNN properties on featurized graphs.

GraphMETRO: Mitigating Complex Graph Distribution Shifts via Mixture of Aligned Experts by Shirley Wu et al feat. Bruno Ribeiro and Jure Leskovec

Scalable Message Passing Neural Networks: No Need for Attention in Large Graph Representation Learning by Haitz Borde et al feat. Michael Bronstein - a transformer-like block where multi-head attention is replaced with a GCN (keeping layer norms, residual stream and MLPs intact) is suprisingly competitive on very large graphs

GLOW Reading Group on Nov 20th

🌟 Graph Learning on Wednesdays (GLOW) is a new reading group about foundations and latest developments in Graph Machine Learning - its inaugural meeting was held on Oct 9th, and the next one is happening on Wednesday, Nov 20th at 5pm CET (11am Eastern) featuring two papers presented by the first authors:

- On the Expressivity and Sample Complexity of Node-Individualized Graph Neural Networks by Paolo Pelizzoni
- Graph Neural Networks Use Graphs When They Shouldn’t by Maya Bechler-Speicher

The format of the RG is rather interactive: short presentations by the authors are followed by discussions and brainstorms, and sometimes expert panels study the paper together. Consider joining on Wednesday!

GraphML News (Nov 23rd) - LOG 2024, Boltz-1 and Chai-1r, OCx24, cuEquivariance

🎙️ LOG 2024 starts next Tuesday! The schedule of orals and posters is already available, registration is free, we’ll be happy to see you there online or at one of many local meetups!

🧬 The “Stable Diffusion moment” in generative models for structural biology is spinning up: two models released during the past week starting with fully open-source Boltz-1 from MIT (tech report, code) that achieves AF3-like quality and outperforms a recent Chai-1 (open inference code). Meanwhile, Chai folks released Chai-1r that supports complexes with user-specified restraints - and already compared with Boltz-1. Open source and competition really drive the field forward 👏 The next step for AF3-like models seems to be integrating the recent GPU-enabled MSA tool MMSeqs2-GPU that might shave off another order of magnitude of inference time of protein structure prediction models.

🧊 FAIR Chemistry at Meta, University of Toronto, and VSParticle presented Open Catalyst experiments (OCx24) - a new dataset of 600 mixed metal catalysts synthesized and probed physically in the lab (a huge step beyond DFT-only simulations) along with analytical data for 19k catalyst materials with 685M relaxations. It’s already the 3rd huge dataset openly published by Meta in addition to OpenDAC and OMAT - Meta is a firm leader in this area of AI 4 Science. Fun fact: models from the Open Catalyst ecosystem are directly used in Meta’s products like recent Orion AR glasses.

🔱 Faster spherical harmonics and tensor products for geometric GNNs: following EquiTriton, an open-source collection of Triton kernels for fast and memory-efficient computation of spherical harmonics developed by Intel Labs (enabling harmonics of up to the 10th order), NVIDIA released cuEquivariance - CUDA kernels (closed-source kernels with public bindings for PyTorch, JAX, and numpy) for spherical harmonics and tensor products which speed up DiffDock, MACE and other models by 2-20x, this is especially useful in tasks where a model is called multiple times like a generative model or MD calculations. cuEquivariance is a part of the new BioNeMo suite for drug discovery.

Weekend reading:

📚 Check out accepted orals and posters of LOG 2024 on OpenReview

GraphML News (Nov 30th) - LOG recordings, The Chip Has Sailed, Illustrated Flow Matching

📺 LOG 2024 has just wrapped up, and all recordings are now available on the official YouTube channel, featuring:

- Keynotes from Yusu Wang (UCSD), Zach Ulissi (Meta), Xavier Bresson (NUS), Alden Hung (Isomorphic Labs)
- Tutorials on geometric generative models, neural algorithmic reasoning, GNNs for time series, heterophilic graph learning, and KGs + LLMs
- All oral presentations

A lot of stuff to digest over the weekend if you missed it!

⛵️ The GNNs for chip design saga continues:

- the seminal 2021 Nature paper (now known as AlphaChip) spun off some controversy from the chip design community about reproducibility (however, other people say it’s rather a skill issue and the CD community just doesn’t have proper deep learning chops),
- several rounds of message exchange led to the official Addendum on the Nature paper clarifying the concerns and adding that pre-training is a must.
- In Oct 2024, Igor Markov (Synopsys) published an article at ACM Communications with a further criticism of the approach.
- Recently, in Nov 2024, the lead authors of AlphaChip (Anna Goldie, Azalia Mirhoseini, and Jeff Dean himself) recently put an arxiv paper The Chip Has Sailed with the full timeline and emphasizing that the approach is actually already working within Google and AlphaChip has been used in several generations of chips used in production.

Perhaps one of the most important messages from this paper is that while academics and industry debate whether the approach could work, it is actually already working. We’ll keep you posted!

🖌️ A Visual Dive into Conditional Flow Matching by Anne Gagneux, Ségolène Martin, Rémi Emonet, Quentin Bertrand, and Mathurin Massias to flow matching is what the Illustrated Transformer to transformers - a detailed visual guide on the inner workings of flow matching and math behind it, a highly recommended reading.

🪠 PLUMBER from Bioptic is the new protein-ligand benchmark of 1.8M data points based on PLINDER and enriched with more data from BindingDB, ChEMBL, and BioLip 2 to probe robustness of protein-ligand binding models in more diverse compositional generalization tests.

GraphML News (Dec 8th) - NeurIPS’24, ESM Cambrian, Antiviral Competition, The Well

🍻 NeurIPS 2024 starts next week, the full schedule is available, the main conference is scheduled for Wed-Fri with two days of workshops (Sat-Sun) and unknown amount of private parties and gatherings throughout the week. See you in Vancouver!

🧬 EvolutionaryScale announced ESM Cambrian (ESM C), a new family of embedding models replacing ESM-2 with better performance across all sizes (300M, 600M, and 6B), dramatically smaller memory requirements and faster inference (think of Triton kernels here). ESM C was trained on UniRef, MGnify, and JGI data, smaller models are already available on GitHub, the 6B is available through the API service.

💊 Polaris Hub launches the Antiviral Competition together with ASAP discovery and OpenADMET. The competition includes three tracks:

- Predicting ligand poses of MERS-CoV based on SARS-CoV2 structures (metric: RMSD)
- Predicting ligand fluorescence potencies based on SARS and MERS data (metrics: MAE of pIC50 and ranking)
- Predicting ligands’ ADMET properties (MAE and ranking)

The competition starts on Jan 13th and ends on March 25th, prepare your big GNNs ⚔️

⚛️ After the announcement in May, MSR released the code and weights of MatterSim, a universal ML potential akin to MACE-MP-0 and Orb models. MatterSim is based on the M3GNet message passing GNN and is available in 1M and 5M params versions.

🪣 Polymathic AI, Flatiron Institute, and a collab of universities and national labs released The Well, a 15 TB dataset of physical simulations (think PDEs and Neural Operators) covering 16 different areas from fluid dynamics to supernova explosions. In the accompanying preprint, the authors compared several variants of Fourier Neural Operators (FNO) and U-Nets. A great resource for scientific and industrial applications where expensive simulations eat up a huge bulk of supercomputers time.

GraphML News (Dec 20th) - Thoughts after NeurIPS, ICLR workshops, new blogposts

In the last op-ed of this year and returning back from NeurIPS, it’s about time to reflect on the state of graph learning research. Comments to this post should be open (hopefully?)

🤔 At the age when o3 solves some of incredibly hard FrontierMath problems, when NotebookLM allows to call into a podcast generated about a paper of your choice, and Veo2 generates 4K videos with increasingly correct physics, it is somewhat frustrating to see that the graph learning (meaning vanilla 2D graph learning here, because geometric DL is flourishing in the AI 4 Science areas) community is still obsessed with WL tests, node classification on OGB, and other toy tasks that increasingly lose relevance in the modern deep learning world. Is it the issue of too toyish benchmarks, the lack of cool applications, or something else? Is Graph ML to be confined in the recsys and retail predictions domain or it could get its own “RLHF revival moment”?

🏗️ ICLR 2025 started announcing the accepted workshops, you might find some of those interesting:

- Frontiers in Probabilistic Inference: Sampling Meets Learning
- Generative and Experimental Perspectives for Biomolecular Design
- Weight Space Learning
- Learning Meaningful Representations of Life
- AI for Accelerated Materials Design will be back, too

📝 New blogposts! Understanding Transformer reasoning capabilities via graph algorithms by Google Research elaborating on the NeurIPS 2024 paper on when and where transformers can outperform GNNs on graph tasks. A massive 3-part study of pooling in GNNs by Filippo Maria Bianchi (Arctic University of Norway) introduces the common pooling framework (select-reduce-connect), studies a variety of pooling methods and their evaluation protocols.

Let's use this post for comments!

GraphML News (Jan 18th) - MatterGen release, Aviary, Metagene-1

We are getting back to the regular schedule after the winter break!

⚛️ MSR AI 4 Science has finally released code and weights of MatterGen, the generative model for inorganic materials, together with its publication on Nature (free and no paywall). The final version includes new evaluation pipeline that accounts for compositional disorder (which unrealistically increases performance metrics of recent generative models) and experimental validation of the first generated material TaCr2O6. MatterSim, an ML potential model, was also used during the filtering stages along with standard DFT calculations. Great result for MSR, materials science community, and diffusion model experts who can apply a bag of tricks to a new domain 👏

📜 Together with MatterGen, MaSIF-neosurf from EPFL got published on Nature as well. MaSIF-neosurf is a geometric model for studying surfaces of protein-ligand complexes which was experimentally evaluated on binders agains three real-world protein complexes. To conclude the celebration of massive scientific publications, ESM 3 - the foundation model for proteins - got accepted at Science.

🕊️ FutureHouse released Aviary, the agentic framework for scientific tasks like molecular cloning, protein stability, and scientific QA. Aviary extends the framework of PaperQA (the best open source scientific RAG tool) with learnable RL environments and demonstrated that even small LLMs like LLaMa 3.1 7B excel at these tasks with enough inference compute. Get ready to hear about inference time scaling all over 2025 😉

✏️ The AI 4 Science consortium published a blog post AI 4 Science 2024 highlighting AF3 and its replications, the success of non-equivariant models, scientific foundation models, new progress in small molecules and quantum chemistry. A short but insightful read.

🪣 Metagene-1 (by USC and Prime Intellect) is a foundation model trained on metagenomic sequences (”human wastewater samples” we all know what it means 💩) that might help in pandemic monitoring and pathogen detection. It’s a standard LLaMa 2-7B architecture but, interestingly, outperforms some state space models like HyenaDNA on genome understanding.

Weekend reading:

GenMol: A Drug Discovery Generalist with Discrete Diffusion by Seul Lee, NVIDIA and KAIST - a generalist generative model for a suite of drug discovery tasks like de-novo generation, fragment-conditioned generation, and lead optimization.

The Jungle of Generative Drug Discovery: Traps, Treasures, and Ways Out by Riza Özçelik and Francesca Grisoni - on metrics and benchmarking for generative models for molecules.

Explaining k-Nearest Neighbors: Abductive and Counterfactual Explanations by Pablo Barceló and CENIA team from Chile - a theoretical work tackling classical (but still important) kNN classifiers and how their predictions can be explained. Experiments on MNIST and can run on a laptop

​​GraphML News (Jan 26th) - Graph learning overtakes ⚽, GPT-4b

⚽ TacticAI, the system for analyzing football games using MPNNs and geometric learning developed by Google DeepMind for Liverpool FC (one of the coolest applications highlighted in the post about 2023 graph learning advancements), is making rounds in the UK sports industry: now Arsenal FC is looking for Research Engineers to work on “spatiotemporal transformer with built-in geometric equivariances”. This hints about the success of the original TacticAI (as well as about the small world of English Premier League), and makes us wonder:

- Would graph learning get a surprising vanity boost from a rather unexpected place? Transfers of best GraphML researchers from one club to another along with the head coach team, eg, Christopher Morris to Borussia Dortmund or Stephan Günnemann to Bayern Munich?
- Would 💰 from sheikhs and rich owners of football clubs investing into GPUs result in more expected scaling laws (instead of ridiculously inflated players’ contracts)?
- When will Mikel Arteta (head coach of Arsenal) tell English newspapers “Equivariance rules!” like Geoff Hinton?

We will keep you posted about those important matters. Meanwhile, post your fantasy teams of graph researchers and FCs in the comments.

🧬 OpenAI reportedly finished training of GPT-4b, the protein LLM, together with Retro Biosciences (where Sam Altman conveniently invested $180M) that focuses on longevity studies. GPT-4b aims at re-engineering a specific set of proteins, Yamanaka factors, that can turn human skin cells into young stem cells. We don’t know more details, the model is likely to stay closed - one could hypothesize it might look like the ESM family of protein models with the knowledge of protein function and trained on a massive dataset of proprietary data (the key to a successful biotech startup in 2025).

🎙️ The Graph Signal Processing Workshop 2025 will take place on May 14-16 at Mila in Montreal supported by Centre de recherches mathématiques (CRM) and Valence Labs. The workshop invites theoretical works in signal processing on graphs and will showcase examples of applications in gene expression patterns defined on top of gene networks, the spread of epidemics over a social network, the congestion level at the nodes of a telecommunication network, and patterns of brain activity defined on top of a brain network. Submission deadline is Feb 1st.

Weekend reading:

ICLR 2025 announced accepted papers but the full list is not yet available. Moreover, expect a flurry of the announcements next week after the ICML submission deadline.

GraphML News (Feb 8th) - EEML 2025, Weather models, Science agents

Yaay, we got a handful of news this week worth writing about.

🇧🇦 The Eastern European Summer School (EEML) 2025 is coming to Sarajevo July 21-26 (right after ICML) and features a stellar group of speakers, tutorial heads, and organizers. Invited talks include Aaron Courville (Mila, UdeM), Aldan Hung (Isomorphic Labs), Diana Borsa (Google DeepMind), Samy Bengio (Apple) with tutorials led by the Oxford and DeepMind crew - which indicates the presence of graph and geometric learning will be quite strong 📈 There is a plenty of time to apply: the deadline is March 31st, it’s worth attending if you have an opportunity.

🌍 ML-based weather prediction models permeate more into everyday use: first, a few weeks ago Silurian released the Earth API to their Generative Forecasting Transformer (1.5B params) capable of short- and long-range predictions. And DeepMind released the WeatherNext bundle of GraphCast and GenCast (featured many times in media) on Google Cloud. Competition drives the progress (looking at you, DeepSeek-R1, hehe), and weather prediction models are gaining the momentum.

🤖 Andrew White (FutureHouse) published an interesting piece AI for science with reasoning models discussing how frontier models with reasoning and agentic capabilities improve scientific workflows (spoiler: by a good margin). Fast-forward to February 2025, and all major LLM providers offer their Deep Research agents who automatically digest enormous amounts of internet to create reports about your problem: Google offered Gemini Deep Research already in Dec 2024 (powered by Gemini 2.0 Flash Thinking model), OpenAI added Deep Research this week (powered by o3), and HuggingFace is building an open source version of that. One more moat is gone which could be both sad for agentic startups and happy for users who can enjoy the ecosystem they prefer.

Weekend reading:

On the Emergence of Position Bias in Transformers by Xinyi Wu et al. feat Stefanie Jegelka - a graph-based approach to analyzing positional encodings in Transformers, well in line with Round and Round we Go and other recent works on Transformer PEs

Do Graph Diffusion Models Accurately Capture and Generate Substructure Distributions? by Xiyuan Wang et al. feat Muhan Zhang - the answer is no, but if you use more expressive GNNs, then maybe. A similar finding is in HOG-Diff: Higher-Order Guided Diffusion for Graph Generation by Yiming Huang and Tolga Birdal.

GFM-RAG: Graph Foundation Model for Retrieval Augmented Generation by Linhan Luo et al - an approach based on our ULTRA can be very effective in RAG

GraphML News (Feb 15th) - ICLR 2025 papers, Upcoming Workshops, Latent Labs

📚 All ICLR 2025 accepted papers and their respective categories (orals, spotlights, posters) are now visible on OpenReview (as well as rejected papers) - we’ll make an (opinionated) list with the most interesting works. A plenty of weekend reading meanwhile.

🏁 A few announcements and upcoming deadlines: the Helmholtz-ELLIS Workshop on Foundation Models in Science will take place on March 18-19 in Berlin, speakers include a nice mix of AI 4 Science researchers like Tian Xie (MSR AI 4 Science), Shirley Ho (Flatiron), as well as hardcore LLMers like Michal Valko (Meta), Tim Dettmers (AI2), and many others.

The application deadline for the LOGML Summer School 2025 (London Geometry and ML) is February 16th. The summer school will take place July 7-11 in London.

💸 Latent Labs, a startup in generative protein design, came out of stealth with $50M funding from Radical Ventures, Sofinnova Partners, Jeff Dean, and Aidan Gomez. Latent Labs is founded largely by ex-DeepMind researchers who worked on AlphaFold 2 and 3, so the technical expertise is definitely there. We’ll keep an eye on their progress!

What else to read (other than your 10 ICML papers to review):

Spectral Journey: How Transformers Predict the Shortest Path by Andrew Cohen and Meta AI - turns out that a 2-layer transformer, when asked to find the shortest path on a graph, computes a Laplacian of the line graph and selects the next edge based on its distance to the target in the latent space.

MDCrow: Automating Molecular Dynamics Workflows with Large Language Models by Quintina Campbell et al feat. Andrew White - a new crow in the aviary - an LLM agent that can perform molecular dynamics with OpenMM, you can query it with questions like “Simulate protein 1ZNI at 300 K for 0.1 ps and calculate the RMSD over time.” and generate analytical charts. The code is available.

Scalable Discrete Diffusion Samplers: Combinatorial Optimization and Statistical Physics by Sebastian Sanokowski and JKU Linz - having tried discrete diffusion for combinatorial optimization myself, I could second that it’s hard to make it work. This paper makes the application of diffusion models in CO much easier.

GraphML News (Feb 23rd) - Achira, AI Co-Scientist, Evo-2,

The announcements of Thinking Machines, Grok-3, and Majorana-1 saturated the media this week, but there has been a good bunch of science-related news, too.

💸 Achira AI, a startup focusing on foundation simulation models for drug discovery, came out of the stealth mode with $30M seed funding from Dimension, Amplify, NVIDIA, and Compound. Founded by John Chodera (Sloan Kettering Institute), Achira aims at “creating a new class of simulation models that blend geometric deep learning, physics, quantum chemistry, and statistical mechanics into advanced potentials and generative models”. Achira will be competing with DE Shaw Research on the market of fast MD simulations, sounds interesting 📈

🧬 Arc Institute announced Evo-2, a family of foundation models trained on DNA of 100k species. Based on StripedHyena 2 (a hybrid attention-convolution architecture), Evo-2 ingests sequences of up to 1M of context length and packs most modern LLM engineering practices (no surpise that the CTO of OpenAI spent his sabbatical at Arc) - training on 9T tokens on 2048 H100s using a custom framework for hybrid models, available in 7B and 40B sizes. Everything is available on Github including two accompanying preprints focusing on the ML side of things and computational genomics side.

🤖 Google Cloud AI, Google Research, and Google DeepMind announced AI Co-Scientist - a multi-agent based system for hypothesis generation, providing research overviews, and generating plans for experiments. Powered by Deep Research and equipped with tool usage, AI Co-Scientist demonstrates benefits of test-time compute scaling and was probed in three applications: drug repurposing, treatment target discovery, and explaining mechanisms for antimicrobial resistance. There will be more published results in specialized journals in the next few weeks.

In a similar vein, Stanford researchers announced Popper agent for automated hypothesis validation and sequential falsification with several practical examples in biology, economics, and sociology. It’s fully open-source so you can plug in any LLM via vLLM / SGLang / llama.cpp

Weekend reading:

On Vanishing Gradients, Over-Smoothing, and Over-Squashing in GNNs: Bridging Recurrent and Graph Learning by Álvaro Arroyo, Alessio Gravina et al feat. Michael Bronstein - an approach to GNN theory from the SSM and recurrence point of view

Learning Smooth and Expressive Interatomic Potentials for Physical Property Prediction by Xiang Fu and FAIR Chemistry - a new SOTA on MatBench Discovery, MPTraj, and MDR Phonon

Towards Mechanistic Interpretability of Graph Transformers via Attention Graphs by Batu El, Deepro Choudhury feat. our own Chaitanya K. Joshi - among other things they found that GTs learn attention matrices quite different from the input graph you train the model on.

GraphML News (March 2nd) - GNNs at SnapChat, Lab in the Loop, Flow matching in scRNA

📸 SnapChat released a paper about GiGL - Gigantic Graph Learning library - together with the success story of recsys GNN use-cases at Snapchat powering 30+ launches and spanning applications from friend recommendations to fraud and abuse detection. GiGL adopts the graph sampling strategy around target nodes and scales to internal graphs of ~900M nodes, 16.8B edges, and dozens of node/edge features. Some interesting technical details: most of the performance improvements is brought by feature cleanup and engineering (+20 MRR points) while engineering GATv2 with more layers and neighboring nodes brings about +10 MRR points. Some lessons learned and insights: increasing off-line metrics hurts online metrics, graph sparsification is more beneficial than densification, shallow graph embeddings can sometimes be useful, too. More technical details are in the paper, the code is also published

Congrats to SnapChat on joining the elite group of GNN connoisseurs together with Pinterest, Google Maps, Amazon, Spotify, LinkedIn, Alibaba, and many others

🧬 Genentech released a massive paper describing the lab-in-the-loop process for antibody design with deep learning. Targeting clinically relevant antigens, the lab-in-the-loop includes all the recent Genentech papers including those on generative models (like recent discrete walk-jump sampling) for candidate generation and property optimization (eg, Lambo-2), property predictors (protein LMs), filtering and ranking, and experimental (in vivo) validation on 🐀. Experimentally, the active loop pipeline produced 10 candidates per target with 3-100x better binding.

✍️ Finally, Karin Hrovatin wrote an overview blog post of recent uses of flow matching models in single-cell gene expression data (scRNA) covering CFGen, Wasserstein Flow Matching, and Meta Flow Matching.

GraphML News (March 16th) - ICLR 2025 Workshops, Mediterranean Summer School

🍻 ICLR 2025 is approaching and it’s time to select some workshops to attend to chat with friends and hide from the heat of Singapore. Graph learning aficionados might be interested in a various bio / health / science workshops:

- Neural Network Weights as a New Data Modality
- GemBio
- ML for Genomics
- Agentic AI 4 Science
- AI for Nucleic Acids
- Learning Meaningful Representations of Life (LMRL)
- AI 4 Material Discovery
- Frontiers in Probabilistic Inference: learning meets Sampling

Some of them have already published the accepted papers on OpenReview - here is a usual reminder to go find some hidden gems as most workshop papers evolve into full conference papers.

🇭🇷 The Balkans get all the fancy machine learning summer schools in 2025: we know that EEML 2025 will take place in Sarajevo, July 21-26. If you won’t make it, the Mediterranean Machine Learning Summer School (M2ML) will open its doors in Split, Sept 8-12 (just 6 hours by bus from Sarajevo). The school is organized by Google DeepMind folks together with the University of Split. The application deadline is March 28th, and there are some rumors about the GNN tutorial held by some familiar faces in this channel 😉

Weekend reading is brought to you by non-equivariant transformers that go brrr in various domains:

Erwin: A Tree-based Hierarchical Transformer for Large-scale Physical Systems by Maksim Zhdanov, Max Welling, and Jan-Willem van de Meent - introduces a smart variant of local attention - ball tree attention - for large particle systems where nearest neighbors are progressively encoded into a ball tree, and the attention is only computed wrt the given ball. Excels in MD and fluid dynamics sims. Code

Proteina: Scaling Flow-based Protein Structure Generative Models by Tomas Geffner, Kieran Didi, Zuobai Zhang, and NVIDIA folks (ICLR 2025 Oral) - scaling flow matching with non-equivariant transformers to 400M params yields significant improvements over Genie2, Chroma, RFDiffusion across many protein design tasks. You only need 128 GPUs 😉 Code and checkpoints

All-atom Diffusion Transformers: Unified generative modelling of molecules and materials by Chaitanya K. Joshi, Xiang Fu, and FAIR at Meta - latent diffusion with a single all-atom transformer for both molecules (on QM9) and periodic structures (MP20) scaled up to 500M params is very competitive with the current equivariant SOTA while being much faster.

​​GraphML News (March 23rd) - Neo-1 and Lila Sciences round

🧬 VantAI announced Neo-1, a foundation model for structure prediction and de novo generation capable of doing a bunch of protein design tasks (folding, co-folding, docking, all-atom molecule design, fragment linking, and more) at once instead of different modules. While we are waiting for the tech report, we could guesstimate that Neo-1 is an all-atom latent generative model (perhaps a Diffusion Transformer like in other competitors as it’s powered by a hefty cluster of H100s) with some advanced sampling techniques beyond standard guidance - the blog post talks about optimizing for non-differentiable properties with reward-like models and it sounds quite similar to the ICLR 2025 paper on posterior prediction.

As impressive as the modeling advances are, true aficionados know that data diversity and distribution is even more important at scale - on that front VantAI introduce NeoLink, a massive data generation flywheel based on cross-linking mass-spectrometry (XLMS). Reported experiments suggest it brings massive improvements in quality, so it’s likely to be the key innovation and the point of further scaling up. The graphics in the blog post are amazing and the graphic designer should get a raise 📈.

💸 Lila Sciences went out of stealth with $200M seed funding. Lila will focus on materials discovery and automated self-driving labs while alluding to Superscience, an AI 4 Science equivalent of Super Intelligence you often hear from LLM folks which would massive speed up exploration pipelines. Lila is part of the Flagship Pioneering ecosystem (you might know Generate Biomedicines and their Chroma generative model made some noise last year) and attracted funding from General Catalyst, March Capital, ARK, and other famous VCs (even Abu Dhabi Investment Authority). Knowing that the OpenAI VP of post-training William Fedus left to start his own AI 4 Science company, the area is likely to attract even more VC funding in the near future.

Weekend reading:

Towards Quantifying Long-Range Interactions in Graph Machine Learning: a Large Graph Dataset and a Measurement by Huidong Liang and Oxford folks - introduces new long-range graph datasets extracted from road networks in OpenStreetMap. Good news: graphs are quite large and sparse (100k nodes with 100+ diameter). Less good news: GraphSAGE is still SOTA 🫠

No Metric to Rule Them All: Toward Principled Evaluations of Graph-Learning Datasets by Corinna Coupette, Jeremy Wayland, et al - studies the quality of 11 graph classification datasets, only NC11, MolHIV, and LRGB datasets are ok, others should be thrown to garbage.

A Materials Foundation Model via Hybrid Invariant-Equivariant Architectures by Keqiang Yan and large Texas A&M collab - introduces HIENet, an ML potential rivaling MACE-MP0, Equiformer, and ORB on energy, forces, and stresses predictions.

Survey on Generalization Theory for Graph Neural Networks by Antonis Vasileiou, Stefanie Jegelka, Ron Levie, and Christopher Morris - everything you wanted to know about GNNs linked to VC dimension, Rademacher complexity, PAC-Bayes, and learning theory. MATH ALERT

GraphML News (April 5th) - Isomorphic Round, Graph Transformers at Kumo, new blogs

Got some news!

💸 Isomorphic Labs raised a generous $600M from Thrive Capital, GV, and Alphabet in the first external round. The attached press release also mentions collaborations with pharma giants Eli Lilly and Novartis - seems like whatever comes next after AlphaFold 3 looks quite appealing to the industry. We’ll keep you posted in our Geometric Wall Street Bulletin.

🏵️ Looking at LLM guts from the graph learning perspective becomes popular: Anthropic posted a massive study in two papers and lots of visual material on AI biology with strong graph vibes - LLMs perform multi-hop reasoning with concept graphs in mind, and you can actually identify circuits (DAGs) of activations doing certain kind of computation.

🚚 Kumo published a nice blog post on using graph transformers at scale in relational DL tasks. Perhaps the most insightful part is about positional encodings - as graphs are large (10M+ nodes in RelBench), global PEs don’t really scale up so they have to resort to more local options like hop encoding or relative PEs. Besides, there is a need for time encoding as the e-commerce graphs are always temporal. Experiments on RelBench bring some noticeable improvements.

⌛ Bryan Perozzi from Google Research (the OG of DeepWalk) wrote a review post looking at the milestones of graph learning from pre-historic times (pre-2013) to the most recent applications (those we often highlight in this channel).

Weekend reading:

Why do LLMs attend to the first token? by Federico Barbero, Alvaro Arroyo, and all the familiar folks from Transformers Need Glasses - here the authors study attention sinks and draw parallels to over-squashing and representational collapse.

On that note, don’t miss the talk by Petar Veličković on LLMs as Graph Neural Networks at the recent GLOW reading group - it adds much more context to the research area and hints at the above paper.

Affordable AI Assistants with Knowledge Graph of Thoughts by Maciej Besta and 13 (👀) co-authors. Maciej is the author of the famous Graph of Thoughts, here it’s extended to KGs and agentic environments.

GraphML News (April 13th) - Orb V3, RF Diffusion 2, Breakthrough Prizes, ICML 2025 Workshops

🔮 Orbital Materials released Orb V3, the next version of the universal ML potential. Some improvements include training a wider but shallower model (5-layer MPNN with 1024d MLP instead of 15-layer with 512d in v2), having both versions where forces are predicted directly (non-conservative) or as a gradient of energy (conservative force field), and a good bunch of training tricks listed on github. ORB v3 shows top results on MatBench Discovery and now has a confidence prediction head akin to pLDDT in AlphaFold. The accompanying paper and model checkpoint are available, plug them in right away.

🧬 The Baker Lab released RF Diffusion 2 (as a pre-print right now) focusing on de novo enzyme design. RFD2 is a Riemannian flow matching model in both frame and coordinate space (the frame part is very much like FrameFlow) and was trained for 17 days on 24 A100s (rather average compute those days). RFD2 outperforms original RFDiffusion on older and newly constructed benchmarks, the code is coming soon.

🏆 The Breakthrough Prize (founded by Sergey Brin, Mark Zuckerberg, Yuri Milner, and other famous tech folks) announced 2025 laureates: the authors of GLP-1 (aka Ozempic) got the Life Sciences award, CERN and Large Hadron Collider got the physics award, and Dennis Gaitsgory got the mathematics prize for proving the geometric Langlands conjecture (from which a proof of Ferma’s theorem stems naturally). Check out also the New Frontiers and New Horizons categories dedicated for younger scientists.

Finally, ICML 2025 announced a list of accepted workshops, all the usual suspects are there: generative models, comp bio, AI 4 Science, a handful of LLM and world models workshops, should be a nice selection for those attending in Vancouver in summer.

Weekend reading:

Orb-v3: atomistic simulation at scale by Benjamin Rhodes, Sander Vandenhaute, and folks from Orbital Materials

Atom level enzyme active site scaffolding using RFdiffusion2 by Woody Ahern, Jason Yim, Doug Tischer, UW and Baker Lab (including the man himself)

Graph Learning Will Lose Relevance Due To Poor Benchmarks

by Maya Bechler-Speicher, Ben Finkelshtein, Fabrizio Frasca, Luis Müller, Jan Tönshoff, Antoine Siraudin, Viktor Zaverkin, Michael M. Bronstein, Mathias Niepert, Bryan Perozzi, Mikhail Galkin, Christopher Morris

📜 arxiv

📣 Our new spicy ICML 2025 position paper. Graph learning is less trendy in the ML world than it was in 2020-2022. We believe the problem is in poor benchmarks that hold the field back - and suggest ways to fix it!

We identified three problems:
#️⃣ P1: No transformative real-world applications - while LLMs and geometric generative models become more powerful and solve complex tasks every generation (from reasoning to protein folding), how transformative could a GNN on Cora or OGB be?

P1 Remedies: The community is overlooking many significant and transformative applications, including chip design and broader ML for systems, combinatorial optimization, and relational data (as highlighted by RelBench). Each of them offers $billions in potential outcomes.

#️⃣ P2: While everything can be modeled as a graph, often it should not be. We made a simple experiment and probed a vanilla DeepSet w/o edges and a GNN on Cayley graphs (fixed edges for a certain number of nodes) on molecular datasets and the performance is quite competitive.

#️⃣ P3: Bad benchmarking culture (this one hits hard) - it’s a mess 🙂
Small datasets (don’t use Cora and MUTAG in 2025), no standard splits, and in many cases recent models are clearly worse than GCN / Sage from 2020. It gets worse when evaluating generative models.

Remedies for P3: We need more holistic benchmarks which are harder to game and saturate - while it’s a common problem for all ML fields, standard graph learning benchmarks are egregiously old and rather irrelevant for the scale of problems doable in 2025.

💡 As a result, it’s hard to build a true foundation model for graphs. Instead of training each model on each dataset, we suggest using GNNs / GTs as processors in the “encoder-processor-decoder” blueprint, train them at scale, and only tune graph-specific encoders/decoders.
For example, we pre-trained several models on PCQM4M-v2, COCO-SP, and MalNet Tiny, and fine-tuned them on PascalVOC, Peptides-struct, and Stargazers to find that graph transformers benefit from pre-training.

The project started around NeurIPS 2024 when Christopher Morris gathered us to discuss the peeve points of graph learning and how to continue to do impactful research in this area. I believe the outcomes appear promising, and we can re-imagine graph learning in 2025 and beyond!