DGL: Billion-Scale Graphs and Sparse Matrix API

In a new release 0.9.1 DGL accelerated the pipeline of working with very large graphs (5B edges). Before it was taking 10 hours and 4TB of RAM and now 3 hours and 500GB of RAM, which also reduces the cost by 4x.

Also, if you use or would like to use sparse API for your GNNs, you can provide the feedback and use cases to the DGL team (feel free to reach out to @ivanovserg990 to connect). They are looking for the following profiles:

* Researchers/students who are familiar with sparse matrix notations or linear algebra.
* May have math or geometry backgrounds.
* Work majorly on innovating GNN architecture; less on domain applications.
* May have PyG/DGL experience.

Hot New Graph ML Submissions from ICLR

🧬 Diffusion remains the top trend in AI/ML venues this year, including the graph domain. Ben Blaiszik compiled a Twitter thread of interesting papers in AI 4 Science domain including material discovery, catalyst discovery, and crystallography. Particularly cool works:

- Protein structure generation via folding diffusion by the collab between Stanford and MSR - Kevin E. Wu, Kevin K. Yang, Rianne van den Berg, James Y. Zou, Alex X. Lu, Ava P. Amini - why do you need AlphaFold and MSAs if you can just train a diffusion model to predict all the structure? 😉

- Dynamic-Backbone Protein-Ligand Structure Prediction with Multiscale Generative Diffusion Models by NVIDIA and Caltech - Zhuoran Qiao, Weili Nie, Arash Vahdat, Thomas F. Miller III,  Anima Anandkumar

- DiffDock: Diffusion Steps, Twists, and Turns for Molecular Docking by MIT - Gabriele Corso, Hannes Stärk, Bowen Jing, Regina Barzilay, Tommi Jaakkola - the next version of the famous EquiDock and EquiBind combined with the recent Torsional Diffusion.

- We’d include here a novel benchmark work Tartarus: A Benchmarking Platform for Realistic And Practical Inverse Molecular Design by Stanford and University of Toronto - AkshatKumar Nigam, Robert Pollice, Gary Tom, Kjell Jorner, Luca A. Thiede, Anshul Kundaje, Alan Aspuru-Guzik

📚 In a more general context, Yilun Xu shared a Google Sheet with ICLR submissions on diffusion papers and score-based generative modeling including trendy text-to-video models announced by FAIR and Google.

🤖 Derek Lim compiled a Twitter thread on 10+ ICLR submissions on Graph Transformers - the field looks a bit saturated at the moment, let’s see what reviewers say.

🪓 Michael Bronstein’s lab at Twitter announced two cool papers:

- Gradient Gating for Deep Multi-Rate Learning on Graphs by the collab between ETH Zurich, Oxford, and Berkley - T. Konstantin Rusch, Benjamin P. Chamberlain, Michael W. Mahoney, Michael M. Bronstein, Siddhartha Mishra. A clever trick improving a standard residual connection to allow nodes to get updated ad different speeds. A blast from the past - GraphSAGE from 2017 with gradient gating becomes a unanimous leader by a large margin in heterophilic graphs 👀

- Graph Neural Networks for Link Prediction with Subgraph Sketching by Benjamin Paul Chamberlain, Sergey Shirobokov, Emanuele Rossi, Fabrizio Frasca, Thomas Markovich, Nils Hammerla, Michael M. Bronstein, Max Hansmire. A neat usage of sketching to encode subgraphs in ELPH and its more scalable buddy BUDDY for solving link prediction in large graphs.

📏 Long Range Graph Benchmark

Vijay Dwivedi (NTU, Singapore) published a new blogpost on long-range graph benchmarks introducing 5 new challenging tasks in node classification, link prediction, graph classification, and graph regression.

“Many of the existing graph learning benchmarks consist of prediction tasks that primarily rely on local structural information rather than distant information propagation to compute a target label or metric. This can be observed in datasets such as ZINC, ogbg-molhiv and ogbg-molpcba where models that rely significantly on encoding local (or, near-local) structural information continue to be among leaderboard toppers.”

LRGB, a new collection of datasets, aims at evaluating long-range capabilities of MPNNs and graph transformers. Particularly, the node classification tasks were derived from image-based Pascal-VOC and COCO, the link prediction task is derived from PCQM4M asking about links between atoms distant in the 2D space (5+ hops away) but close in the 3D space where only 2D features are given, and the graph-level tasks focus on predicting structures and functions of small proteins (peptides).

Message passing nets (MPNNs) are known to suffer from the bottleneck effects and oversquashing and, hence, underperform in long-range tasks. First LRGB experiments confirm that showing that fully-connected graph transformers quite significantly outperform MPNNs. A big room for improving MPNNs!



Paper, Code, Leaderboard

Graph Papers of the Week

Expander Graph Propagation by Andreea Deac, Marc Lackenby, Petar Veličković. A clever approach to bypass bottlenecks without fully-connected graph transformers. Turns out that sparse but well-connected 4-regular Cayley graphs (expander graphs) can be a helpful template for message propagation. Cayley graphs of a desired size can be pre-computed w/o looking at the original graph. Practically, you can add a GNN layer propagating along a Cayley graph after each normal GNN layer over the original graph.

The anonymous ICLR 2023 submission Exphormer: Scaling Graph Transformers with Expander Graphs applies the same idea of expander graphs as a sparse attention in Graph Transformers allowing them to scale to ogb-arxiv (170k nodes)

Rethinking Knowledge Graph Evaluation Under the Open-World Assumption by Haotong Yang, Zhouchen Lin, Muhan Zhang. When evaluating KG link prediction tasks, there is no guarantee that the test set contains really all missing triples. The authors show that if there is an additional set of true triples (not labeled as true in the test), as small as 10% of the test set, MRR on the original test set only log-correlates with the MRR on the true test set. It means that if your model shows 40% MRR on the test set and you think it’s incomplete, chances are the true MRR can be much higher, you should inspect the top predictions as possibly new unlabeled true triples.

Pre-training via Denoising for Molecular Property Prediction by Sheheryar Zaidi, Michael Schaarschmidt, and DeepMind team. The paper takes the NoisyNodes SSL objective to the next level (aka NoisyNodes on steroids). NoisyNodes takes a molecular graph with 3D coordinates, adds Gaussian noise to those 3D features, and asks to predict this noise as a loss term. NoisyNodes, as an auxiliary objective, was used in many OGB Large-Scale Challenge winning approaches, but now the authors study NoisyNodes as the sole pre-training SSL objective. Theory-wise, the authors find a link between denoising and score-matching (commonly used in generative diffusion models) and find that denoising helps to learn force fields. MPNN pre-trained on PCQM4Mv2 with this objective transfers well to QM9 and OC20 datasets and often outperforms fancier models like DimeNet++ and E(n)-GNN.

Blog Posts of the Week

A few fresh blog posts to add to your weekend reading list.

Graph Neural Networks as gradient flows by Michael Bronstein, Francesco Di Giovanni, James Rowbottom, Ben Chamberlain, and Thomas Markovich. The blog summarizes recent efforts in understanding GNNs from the physics perspective. Particularly, the post describes how GNNs can be seen as gradient flows that help in heterophilic graphs. Essentially, the approach implies having one symmetric weight matrix W shared among all GNN layers, residual connections, and non-linearities can be dropped. Under this sauce, classic GCNs by Kipf & Welling strike back!

Graph-based nearest neighbor search by Liudmila Prokhorenkova and Dmitry Baranchuk. The post gives a nice intro to the graph-based technology (eg, HNSW) behind many vector search engines and reviews recent efforts in improving scalability and recall. Particularly, the authors show that non-Euclidean hyperbolic space might have a few cool benefits unattainable by classic Euclidean-only algorithms.

Long Range Graph Benchmark by Vijay Dwivedi. Covered in one the previous posts in this channel, the post introduces a new suite of tasks designed for capturing long-range interactions in graphs.

Foundation Models are Entering their Data-Centric Era by Chris Ré and Simran Arora. The article is very relevant to any large-scale model pre-training in any domain, be it NLP, Vision, or Graph ML. The authors observe that in the era of foundation models we have to rethink how we train such big models, and data diversity becomes the single most important factor of inference capabilities of those models. Two lessons learned by the authors: “Once a technology stabilizes the pendulum for value swings back to the data” and “We can (and need to) handle noise”.

GraphML News

Today (Oct 21st) MIT hosts the first Molecular ML Conference (MoML 2022).

OpenBioML backed by StabilityAI (creators of Stable Diffusion) launches an open-source initiative to improve protein structure prediction. The base implementation will be OpenFold — powered by the cluster behind Stable Diffusion, we could expect full reproduction of AlphaFold experiments, ablations, and, of course, better interfaces thanks to the open-source community!

OpenMM, one of the most popular Python frameworks for molecular modeling, released a new version 8.0.

The workshop on Geometric Deep Learning in Medical Image Analysis (GeoMediA), to be held on Nov 18th in Amsterdam, published the list of accepted papers and a program including keynotes by Emma Robinson and Michael Bronstein.

Wednesday Papers

Something you might be interested in while waiting for the LOG reviews (unless you are writing emergency reviews, hehe)

- Expander Graphs Are Globally Synchronising by Pedro Abdalla, Afonso S. Bandeira, Martin Kassabov, Victor Souza, Steven H. Strogatz, Alex Townsend. In the previous posts, we covered interesting properties of Expander Graphs (Cayley graphs). This new work on the theory side employs expander graphs to demonstrate that random Erdos-Renyi graphs G(n,p) are connected if p ≥ (1 + eps)(log n)/n 👏

- On Classification Thresholds for Graph Attention with Edge Features by Kimon Fountoulakis, Dake He, Silvio Lattanzi, Bryan Perozzi, Anton Tsitsulin, Shenghao Yang

- Simplifying Node Classification on Heterophilous Graphs with Compatible Label Propagation by Zhiqiang Zhong, Sergei Ivanov, Jun Pang

- Graphein - a Python Library for Geometric Deep Learning and Network Analysis on Protein Structures and Interaction Networks by Arian R. Jamasb, Ramon Viñas, Eric J. Ma, Charlie Harris, Kexin Huang, Dominic Hall, Pietro Lió, Tom L. Blundell

- Annotation of spatially resolved single-cell data with STELLAR by Maria Brbić, Kaidi Cao, John W. Hickey, Yuqi Tan, Michael P. Snyder, Garry P. Nolan & Jure Leskovec

Webinar on Fraud Detection with GNNs

Graph neural networks (GNN) are increasingly being used to identify suspicious behavior. GNNs can combine graph structures, such as email accounts, addresses, phone numbers, and purchasing behavior to find meaningful patterns and enhance fraud detection.

Join the webinar by Nikita Iserson, Senior ML/AI Architect at TigerGraph, to learn how graphs are used to uncover fraud on Thursday, Oct 27th, 6pm CET.

Agenda:

• Introduction to TigerGraph
• Fraud Detection Challenges
• Graph Model, Data Exploration, and Investigation
• Visual Rules, Red Flags, and Feature Generation
• TigerGraph Machine Learning Workbench
• XGBoost with Graph Features
• Graph Neural Network and Explainability

GraphML News

It’s Friday - time to look back at what happened in the field this week.

📚 Blogs & Books

(Editors’ Choice 👍) An Introduction to Poisson Flow Generative Models by Ryan O’Connor. Diffusion models are the hottest topic in Geometric Deep Learning but have an important drawback - the sampling is slow 🐌 due to necessity of performing 100-1000 forward passes. Poisson Flow generative models take inspiration from physics and offer another look at the generation process that allows much much faster sampling. This blog gives a very detailed and pictorial explanation of Poisson Flows.

Awesome GFlowNets by Narsil-Dinghuai Zhang. Generative Flow Networks (GFlowNets) bring together generative modeling with ideas from reinforcement learning and show especially promising results in drug discovery. This Awesome repo will get you acquainted with the main ideas, most important papers, and some implementations

Sheaf Theory through Examples - a book by Daniel Rosiak on the sheaf theory. If you felt you want to know more after reading the Sheaf Diffusion paper - this would be your next step.

🗞️ News & Press

Elon Musk finally acquired Twitter so it’s time to move to Telegram

Mila and Helmholtz Institute announced a new German-Canadian partnership on developing causal models of the cell. As Geometric DL is in the heart of modern structural biology, we’ll keep an eye on the future outcomes.

🛠️ Code & Data

We somehow missed that but catching up now - the DGL team at Amazon published the materials of the KDD’2022 tutorial on GNNs in Life Sciences.

Geometric Kernels - a new fresh framework for kernels and Gaussian processes on non-Euclidean spaces (including graphs, meshes, and Riemannian manifodls). Supports PyTorch, TensorFlow, JAX, and Numpy.

A post with the hot new papers for your weekend reading will be arriving shortly!

Halloween Paper Reading 🎃

We hope you managed to procure enough candies and carve spooky faces on a bunch of pumpkins those days so now you can relax and read a few papers (not that spooky).

Molecular dynamics is one of the booming Geometric DL areas where equivariant models show the best qualities. The two cool recent papers on that topic:

⚛️ Forces are not Enough: Benchmark and Critical Evaluation for Machine Learning Force Fields with Molecular Simulations by Fu et al. introduces a new benchmark for molecular dynamics - in addition to MD17, the authors add datasets on modeling liquids (Water), peptides (Alanine dipeptide), and solid-state materials (LiPS). More importantly, apart from Energy as the main metric, the authors consider a wide range of physical properties like Stability, Diffusivity, and Radial Distribution Functions. Most SOTA molecular dynamics models were probed including SchNet, ForceNet, DimeNet, GemNet (-T and -dT), NequIP.

Density Functional Theory (DFT) calculations are one of the main workhorses of molecular dynamics (and account for a great deal of computing time in big clusters). DFT is O(n^3) to the input size though, so can ML help here? Learned Force Fields Are Ready For Ground State Catalyst Discovery by Schaarschmidt et al. present the experimental study of models of learned potentials - turns out GNNs can do a very good job in O(n) time! Easy Potentials (trained on Open Catalyst data) turns out to be quite a good predictor especially when paired with a subsequent postprocessing step. Model-wise, it is an MPNN with the NoisyNodes self-supervised objective that we covered a few weeks ago.

🪐 For astrophysics aficionados: Mangrove: Learning Galaxy Properties from Merger Trees by Jespersen et al. apply GraphSAGE to merger trees of dark matter to predict a variety of galactic properties like stellar mass, cold gas mass, star formation rate, and even black hole mass. The paper is heavy on the terminology of astrophysics but pretty easy in terms of GNN parameterization and training. Mangrove works 4-9 orders of magnitude faster than standard models (that is, 10 000 - 1 000 000 000 times faster). Experimental charts are pieces of art that you can hang on a wall.

🤖 Compositional Semantic Parsing with Large Language Models by Drozdov, Schärli et al. pretty much solve the compositional semantic parsing task (natural language query - structured query like SPARQL) using only code-davinci-002 language model from OpenAI (which is InstructGPT fine-tuned on code). No need for hefty tailored semantic parsing models - turns out a smart extension of the Chain-of-thought prompting (aka "let's think step by step") devised as Least-to-Most prompting (where we first answer easy subproblems before generating a full query) yields whopping 95% accuracy even on hardest Compositional Freebase Questions (CFQ) dataset. CFQ was introduced at ICLR 2020, and just after two years LMs cracked this task - looks like it's time for the new, even more complex dataset.

ESM Metagenomic Atlas

Meta AI just published the ESM Metagenomic Atlas - a collection of >600M metagenomic protein structures built with ESMFold - the most recent model from Meta for protein folding. We covered ESMFold a few months ago, and both ESM-2 and ESMFold are available in the recent 🤗 Transformers 4.24 release (checkpoints for 8M - 3B models for ESM2, a full checkpoint for ESMFold). That’s a nice flex from Meta AI after DeepMind released 200M AlphaFold predictions for PDB, the community definitely benefits from the competition.

Weekend Reading

For those who are not busy with ICLR rebuttals — you can now have a look at all accepted NeurIPS’22 papers on OpenReview (we will have a review of graph papers at NeurIPS a bit later). Meanwhile, the week brought several cool new works:

Are Defenses for Graph Neural Networks Robust? by Felix Mujkanovic, Simon Geisler, Stephan Günnemann, Aleksandar Bojchevski. Probably THE most comprehensive work of 2022 on adversarial robustness of GNNs.

TuneUp: A Training Strategy for Improving Generalization of Graph Neural Networks by Weihua Hu, Kaidi Cao, Kexin Huang, Edward W Huang, Karthik Subbian, Jure Leskovec. The paper introduces a new self-supervised strategy by asking the model to generalize better on tail nodes of the graph after some synthetic edge dropout. Works in node classification, link prediction, and recsys.

Neural Set Function Extensions: Learning with Discrete Functions in High Dimensions by Nikolaos Karalias, Joshua David Robinson, Andreas Loukas, Stefanie Jegelka. Insightful theoretical work on set functions and discrete learning. Particularly good results on combinatorial optimization problems like max clique and max independent set.

GraphML News (11.11.22)

This week was dominated by molecular ML and drug discovery events:

- Broad Institute published a YouTube playlist of talks from the recent Machine Learning and Drug Discovery Symposium and leading drug discovery researchers.

- ELLIS Machine Learning for Drug Discovery Workshop will take place online in Zoom and GatherTown on Nov 28th, registration is free!

- Valence Discovery launched a blog platform for Drug Discovery related posts, the inaugural post by Clemens Isert talks about Quantum ML for drug-like molecules.

And a new work from GemNet authors: How robust are modern graph neural network potentials in long and hot molecular dynamics simulations? The work supports a recent line of works (see the Halloween post) on ML force fields particularly focusing on hot dynamics where low test MAE does not necessarily correspond to good simulations. The most important robustness factor seems to be more training data!

​​Max Welling, Regina Barzilay, Michael Bronstein @ AI Helps Ukraine Charity Conference

Mila hosts a fundraising conference *AI Helps Ukraine: Charity Conference*. The goal of the conference is to raise funds to support Ukraine with medicines and humanitarian aid. It will consist of a series of online talks during the month of November and an in-person event on the 8th of December at Mila under the theme AI for Good. The online part will feature the talks of renowned AI researchers including Yoshua Bengio, Max Welling, Alexei Efros, Regina Barzilay, Timnit Gebru and Michael Bronstein.

A stellar lineup for Graph ML research! We encourage you to support this wonderful initiative.

- Tomorrow (Nov 14th) 4pm UTC Max Welling will be giving a talk on Generating and Steering Molecules with ML and RL.
- Nov 21st - Regina Barzilay will talk about Expanding the reach of molecular models in the drug discovery space
- Finally, Michael Bronstein (as recurring Graph Santa) will give a talk after LOG closer to Christmas.

Recordings of previous talks by Timnit Gebru, Yoshua Bengio, Irina Rish are already available, the full schedule with all speakers and the in-person event is up to date on the website.

Denoising Diffusion Is Still All You Need (Weekend Reading)

In the previous post in June we covered the emergence of denoising diffusion models (DDPMs) in generative Geometric DL tasks. After 5 months, we can acknowledge that diffusion models became the first-class citizen in Geometric DL with numerous works appearing on arxiv every week. Here are 4 recent and very interesting works you might want to check:

1️⃣ DiGress by Clemént Vignac, Igor Krawczuk, and the EPFL team is the unconditional graph generation model (although with the possibility to incorporate a score-based function for conditioning on graph-level features like energy MAE). DiGress is a discrete diffusion model, that is, it operates on discrete node types (like atom types C, N, O) and edge types (like single / double / triple bond) where adding noise corresponds to multiplication with the transition matrix (from one type to another) mined as marginal probabilities from the training set. The denoising neural net is a modified Graph Transformer. Works for many graph families - planar, SBMs, and molecules, code is available, and check the video from the reading group presentation!

2️⃣ DiffDock by Gabriele Corso, Hannes Stärk, Bowen Jing, and the MIT team is the score-based generative model for molecular docking, eg, given a ligand and a protein, predicting how a ligand binds to a target protein. DiffDock runs the diffusion process over translations T(3), rotations SO(3), and torsion angles SO(2)^m in the product space: (1) positioning of the ligand wrt the protein (often called binding pockets), the pocket is unknown in advance so it is blind docking, (2) defining rotational orientation of the ligand, and (3) defining torsion angles of the conformation.

DiffDock trains 2 models: the score model for predicting actual coordinates and the confidence model for estimating the likelihood of the generated prediction. Both models are SE(3)-equivariant networks over point clouds, but the heavier score model works on protein residues from alpha-carbons (initialized from the now-famous ESM2 protein LM) while the confidence model uses the fine-grained atom representations. Initial ligand structures are generated by RDKit. DiffDock dramatically improves the prediction quality, code is available, and you can even upload your own proteins (PDB) and ligands (SMILES) in the online demo on HuggingFace spaces to test it out!

3️⃣ DiffSBDD by Schneuing, Du, and the team from EPFL, Cornell, Cambridge, MSR, USTC, Oxford is the diffusion model for generating novel ligands conditioned on the protein pocket. DiffSBDD can be implemented with 2 approaches: (1) pocket-conditioned ligand generation when the pocket is fixed; (2) inpainting-like generation that approximates the joint distribution of pocket-ligand pairs. In both approaches, DiffSBDD relies on the tuned equivariant diffusion model (EDM, ICML 2022) and equivariant EGNN as the denoising model. Practically, ligands and proteins are represented as point clouds with categorical features and 3D coordinates (proteins can be alpha-carbon residues or full atoms, one-hot encoding of residues — ESM2 could be used here in future), so diffusion is performed over the 3D coordinates ensuring equivariance. The code is already available!

Denoising Diffusion Is Still All You Need (Part 2)

4️⃣ DiffLinker from Igashov, Stärk, and EPFL / MSR / Oxford co-authors is the diffusion model for generating molecular linkers conditioned on 3D fragments. While previous models are autoregressive (hence not permutation equivariant) and can only link 2 fragments, DiffLinker generates the whole structure and can link 2+ fragments. In DiffLinker, each point cloud is conditioned on the context (all other known fragments and/or protein pocket), the context is usually fixed. The diffusion framework is similar to EDM but is now conditioned on the 3D data rather than on scalars. The denoising model is the same equivariant EGNN. Interestingly, DiffLinked has an additional module to predict the linker size (number of molecules) so you don’t have to specify it beforehand. The code is available, too!

Even more: SMCDiff for generating protein scaffolds conditioned on a desired motif (also with EGNN). Generally, in graph and molecule generation we’d like to support some discreteness, so any improvements to the discrete diffusion are very welcome, eg, Richemond, Dieleman, and Doucet propose a new simplex diffusion for categorical data with the Cox-Ingersoll-Ross SDE (rare find!). Discrete diffusion is also studied for text generation in the recent DiffusER.

We’ll spare your browser tabs for now 😅 but do expect more diffusion models in Geometric DL!

OGB Large Scale Challenge 2022 Winners

The OGB team has just announced the winners of the annual Large Scale Challenge - the Graphcore team celebrates a double win: top-1 entries in graph regression and link prediction!

Notably, most of the winning top-3 models in all tracks are ensembles. The final results:

Track 1: Node Classification on MAG240M:

1) Baidu - RUniMP and positional encodings
2) Michigan State Uni / TigerGraph - APPNP + relational GAT, 10-model ensemble
3) Beijing Institute of Technology / Zhipu AI / Tsinghua University - MDGNN, 15-model ensemble

Track 2: Link prediction and KG completion on WikiKG90M v2:

1) Graphcore - ensemble of 85 shallow embedding models
2) Microsoft Research Asia - ensemble of 13 shallow models and 10 manual features
3) Tencent - ensemble of 6 models

Track 3: Graph Regression on PCQM4M v2:

1) Graphcore / Mila / Valence Discovery - GPS++ (the improved version of GraphGPS covered in this channel), 112 model ensemble
2) Microsoft Research AI4Science - Transformer-M + ViSNet, 22 model ensemble
2) NVIDIA / UCLA - ensemble of Transformer-M, GNN, and ResNet
MSR and NVIDIA share the joint 2nd place due to reporting exactly the same MAE on the test set.

Friday News: LOG Accepted Papers, NeurIPS

The inaugural event of the Learning of Graphs (LOG) conference announced accepted papers, extended abstracts, and spotlights - the acceptance rate this year is pretty tough (<25%) but we have heard multiple times that the quality of reviews is on average higher than in other big conferences. Is it the impact of the $$$ rewards for the best reviewers?

Tech companies summarize their presence at NeurIPS’22 that starts next week: have a look at works from DeepMind, Amazon, Microsoft, and the GraphML team from Google.

A new blog post by Petar Veličković and Fabian Fuchs on universality of neural networks on sets and graphs - the authors identify a direct link between permutation-invariant DeepSets and permutation-invariant aggregations in GNNs like GIN. However, when it comes to multisets (such as nodes sending exactly the same message), PNA might be more expressive thanks to the link to the theoretical findings - given a set of n elements, that the width of the encoder should be at least n - recall that PNA postulates that it is necessary to have n aggregators. Nice read with references!

Denoising Diffusion Is All You Need in Graph ML? - Now on Medium

We just published the extended version of the posts on diffusion models on Medium with more spelled out intro and newly generated images by Stable Diffusion! A good option to spend the time if you are on the way to New Orleans and NeurIPS.

GPS++ (OGB LSC’22 Winner) is Available on IPUs

GPS++, the model by Graphcore, Mila, and Valence Discovery that won the OGB Large-Scale Challenge 2022 in the PCQM4M v2 track (graph regression) is now publicly available on Paperspace with simple training and inference examples in Jupyter Notebooks. Actually, you can try it on powerful IPUs — custom chips and servers built by Graphcore for optimized sparse operations. Raw checkpoints are also available in the official Github repo.

Friday News: PyG 2.2 and Protein Diffusion Models

For those who are at the NeurIPS workboat 2022, Saturday and Sunday are days of workshops on graph learning, structural biology, physics, and material discovery. Apart from that,

The PyG team has finally released PyG 2.2.0, the first version to feature the super-optimized pyg-lib that speeds up GNNs and sampling on both CPUs and GPUs (sometimes up to 20x!). The 2.2 update also includes new FeatureStore and GraphStore with which you can set up communications with large databases and graphs too big to store in memory. Time to update your envs ⏰

Generate Biomedicines releases Chroma, an equivariant conditional diffusion model for generating proteins. The conditional part is particularly cool as we usually want to generate proteins with certain properties and functions - Chroma allows to impose functional and geometric constraints, and even use natural language queries like “Generate a protein with CHAD domain” thanks to a small GPT-Neo trained on protein captioning. The 80-pager paper is on the website, and you can have a look at the thread by Andrew Beam.

Simultaneously, the Baker Lab releases RoseTTa Fold Diffusion (RF Diffusion) packed with the similar functionality also allowing for text prompts like “Generate a protein that binds to X”. Check out the Twitter thread by Joseph Watson, the 1st author. The 70-pager preprint is available, so here is your casual weekend reading of two papers 🙂