While AI systems like ChatGPT or Diffusion models for Generative AI have been in the limelight in the past months, Graph Neural Networks (GNN) have been rapidly advancing. In the last couple of years Graph Neural Networks have quietly become the dark horse behind a wealth of exciting new achievements that have made it all the way from purely academic research breakthroughs to actual solutions actively deployed at large-scale.
Companies like Uber, Google, Alibaba, Pinterest, Twitter and many others have been already shifting to GNN-based approaches in some of their core products, motivated by the substantial performance improvements exhibited by these methods compared to the previous state-of-the-art AI architectures.
Despite the diversity in the type of problems and the differences of their underlying datasets, all these breakthroughs use the single unifying framework of GNNs to operate at their core. This suggests a potential shift of perspective: graph-structured data provides a general and flexible framework for describing and analyzing any possible set of entities and their mutual interactions.
What are the actual advantages of Graph Machine Learning? And why do Graph Neural Networks matter in 2023? This article will recap on some highly impactful applications of GNNs, the first article in a series that will take a deep dive into Graph Machine Learning, giving you everything you need to know to get up to speed on the next big wave in AI.
Graph data is everywhere in the world: any system consisting of entities and relationships between them can be represented as a graph. Although deep learning algorithms in the last decade have made outstanding progress in areas such as natural language processing, computer vision, and speech recognition, due to their ability to extract high-level features from data by passing it through non-linear layers, most deep learning architectures are specifically tailored for Euclidean-structured data, such as tabular data, images, text, and audio, while graph data has been largely ignored.
Traditional AI methods have been designed to extract information from objects encoded by somewhat “rigid” structures. For example, images are typically encoded as fixed-size 2-dimensional grids of pixels, and text as a 1-dimensional sequence of words (or “tokens”). On the other hand, representing data in a graph-structured way may reveal valuable information that emerges from a higher-dimensional representation of these entities and their relationships, and would otherwise be lost.
Unfortunately, the high flexibility of graphs in allowing for a large number of possibilities to represent the same piece of data comes at the expense of the complexity to design homogeneous frameworks that are able to learn on that data and generalize across different domains. A variety of approaches have been proposed in the past two decades for AI systems capable of dealing with graph data at scale, but these advances were often tied to the specific case and settings for which they had been developed.
In some ways, this reflects what happened during the Deep Learning revolution a decade ago, when speech recognition systems that used to consist of combinations of Hidden Markov Models, Gaussian Mixture Models, and computer vision systems that relied heavily on traditional signal processing have progressively converged to end-to-end deep learning systems that even often use the same fundamental architectures: a standard example for this being Transformers and the Attention mechanism, originated in the field of Natural Language Processing and more recently spreading across so many different domains.
In recent years, a relatively small community of high-level deep learning researchers has been making great strides in showing how various data problems in different domains are best cast as graph problems, where Graph Neural Networks –together with some of their variations– have been shown to outperform mainstream methods in a variety of deep learning tasks. GNNs have de facto become a crucial tool for solving real-world problems in many totally different and seemingly unrelated areas such as drug discovery, recommendation systems, traffic predictions, and many more, in which more traditional and case-specific methods fail.
What is the current role of GNNs in the broader AI research landscape? Let’s take a look at some numbers revealing how GNNs have seen a spectacular rise within the research community.
Graph Neural Networks in the AI Research Landscape
A non-negligible implication of GNNs rapidly establishing themselves as a general paradigm for learning on any graph structure is the fact that any new improvement in this area will propagate advancements across a wide range of domains. It may come as no surprise that the attention to this topic of both academic and industry researchers around the world has experienced a growth explosion in the last few years.
We can get a first glimpse of this impressive growth if we look at the accepted research publications from the last 3 years for both ICLR and NeurIPS, the two major annual conferences which focus on cutting edge AI research. We find that the term Graph Neural Network consistently ranked in the top 3 keywords year over year.
A recent bibliometric study systematically analysed this research trend, revealing an exponential growth of published research involving GNNs, with a striking +447% average annual increase in the period 2017-2019. The State of AI Report 2021 further confirmed Graph Neural Network to be the keyword in AI research publications “with the largest increase in usage from 2019 to 2020”.
We can also examine the versatility of Graph Neural Networks by looking at their impact across different domains of application. The following figure aims at illustrating the distribution of GNN papers across 22 categories.
As we can see, GNNs have seen impressive growth in AI research and their impact is distributed across vastly different domains. Let's now look at a few of these use cases and learn how GNNs have made a (quantifiable) impact there.
Graph Neural Networks: Selected Use Cases
Graph Neural Networks have become a key ingredient behind many recent exciting projects. Let’s take a closer look at some examples and the results of applying GNNs to large-scale models in production.
Boosting Recommender Systems (Uber & Pinterest)
The team at Uber Eats develops an application for food delivery and have recently started to introduce graph learning techniques into the recommendation system empowering the app, which aims to surface the foods that are most likely to appeal to an individual user. Given the large size of graphs dealt with in such settings (Uber Eats serves as a portal to more than 320,000 restaurants in over 500 cities globally), Graph Neural Networks represent a very appealing choice. In fact, as we are going to see in a following article, GNNs require only a fixed amount of parameters that do not depend on the size of the input graph, making learning scalable to large graphs.
On a first test for the model on recommending dishes and restaurants the team reported a performance boost of over 20% compared to the existing production model on key metrics like Mean Reciprocal Rank, Precision@K, and NDCG. After the integration of the GNN model into the Uber Eats recommendation system (which incorporates other non graph-based features) the developers observed a jump from 78% to 87% in AUC compared to the existing productionized baseline model, and a later analysis of impact revealed that the GNN-based feature was by far the most influential feature in the recommendation model as a whole.
Pinterest is a visual discovery engine that operates as a social network, where users interact with visual bookmarks, known as pins, that link to web-based resources. The platform provides users with the ability to organize pins into thematic collections, called boards. The "Pinterest graph", which encompasses 2 billion pins, 1 billion boards, and over 18 billion edges, represents a rich and complex visual ecosystem with potential implications for understanding user behaviour and preferences.
Pinterest is currently actively deploying PinSage, a GNN-powered recommendation system scaled-up to operate on the Pinterest graph. PinSage is able to predict in novel ways which visual concepts that users have found interesting can map to new things they might appeal to them.
In order to measure its accuracy, the research team has evaluated the performance of PinSage against other state-of-the-art content-based deep learning baselines (based on nearest neighbours of visual or annotation embedding methods) and used the following two key metrics for their recommendation task:
- Hit-rate – directly measuring the probability that recommendations made by the algorithm contain the items related to the query.
- Mean Reciprocal Rank (MRR) – measuring the degree to which relevant results are close to the top of search results.
Overall, PinSage yields 150% improvement in hit-rate and 60% improvement in MRR over the best baseline productionized model.
Both Uber Eats and Pinterest teams have built their AI systems on top of a specific flavour of Graph Neural Network called GraphSAGE, because of its strong scalability properties. GraphSAGE (an open-source GNN framework developed by Stanford researchers) is explicitly designed to efficiently generate useful node embeddings on unseen data and represents a major breakthrough in recommendation systems. When evaluated against three hard classification benchmarks, it outperformed all relevant baselines by an average of 51% (classification F1-score) compared to using node features alone, and also consistently outperformed a strong baseline (DeepWalk), with a 100x decrease in inference time.
Next-generation traffic prediction algorithm (Google Maps)
Another highly impactful application of Graph Neural Networks came from a team of researchers from DeepMind who showed how GNNs can be applied to transportation maps to improve the accuracy of estimated time of arrival (ETA). The idea is to use a GNN to learn representations of the transportation network that capture the underlying structure of the network and its dynamics.
This system is already actively deployed at scale by Google Maps in several major cities around the world, and the new approach has led to drastic reductions in the proportion of negative user outcomes when querying the ETA (with up to 50% accuracy improvements compared to the prior approach deployed in production).
New GNN-powered drug discovery algorithm (MIT Lab)
Perhaps one of the most famous recent applications of AI methods in the pharmaceutical domain came out of a research project from the Massachusetts Institute of Technology that turned into a publication in the prestigious scientific journal Cell.
The goal was to use AI models to predict the antibiotic activity of molecules by learning their graph representations, this way capturing their potential antibiotic activity. The choice of encoding the information with graphs is very natural in this setting, since antibiotics can be represented as small molecular graphs, where the nodes are atoms and the edges correspond to their chemical bonds.
The AI model learns from this data to predict the most promising molecules subject to certain desirable conditions, and subsequently these predictions get tested and validated in the lab, this way helping biologists to prioritize the molecules to be analyzed, from a pool of billions of possible candidates.
This led to the identification of a previously unknown compound, named Halicin, found to be a highly potent antibiotic, also effective against antibiotic-resistant bacteria, an outcome regarded as a major breakthrough by field experts in antibiotic discovery research.
The news hit the media, with featured articles by the BBC and Financial Times among others, but barely any consideration was given to the fact that a GNN-based approach lies at the backbone of the specific AI model deployed. On the other hand, the researchers reported how using a directed-message passing deep neural network approach, a core feature of GNNs, was essential for the discovery: the other state-of-the-art models that were also tested against Halicin, in fact, failed to output a high prediction rank, contrary to the graph-based AI model.
Diffusion-based Generative AI for protein design (Baker Lab)
The goal of protein design is to create proteins that have desired properties, and can be done through (typically high cost) experimental methods that allow researchers to engineer new proteins by directly manipulating the amino acid sequence of a protein. The design of new proteins has huge potential applications, such as developing new drugs, enzymes, or materials.
Baker Lab recently combined Graph Neural Networks and Diffusion techniques, creating an AI system named RosettaFoldDiffusion (RFDiffusion), which has turned out to be capable of designing protein structures satisfying custom constraints. The AI model operates via an E(n)-Equivariant Graph Neural Network, a special kind of GNN expressly designed to process data structures with rigid motion symmetries (such as translation, rotation and reflection in space), and was fine-tuned as a denoiser, i.e. a diffusion model.
Released in November 2022, RFDiffusion is a highly complex system capable of tackling a wide multitude of specific tasks in protein design, and accordingly has been tested against a variety of metrics and benchmarks. The results have shown dramatic improvements over state-of-the-art competitors: RFDiffusion solved 100% more benchmark problems (23 out of 25) than the previous state-of-the-art deep neural network model in designing scaffolds for protein structural motifs, and achieved an 18% success rate in designing protein binders –a problem referred to as a “grand challenge” in protein design. Moreover, RFDiffusion's experimental success rate ranges from 5x to 214x improvement depending on the target protein.
Some experts in the field believe that RFDiffusion could be one of the “biggest advances in structural biology of this decade together with AlphaFold”, an advancement that relies critically on the latest advances in Graph Neural Networks.
Graph Neural Networks are a rapidly evolving field with many exciting developments, and the AI research community has substantially increased attention to this area in the last few years.
In industry, applications of graph machine learning in remarkably different domains have only recently started to appear and GNNs already established themselves as a game-changer in some state-of-the-art production-ready models deployed at large-scale. These recent successes bring about an opportunity for a new range of applications, and it will be fascinating to see what the field will bring this year.
This article is the first one of a series on Graph Neural Networks. In the upcoming blog posts we are going to provide an in-depth overview of GNNs and their related architectures, as well as going into the details of some of the latest exciting applications of Graph Neural Networks.
Make sure to follow our newsletter or follow us on Twitter to stay in the loop when we release new content!
- These consisted of two evolving document graphs based on citation data and Reddit post data (predicting paper and post categories, respectively), and a multigraph generalization experiment based on a dataset of protein-protein interactions (predicting protein functions).