Diffusion-Convolutional Neural Networks

James AtwoodDon Towsley

article2015NeurIPS1,387 citations

Introduces Diffusion-Convolutional Neural Networks, an efficient framework that models information diffusion across graph structures to learn isomorphism-invariant representations and outperform traditional relational models on node classification tasks.

Listen

Real-world data increasingly takes the form of complex networks and relational graphs, such as citation indices, social networks, and molecular structures. While exploiting graph structure can significantly improve predictive accuracy, traditional machine learning methods often face a difficult trade-off between predictive power and computational tractability. Existing relational models frequently suffer from exponential computational complexity during training and inference, while standard deep learning architectures are primarily designed for regular, grid-like inputs such as images.

The article demonstrates and evaluates Diffusion-Convolutional Neural Networks (DCNNs), a new neural framework for graph-structured data. The main objective is to establish whether scanning information diffusion processes across graph neighborhoods can capture relational patterns effectively without incurring the prohibitive computational overhead of traditional probabilistic relational models.

To evaluate this framework, the authors conducted controlled experiments across standard benchmark datasets. They tested node-level categorization on citation networks (Cora and Pubmed) containing thousands of documents and links, and whole-graph classification across chemical and biological collections (including NCI1, NCI109, MUTAG, PTC, and ENZYMES). The approach compares DCNN performance against established baselines, such as regularized logistic regression, graph kernels, and partially-observed conditional random fields.

The findings show that DCNNs achieve superior performance on node-level classification. On citation networks, a two-hop diffusion architecture achieved statistically significant gains over all baseline methods, reaching approximately 86.8% accuracy on Cora and 89.8% on Pubmed. Second, most of the performance gains were realized quickly within a local neighborhood search depth of two to three hops before leveling off. Third, DCNNs achieved these accuracy gains while maintaining polynomial-time computational complexity, executing substantially faster than exponential probabilistic alternatives. However, for whole-graph classification tasks, DCNNs did not consistently outperform specialized graph kernel methods, demonstrating that simply averaging node-level diffusion states is insufficient for summarizing entire graphs.

These results demonstrate that DCNNs provide an efficient, high-performance solution for entity-level prediction tasks in relational networks, allowing organizations to leverage GPU acceleration without expensive inference routines. For decision-makers, DCNNs are recommended for node-level relational problems such as document tagging, entity labeling, or user categorization. When dealing with whole-graph categorization, such as molecular property prediction, established graph kernel methods remain preferable until more sophisticated aggregation mechanisms are developed.

Confidence in node classification is high given the statistically validated benchmarks. However, key operational limitations remain. Because the framework represents diffusion states as dense tensors, GPU memory requirements scale quadratically with node count, making the current implementation practical for networks with tens to hundreds of thousands of nodes, but unsuitable for massive graphs with millions or billions of nodes without further optimization.

arXiv: 1511.02136
  • Paper: Spectral Networks and Locally Connected Networks on Graphs, Joan Bruna et al. (2014). This foundational paper establishes spectral and spatial neural network formulations on graphs, providing essential background for understanding how diffusion processes adapt convolution to non-Euclidean domains.
  • Paper: The Graph Neural Network Model, Franco Scarselli et al. (2009). It introduces the foundational framework of processing graph-structured data through iterative information diffusion mechanisms across neighboring nodes.
  • Paper: Learning with Local and Global Consistency, Dengyong Zhou et al. (2003). It formalizes label propagation and consistency via iterative graph diffusion operators, which directly underpins diffusion-based node classification in graphs.
  • Paper: Discrete Signal Processing on Graphs, Aliaksei Sandryhaila et al. (2012). It defines shift operators and filtering on graph adjacency structures, providing fundamental discrete signal processing concepts used to model graph convolutions.
  • Paper: Deep Convolutional Networks on Graph-Structured Data, Mikael Henaff et al. (2015). It explores extending spectral graph convolutions to unstructured data, establishing the groundwork for GPU-efficient graph-based deep learning.
Cover for Diffusion-Convolutional Neural Networks

Abstract

We present diffusion-convolutional neural networks (DCNNs), a new model for graph-structured data. Through the introduction of a diffusion-convolution operation, we show how diffusion-based representations can be learned from graph-structured data and used as an effective basis for node classification. DCNNs have several attractive qualities, including a latent representation for graphical data that is invariant under isomorphism, as well as polynomial-time prediction and learning that can be represented as tensor operations and efficiently implemented on the GPU. Through several experiments with real structured datasets, we demonstrate that DCNNs are able to outperform probabilistic relational models and kernel-on-graph methods at relational node classification tasks.

Table of Contents

  • 1 Introduction
  • 2 Model
  • 3 Experiments
  • 3.1 Node classification
  • 3.2 Graph Classification
  • 4 Limitations
  • 5 Related Work
  • 6 Conclusion and Future Work
  • 7 Appendix: Representation Invariance for Isomorphic Graphs
  • References

Knowls

  1. Knowl 1 — Diffusion-Convolution Operation for Node Classification

    model/method

    The diffusion-convolution operation defines a latent representation for graph nodes by mapping entity features through a graph diffusion process. Let G=(V,E)G = (V, E) be a graph with NN vertices, feature matrix X∈RN×FX \in \mathbb{R}^{N \times F}, and degree-normalized transition probability matrix P∈RN×NP \in \mathbb{R}^{N \times N} (where PijP_{ij} denotes the one-step transition probability from node ii to node jj). Let P∗∈RN×H×NP^* \in \mathbb{R}^{N \times H \times N} denote the 3D tensor containing the power series of PP across HH diffusion hops, where slice j∈{0,…,H−1}j \in \{0, \dots, H-1\} corresponds to PjP^j.

    The diffusion-convolutional activation ZijkZ_{ijk} for node ii, hop jj, and feature kk is given by: Zijk=f(Wjkc∑l=1NPijl∗Xlk)Z_{ijk} = f\left(W^c_{jk} \sum_{l=1}^N P^*_{ijl} X_{lk}\right)

    In tensor notation: Z=f(Wc⊙(P∗X))Z = f\left(W^c \odot (P^* X)\right) where Wc∈RH×FW^c \in \mathbb{R}^{H \times F} is a learnable real-valued weight tensor, ⊙\odot denotes element-wise multiplication broadcast across the node dimension, and ff is a differentiable activation function (such as tanh⁡\tanh). The model requires O(H×F)\mathcal{O}(H \times F) parameters, making the parameter count independent of the number of nodes NN.

    Node label predictions Y^\hat{Y} and class conditional probabilities P(Y∣X)\mathbb{P}(Y \mid X) are obtained via a dense classification layer with weight tensor WdW^d: Y^=arg⁡max⁡f(Wd⊙Z),P(Y∣X)=softmax(f(Wd⊙Z))\hat{Y} = \arg\max f(W^d \odot Z), \qquad \mathbb{P}(Y \mid X) = \text{softmax}(f(W^d \odot Z))

  2. Knowl 2 — Graph-Level Classification with Diffusion-Convolutional Networks

    model/method

    Diffusion-Convolutional Neural Networks (DCNNs) extend to whole-graph classification by aggregating node diffusion activations across all vertices via mean pooling.

    For an input graph Gt=(Vt,Et)G_t = (V_t, E_t) with NtN_t nodes, feature design matrix Xt∈RNt×FX_t \in \mathbb{R}^{N_t \times F}, and transition matrix power series tensor Pt∗∈RNt×H×NtP^*_t \in \mathbb{R}^{N_t \times H \times N_t}, the graph-level latent representation Zt∈RH×FZ_t \in \mathbb{R}^{H \times F} is defined as: Zt=f(Wc⊙(1Nt1NtTPt∗Xt))Z_t = f\left(W^c \odot \left(\frac{1}{N_t} \mathbf{1}_{N_t}^T P^*_t X_t\right)\right) where 1Nt\mathbf{1}_{N_t} is an Nt×1N_t \times 1 all-ones vector, Wc∈RH×FW^c \in \mathbb{R}^{H \times F} is the learnable weight tensor, ⊙\odot is element-wise multiplication, and ff is a differentiable activation function.

    Predictions Y^t\hat{Y}_t or class posterior distributions P(Yt∣Xt)\mathbb{P}(Y_t \mid X_t) for the whole graph are then computed using a dense weight tensor Wd∈RH×FW^d \in \mathbb{R}^{H \times F}: Y^t=arg⁡max⁡f(Wd⊙Zt),P(Yt∣Xt)=softmax(f(Wd⊙Zt))\hat{Y}_t = \arg\max f(W^d \odot Z_t), \qquad \mathbb{P}(Y_t \mid X_t) = \text{softmax}(f(W^d \odot Z_t))

  3. Knowl 3 — Edge Classification and Feature Incorporation via Adjacency Augmentation

    model/method

    To handle edge classification and incorporate edge features in a Diffusion-Convolutional Neural Network, each edge is transformed into an intermediate node connected to its head and tail vertices.

    Given a graph with NtN_t vertices, vertex adjacency matrix At∈RNt×NtA_t \in \mathbb{R}^{N_t \times N_t}, and vertex-edge incidence matrix Bt∈RMt×NtB_t \in \mathbb{R}^{M_t \times N_t} for MtM_t edges, an augmented adjacency matrix At′∈R(Nt+Mt)×(Nt+Mt)A'_t \in \mathbb{R}^{(N_t + M_t) \times (N_t + M_t)} is defined as: At′=(AtBtTBt0)A'_t = \begin{pmatrix} A_t & B_t^T \\ B_t & 0 \end{pmatrix}

    The degree-normalized transition probability matrix Pt′P'_t and its power series tensor (Pt′)∗(P'_t)^* are derived from At′A'_t. Applying the diffusion-convolution operation over this augmented graph produces a latent activation tensor Zt∈RMt×H×FZ_t \in \mathbb{R}^{M_t \times H \times F} for the edge nodes, enabling edge label prediction and joint representation learning over node and edge attributes.

  4. Knowl 4 — Isomorphism Invariance of Diffusion-Convolutional Representations

    theoretical result

    The latent diffusion-convolutional representations produced by a Diffusion-Convolutional Neural Network (DCNN) are invariant under graph isomorphism.

    Let G1=(V1,E1)G_1 = (V_1, E_1) and G2=(V2,E2)G_2 = (V_2, E_2) be two isomorphic graphs with identical vertex feature mappings such that ∣V1∣=∣V2∣=N|V_1| = |V_2| = N, X1=X2=X∈RN×FX_1 = X_2 = X \in \mathbb{R}^{N \times F}, and transition power series tensors P1∗=P2∗=P∗P^*_1 = P^*_2 = P^*. The graph-level diffusion-convolutional activations Z1,Z2∈RH×FZ_1, Z_2 \in \mathbb{R}^{H \times F} computed by: Zjk=f(Wjkc⊙1N∑v∈V∑v′∈VPvjv′∗Xv′k)Z_{jk} = f\left(W^c_{jk} \odot \frac{1}{N} \sum_{v \in V} \sum_{v' \in V} P^*_{v j v'} X_{v' k}\right) satisfy Z1=Z2Z_1 = Z_2 identically for all diffusion hops j∈{0,…,H−1}j \in \{0, \dots, H-1\} and features k∈{1,…,F}k \in \{1, \dots, F\}. Because parameters are tied across diffusion search depth rather than node indices or spatial grid locations, DCNN activations are permutation-invariant across isomorphic graphs.

  5. Knowl 5 — DCNN Training and Optimization Procedure

    algorithm

    Diffusion-Convolutional Neural Networks are trained end-to-end via stochastic minibatch gradient descent on backpropagated multiclass hinge loss using AdaGrad optimization and windowed early stopping.

    Input: Dataset G={Gt=(Vt,Et,Xt,Yt)}t=1TG = \{G_t = (V_t, E_t, X_t, Y_t)\}_{t=1}^T, diffusion depth HH, learning rate α=0.05\alpha = 0.05, early stopping window ww
    Output: Optimized weight tensors WcW^c and WdW^d
    Initialize weight tensors Wc,Wd∼N(0,0.01)W^c, W^d \sim \mathcal{N}(0, 0.01)
    Compute degree-normalized transition matrices PtP_t and power series tensors Pt∗P^*_t for each graph GtG_t
    repeat
        Randomly partition entity indices into minibatches
        for each minibatch BB do
            Compute diffusion-convolution activations Z=f(Wc⊙(PB∗X))Z = f(W^c \odot (P^*_B X))
            Compute dense predictions Y^=f(Wd⊙Z)\hat{Y} = f(W^d \odot Z)
            Compute multiclass hinge loss L(Y^,Y)\mathcal{L}(\hat{Y}, Y)
            Compute gradients ∇WcL\nabla_{W^c} \mathcal{L} and ∇WdL\nabla_{W^d} \mathcal{L} via backpropagation
            Update parameters WcW^c and WdW^d using AdaGrad with learning rate α\alpha
        end for
        Compute validation loss Lval\mathcal{L}_{\text{val}} on the validation set
    until Lval\mathcal{L}_{\text{val}} exceeds the moving average of validation losses over the previous ww epochs
    return Wc,WdW^c, W^d
  6. Knowl 6 — Relational Node Classification Performance on Cora and Pubmed

    data/table

    Node classification was evaluated on two citation network datasets: Cora (2,708 scientific papers across 7 classes, 5,429 citation links, 1,433 binary word features) and Pubmed (19,717 diabetes papers across 3 classes, 44,338 citation links, 500 TF-IDF features). In each trial, nodes were randomly split into equal-sized training, validation, and test partitions.

    A 2-hop DCNN was benchmarked against feature-only ℓ1\ell_1- and ℓ2\ell_2-regularized logistic regression, graph-structure-only kernel methods (Exponential Diffusion Kernel KED and Laplacian Exponential Diffusion Kernel KLED), and a partially-observed Conditional Random Field with Loopy Belief Propagation (CRF-LBP).

    Cora Pubmed
    Model Accuracy F (micro) F (macro) Accuracy F (micro) F (macro)
    l1logistic 0.7087 0.7087 0.6829 0.8718 0.8718 0.8698
    l2logistic 0.7292 0.7292 0.7013 0.8631 0.8631 0.8614
    KED 0.8044 0.8044 0.7928 0.8125 0.8125 0.7978
    KLED 0.8229 0.8229 0.8117 0.8228 0.8228 0.8086
    CRF-LBP 0.8449 – 0.8248 – – –
    2-hop DCNN 0.8677 0.8677 0.8584 0.8976 0.8976 0.8943

    The 2-hop DCNN achieved statistically significant improvements over all baselines on both datasets across accuracy, micro-averaged F1, and macro-averaged F1.

  7. Knowl 7 — Graph Classification Benchmark Performance across Chemical and Biological Datasets

    data/table

    Whole-graph classification was evaluated on five standard benchmarks: NCI1 (4,100 chemical compounds, 37 node labels), NCI109 (4,127 chemical compounds, 38 node labels), MUTAG (188 nitro compounds, 7 node features), PTC (344 carcinogenicity compounds, 19 node features), and ENZYMES (600 protein structures, 3 node features). Graphs were randomly partitioned into equal-sized training, validation, and test splits.

    DCNN models with diffusion breadths H=2H=2 and H=5H=5 were compared against linear classifiers on graph-averaged features (l1logistic, l2logistic) and the Weisfeiler-Lehman subtree deep graph kernel (deepwl).

    NCI1 NCI109
    Model Accuracy F (micro) F (macro) Accuracy F (micro) F (macro)
    l1logistic 0.5728 0.5728 0.5711 0.5555 0.5555 0.5411
    l2logistic 0.5688 0.5688 0.5641 0.5586 0.5568 0.5402
    deepwl 0.6215 0.6215 0.5821 0.5801 0.5801 0.5178
    2-hop DCNN 0.6250 0.5807 0.5807 0.6275 0.5884 0.5884
    5-hop DCNN 0.6261 0.5898 0.5898 0.6286 0.5950 0.5899
    MUTAG PTC
    Model Accuracy F (micro) F (macro) Accuracy F (micro) F (macro)
    l1logistic 0.7190 0.7190 0.6405 0.5470 0.5470 0.4272
    l2logistic 0.7016 0.7016 0.5795 0.5565 0.5565 0.4460
    deepwl 0.6563 0.6563 0.5942 0.5113 0.5113 0.4444
    2-hop DCNN 0.6635 0.7975 0.79747 0.5660 0.0500 0.0531
    5-hop DCNN 0.6698 0.8013 0.8013 0.5530 0.0 0.0526
    ENZYMES
    Model Accuracy F (micro) F (macro)
    l1logistic 0.1640 0.1640 0.0904
    l2logistic 0.2030 0.2030 0.1110
    deepwl 0.2155 0.2155 0.1431
    2-hop DCNN 0.1590 0.1590 0.0809
    5-hop DCNN 0.1810 0.1810 0.0991

    Unlike in node classification, no single model dominated across all graph classification datasets. DCNNs performed competitively on NCI1, NCI109, and MUTAG, but underperformed deepwl on ENZYMES, demonstrating that mean-pooling node diffusion activations is suboptimal for summarizing whole-graph structures.

  8. Knowl 8 — Computational Complexity of DCNN vs. Partially-Observed CRFs

    theoretical result

    Training a Diffusion-Convolutional Neural Network involves numerical gradient descent requiring a single forward and backward pass per instance, resulting in polynomial time complexity dominated by dense tensor multiplications: O(Nt2F)\mathcal{O}(N_t^2 F) for a graph with NtN_t vertices and FF features.

    In contrast, learning parameters for partially-observed Conditional Random Fields (CRFs) involves minimizing negative marginal log-likelihood using a contrast-of-partition-functions objective that requires running loopy belief propagation twice per step (once over the entire graph and once with observed labels conditioned). This induces an exponential time complexity per optimization step of: O(EtNtCt)\mathcal{O}(E_t N_t^{C_t}) where EtE_t is the number of edges and CtC_t is the size of the maximal clique in graph GtG_t. DCNNs thereby achieve superior classification accuracy without incurring the combinatorial inference costs of probabilistic relational graphical models.

  9. Knowl 9 — Scalability and Locality Limitations in DCNNs

    limitation

    Diffusion-Convolutional Neural Networks have two primary structural limitations:

    1. Memory Scalability: Storing the full transition probability power series tensor P∗∈RNt×H×NtP^* \in \mathbb{R}^{N_t \times H \times N_t} as a dense tensor requires O(Nt2H)\mathcal{O}(N_t^2 H) memory. On GPU hardware, this results in out-of-memory constraints for graphs larger than tens to hundreds of thousands of nodes, preventing application to million- or billion-node graphs.
    2. Locality Bias: Constructing latent representations from diffusion processes centered at individual nodes restricts the model to local topological neighborhoods. Consequently, DCNNs may fail to capture non-local graph behaviors or long-range spatial dependencies between distant entities.
  10. Knowl 10 — Impact of Diffusion Hop Count on Classification Accuracy

    empirical result

    Evaluating DCNN performance across varying diffusion depths HH demonstrates that for node classification tasks (e.g., on Cora and Pubmed), accuracy rises sharply as the search breadth increases from H=0H=0 (isolated node features with no graph structure) to H=3H=3 hops. Beyond 3 hops, performance gains plateau as the random walk transition matrix powers converge toward stationarity. For whole-graph classification tasks, increasing the diffusion breadth beyond H=2H=2 to H=5H=5 or H=10H=10 provides no significant gain in predictive accuracy.

Coverage note — All core contributions, including the node, graph, and edge DCNN model formulations, isomorphism invariance property, optimization algorithm, complexity analysis, limitations, and empirical benchmark evaluations, have been included; no substantial contributed material was omitted.

References

  1. 1.John Duchi, Elad Hazan, and Yoram Singer. Adaptive subgradient methods for online learning and stochastic optimization. The Journal of Machine Learning Research, 2011.
  2. 2.James Bergstra, Olivier Breuleux, Frédéric Bastien, Pascal Lamblin, Razvan Pascanu, Guillaume Desjardins, Joseph Turian, David Warde-Farley, and Yoshua Bengio. Theano: a CPU and GPU math expression compiler. In Proceedings of the Python for Scientific Computing Conference (SciPy), 2010.
  3. 3.P Sen and L Getoor. Link-based classification. Technical Report, 2007.
  4. 4.François Fouss, Kevin Francoisse, Luh Yen, Alain Pirotte, and Marco Saerens. An experimental investigation of kernels on graphs for collaborative recommendation and semisupervised classification. Neural Networks, 31:53–72, July 2012.
  5. 5.Prithviraj Sen, Galileo Mark Namata, Mustafa Bilgic, Lise Getoor, Brian Gallagher, and Tina Eliassi-Rad. Collective Classification in Network Data. AI Magazine, 2008.
  6. 6.Pinar Yanardag and S V N Vishwanathan. Deep Graph Kernels. In the 21th ACM SIGKDD International Conference, pages 1365–1374, New York, New York, USA, 2015. ACM Press.
  7. 7.Nikil Wale, Ian A Watson, and George Karypis. Comparison of descriptor spaces for chemical compound retrieval and classification. Knowledge and Information Systems, 14(3):347–375, August 2007.
  8. 8.Asim Kumar Debnath, Rosa L Lopez de Compadre, Gargi Debnath, Alan J Shusterman, and Corwin Hansch. Structure-activity relationship of mutagenic aromatic and heteroaromatic nitro compounds. correlation with molecular orbital energies and hydrophobicity. Journal of medicinal chemistry, 34(2):786–797, 1991.
  9. 9.Hannu Toivonen, Ashwin Srinivasan, Ross D King, Stefan Kramer, and Christoph Helma. Statistical evaluation of the predictive toxicology challenge 2000–2001. Bioinformatics, 19(10):1183–1193, 2003.
  10. 10.Karsten M Borgwardt, Cheng Soon Ong, Stefan Schönauer, SVN Vishwanathan, Alex J Smola, and Hans-Peter Kriegel. Protein function prediction via graph kernels. Bioinformatics, 21(suppl 1):i47–i56, 2005.
  11. 11.Joan Bruna, Wojciech Zaremba, Arthur Szlam, and Yann LeCun. Spectral networks and locally connected networks on graphs. arXiv.org, 2014.
  12. 12.M Henaff, J Bruna, and Y LeCun. Deep Convolutional Networks on Graph-Structured Data. arXiv.org, 2015.
  13. 13.F Scarselli, M Gori, Ah Chung Tsoi, M Hagenbuchner, and G Monfardini. The Graph Neural Network Model. IEEE Transactions on Neural Networks, 2009.
  14. 14.A Micheli. Neural Network for Graphs: A Contextual Constructive Approach. IEEE Transactions on Neural Networks, 2009.
  15. 15.David K Duvenaud, Dougal Maclaurin, Jorge Aguilera-Iparraguirre, Rafael Gómez-Bombarelli, Timothy Hirzel, Alán Aspuru-Guzik, and Ryan P Adams. Convolutional Networks on Graphs for Learning Molecular Fingerprints. NIPS, 2015.
  16. 16.Daphne Koller and Nir Friedman. Probabilistic Graphical Models: Principles and Techniques. The MIT Press, 2009.
  17. 17.Jakob Verbeek and William Triggs. Scene segmentation with crfs learned from partially labeled images. NIPS, 2007.
  18. 18.Trevor Cohn. Efficient Inference in Large Conditional Random Fields. ECML, 2006.

Citation

MLA
Atwood, J., and D. Towsley. “Diffusion-Convolutional Neural Networks”. arXiv, 2015, http://arxiv.org/abs/1511.02136v6.
APA
Atwood, J., & Towsley, D. (2015). Diffusion-Convolutional Neural Networks. arXiv. http://arxiv.org/abs/1511.02136v6
Chicago
Atwood, J., and D. Towsley. 2015. “Diffusion-Convolutional Neural Networks”. arXiv. http://arxiv.org/abs/1511.02136v6.
Harvard
Atwood, J. and Towsley, D. (2015) “Diffusion-Convolutional Neural Networks”, arXiv [Preprint]. Available at: http://arxiv.org/abs/1511.02136v6.
Vancouver
1. Atwood J, Towsley D (2015) Diffusion-Convolutional Neural Networks. arXiv

BibTeX

@article{atwood2015diffusion,
  title = {Diffusion-Convolutional Neural Networks},
  author = {Atwood, James and Towsley, Don},
  year = {2015},
  journal = {arXiv},
  url = {http://arxiv.org/abs/1511.02136v6},
  eprint = {1511.02136}
}
Metadata:arXiv

Access the Paper

This paper is available from its original source. Click below to access the PDF.

Open PDF
License: Authors