-
HoloNets: Spectral Convolutions do extend to Directed Graphs
Authors:
Christian Koke,
Daniel Cremers
Abstract:
Within the graph learning community, conventional wisdom dictates that spectral convolutional networks may only be deployed on undirected graphs: Only there could the existence of a well-defined graph Fourier transform be guaranteed, so that information may be translated between spatial- and spectral domains. Here we show this traditional reliance on the graph Fourier transform to be superfluous a…
▽ More
Within the graph learning community, conventional wisdom dictates that spectral convolutional networks may only be deployed on undirected graphs: Only there could the existence of a well-defined graph Fourier transform be guaranteed, so that information may be translated between spatial- and spectral domains. Here we show this traditional reliance on the graph Fourier transform to be superfluous and -- making use of certain advanced tools from complex analysis and spectral theory -- extend spectral convolutions to directed graphs. We provide a frequency-response interpretation of newly developed filters, investigate the influence of the basis used to express filters and discuss the interplay with characteristic operators on which networks are based. In order to thoroughly test the developed theory, we conduct experiments in real world settings, showcasing that directed spectral convolutional networks provide new state of the art results for heterophilic node classification on many datasets and -- as opposed to baselines -- may be rendered stable to resolution-scale varying topological perturbations.
△ Less
Submitted 10 November, 2023; v1 submitted 3 October, 2023;
originally announced October 2023.
-
ResolvNet: A Graph Convolutional Network with multi-scale Consistency
Authors:
Christian Koke,
Abhishek Saroha,
Yuesong Shen,
Marvin Eisenberger,
Daniel Cremers
Abstract:
It is by now a well known fact in the graph learning community that the presence of bottlenecks severely limits the ability of graph neural networks to propagate information over long distances. What so far has not been appreciated is that, counter-intuitively, also the presence of strongly connected sub-graphs may severely restrict information flow in common architectures. Motivated by this obser…
▽ More
It is by now a well known fact in the graph learning community that the presence of bottlenecks severely limits the ability of graph neural networks to propagate information over long distances. What so far has not been appreciated is that, counter-intuitively, also the presence of strongly connected sub-graphs may severely restrict information flow in common architectures. Motivated by this observation, we introduce the concept of multi-scale consistency. At the node level this concept refers to the retention of a connected propagation graph even if connectivity varies over a given graph. At the graph-level, multi-scale consistency refers to the fact that distinct graphs describing the same object at different resolutions should be assigned similar feature vectors. As we show, both properties are not satisfied by poular graph neural network architectures. To remedy these shortcomings, we introduce ResolvNet, a flexible graph neural network based on the mathematical concept of resolvents. We rigorously establish its multi-scale consistency theoretically and verify it in extensive experiments on real world data: Here networks based on this ResolvNet architecture prove expressive; out-performing baselines significantly on many tasks; in- and outside the multi-scale setting.
△ Less
Submitted 30 October, 2023; v1 submitted 30 September, 2023;
originally announced October 2023.
-
Graph Scattering beyond Wavelet Shackles
Authors:
Christian Koke,
Gitta Kutyniok
Abstract:
This work develops a flexible and mathematically sound framework for the design and analysis of graph scattering networks with variable branching ratios and generic functional calculus filters. Spectrally-agnostic stability guarantees for node- and graph-level perturbations are derived; the vertex-set non-preserving case is treated by utilizing recently developed mathematical-physics based tools.…
▽ More
This work develops a flexible and mathematically sound framework for the design and analysis of graph scattering networks with variable branching ratios and generic functional calculus filters. Spectrally-agnostic stability guarantees for node- and graph-level perturbations are derived; the vertex-set non-preserving case is treated by utilizing recently developed mathematical-physics based tools. Energy propagation through the network layers is investigated and related to truncation stability. New methods of graph-level feature aggregation are introduced and stability of the resulting composite scattering architectures is established. Finally, scattering transforms are extended to edge- and higher order tensorial input. Theoretical results are complemented by numerical investigations: Suitably chosen cattering networks conforming to the developed theory perform better than traditional graph-wavelet based scattering approaches in social network graph classification tasks and significantly outperform other graph-based learning approaches to regression of quantum-chemical energies on QM7.
△ Less
Submitted 26 January, 2023;
originally announced January 2023.
-
Limitless stability for Graph Convolutional Networks
Authors:
Christian Koke
Abstract:
This work establishes rigorous, novel and widely applicable stability guarantees and transferability bounds for graph convolutional networks -- without reference to any underlying limit object or statistical distribution. Crucially, utilized graph-shift operators (GSOs) are not necessarily assumed to be normal, allowing for the treatment of networks on both undirected- and for the first time also…
▽ More
This work establishes rigorous, novel and widely applicable stability guarantees and transferability bounds for graph convolutional networks -- without reference to any underlying limit object or statistical distribution. Crucially, utilized graph-shift operators (GSOs) are not necessarily assumed to be normal, allowing for the treatment of networks on both undirected- and for the first time also directed graphs. Stability to node-level perturbations is related to an 'adequate (spectral) covering' property of the filters in each layer. Stability to edge-level perturbations is related to Lipschitz constants and newly introduced semi-norms of filters. Results on stability to topological perturbations are obtained through recently developed mathematical-physics based tools. As an important and novel example, it is showcased that graph convolutional networks are stable under graph-coarse-graining procedures (replacing strongly-connected sub-graphs by single nodes) precisely if the GSO is the graph Laplacian and filters are regular at infinity. These new theoretical results are supported by corresponding numerical investigations.
△ Less
Submitted 30 September, 2023; v1 submitted 26 January, 2023;
originally announced January 2023.
-
The Operating System of the Neuromorphic BrainScaleS-1 System
Authors:
Eric Müller,
Sebastian Schmitt,
Christian Mauch,
Sebastian Billaudelle,
Andreas Grübl,
Maurice Güttler,
Dan Husmann,
Joscha Ilmberger,
Sebastian Jeltsch,
Jakob Kaiser,
Johann Klähn,
Mitja Kleider,
Christoph Koke,
José Montes,
Paul Müller,
Johannes Partzsch,
Felix Passenberg,
Hartmut Schmidt,
Bernhard Vogginger,
Jonas Weidner,
Christian Mayr,
Johannes Schemmel
Abstract:
BrainScaleS-1 is a wafer-scale mixed-signal accelerated neuromorphic system targeted for research in the fields of computational neuroscience and beyond-von-Neumann computing. The BrainScaleS Operating System (BrainScaleS OS) is a software stack giving users the possibility to emulate networks described in the high-level network description language PyNN with minimal knowledge of the system. At th…
▽ More
BrainScaleS-1 is a wafer-scale mixed-signal accelerated neuromorphic system targeted for research in the fields of computational neuroscience and beyond-von-Neumann computing. The BrainScaleS Operating System (BrainScaleS OS) is a software stack giving users the possibility to emulate networks described in the high-level network description language PyNN with minimal knowledge of the system. At the same time, expert usage is facilitated by allowing to hook into the system at any depth of the stack. We present operation and development methodologies implemented for the BrainScaleS-1 neuromorphic architecture and walk through the individual components of BrainScaleS OS constituting the software stack for BrainScaleS-1 platform operation.
△ Less
Submitted 2 February, 2022; v1 submitted 30 March, 2020;
originally announced March 2020.
-
Accelerated physical emulation of Bayesian inference in spiking neural networks
Authors:
Akos F. Kungl,
Sebastian Schmitt,
Johann Klähn,
Paul Müller,
Andreas Baumbach,
Dominik Dold,
Alexander Kugele,
Nico Gürtler,
Luziwei Leng,
Eric Müller,
Christoph Koke,
Mitja Kleider,
Christian Mauch,
Oliver Breitwieser,
Maurice Güttler,
Dan Husmann,
Kai Husmann,
Joscha Ilmberger,
Andreas Hartel,
Vitali Karasenko,
Andreas Grübl,
Johannes Schemmel,
Karlheinz Meier,
Mihai A. Petrovici
Abstract:
The massively parallel nature of biological information processing plays an important role for its superiority to human-engineered computing devices. In particular, it may hold the key to overcoming the von Neumann bottleneck that limits contemporary computer architectures. Physical-model neuromorphic devices seek to replicate not only this inherent parallelism, but also aspects of its microscopic…
▽ More
The massively parallel nature of biological information processing plays an important role for its superiority to human-engineered computing devices. In particular, it may hold the key to overcoming the von Neumann bottleneck that limits contemporary computer architectures. Physical-model neuromorphic devices seek to replicate not only this inherent parallelism, but also aspects of its microscopic dynamics in analog circuits emulating neurons and synapses. However, these machines require network models that are not only adept at solving particular tasks, but that can also cope with the inherent imperfections of analog substrates. We present a spiking network model that performs Bayesian inference through sampling on the BrainScaleS neuromorphic platform, where we use it for generative and discriminative computations on visual data. By illustrating its functionality on this platform, we implicitly demonstrate its robustness to various substrate-specific distortive effects, as well as its accelerated capability for computation. These results showcase the advantages of brain-inspired physical computation and provide important building blocks for large-scale neuromorphic applications.
△ Less
Submitted 1 April, 2020; v1 submitted 6 July, 2018;
originally announced July 2018.
-
Pattern representation and recognition with accelerated analog neuromorphic systems
Authors:
Mihai A. Petrovici,
Sebastian Schmitt,
Johann Klähn,
David Stöckel,
Anna Schroeder,
Guillaume Bellec,
Johannes Bill,
Oliver Breitwieser,
Ilja Bytschok,
Andreas Grübl,
Maurice Güttler,
Andreas Hartel,
Stephan Hartmann,
Dan Husmann,
Kai Husmann,
Sebastian Jeltsch,
Vitali Karasenko,
Mitja Kleider,
Christoph Koke,
Alexander Kononov,
Christian Mauch,
Eric Müller,
Paul Müller,
Johannes Partzsch,
Thomas Pfeil
, et al. (11 additional authors not shown)
Abstract:
Despite being originally inspired by the central nervous system, artificial neural networks have diverged from their biological archetypes as they have been remodeled to fit particular tasks. In this paper, we review several possibilites to reverse map these architectures to biologically more realistic spiking networks with the aim of emulating them on fast, low-power neuromorphic hardware. Since…
▽ More
Despite being originally inspired by the central nervous system, artificial neural networks have diverged from their biological archetypes as they have been remodeled to fit particular tasks. In this paper, we review several possibilites to reverse map these architectures to biologically more realistic spiking networks with the aim of emulating them on fast, low-power neuromorphic hardware. Since many of these devices employ analog components, which cannot be perfectly controlled, finding ways to compensate for the resulting effects represents a key challenge. Here, we discuss three different strategies to address this problem: the addition of auxiliary network components for stabilizing activity, the utilization of inherently robust architectures and a training method for hardware-emulated networks that functions without perfect knowledge of the system's dynamics and parameters. For all three scenarios, we corroborate our theoretical considerations with experimental results on accelerated analog neuromorphic platforms.
△ Less
Submitted 3 July, 2017; v1 submitted 17 March, 2017;
originally announced March 2017.
-
Neuromorphic Hardware In The Loop: Training a Deep Spiking Network on the BrainScaleS Wafer-Scale System
Authors:
Sebastian Schmitt,
Johann Klaehn,
Guillaume Bellec,
Andreas Gruebl,
Maurice Guettler,
Andreas Hartel,
Stephan Hartmann,
Dan Husmann,
Kai Husmann,
Vitali Karasenko,
Mitja Kleider,
Christoph Koke,
Christian Mauch,
Eric Mueller,
Paul Mueller,
Johannes Partzsch,
Mihai A. Petrovici,
Stefan Schiefer,
Stefan Scholze,
Bernhard Vogginger,
Robert Legenstein,
Wolfgang Maass,
Christian Mayr,
Johannes Schemmel,
Karlheinz Meier
Abstract:
Emulating spiking neural networks on analog neuromorphic hardware offers several advantages over simulating them on conventional computers, particularly in terms of speed and energy consumption. However, this usually comes at the cost of reduced control over the dynamics of the emulated networks. In this paper, we demonstrate how iterative training of a hardware-emulated network can compensate for…
▽ More
Emulating spiking neural networks on analog neuromorphic hardware offers several advantages over simulating them on conventional computers, particularly in terms of speed and energy consumption. However, this usually comes at the cost of reduced control over the dynamics of the emulated networks. In this paper, we demonstrate how iterative training of a hardware-emulated network can compensate for anomalies induced by the analog substrate. We first convert a deep neural network trained in software to a spiking network on the BrainScaleS wafer-scale neuromorphic system, thereby enabling an acceleration factor of 10 000 compared to the biological time domain. This mapping is followed by the in-the-loop training, where in each training step, the network activity is first recorded in hardware and then used to compute the parameter updates in software via backpropagation. An essential finding is that the parameter updates do not have to be precise, but only need to approximately follow the correct gradient, which simplifies the computation of updates. Using this approach, after only several tens of iterations, the spiking network shows an accuracy close to the ideal software-emulated prototype. The presented techniques show that deep spiking networks emulated on analog neuromorphic devices can attain good computational performance despite the inherent variations of the analog substrate.
△ Less
Submitted 6 March, 2017;
originally announced March 2017.
