Sparse Connectivity¶
This guide covers the packages that replace the former sparse layers of
btorch.models.linear:
| Package | Purpose | Needed for |
|---|---|---|
btorch.models.connection |
SparseConnection, an nn.Module that maps pre-synaptic activity to post-synaptic input; Synapse, the description of weights, receptors and delays; Projection and the connection rules, which build a connection from two populations; HardDeepR rewiring. |
Building models. Part 1. |
btorch.sparse |
Sparse arrays with a SciPy/PyTorch-like API: create, convert, multiply. Matrix-free linear operators. | Numerical work with sparse matrices. Part 2. |
btorch.sparse.runtime |
Execution layer: registered operators, kernel backends, planner. | Nothing in model code. Part 3. |
If you are porting code that used SparseConn or SparseConstrainedConn, go
to Migration.
All Python blocks on this page run in order as one script.
Part 1: building models¶
Orientation: which way round?¶
A source (pre-synaptic) neuron sends spikes; a target (post-synaptic) neuron receives current. A connectivity matrix can be written in two ways that are transposes of each other, and every function in this guide that takes or returns a matrix names the one it means with one of two strings:
| String | Shape | Entry [i, j] |
Product | Where it comes from |
|---|---|---|---|---|
"pre_post" |
(n_pre, n_post) |
weight from source i to target j |
current = spikes @ W |
connectome tables, btorch.connectome.connection |
"post_pre" |
(n_post, n_pre) |
weight from source j to target i |
current = A @ spikes |
linear algebra, btorch.sparse |
A "pre_post" matrix is called an adjacency matrix below and a "post_pre"
matrix an operator. Whatever the orientation of the input, a connection is
called the same way: conn(x) maps [..., in_features] to
[..., out_features].
import scipy.sparse
import torch
from torch import nn
from btorch import sparse
from btorch.models.connection import SparseConnection, Synapse
# W[i, j]: weight from source i to target j (3 sources, 2 targets).
W = scipy.sparse.coo_array(([0.5, -1.0], ([0, 2], [1, 0])), shape=(3, 2))
spikes = torch.tensor([[1.0, 0.0, 1.0]])
conn = SparseConnection.from_adjacency(W) # orientation="pre_post" is the default
assert (conn.n_pre, conn.n_post) == (3, 2)
assert torch.allclose(conn(spikes), spikes @ torch.tensor(W.toarray(), dtype=torch.float32))
# The same connection from the operator A = W.T, shape (n_post, n_pre).
conn_op = SparseConnection(W.T)
conn_op2 = SparseConnection.from_adjacency(W.T, orientation="post_pre")
assert torch.equal(conn_op(spikes), conn(spikes))
assert torch.equal(conn_op2(spikes), conn(spikes))
The table below lists every entry point and accessor that has an orientation. "Fixed" means there is no argument to change it.
| Entry point or accessor | What it takes or returns | Orientation |
|---|---|---|
SparseConnection(A, synapse) |
matrix A of shape (n_post, n_pre) |
fixed "post_pre" |
SparseConnection.from_adjacency(W, synapse, orientation=...) |
matrix W |
default "pre_post": (n_pre, n_post); "post_pre" on request |
SparseConnection.from_edges(pre, post, n_pre, n_post, synapse) |
two index tensors and two sizes | source first, then target |
SparseConnection.from_hetersynapse(M, ...) |
expanded matrix (n_pre * n_delay, n_post * n_receptor) |
fixed "pre_post" |
FromSparse(A, orientation=...) (rule) |
matrix A |
default "pre_post"; "post_pre" on request |
FromEdges(pre, post, values) (rule), rule.edges(n_pre, n_post) |
index tensors; edges returns (pre, post) |
source first, then target |
Projection(pre, post, rule, synapse) |
two populations | source first, then target |
ConstrainedWeight(group=G) with a matrix G |
group-id matrix | the orientation of the matrix the connection is built from |
conn.to_sparse(orientation=None) |
effective weight matrix | default conn.orientation, the orientation the connection was built from |
conn.orientation |
"pre_post" or "post_pre" |
"post_pre" after SparseConnection(A); the argument after from_adjacency; "pre_post" after from_edges, from_hetersynapse and for the connection of a Projection |
conn.indices |
[2, n_edge] buffer |
row 0 is the target, row 1 the source: [post, pre] |
conn.pre, conn.post |
[n_edge] source and target of every edge |
named, no order to remember |
conn.edge_table() |
dict with the keys pre, post, weight, ... |
named |
conn.find_edges(pre, post) |
edge slots between the given neurons | source first, then target |
conn.set_edges_(slots, *, pre, post) |
new source and target of edge slots | keyword-only |
HardDeepROptions(candidate=f) |
f(pre, post) returning a boolean mask |
source first, then target |
sparse.from_edges(rows, cols, values, shape) |
numerical sparse array, no neuron semantics | used as a product, rows are the outputs (targets) and cols the inputs (sources) |
A @ x, A.matvec(x) on a Sparse |
product | A is (M, N) and maps length N to length M: "post_pre" |
Linear(in_features, out_features, weight=W) |
dense W of shape (out_features, in_features) |
standard PyTorch nn.Linear; its mask has the same shape as weight |
Three rules summarise the table:
btorch.sparsenever transposes. What you put in is what is multiplied.- In
btorch.models.connection, every function that takes two index tensors or two populations takes the source first. Matrices default to"pre_post", with one exception: the bare constructorSparseConnection(A)takes the operator. - The stored buffer
conn.indicesis[post, pre]. Preferconn.preandconn.post.
A wrong orientation of a square matrix is not an error
For a recurrent connection n_pre == n_post, so a matrix in the wrong
orientation has a valid shape. The connection is built without complaint
and computes with the transposed network. Check one known edge:
conn.find_edges(pre=i, post=j) must be non-empty for a pair you know is
connected from i to j.
The two product forms of a sparse array differ in the same silent way.
A @ X follows torch.matmul and reads the vectors along the
second-to-last axis of X; A.matvec(X) reads them along the last axis.
For a square A and a square batch X both run and return different
numbers.
# Edge 0 -> 1 with weight 0.5, edge 2 -> 0 with weight -1.0.
assert conn.find_edges(pre=0, post=1).numel() == 1
assert conn.find_edges(pre=1, post=0).numel() == 0
# A square operator and a square batch: both products run, the numbers differ.
A_square = sparse.from_edges(
rows=torch.tensor([0, 1]), cols=torch.tensor([1, 2]), values=torch.tensor([1.0, 2.0]), shape=(3, 3)
)
X_square = torch.rand(3, 3)
assert torch.allclose(A_square @ X_square, A_square.to_dense() @ X_square) # columns are vectors
assert torch.allclose(A_square.matvec(X_square), X_square @ A_square.to_dense().T) # rows are vectors
assert not torch.allclose(A_square @ X_square, A_square.matvec(X_square))
Use matvec (or a connection) for activity tensors such as
[time, batch, n_neuron].
SparseConnection¶
SparseConnection stores an explicit list of edges. Three constructors exist:
| Constructor | Input |
|---|---|
SparseConnection(A, synapse) |
operator (n_post, n_pre) |
SparseConnection.from_adjacency(W, synapse, orientation="pre_post") |
adjacency matrix, or the operator with orientation="post_pre" |
SparseConnection.from_edges(pre, post, n_pre, n_post, synapse, values=None) |
edge list; weights default to 1; ids outside the populations raise ValueError |
A matrix may be a btorch Sparse, a PyTorch sparse tensor or a SciPy sparse
array. All constructors accept:
bias=:Truefor a zero-initialised bias of shape[out_features], or a tensor of that shape as the initial value (a wrong shape raisesValueError). The bias is a parameter.hints=: see Performance hints.device=anddtype=. Withoutdtype, weights from btorch and PyTorch input keep their dtype; weights from SciPy input and from integer matrices use the PyTorch default dtype.
An edge slot is one stored edge. The connection keeps the slots sorted by target and then source, and merges duplicate entries of the input into one slot whose weight is their sum. The following members describe the slots:
print(conn.nnz) # number of edge slots: 2
print(conn.pre, conn.post) # tensor([2, 0]) tensor([0, 1])
print(conn.edge_table()) # {'pre': ..., 'post': ..., 'weight': ...}; the weight is a detached snapshot
print(conn.orientation) # 'pre_post': how this connection was built
# to_sparse() returns the effective weights in the orientation of the input ...
assert conn.to_sparse().shape == (3, 2)
assert torch.equal(conn.to_sparse().to_dense(), torch.tensor(W.toarray(), dtype=torch.float32))
# ... or in the orientation you ask for.
assert conn.to_sparse("post_pre").shape == (2, 3)
assert conn_op.to_sparse().shape == (2, 3) # built from the operator
with_bias = SparseConnection.from_adjacency(W, bias=True)
assert with_bias.bias.shape == (2,) and not with_bias.bias.any()
Each call to to_sparse() snapshots the current effective weights while
preserving their autograd history, so gradients flow through
them to the weight parameters.
Synapse and weights¶
A Synapse describes what the edges of a connection do: their weight, the
receptor channel they act on and their transmission delay. The three are
independent of each other.
Per-edge arrays inside a Synapse (weights, receptors, delays, group ids) are
aligned with the edges as you pass them: the stored entries of the matrix in
their stored order, or the edge list. The connection reorders them together
with the edges.
Synapse(weight=...) |
Weight module | Trainable |
|---|---|---|
None (default) |
EdgeWeight initialised from the matrix values |
yes |
| a number | ConstantWeight: the same weight on every edge |
no |
a [n_edge] tensor |
EdgeWeight with these values |
no |
an nn.Parameter of shape [n_edge] |
EdgeWeight with these values |
follows requires_grad |
a callable f(n_edge) -> Tensor |
EdgeWeight holding f(n_edge), called once the edges are known |
yes |
a torch.distributions.Distribution |
EdgeWeight holding dist.sample((n_edge,)) |
yes |
a Weight module |
that module, for example ConstrainedWeight |
module-defined |
A weight module is an nn.Module that owns everything trainable about the
strength of the edges. It is conn.weight. Calling it returns the effective
weight of every edge slot, a tensor of shape [n_edge] aligned with
conn.pre and conn.post. The tensors it stores are its attributes:
| Expression | Meaning |
|---|---|
conn.weight |
the weight module (EdgeWeight, ConstantWeight, ConstrainedWeight) |
conn.weight() |
effective weight of every edge slot, [n_edge] ([G, n_edge] for a network batch) |
conn.weight.shape, .dtype, .device |
those of conn.weight() |
conn.weight.value |
EdgeWeight: the stored per-edge weights, an nn.Parameter if trainable and a buffer otherwise. ConstantWeight: the scalar |
conn.weight.sign |
EdgeWeight with Dale's law: the reference sign of every edge |
conn.weight.scale, .base, .group |
ConstrainedWeight: learnable scale per group, fixed weight per edge, group id per edge |
Recipes¶
rng_matrix = scipy.sparse.random(6, 6, density=0.4, format="csr", random_state=0)
rng_matrix.data -= 0.5 # mixed signs
n_edge = rng_matrix.nnz
# 1. Matrix values as trainable per-edge weights (the default).
per_edge = SparseConnection.from_adjacency(rng_matrix)
assert isinstance(per_edge.weight.value, nn.Parameter)
# 2. One fixed number on every edge: nothing to train.
constant = SparseConnection.from_adjacency(rng_matrix, Synapse(weight=0.1))
assert not list(constant.parameters())
# 3. A tensor: fixed per-edge weights (a buffer), aligned with the stored entries.
fixed = SparseConnection.from_adjacency(rng_matrix, Synapse(weight=torch.ones(n_edge)))
assert not list(fixed.parameters())
# 4. A callable or a distribution: trainable weights drawn once the edges are known.
drawn = SparseConnection.from_adjacency(rng_matrix, Synapse(weight=lambda n: 0.1 * torch.randn(n)))
sampled = SparseConnection.from_adjacency(
rng_matrix, Synapse(weight=torch.distributions.Uniform(0.0, 0.2))
)
assert drawn.weight.shape == sampled.weight.shape == (n_edge,)
assert isinstance(sampled.weight.value, nn.Parameter)
A callable receives the number of edges as supplied, before duplicates are merged, and its result is aligned with them. A distribution is sampled with the global PyTorch random generator.
Optimizer and nn.init. The trainable tensors are ordinary parameters of
the connection, so conn.parameters() (or model.parameters()) is all an
optimizer needs. To address the weights alone use conn.weight.value (or
conn.weight.scale for a ConstrainedWeight). In-place initialisers work on
the same tensors:
optimizer = torch.optim.Adam([per_edge.weight.value], lr=1e-3) # same as per_edge.parameters()
nn.init.normal_(per_edge.weight.value, std=0.1)
With Dale's law (below) the reference signs are taken when the connection is
built. Initialise such weights through a callable or a distribution instead
of nn.init, or the new values are projected back onto the old signs.
Passing your own nn.Parameter. The connection stores edges sorted by
target and then source. If the parameter you pass is already in that order and
no duplicates had to be merged, the connection adopts it: conn.weight.value
is the same object, and an optimizer built from your parameter trains the
connection. Otherwise the values are copied into a new parameter and a
UserWarning says so; an optimizer must then use conn.weight.value.
import warnings
# Adopted: a CSR operator is already sorted by target, then source.
operator_csr = rng_matrix.T.tocsr()
mine = nn.Parameter(torch.rand(n_edge))
adopted = SparseConnection(operator_csr, Synapse(weight=mine))
assert adopted.weight.value is mine
# Copied: the stored order of a CSR adjacency matrix is source, then target.
with warnings.catch_warnings(record=True) as caught:
warnings.simplefilter("always")
copied = SparseConnection.from_adjacency(rng_matrix, Synapse(weight=mine))
assert copied.weight.value is not mine
assert any("could not be adopted" in str(w.message) for w in caught)
The order is kept by SparseConnection(A) with a canonical CSR operator, by
from_adjacency(W) with a canonical CSC adjacency matrix, and by from_edges
with edges sorted by (post, pre). The edges of a random rule in a
Projection are generally not in that
order. Code that does not control the order should always read the parameter
back from conn.weight.value.
Passing a Weight module. The module you pass is conn.weight. It
holds the parameters of that one connection, so it can be used once; building
a second connection with it raises a RuntimeError. A Synapse with any
other kind of weight can be reused for several connections, and each gets its
own weight module.
from btorch.models.connection import EdgeWeight
module = EdgeWeight(torch.rand(n_edge))
shared_synapse = Synapse(weight=module)
first = SparseConnection.from_adjacency(rng_matrix, shared_synapse)
assert first.weight is module
try:
SparseConnection.from_adjacency(rng_matrix, shared_synapse)
except RuntimeError as err:
print(err) # This weight module is already bound to a connection. ...
Synapse(plasticity=...) is reserved for online plasticity rules. Any value
other than None raises NotImplementedError.
ConstrainedWeight: tied weights¶
ConstrainedWeight ties edges into groups. Each edge has a fixed base
weight and each group one learnable scale; the effective weight of edge e
is base[e] * scale[group[e]], so the edges of a group keep their relative
strengths.
import numpy as np
from btorch.models.connection import ConstrainedWeight
# Group ids as a matrix with the same orientation and pattern as the
# connection matrix. Ids in a matrix are 1-based because 0 means "no entry".
group_matrix = rng_matrix.copy()
group_matrix.data = np.random.default_rng(0).integers(1, 4, n_edge).astype(float)
tied = SparseConnection.from_adjacency(
rng_matrix, Synapse(weight=ConstrainedWeight(group=group_matrix))
)
assert tied.weight.scale.shape == (3,) # the only trainable tensor
print(tied.weight.group_info()) # DataFrame: group_id, num_connections, scale
tied.weight.set_scale(0, 2.0) # double every edge of group 0
by_group = tied.weight.weights_by_group() # {group id: effective weights}
# Alternatively pass 0-based ids as a [n_edge] tensor aligned with the entries.
ids = torch.from_numpy(group_matrix.data).long() - 1
tied2 = SparseConnection.from_adjacency(rng_matrix, Synapse(weight=ConstrainedWeight(group=ids)))
assert torch.equal(tied2.weight.group, tied.weight.group)
Dale's law¶
Dale's law states that all outgoing synapses of a neuron have the same sign. btorch enforces the per-edge form: an edge keeps the sign it had at construction.
Synapse(dale=True)with per-edge weights stores the reference sign inconn.weight.sign.constrain()sets weights that crossed zero to zero.ConstrainedWeight(group=..., dale=True)keeps every scale non-negative.Synapse(weight=module, dale=True)with aWeightmodule switches thedaleflag of that module on, soSynapse(weight=ConstrainedWeight(group=...), dale=True)is the same as the previous line. A module without adaleflag (ConstantWeight) raises aValueError. Withdale=Falsethe module is used as given.Synapse(weight=<number>, dale=True)raises aValueError: the sign of one fixed number cannot change, so there is nothing to enforce.
The projection onto the constraint is not part of the forward pass. Call
btorch.models.constrain.constrain_net(model) after every optimizer step; it
calls constrain() on every weight module of the model. For a single
connection, conn.constrain() does the same. dale defaults to False.
from btorch.models.constrain import constrain_net
dale = SparseConnection.from_adjacency(rng_matrix, Synapse(dale=True))
dale_opt = torch.optim.SGD(dale.parameters(), lr=10.0)
loss = dale(torch.rand(4, 6)).sum()
loss.backward()
dale_opt.step()
constrain_net(dale) # weights that changed sign are now 0
w, s = dale.weight.value, dale.weight.sign
assert bool((w * s >= 0).all())
Receptors and delays¶
A receptor is an input channel of the target neuron (for example AMPA and GABA); a delay is the number of time steps a spike takes to arrive. Both are attributes of an edge:
pre = torch.tensor([0, 1, 2, 2])
post = torch.tensor([1, 2, 0, 1])
synapse = Synapse(
receptor=torch.tensor([0, 1, 0, 1]), # channel on the target, per edge
delay=torch.tensor([0, 2, 1, 0]), # time steps, per edge
n_receptor=2,
n_delay=3,
)
routed = SparseConnection.from_edges(pre, post, n_pre=3, n_post=3, synapse=synapse)
assert routed.in_features == 3 * 3 # n_pre * n_delay
assert routed.out_features == 3 * 2 # n_post * n_receptor
print(routed.edge_table()) # dict with pre, post, weight, receptor, delay
An int instead of a tensor assigns the same receptor or delay to all edges.
n_receptor and n_delay default to the largest id plus one. The connection
stores the edge list with its attributes. Its input and output are flat
layouts derived from them:
| Side | Size | Index of (neuron, attribute) |
|---|---|---|
| input | n_pre * n_delay |
pre * n_delay + delay |
| output | n_post * n_receptor |
post * n_receptor + receptor |
Delay d reads the spikes from d steps ago, so delay 0 is the current step.
The input layout is the one SpikeHistory.get_flattened(n_delay) produces; the
output can be reshaped to [..., n_post, n_receptor].
from btorch.models.history import SpikeHistory
history = SpikeHistory(n_neuron=3, max_delay_steps=3)
history.init_state(batch_size=1)
for _ in range(4):
history.update((torch.rand(1, 3) < 0.5).float())
current = routed(history.get_flattened(3)) # [1, n_post * n_receptor]
per_receptor = current.reshape(1, 3, 2) # [batch, n_post, n_receptor]
The helpers in btorch.connectome.connection (make_hetersynapse_conn,
expand_conn_for_delays, stack_hetersynapse, ...) are unchanged. They
return a physically expanded adjacency matrix of shape
(n_pre * n_delay, n_post * n_receptor) that uses exactly these layouts. Two
ways to use such a matrix give the same result:
SparseConnection.from_adjacency(expanded)treats it as an ordinary matrix betweenn_pre * n_delaysources andn_post * n_receptortargets.SparseConnection.from_hetersynapse(expanded, n_receptor=None, n_delay=1, receptor_type_index=None)decodes the expansion into per-edgereceptoranddelayattributes, soconn.n_pre,conn.n_post,conn.edge_table()and the checkpoint describe neurons and not expanded rows and columns. Pass eithern_receptoror the receptor index table returned by the helper asreceptor_type_index; only the length of the table is used, and the two must agree if both are given.
import pandas as pd
from btorch.connectome.connection import make_hetersynapse_conn
neurons = pd.DataFrame({"simple_id": range(4), "EI": ["E", "E", "I", "I"]})
edges = pd.DataFrame(
{
"pre_simple_id": [0, 1, 2, 3, 0],
"post_simple_id": [1, 2, 3, 0, 2],
"syn_count": [3, 1, 2, 4, 1],
"delay_steps": [0, 1, 2, 0, 1],
}
)
expanded, receptor_idx = make_hetersynapse_conn(
neurons, edges, receptor_type_col="EI", delay_col="delay_steps", n_delay_bins=3
)
n_receptor = len(receptor_idx) # (pre type, post type) pairs: 4
assert expanded.shape == (4 * 3, 4 * n_receptor)
hetero = SparseConnection.from_hetersynapse(expanded, receptor_type_index=receptor_idx, n_delay=3)
flat = SparseConnection.from_adjacency(expanded)
assert (hetero.n_pre, hetero.n_post) == (4, 4) and (flat.n_pre, flat.n_post) == (12, 16)
assert (hetero.n_receptor, hetero.n_delay) == (n_receptor, 3)
z = torch.rand(2, 12)
assert torch.allclose(hetero(z), flat(z))
assert hetero.to_sparse().shape == expanded.shape # the expanded "pre_post" layout
The connection does not store the receptor index table. Keep it next to the
model to map channel ids to receptor names. A ConstrainedWeight group matrix
passed to from_hetersynapse uses the same expanded layout as the connection
matrix (this is what make_hetersynapse_constrained_conn returns).
Connection rules and Projection¶
Instead of a matrix, a connection can be specified the way NEST does it: two
populations, a rule that says which pairs of neurons are connected, and a
Synapse that says what the connections do. Projection combines the three
and builds the connection.
A rule returns an edge list (pre, post) in population-local indices and
nothing else. Two terms from NEST: an autapse is a connection of a neuron
onto itself, a multapse is a pair that is connected more than once.
| Rule | Which pairs are connected |
|---|---|
OneToOne() |
source i to target i; needs equal population sizes |
AllToAll(allow_autapses=True) |
every source to every target |
FixedIndegree(k, allow_autapses=True, allow_multapses=True) |
every target receives exactly k connections from random sources |
FixedOutdegree(k, allow_autapses=True, allow_multapses=True) |
every source makes exactly k connections onto random targets |
PairwiseBernoulli(p, allow_autapses=True) |
every pair independently with probability p |
DistanceDependent(pre_pos, post_pos, probability, max_distance=None) |
every pair independently with probability probability(distance) |
FromEdges(pre, post, values=None) |
the given edge list |
FromSparse(A, orientation="pre_post") |
the stored entries of a sparse matrix; "pre_post" is the adjacency (n_pre, n_post), "post_pre" the operator (n_post, n_pre) |
allow_autapses=False is only defined when a population projects onto itself
(n_pre == n_post). All randomness comes from the generator argument, so a
seeded torch.Generator reproduces the edges.
from btorch.models.connection import (
AllToAll,
FixedIndegree,
FixedOutdegree,
FromSparse,
Projection,
StructuredConnection,
)
from btorch.models.neurons.lif import LIF
exc, inh = LIF(n_neuron=80), LIF(n_neuron=20)
# Every inhibitory neuron receives 10 excitatory inputs, two steps delayed.
e_to_i = Projection(
pre=exc,
post=inh,
rule=FixedIndegree(10),
synapse=Synapse(weight=0.1, delay=2),
generator=torch.Generator().manual_seed(0),
)
assert (e_to_i.n_pre, e_to_i.n_post) == (80, 20)
assert (e_to_i.in_features, e_to_i.out_features) == (80 * 3, 20) # delays 0..2
assert (e_to_i.n_delay, e_to_i.n_receptor) == (3, 1)
# Mutual inhibition without self-connections.
i_to_i = Projection(inh, inh, AllToAll(allow_autapses=False), Synapse(weight=-0.5))
assert i_to_i(torch.ones(20)).tolist() == [-0.5 * 19] * 20
pre and post are neuron counts or modules that know their size (size or
n_neuron, as every btorch neuron model has); only the size is read. A
projection is an nn.Module and is called like a connection. The connection
it built is proj.connection; proj.n_delay, proj.n_receptor,
proj.weight, proj.edge_table() and proj.to_sparse() forward to it. The
last three need explicit edges and raise AttributeError for a projection
that was realised without them (see below).
Synapse arrays align with the rule's edges. Entry e of a per-edge
weight, delay, receptor or group array belongs to edge e of
rule.edges(n_pre, n_post, generator=...). To build such arrays, ask the rule
for its edges with a generator of the same seed as the one given to the
projection:
out_rule = FixedOutdegree(3, allow_multapses=False)
rule_pre, rule_post = out_rule.edges(80, 20, generator=torch.Generator().manual_seed(1))
# Even sources act on receptor 0, odd sources on receptor 1, with other weights.
by_source = Projection(
80,
20,
out_rule,
Synapse(weight=0.1 * (1 + rule_pre % 2), receptor=rule_pre % 2),
generator=torch.Generator().manual_seed(1),
)
table = by_source.edge_table()
assert torch.equal(table["receptor"], table["pre"] % 2)
assert torch.allclose(table["weight"], 0.1 * (1 + table["pre"] % 2))
A weight that does not depend on the individual edge needs no such alignment: pass a number, a callable or a distribution.
Without a synapse, FromEdges and FromSparse use the values they carry as
trainable per-edge weights, and all other rules start from trainable unit
weights. A weight given in the synapse replaces the values of the rule.
# The 3 x 2 adjacency W of the orientation section (rows are sources).
from_matrix = Projection(3, 2, FromSparse(sparse.asarray(W, dtype=torch.float32)))
assert torch.allclose(from_matrix(spikes), conn(spikes))
assert isinstance(from_matrix.weight.value, nn.Parameter)
assert from_matrix.to_sparse().shape == (3, 2)
Multapses are merged. Parallel edges that agree on source, target,
receptor and delay become one stored edge. Their weights add: per-edge weights
are summed, and a fixed scalar weight counts with the multiplicity (a pair
connected three times carries three times the weight). Edges between the same
pair with different receptors or delays stay separate. Merged edges must
belong to the same ConstrainedWeight group; otherwise a ValueError is
raised.
Rules never decide the execution format. A rule is not a sparse array and has no format, algorithm or backend; the same rule object can be used for several projections. How a projection is realised follows from the synapse:
- If the rule has a closed form (
OneToOne,AllToAll,PairwiseBernoulliwithp == 1) and the synapse is one fixed number without receptor, delay and Dale's law, the projection builds aStructuredConnectionover a linear operator. No edge is stored. - In every other case (trainable or per-edge weights, receptors, delays, any
other rule) it builds a
SparseConnectionwith explicit edges.realization="sparse"forces this.
assert isinstance(i_to_i.connection, StructuredConnection)
assert isinstance(e_to_i.connection, SparseConnection)
explicit = Projection(
inh, inh, AllToAll(allow_autapses=False), Synapse(weight=-0.5), realization="sparse"
)
assert explicit.connection.nnz == 20 * 19
xi = torch.rand(5, 20)
assert torch.allclose(explicit(xi), i_to_i(xi), atol=1e-5)
A random rule draws new edges every time a projection is constructed. The
edges are part of the state_dict of a sparse projection, so
load_state_dict restores the saved network. It requires the same number of
stored edges, as described under Checkpoints: that holds for
FixedIndegree and FixedOutdegree with allow_multapses=False and for the
deterministic rules. With multapses, PairwiseBernoulli or
DistanceDependent the number of stored edges is random, so construct the
receiving projection with the same generator seed as the saved one.
Wiring into neuron, PSC and RNN modules¶
A connection is the linear argument of a post-synaptic current (PSC) module
from btorch.models.synapse. The PSC module filters the output of the
connection in time; RecurrentNN feeds the spikes of a neuron module back
through the PSC module and unrolls the loop over time steps.
The example below builds a recurrent network of 80 excitatory (E) and 20 inhibitory (I) neurons from four projections. One neuron module holds both populations, E first. A small module splits the spike vector by population, applies the four projections and concatenates the two target populations again. Weights are drawn from a distribution (excitatory) and a callable (inhibitory), and Dale's law keeps their signs during training.
from btorch.models import environ, functional
from btorch.models.connection import PairwiseBernoulli
from btorch.models.rnn import RecurrentNN
from btorch.models.synapse import ExponentialPSC
torch.manual_seed(0) # seeds the weight draws; the generator below seeds the edges
n_exc, n_inh = 80, 20
n_all = n_exc + n_inh
excitatory = Synapse(weight=torch.distributions.Uniform(0.0, 0.2), dale=True)
inhibitory = Synapse(weight=lambda n: -0.8 * torch.rand(n), dale=True)
class EIRecurrence(nn.Module):
"""Recurrent input of an E/I network: [..., n_exc + n_inh] -> the same."""
def __init__(self, n_exc, n_inh, generator):
super().__init__()
self.n_exc = n_exc
in_exc = FixedIndegree(8, allow_multapses=False)
self.ee = Projection(
n_exc,
n_exc,
FixedIndegree(8, allow_autapses=False, allow_multapses=False),
excitatory,
generator=generator,
)
self.ei = Projection(n_exc, n_inh, in_exc, excitatory, generator=generator)
self.ie = Projection(n_inh, n_exc, PairwiseBernoulli(0.25), inhibitory, generator=generator)
self.ii = Projection(
n_inh,
n_inh,
PairwiseBernoulli(0.25, allow_autapses=False),
inhibitory,
generator=generator,
)
def forward(self, spikes):
z_exc, z_inh = spikes[..., : self.n_exc], spikes[..., self.n_exc :]
to_exc = self.ee(z_exc) + self.ie(z_inh)
to_inh = self.ei(z_exc) + self.ii(z_inh)
return torch.cat([to_exc, to_inh], dim=-1)
recurrence = EIRecurrence(n_exc, n_inh, torch.Generator().manual_seed(0))
net = RecurrentNN(
neuron=LIF(n_neuron=n_all, v_threshold=1.0, v_reset=0.0, tau=20.0, tau_ref=2.0, step_mode="s"),
synapse=ExponentialPSC(n_all, tau_syn=5.0, linear=recurrence, step_mode="s"),
step_mode="m", # the module consumes a whole [time, batch, n_all] sequence
update_state_names=("neuron.v",),
)
functional.init_net_state(net, batch_size=4, dtype=torch.float32)
# The four weight tensors are the only parameters of this network.
assert len(list(net.parameters())) == 4
ei_opt = torch.optim.Adam(net.parameters(), lr=1e-2)
drive = 0.08 + 0.05 * torch.randn(50, 4, n_all) # external input [time, batch, n_all]
with environ.context(dt=1.0):
for _ in range(3):
functional.reset_net(net, batch_size=4)
net_spikes, states = net(drive) # [50, 4, 100], {"neuron.v": [50, 4, 100]}
rate_loss = (net_spikes.mean(dim=0) - 0.05).pow(2).mean() # target rate per step
ei_opt.zero_grad()
rate_loss.backward() # surrogate gradient through spikes, PSC and connections
ei_opt.step()
constrain_net(net) # Dale's law: project the weights after every step
assert net_spikes.shape == states["neuron.v"].shape == (50, 4, n_all)
for name in ("ee", "ei", "ie", "ii"):
projection = getattr(recurrence, name)
assert projection.weight.value.grad is not None
assert bool((recurrence.ee.weight.value >= 0).all()) and bool((recurrence.ie.weight.value <= 0).all())
The same Synapse object describes two projections each: a Synapse with a
distribution or a callable creates a new weight module per connection. The
synapses of this example have no delay.
A connection with receptor channels has n_post * n_receptor outputs.
HeterSynapsePSC takes such a connection, keeps one PSC state per channel and
sums the channels of every neuron. It needs the receptor index table of the
connectome helper:
from btorch.models.synapse import AlphaPSC, HeterSynapsePSC
# The neurons and edges of the receptor section, without the delay column.
by_type, type_idx = make_hetersynapse_conn(
neurons, edges.drop(columns="delay_steps"), receptor_type_col="EI"
)
typed = SparseConnection.from_hetersynapse(by_type, receptor_type_index=type_idx)
assert (typed.out_features, typed.n_receptor) == (4 * 4, 4)
with environ.context(dt=1.0):
receptor_psc = HeterSynapsePSC(
n_neuron=4,
n_receptor=typed.n_receptor,
receptor_type_index=type_idx,
linear=typed,
base_psc=AlphaPSC,
tau_syn=5.0,
)
functional.init_net_state(receptor_psc, batch_size=2)
summed = receptor_psc(torch.ones(2, 4)) # [batch, n_neuron]
assert summed.shape == (2, 4)
Network batch versus sample batch¶
Two kinds of leading dimension occur in the input of a connection:
- A sample batch is a set of inputs for one network: time steps, trials,
mini-batch entries. A connection accepts any number of them:
[..., in_features]. - A network batch is a set of
Gnetworks (an ensemble or a parameter sweep). It exists when the weights have shape[G, n_edge], or when the matrix is asparse.stackof different patterns. The input is then[G, ..., in_features]: network dimensions come first.
# Four networks that share the pattern of rng_matrix but not the weights.
ensemble_matrix = sparse.asarray(rng_matrix, dtype=torch.float32)
ensemble_matrix = ensemble_matrix.with_values(torch.randn(4, n_edge))
ensemble = SparseConnection.from_adjacency(ensemble_matrix)
assert ensemble.batch_shape == (4,)
x = torch.rand(4, 10, 32, 6) # [G, time, batch, n_pre]
assert ensemble(x).shape == (4, 10, 32, 6)
# The same samples for every network: a leading dimension of size 1 broadcasts.
shared_x = torch.rand(10, 32, 6)
assert ensemble(shared_x.unsqueeze(0)).shape == (4, 10, 32, 6)
try:
ensemble(torch.rand(6)) # no network dimension
except ValueError as err:
print(err)
A network dimension is never inferred from, or merged with, a sample
dimension. With ConstrainedWeight, batch the scales (scale= of shape
[G, n_group]), not the base weights. Receptors and delays are not supported
for a batch of different patterns.
For a batch of different patterns, conn.edge_table() and conn.to_sparse()
report network coordinates: the table has an additional network entry with
the batch index of every edge, pre and post are neuron ids within that
network, and to_sparse() returns a batched COO. The stored indices buffer
(and with it conn.pre and conn.post) instead addresses the block-diagonal
operator over all networks that is used for execution.
S_a = scipy.sparse.random(5, 4, density=0.4, format="csr", random_state=0)
S_b = scipy.sparse.random(5, 4, density=0.2, format="csr", random_state=1)
ragged = sparse.stack([sparse.asarray(S_a, dtype=torch.float32), sparse.asarray(S_b, dtype=torch.float32)])
ragged_conn = SparseConnection(ragged) # two 5 x 4 operators
ragged_table = ragged_conn.edge_table()
assert set(ragged_table["network"].flatten().tolist()) == {0, 1}
assert int(ragged_table["post"].max()) < 5 and int(ragged_table["pre"].max()) < 4
assert ragged_conn.to_sparse().shape == (2, 5, 4)
assert ragged_conn(torch.rand(2, 7, 4)).shape == (2, 7, 5)
torch.compile¶
There is no separate preparation step and no backend to choose: compile the
connection, or the model that contains it, with torch.compile.
fullgraph=True works on CPU and GPU and does not need torch_sparse. The
compiled graph contains one registered operator per connection call; the
sparse layout is not visible to the tracer.
compiled = torch.compile(per_edge, fullgraph=True)
x = torch.rand(5, 6)
assert torch.allclose(compiled(x), per_edge(x), atol=1e-6)
Gradients with respect to the weights and the input are available in compiled and uncompiled mode. The input stays a dense tensor, so surrogate-gradient training needs no change.
CUDA graphs and reduce-overhead¶
A CUDA graph records the GPU kernel launches of a piece of code once and replays them later without running the Python code again. Replay reads the contents of the tensors it recorded, but everything decided in Python at capture time is frozen: which kernels run, with which launch sizes, and on which device buffers.
For a connection this means:
- Weight updates are safe. The weights are read at replay time. An
optimizer step,
constrain()or an in-place edit does not invalidate a graph. - A graph captured around a connection is invalid after
set_edges_(and so after aHardDeepRupdate that moved an edge), after aload_state_dictthat brings other edges, after.to(...)(also.cuda(),.float(), ...) and after aset_hintsthat changes the planned algorithm. Replaying it then computes with the old wiring or the old execution path, without an error. conn.capture_versionis a hashable value that changes on each of these events (on everyload_state_dict, whether or not the edges differ) and does not change on weight updates. Compare it before a replay and capture again if it differs.btorch.models.cudagraph.capture_versions(model)collects it for every connection of a model.conn.capture_incompatibility()returnsNone, or the reason why the connection cannot be captured at all. Today there is one reason: a connection with a density hint that is executed by the reference backend (algorithm"adaptive-push", see Performance hints) packs the spikes on the host, which a graph cannot record.
conn.value_version is the counterpart for weights: it is derived from the
in-place version counters of the weight tensors, so it changes whenever a
weight tensor is written in place (an optimizer step, a constrain() that
clamps, a checkpoint load).
from btorch.models.cudagraph import capture_incompatibilities, capture_versions
graph_conn = SparseConnection.from_adjacency(rng_matrix)
captured_at = capture_versions(graph_conn) # (graph_conn.capture_version,)
weights_at = graph_conn.value_version
with torch.no_grad():
graph_conn.weight.value.mul_(0.5) # a weight update ...
assert capture_versions(graph_conn) == captured_at # ... keeps a captured graph valid
assert graph_conn.value_version != weights_at
slot = torch.tensor([0])
graph_conn.set_edges_(slot, pre=torch.tensor([1]), post=torch.tensor([4])) # rewiring ...
assert capture_versions(graph_conn) != captured_at # ... does not
assert capture_incompatibilities(graph_conn) == {} # no density hint: can be captured
RecurrentNN(cudagraph=True) checks these versions on every call and
captures again by itself when one of them changed, so rewiring and loading a
checkpoint between calls need no extra code there; it raises with the reason
when a connection reports a capture incompatibility.
torch.compile(mode="reduce-overhead") uses CUDA graphs internally and has no
such check. Two rules follow:
- Do not combine
mode="reduce-overhead"with rewiring or with loading a different connectivity pattern. Afterset_edges_, aHardDeepRupdate or aload_state_dictwith other edges, the compiled module keeps replaying the graph of the old pattern. Use the default compile mode for such models, orRecurrentNN(cudagraph=True)for inference. - Call the connection once in eager mode before compiling in that mode, on its final device and with its final hints. The first call lets the backend finish its one-time setup outside any capture.
When you drive torch.cuda.CUDAGraph by hand, do the same and keep the
versions next to the graph:
if torch.cuda.is_available():
cuda_conn = SparseConnection.from_adjacency(rng_matrix).to("cuda")
static_x = torch.rand(8, 6, device="cuda")
assert not capture_incompatibilities(cuda_conn)
def capture():
graph = torch.cuda.CUDAGraph()
with torch.no_grad():
for _ in range(3): # warm-up in eager mode, outside the capture
cuda_conn(static_x)
with torch.cuda.graph(graph):
static_y = cuda_conn(static_x)
return graph, static_y, capture_versions(cuda_conn)
graph, static_y, versions = capture()
static_x.copy_(torch.rand(8, 6)) # new input, same buffer
graph.replay()
with torch.no_grad():
assert torch.allclose(static_y, cuda_conn(static_x), atol=1e-6)
cuda_conn.set_edges_(torch.tensor([0]), pre=torch.tensor([1]), post=torch.tensor([4]))
if capture_versions(cuda_conn) != versions: # the old graph is stale
graph, static_y, versions = capture()
graph.replay()
with torch.no_grad():
assert torch.allclose(static_y, cuda_conn(static_x), atol=1e-6)
Checkpoints¶
The state_dict of a connection contains the edge list, the sizes it was
built with and everything needed to reproduce the weights. Execution layouts
(CSR pointers, permutations) are derived, never saved, and rebuilt after
load_state_dict.
| Key | Content | Present |
|---|---|---|
indices |
[2, n_edge]: target (row 0) and source (row 1) neuron of every edge |
always |
layout |
[n_post, n_pre, n_receptor, n_delay], followed by the batch shape for a batch of different patterns |
always |
receptor, delay |
[n_edge] ids |
when the synapse sets them |
bias |
[out_features] |
when bias= is given |
weight.value |
per-edge weights | EdgeWeight, ConstantWeight (scalar) |
weight.multiplicity |
[n_edge] number of parallel input edges merged into each edge |
ConstantWeight |
weight.sign |
reference signs for Dale's law | EdgeWeight with dale=True |
weight.scale, weight.group, weight.base |
scales, group ids, base weights | ConstrainedWeight |
print(list(dale.state_dict())) # ['indices', 'layout', 'weight.value', 'weight.sign']
print(list(tied.state_dict())) # ['indices', 'layout', 'weight.scale', 'weight.group', 'weight.base']
print(list(routed.state_dict())) # ['indices', 'receptor', 'delay', 'layout', 'weight.value']
# A module with the same number of edges, but other edges, weights and signs.
other = rng_matrix.copy()
other.indices = np.random.default_rng(1).permutation(other.indices)
other.data = -other.data
restored = SparseConnection.from_adjacency(other, Synapse(dale=True))
restored.load_state_dict(dale.state_dict())
x = torch.rand(3, 6)
assert torch.equal(restored(x), dale(x))
assert torch.equal(restored.weight.sign, dale.weight.sign)
To load a checkpoint, construct the connection with the same options, the
same population sizes and the same number of edges, then call
load_state_dict. The edges, Dale signs and group structure all come from the
checkpoint, not from the matrix used to construct the module.
load_state_dict validates what it is given, and raises a RuntimeError
without copying anything from that checkpoint into the connection when a check
fails. The checks also apply with strict=False:
- The population sizes and the numbers of receptors and delays in
layoutmust equal those of the module. indices,receptoranddelaymust address neurons and channels inside the module.- A checkpoint that has
indicesbut nolayoutwas written by the removedSparseConn/SparseConstrainedConnlayers. It is refused with a message that says how to migrate: rebuild the connection from its matrix withSparseConnection.from_adjacencyand copy the weights.
wider = SparseConnection.from_adjacency(
scipy.sparse.random(7, 6, density=0.4, format="csr", random_state=0)
)
try:
wider.load_state_dict(per_edge.state_dict()) # 6 sources into a module with 7
except RuntimeError as err:
print(err)
legacy = {"indices": per_edge.indices.clone(), "magnitude": torch.rand(n_edge)}
try:
per_edge.load_state_dict(legacy, strict=False)
except RuntimeError as err:
print(err)
Performance hints and explain¶
Hints describe how a connection will be used. They do not change what is
computed, only how the product is executed (see the note on rounding below).
from btorch.sparse import Hints
hinted = SparseConnection.from_adjacency(rng_matrix, hints=Hints(expected_density=0.001))
x = (torch.rand(8, 6) < 0.001).float()
print(hinted.explain(x)) # on CPU: "algorithm = adaptive-push (expected_density 0.001 <= 0.002)"
expected_density is the expected fraction of non-zero entries in the input
(the spike probability per neuron and step), a number in [0, 1]. The
planner chooses one of three algorithms from it when the connection is
built, moved to a device, or given new hints:
| Algorithm | When it is planned | What it does |
|---|---|---|
"pull" |
no density hint; a hint above the limit of the device; a network batch of weights | Visits every edge: each target sums over its incoming edges. Deterministic. |
"push" |
a hint at or below the limit, on a backend that compacts the spikes on the device (Triton on CUDA) | Visits only the outgoing edges of active sources, in one launch without host synchronisation. Used for every call, whatever the density of an individual input. |
"adaptive-push" |
a hint at or below the limit, on the reference backend (CPU, or CUDA without Triton) | Packs the active sources on the host and visits only their outgoing edges. Each call whose input is denser than the limit falls back to pull. |
The limits are measured crossover points: 0.2 % on CPU and 2 % on CUDA. They
are stored in btorch.sparse.runtime.planner.push_max_density.
Points to know before setting a hint:
"push"is not bitwise reproducible. Several sources add into the same target concurrently, so the order of the floating-point additions varies between runs. Results agree with pull to about4e-7relative error. Undertorch.use_deterministic_algorithms(True)the planner plans"pull"instead; the setting is read when the plan is made, so set it before the connection is built or moved."push"takes the hint at its word. It is used for every input, also for one that is far denser than the hint; it stays correct and becomes slower than pull."push"hides non-finite weights of silent sources. Pull multiplies every weight with its input, so aninfornanweight reaches the output even when its source is silent (inf * 0isnan). Push never visits the edges of a silent source. A diverged weight can therefore go unnoticed until its source fires."adaptive-push"cannot be captured in a CUDA graph, because it synchronises with the host to pack the spikes (see CUDA graphs).- The gradient is the same deterministic one for all three algorithms.
conn.explain(x) (also available as sparse.explain(conn, x)) returns a text
report: the measured density of x, the edge count, the chosen algorithm with
the reason, the kernel backend and the weight module. x is optional.
conn.set_hints(Hints(...)) changes the hints of an existing connection and
plans again.
Hints also has expected_calls and expected_batch (positive integers);
the planner does not use them yet. A value outside its range raises a
ValueError when the Hints object is created.
Structural plasticity¶
Structural plasticity changes which neurons are connected during training.
HardDeepR implements hard Deep Rewiring (Bellec et al., 2018) for a
SparseConnection with trainable per-edge weights.
The connection keeps a fixed number of edge slots (conn.nnz). Slot k has
a weight w_k and a reference sign s_k. After an optimizer step, a slot is
dormant if its weight reached or crossed zero relative to that sign
(s_k * w_k <= 0). The connection of a dormant slot is removed and the slot
is reused for a new connection at a random position that is currently
unconnected, with a small initial weight (init). The number of connections
therefore never changes, no tensor changes shape and the weight parameter is
never replaced, so the optimizer and modules compiled with the default mode of
torch.compile stay valid.
from btorch.models.connection import HardDeepR, HardDeepROptions
plastic_matrix = scipy.sparse.random(30, 30, density=0.1, format="csr", random_state=0)
plastic_matrix.data -= 0.5 # mixed signs
plastic = SparseConnection.from_adjacency(plastic_matrix, Synapse(dale=True))
# 1. The controller, 2. torch.compile, 3. attach to the optimizer.
rewire = HardDeepR(
plastic,
HardDeepROptions(l1=1e-3, candidate=lambda pre, post: pre < 20), # new edges from sources 0..19 only
generator=torch.Generator().manual_seed(0),
)
plastic_fast = torch.compile(plastic, fullgraph=True)
plastic_opt = torch.optim.Adam(plastic.parameters(), lr=0.05)
rewire.attach(plastic_opt)
n_slot, weight_param = plastic.nnz, plastic.weight.value
edges_before = plastic.indices.clone()
xp, target = torch.rand(16, 30), torch.rand(16, 30)
for _ in range(20):
plastic_opt.zero_grad()
((plastic_fast(xp) - target) ** 2).mean().backward()
plastic_opt.step() # gradient step, then the rewiring update
assert rewire.n_rewired > 0 and plastic.topology_version > 0
moved = (plastic.indices != edges_before).any(dim=0)
assert bool(moved.any()) and bool((plastic.pre[moved] < 20).all()) # edges moved, as allowed ...
assert plastic.nnz == n_slot and plastic.weight.value is weight_param # ... slots did not
assert torch.allclose(plastic_fast(xp), plastic(xp), atol=1e-5) # no stale graph
- What can be rewired.
HardDeepRtakes aSparseConnection, or aProjectionthat was realised as one. The weight must be a trainable, unbatchedEdgeWeight;ConstantWeight,ConstrainedWeight, fixed weights and a network batch are refused. attach(optimizer)registers a hook that runs after everyoptimizer.step(). It applies the optional L1 shrink (l1) and random walk (noise) of Deep R to the weights and then, on everyevery-th step, rewires the dormant slots. The shrink and the random walk are skipped on steps where the weights received no gradient.rewire.step()runs the structural update by hand;rewire.detach()removes the hook.- Signs. With
Synapse(dale=True)the reference sign is thesignbuffer of the weight, and a new connection takes the sign of its source neuron. Without Dale's law the controller tracks the sign each weight had when its slot was last activated, and a new connection gets a random sign. - Where new connections may appear.
candidate=frestricts the positions:f(pre, post)receives two integer tensors of equal shape, the source and the target of each proposed connection, and returns a boolean mask.allow_autapses=Falseexcludespre == post. - Optimizer state. A rewired slot is a new connection, so its entries in
the optimizer's per-parameter state need a policy (
optimizer_state)."neutral"(the default) zeroes first-moment state (Adam'sexp_avg, SGD momentum) and sets second-moment state (exp_avg_sq, RMSprop'ssquare_avg) to the mean over the other slots, so a new connection takes steps of the same size as established ones."reset"zeroes everything; because Adam's step count is global, the first step of a reset slot is then about 2.5 times larger than normal (10 times for RMSprop)."keep"leaves the state of the removed connection in place. - When no free position is left. On a small or nearly full layer there
may be fewer admissible unconnected positions than dormant slots. The
update then places as many as it can; the remaining slots stay where they
are with weight exactly zero and are retried at later updates.
rewire.n_unplacedreports how many are waiting, and aRuntimeWarningis emitted once per controller. A position vacated in an update becomes available at the next one. - Construct the controller before
torch.compile. The constructor callsconn.enable_rewiring(), which makes the traced forward pass independent of the order of the edges, so rewiring never changes the traced graph. This holds for the default compile mode.mode="reduce-overhead"must not be combined with rewiring, and a hand-captured CUDA graph has to be captured again after an update that moved an edge; see CUDA graphs. conn.topology_versionis incremented by every update that moves at least one edge (and byload_state_dict). The execution layouts are rebuilt at that moment, outside the forward pass.rewire.n_rewiredcounts the rewired slots.- What is persisted. The edges, weights and Dale signs are in the
state_dictof the connection. The controller has its ownstate_dict()/load_state_dict()(counters, tracked signs, waiting slots and the state of its random generator). To resume a run, load the connection, then the optimizer, then the controller; training then continues exactly as an uninterrupted run would.
The remaining options (sign, max_tries, every, init) are described in
the docstring of HardDeepROptions.
The low-level call behind the controller is
conn.set_edges_(slots, *, pre, post, receptor=None, delay=None). The index
arguments are keyword-only. It changes which edge each listed slot stands
for, in place, and rebuilds the execution layouts. Call it outside the
forward pass, and call conn.enable_rewiring() before compiling a connection
you rewire by hand.
manual = SparseConnection.from_adjacency(plastic_matrix)
manual.enable_rewiring()
free_slots = manual.find_edges(pre=manual.pre[:2], post=manual.post[:2])[:2]
manual.set_edges_(free_slots, pre=torch.tensor([3, 4]), post=torch.tensor([5, 6]))
assert manual.find_edges(pre=3, post=5).numel() >= 1
assert manual.topology_version == 1
Two slots may describe the same pair after such a call; their weights then add.
Soft Deep R is not implemented. In soft Deep R a dormant connection keeps its
parameter and can reactivate at its old position, which needs one parameter
for every possible connection, that is dense n_post * n_pre memory. This is
what a sparse connection avoids.
Part 2: numerical API (btorch.sparse)¶
btorch.sparse is a sparse-array library without neuron semantics. Model
code needs it only to prepare matrices; a SparseConnection accepts SciPy and
PyTorch sparse objects directly.
A Sparse array stores only some of its entries. Three storage formats exist:
COO (one coordinate tuple per entry), CSR (compressed rows) and CSC
(compressed columns). Sparse is the common interface; it is a plain Python
object that holds tensors, not a torch.Tensor subclass.
Create¶
# Entries A[0, 2] = 3 and A[1, 0] = 5 of a 2 x 3 matrix.
A = sparse.from_edges(
rows=torch.tensor([0, 1]),
cols=torch.tensor([2, 0]),
values=torch.tensor([3.0, 5.0]),
shape=(2, 3),
)
print(A) # COO(shape=(2, 3), nnz=2, dtype=torch.float32)
# The same matrix through the format-specific constructors.
A_coo = sparse.coo(torch.tensor([[0, 1], [2, 0]]), torch.tensor([3.0, 5.0]), shape=(2, 3))
A_csr = sparse.csr(
torch.tensor([0, 1, 2]), torch.tensor([2, 0]), torch.tensor([3.0, 5.0]), shape=(2, 3)
)
# From the non-zero entries of a dense tensor (never done implicitly).
A_dense = sparse.from_dense(torch.tensor([[0.0, 0.0, 3.0], [5.0, 0.0, 0.0]]))
nnz is the number of stored entries. Duplicate coordinates are allowed in
COO and are summed by every operation, as in SciPy and PyTorch.
Multiply¶
Products follow standard linear algebra: a matrix of shape (M, N) maps a
length-N vector to a length-M vector.
x = torch.tensor([1.0, 0.0, 2.0])
print(A @ x) # tensor([6., 5.])
# A @ X follows torch.matmul: X is [..., N, K] and the result is [..., M, K].
X = torch.rand(3, 4)
assert (A @ X).shape == (2, 4)
# matvec applies A along the LAST axis of a stack of vectors [..., N].
batch = torch.rand(8, 3)
assert A.matvec(batch).shape == (8, 2)
# rmatvec applies the transpose without building it: [..., M] -> [..., N].
assert A.rmatvec(torch.rand(8, 2)).shape == (8, 3)
v = torch.tensor([1.0, 2.0])
assert torch.equal(v @ A, A.rmatvec(v)) # tensor([10., 0., 3.])
Use matvec for activity tensors such as [time, batch, n_neuron]; use @
when you want torch.matmul shape rules. For a square matrix the two accept
the same input and return different results (see the
warning in Part 1). sparse.matmul,
sparse.matvec and sparse.rmatvec are the function forms; they accept
sparse arrays and linear operators. Products are
differentiable with respect to both the stored values and the dense operand.
The product of two sparse arrays is not implemented.
These products run through a reference implementation (a gather and an
index_add over the stored entries). The kernel backends of
Part 3 are reached only through a SparseConnection.
Convert: tocsr() versus as_csr()¶
tocoo(),tocsr(),tocsc()convert. They may sort entries and sum duplicates, and they returnselfwhen the array already has that format.as_coo(),as_csr(),as_csc()assert. They returnselfif the array is stored in that format and raiseTypeErrorotherwise. They never allocate.
Format-specific fields such as indptr only exist on the concrete class, so
obtain it first with one of the two. coalesce() keeps the format and returns
an array with sorted indices in which duplicate coordinates are summed; it
exists for COO, CSR and CSC.
csr = A.tocsr()
print(csr.indptr, csr.indices, csr.data) # tensor([0, 1, 2]) tensor([2, 0]) tensor([3., 5.])
assert csr.as_csr() is csr
try:
A.as_csr() # A is stored as COO
except TypeError as err:
print(err)
# Transposing CSR gives CSC over the same arrays (no copy).
assert A.tocsr().T.format == "csc"
assert torch.equal(A.to_dense(), A.tocsc().to_dense())
Both naming schemes are available: SciPy names (indptr, indices, data,
row, col) and PyTorch names (crow_indices(), col_indices(),
values()).
SciPy and PyTorch interop¶
sparse.asarray (alias sparse.as_sparse) accepts a btorch Sparse, a
PyTorch sparse tensor or a SciPy sparse array/matrix. Conversions keep the
format where possible and never transpose or densify. A dense tensor is
rejected; use sparse.from_dense.
S = scipy.sparse.random(5, 4, density=0.4, format="csr", random_state=0)
B = sparse.asarray(S) # same as sparse.from_scipy(S)
assert B.format == "csr" and B.shape == (5, 4)
assert sparse.asarray(S, format="coo", dtype=torch.float32).format == "coo"
# Back to SciPy (CPU, detached) and to PyTorch (keeps autograd history).
assert (B.to_scipy() != S).nnz == 0
T = B.to_torch() # torch.sparse_csr tensor
assert T.layout == torch.sparse_csr
assert sparse.from_torch(T).shape == (5, 4)
SciPy data is float64; pass dtype= to cast. to_scipy only accepts
unbatched matrices. to_torch(layout="bsr", blocksize=...) and
to_scipy(format="bsr") export block storage, which is not a native btorch
format.
Two details of PyTorch COO tensors:
- A PyTorch COO tensor has no notion of a batch. A tensor of shape
(G, M, N)is read as an array with three sparse dimensions unlessbatch_dim=1says that it isGmatrices.sparse.from_torchandsparse.as_sparseboth takebatch_dim. - An uncoalesced COO tensor that does not require grad keeps its entries as stored, in their order and with duplicates, so arrays aligned with the entries stay aligned. An uncoalesced tensor that requires grad is coalesced on import, because PyTorch exposes its values to autograd only then.
stacked = torch.sparse_coo_tensor(
torch.tensor([[0, 1], [0, 1], [1, 0]]), torch.tensor([1.0, 2.0]), (2, 2, 2)
)
assert sparse.from_torch(stacked).sparse_dim() == 3
as_batch = sparse.from_torch(stacked, batch_dim=1)
assert as_batch.batch_shape == (2,) and as_batch.sparse_dim() == 2
uncoalesced = torch.sparse_coo_tensor(
torch.tensor([[1, 0, 1], [0, 1, 0]]), torch.tensor([1.0, 2.0, 3.0]), (2, 2)
)
kept = sparse.from_torch(uncoalesced)
assert kept.nnz == 3 and kept.row.tolist() == [1, 0, 1] # order and duplicate kept
Shapes and batches¶
The logical shape of a sparse array is [*batch, *sparse, *dense]:
| Group | Meaning | Accessors |
|---|---|---|
| batch | Independent sparse arrays, for example an ensemble of networks. | batch_shape, batch_dim() |
| sparse | The dimensions stored sparsely (two for a matrix). | sparse_shape, sparse_dim() |
| dense | A dense payload attached to every stored entry. | dense_shape, dense_dim() |
A batch comes in two kinds.
# 1. Shared pattern: one set of indices, values [G, nnz]. Works for COO, CSR, CSC.
shared = B.with_values(torch.rand(4, B.nnz))
assert shared.shape == (4, 5, 4) and shared.batch_shape == (4,)
# 2. Different patterns: sparse.stack builds a batched COO without padding.
S2 = scipy.sparse.random(5, 4, density=0.2, format="csr", random_state=1)
stacked_patterns = sparse.stack([sparse.asarray(S), sparse.asarray(S2)])
assert stacked_patterns.shape == (2, 5, 4) and stacked_patterns.format == "coo"
# Batch dimensions align with the LEADING dimensions of the input of matvec.
x = torch.rand(4, 7, 4, dtype=torch.float64) # [G, samples, N]
assert shared.matvec(x).shape == (4, 7, 5)
# A leading dimension of size 1 broadcasts the same samples to every member.
assert shared.matvec(x[:1]).shape == (4, 7, 5)
Batch members are never combined implicitly with input samples: an input with
fewer leading dimensions than the array has batch dimensions raises a
ValueError. sparse.stack returns a shared-pattern batch when all members
have identical indices.
sparse.einsum (experimental)¶
sparse.einsum(subscripts, A, *operands) contracts one sparse array with dense
tensors. The result equals torch.einsum(subscripts, A.to_dense(), *operands),
but A.to_dense() is never built: time and memory are proportional to the
number of stored entries. It is the tool for arrays with more than two sparse
dimensions, which @ does not handle.
# The 2 x 3 matrix A from above: a batched matrix-vector product.
xb = torch.tensor([[1.0, 0.0, 2.0]])
assert torch.equal(sparse.einsum("mn,bn->bm", A, xb), torch.tensor([[6.0, 5.0]]))
assert sparse.einsum("mn->", A) == 8.0 # no dense operand: sum of the entries
# A third-order sparse array (one row of indices per dimension) contracted
# with two dense tensors.
T3 = sparse.coo(
torch.tensor([[0, 1, 1], [2, 0, 2], [1, 1, 0]]),
torch.tensor([1.0, 2.0, 3.0]),
shape=(2, 3, 2),
)
u3, v3 = torch.rand(4, 3), torch.rand(4, 2)
out3 = sparse.einsum("ijk,bj,bk->bi", T3, u3, v3)
assert torch.allclose(out3, torch.einsum("ijk,bj,bk->bi", T3.to_dense(), u3, v3))
The function is experimental and its interface may change. What is supported:
- Exactly one sparse operand, and it comes first. It may have any format and
any number of batch, sparse and dense dimensions; its subscript has one
letter per dimension of
A.shape. - Any number of dense operands (including none).
- A dense output, in any order of the letters, with or without
->.
What raises instead: an ellipsis (...), a second sparse operand, and a letter
used with two different sizes (size-1 dimensions do not broadcast). A sparse
output is not available. The reasons and the cost model are in
Sparse einsum: scope and what a compiler would add.
Linear operators¶
A linear operator is anything that maps a vector of length N to a vector
of length M linearly. A sparse array is one way to do that. The module
btorch.sparse.operator holds the others: operators that compute the product
from a formula and never store a matrix. They are not Sparse arrays: they
have no nnz, no format and no indices.
| Operator | Matrix | Cost of a product |
|---|---|---|
ConstantOperator((M, N), value) |
every entry equals value (all-to-all) |
O(M + N) |
DiagonalOperator(d) |
diag(d) (one-to-one) |
O(N) |
LowRankOperator(U, V) |
U @ V.T with U [M, r], V [N, r] |
O((M + N) r) |
ImplicitOperator((M, N), matvec=f) |
whatever the callable f computes |
that of f |
All of them derive from LinearOperator and share the product interface of a
sparse array: op.matvec(x) applies the operator along the last axis of
x [..., N], op @ x follows torch.matmul, and op.rmatvec(x) applies the
transpose. The function forms sparse.matvec(op, x), sparse.matmul(op, x)
and sparse.rmatvec(op, x) accept operators as well. Operators combine
lazily, with each other and with sparse arrays: + and - give a sum, @ a
composition, multiplication or division by a number a scaled operator, and
.T the transpose. No matrix is formed by any of these.
from btorch.sparse.operator import (
ConstantOperator,
DiagonalOperator,
ImplicitOperator,
LowRankOperator,
)
ones = ConstantOperator((4, 4), 1.0)
eye = DiagonalOperator(torch.ones(4))
low = LowRankOperator(torch.rand(4, 2), torch.rand(4, 2))
# Only matvec is given: a circular shift.
shift = ImplicitOperator((4, 4), matvec=lambda t: t.roll(1, -1))
# All-to-all without the diagonal, plus a low-rank part and a sparse matrix.
ring_edges = sparse.from_edges(
torch.tensor([0, 1]), torch.tensor([1, 2]), torch.tensor([1.0, 2.0]), (4, 4)
)
op = 0.5 * (ones - eye) + low + ring_edges
dense_op = 0.5 * (torch.ones(4, 4) - torch.eye(4)) + low.to_dense() + ring_edges.to_dense()
xo = torch.rand(8, 4)
assert torch.allclose(op.matvec(xo), xo @ dense_op.T)
assert torch.allclose(sparse.matvec(op, xo), op.matvec(xo))
assert torch.allclose(op @ xo[0], dense_op @ xo[0])
assert torch.allclose((ones @ ring_edges).matvec(xo), xo @ (torch.ones(4, 4) @ ring_edges.to_dense()).T)
Not every operator can do everything. A capability is one of "matvec",
"matmat", "rmatvec", "enumerate_edges" and "materialize_sparse";
op.supports(name) tells whether it is available, and using an unavailable
one raises NotImplementedError naming it. A sum or product supports a
capability only if all of its parts do.
# Entries of a structured operator can be listed on request ...
assert (ones - eye).supports("materialize_sparse")
assert (ones - eye).tocsr().shape == (4, 4)
# ... but a low-rank matrix is dense, and a product would need sparse x sparse.
assert not low.supports("materialize_sparse") and not op.supports("materialize_sparse")
assert op.to_dense().shape == (4, 4) # explicit, never implicit
# An ImplicitOperator has exactly the capabilities whose callable was given.
assert shift.supports("matvec") and not shift.supports("rmatvec")
try:
shift.T
except NotImplementedError as err:
print(err)
Operators as connections¶
In a model an operator is wrapped in a connection, so that it is called like
every other connection (current = conn(spikes), any leading dimensions).
OperatorConnection(op) does that for any object with a shape and a
matvec; StructuredConnection (closed-form operators) and
ImplicitConnection (procedural operators) are the two named kinds, and they
behave identically. HybridConnection([...]) adds the outputs of several
connections between the same two populations.
from btorch.models.connection import HybridConnection, ImplicitConnection
inhibition = StructuredConnection(-0.2 * (ones - eye)) # global, no self-inhibition
ring = ImplicitConnection(shift)
local = SparseConnection(ring_edges)
hybrid = HybridConnection([local, inhibition, ring])
assert (hybrid.n_pre, hybrid.n_post) == (4, 4)
reference = xo @ (ring_edges.to_dense() - 0.2 * (torch.ones(4, 4) - torch.eye(4))).T
assert torch.allclose(hybrid(xo), reference + xo.roll(1, -1), atol=1e-6)
assert torch.allclose(torch.compile(hybrid, fullgraph=True)(xo), hybrid(xo), atol=1e-6)
An operator itself is a plain object, not a module. The connection that wraps
it registers the tensors the operator holds (d, U, V, a tensor value,
and those of the parts of a composite operator): an nn.Parameter becomes a
parameter of the connection, every other tensor a buffer. They therefore
follow connection.to(...), appear in parameters() and are saved in the
state_dict. A learnable operator needs no extra module:
gain = nn.Parameter(torch.tensor(-0.2))
global_inhibition = StructuredConnection(ConstantOperator((4, 4), gain))
assert [name for name, _ in global_inhibition.named_parameters()] == ["operator_0_value"]
assert list(global_inhibition.state_dict()) == ["operator_0_value"]
global_inhibition(xo).sum().backward()
assert gain.grad is not None
Two points to keep in mind:
- Inside compiled code call
op.matvec(x), notop @ x. The Dynamo tracer of PyTorch 2.11 does not dispatch@to a user-defined object, soop @ xandA @ xfail undertorch.compile(..., fullgraph=True). The connections above callmatvec.A.matvec(x)and the function formssparse.matvec/sparse.matmulalso work in compiled code. - Rules can produce operators. A
Projectionwith an all-to-all or one-to-one rule and one fixed weight builds aStructuredConnectionby itself.
Runtime (advanced)¶
btorch.sparse.runtime holds the registered torch.library operators
(csr_propagate, spike_propagate), the kernel backend registry
(registry), the planner and the representation cache. Model code does not
import it. Its public names are registry, use_backend, planner,
Planner, BackendRegistry, KernelCache, RepresentationCache,
propagate, csr_propagate, spike_propagate and pack_spikes; the kernel
modules are internal.
A kernel is a low-level routine such as the CSR product; a backend is one
implementation of it. The runtime picks the highest-priority available
backend. The default backend "aten" uses only PyTorch and is the reference.
If torch_sparse is installed it is registered as an additional,
lower-priority backend for the CSR product, and can be selected for a block of
code:
from btorch.sparse import runtime
print(runtime.registry.available("csr_matvec", "cpu")) # ['aten'] or ['aten', 'torch_sparse']
with runtime.use_backend("aten"):
y = per_edge(torch.rand(5, 6))
use_backend is a process-wide override meant for debugging and benchmarks.
A backend that does not implement a kernel is skipped for that kernel. A name
that no kernel knows raises a ValueError listing the registered backends.
Only a SparseConnection reaches these kernels. A product with a Sparse
array (A @ x, A.matvec(x)) always uses the reference implementation of
btorch.sparse.
Triton backend (CUDA)¶
On a GPU, the kernels have a second implementation written in
Triton, registered as backend
"triton" for the device type "cuda":
| Kernel | Role |
|---|---|
"csr_matvec" |
the destination-driven ("pull") product of the forward pass |
"edge_grad" |
the gradient with respect to the edge weights in the backward pass |
"spike_push_dense" |
the source-driven ("push") product that compacts the spikes on the device |
The backend is available when the triton package can be imported and CUDA is
present. It then has a higher priority than "aten", so it is the default on
CUDA and nothing has to be selected. The Triton kernels handle float32 data
only. For every other dtype they call the ATen kernel themselves, without a
message; conn.explain() still reports triton in that case. "aten"
remains the fallback when Triton is missing, the default on CPU, and the
reference the Triton kernels are tested against.
print(runtime.registry.available("csr_matvec", "cuda")) # best first
if torch.cuda.is_available():
gpu_conn = SparseConnection.from_adjacency(rng_matrix).to("cuda")
print(sparse.explain(gpu_conn)) # "backend = triton" when Triton is installed
xg = torch.rand(5, 6, device="cuda")
y_default = gpu_conn(xg)
with runtime.use_backend("aten"): # force the reference kernels
assert runtime.registry.name("csr_matvec", "cuda") == "aten"
y_aten = gpu_conn(xg)
assert torch.allclose(y_default, y_aten, atol=1e-5)
The connection keeps its prepared route until planning inputs change.
Entering or leaving use_backend invalidates that route, so the next call
replans an existing connection; conn.explain() reports the backend that
would run under the current override.
With the Triton push kernel a density-hinted connection is planned as
"push"; without it (CPU, or CUDA without Triton) as "adaptive-push". The
differences between the two are listed under
Performance hints.
Migration from SparseConn¶
SparseConn, SparseConstrainedConn, BaseSparseConn, SparseBackend,
available_sparse_backends and the sparse_backend= argument were removed
from btorch.models.linear without a compatibility layer. Linear is the
its interface.
| Old | New |
|---|---|
SparseConn(conn, bias=b, enforce_dale=E) (default enforce_dale=True) |
SparseConnection.from_adjacency(conn, Synapse(dale=E), bias=b) (default dale=False) |
sparse_backend=, available_sparse_backends() |
removed; the runtime chooses. Advanced: btorch.sparse.runtime.use_backend(...) |
layer.magnitude / layer.initial_sign / get_sparse_matrix() |
conn.weight.value / conn.weight.sign (now persistent) / conn.to_sparse() (the orientation of the input; conn.to_sparse("pre_post") to be explicit) |
SparseConstrainedConn(conn, constraint, enforce_dale=E) |
SparseConnection.from_adjacency(conn, Synapse(weight=ConstrainedWeight(group=constraint, dale=E))) |
.magnitude / .initial_weight / ._constraint_scatter_indices |
conn.weight.scale / .base / .group |
get_group_info / set_group_magnitude / get_weights_by_group |
conn.weight.group_info / set_scale / weights_by_group |
SparseConstrainedConn.from_hetersynapse(conn, constraint, receptor_idx), constraint_info, persist_initial_weight |
SparseConnection.from_adjacency(...) (expanded layout) or SparseConnection.from_hetersynapse(conn, Synapse(weight=ConstrainedWeight(group=constraint)), receptor_type_index=receptor_idx, n_delay=...) (semantic per-edge receptor/delay); the rest removed |
state_dict keys magnitude, indices |
indices, layout, weight.value (+ weight.sign) or weight.scale/weight.group/weight.base, plus receptor/delay |
Points that need attention when porting:
- Dale's law is opt-in.
SparseConnenforced it by default; a plainSparseConnection.from_adjacency(conn)does not. PassSynapse(dale=True)to keep the old behaviour. - Checkpoints are not compatible. The
state_dictkeys changed (last row of the table).load_state_dictrefuses a checkpoint written by the old layers with a message, also withstrict=False. There is no converter: rebuild the connection from its matrix and copy the weights. torch_sparseis optional. It is no longer required fortorch.compile(..., fullgraph=True).- Receptor index tables are not stored on the connection. Keep the
receptor_type_indexDataFrame returned by the connectome helpers next to the model, asHeterSynapsePSCalready requires.
# Before:
# from btorch.models.linear import SparseConn
# layer = SparseConn(conn_matrix, enforce_dale=True, sparse_backend="native")
# w = layer.get_sparse_matrix()
# After:
layer = SparseConnection.from_adjacency(rng_matrix, Synapse(dale=True))
w = layer.to_sparse().to_scipy() # "pre_post", like rng_matrix
assert w.shape == rng_matrix.shape
See also the design note Sparse and connectivity subsystem.