Skip to content

fluct_init — Fluctuation-Driven Weight Initialisation

fluct_init initialises hardware layers close to the fluctuation-driven regime instead of relying on generic ANN schemes such as Xavier or Kaiming. The goal is to place the sub-threshold membrane potential near threshold at initialisation, which increases the number of neurons that contribute surrogate-gradient signal from the first optimisation step.

\[\xi = \frac{\theta - \mu_U}{\sigma_U}\]

Reference: Rossbroich, Gygax & Zenke (2022), Fluctuation-driven initialization for spiking neural network training
DOI: 10.48550/arXiv.2206.10226

Supported hardware: H1v1 and H1v2. All layers in a model must belong to the same family.

Relation to Rossbroich et al.

NWAVE's implementation is derived from Rossbroich et al., not a verbatim port:

  1. The PSP kernel ε(t) is measured numerically from the actual HW layer dynamics, rather than assumed to be exponential (as in the LIF paper). For H1v1 the nonlinear synapse makes a small-probe measurement exact; for H1v2 the linear synapse is exact by design.
  2. NWAVE adds an adaptive binary search on the feed-forward mean weight µ_W to ensure no dead neurons at initialisation. This step is not in the original Rossbroich et al. method.

H1v1/H1v2 Differences

H1v2 weights are weaker than H1v1, causing a lower charge to be added to the membrane of the neuron per unit value. This means that initializations that worked on H1v1 might generate weights out of bounds for H1v2 models. To prevent this we advice to use higher xi_targets and a good number of high firing inputs for each layer!

Import path

from nwavesdk.init.fluct_init import fluct_init

Quick start

H1v1 network (FF)

import torch
import torch.nn as nn
from torch.utils.data import DataLoader, TensorDataset

from nwavesdk.layers import H1v1Synapse, H1v1Layer, prepare_net
from nwavesdk.surrogate import fast_sigmoid
from nwavesdk.init.fluct_init import fluct_init


class TinyH1Net(nn.Module):
    def __init__(self):
        super().__init__()
        self.synapse = H1v1Synapse(nb_inputs=16, nb_outputs=32, device="cpu")
        self.layer   = H1v1Layer(
            n_neurons=32, taus=10e-3, dt=1e-3,
            layer_topology="FF",
            spike_grad=fast_sigmoid(slope=5.0),
            device="cpu",
        )

    def forward(self, x):
        prepare_net(self, collect_metrics=False)
        cur      = self.synapse(x)
        spk, mem = self.layer(cur)
        return spk, mem


x        = (torch.rand(64, 100, 16) < 0.3).float()
train_dl = DataLoader(TensorDataset(x), batch_size=16, shuffle=True)
model    = TinyH1Net()

fluct_init(model, train_dl, xi_target=1.0, alpha=1.0, n_batches=4, verbose=True)

H1v2 network (RC)

from nwavesdk.layers import H1v2Synapse, H1v2Layer, prepare_net
from nwavesdk.init.fluct_init import fluct_init


class TinyH1v2Net(nn.Module):
    def __init__(self):
        super().__init__()
        self.syn = H1v2Synapse(nb_inputs=8, nb_outputs=16, device="cpu")
        self.lyr = H1v2Layer(
            n_neurons=16, taus=10e-3, dt=1e-3,
            layer_topology="RC",
            spike_grad=fast_sigmoid(slope=5.0),
            device="cpu",
        )
        self.layer_pairs = [(self.syn, self.lyr)]   # explicit — recommended for RC nets

    def forward(self, x):
        prepare_net(self, collect_metrics=False)
        cur      = self.syn(x)
        spk, mem = self.lyr(cur)
        return spk, mem


fluct_init(model, train_dl, xi_target=1.0, alpha=0.85, n_batches=4, verbose=True)

fluct_init modifies the model in-place and returns None.

Parameters

Parameter Type Default Description
model nn.Module — Initialised in-place. Must contain at least one (H1v1/H1v2 Synapse, H1v1/H1v2 Layer) dense pair. All pairs must belong to the same hardware family.
dataloader DataLoader or iterable — Input batches (B, T, n_F) or tuples whose first element is that tensor. Used to estimate firing-rate moments and check for dead neurons.
xi_target float 2.0 Target ξ = (θ − µ_U) / σ_U. Recommended range: 1–3. Lower = more fluctuation-driven.
alpha float 1 Fraction of σ_U² budget for FF weights; remaining (1 − alpha) goes to RC weights. Set to 1.0 for FF-only networks.
n_batches int 1 Batches per firing-rate estimate. Raise to 4–8 for noisy datasets.
verbose bool True Print per-layer init summary.

Verbose output

Example for a single FF layer:

[fluct_init] ξ=1.0  α=0.85  dt=1.0ms  (stacked, adaptive µ)  [H1v1]
  Input → ν_mean=392.0Hz  ν_var=392.0Hz  ratio=1.0x
  Layer 1 | ν_in=392.0Hz  µ_W=0.5158  σ_FF=0.0516  µ_U=0.104
[fluct_init] done.

For a frontend-first model:

[fluct_init] ξ=1.0  α=0.85  dt=1.0ms  (stacked, adaptive µ)  [H1v1]
  Frontend → ν_out=145.2Hz  (used as ν_in for layer 1)
  Layer 1 | ν_in=145.2Hz  µ_W=0.3841  σ_FF=0.0384  σ_RC=0.0572  µ_U=0.097
[fluct_init] done.

For multi-layer stacked init, the propagated rate is also printed:

           → ν_2 = 145.3 Hz
Field Meaning
ν_mean Mean input firing rate estimated from data [Hz].
ν_var Second moment used for variance-budget calculation. Equals ν_mean for binary spikes.
ν_in Input firing rate used for this layer (dataloader input or previous layer output).
µ_W Mean FF weight after adaptive search + 10 % safety margin.
σ_FF Std of FF weights. If the FF budget is exhausted the floor 0.1 · µ_W is applied.
σ_RC Std of RC weights (zero-mean). Printed only for recurrent layers.
µ_U Resulting mean membrane potential = n_F · µ_W · ν_in · ε̄.

Warnings

xi_target below 1.0

xi_target < 1.0 triggers a UserWarning. Very small ξ places the mean membrane extremely close to threshold, which can cause very high firing rates and unstable training. Recommended: 1 ≤ ξ ≤ 3.

Single-input layers — exhausted FF variance budget (n_F = 1)

Low fan-in layers may need a large µ_W just to keep all neurons active, exhausting the FF variance budget. fluct_init detects this and applies a symmetry-breaking floor σ_FF = 0.1 · µ_W instead of leaving all FF weights identical (which would stall learning). The warning message calls this init mean-driven and suggests using a smaller ξ to widen the budget.

Hardware weight range

After init, weights are checked against the hardware limits for the detected family: H1v1: [−0.9, 0.9] · H1v2: [−1.66, 1.66]. Out-of-range weights emit a UserWarning and will also be flagged by is_net_deployable(). Add weight_magnitude_loss(model) to your training loss to softly enforce the constraint during training.

Dead neurons after init

If dead neurons remain after the adaptive search and symmetry-breaking floor, fluct_init emits a warning with the layer index and count. Mitigations: smaller xi_target, lower alpha, or more input neurons (increase n_F).

Supported architectures

Feature Supported
H1v1 (H1v1Synapse + H1v1Layer) ✓
H1v2 (H1v2Synapse + H1v2Layer) ✓
Mixed H1v1/H1v2 in one model ✗ — raises ValueError
FF topology ✓
RC topology ✓
Heterogeneous τ per layer ✓
Analog inputs ✓
Binary spike inputs ✓
Multi-layer stacked init ✓ — output ν propagated layer-to-layer
Frontend-first (H1v1/H1v2Frontend → H1v1Layer/H1v2Layer + dense pairs) ✓ — frontend stage skipped, its output ν used as input for first dense pair
Frontend-only (no dense pairs after frontend) ✗ — raises ValueError
LIFSynapse / LIFLayer ✗ — use manual init for LIF networks

Layer pair discovery

fluct_init finds dense (Synapse, Layer) pairs in two ways:

  1. Explicit layer_pairs attribute (recommended for RC nets and complex architectures): python self.layer_pairs = [(self.syn1, self.lyr1), (self.syn2, self.lyr2)]
  2. Auto-discovery: walks model.named_modules() in registration order and collects consecutive (H1v1Synapse|H1v2Synapse, H1v1Layer|H1v2Layer) pairs.

For frontend-first models, expose frontend_stage = (frontend, frontend_layer) for explicit detection, or register the Frontend immediately before its H1v1Layer/H1v2Layer so auto-detection works.