Skip to content

Initializing Your Network

How you initialize a network's weights affects how quickly it converges during training. This page covers three approaches available in the nwave SDK:

  • Default init: PyTorch initializers configured per-layer at construction time
  • fluct_init: Automatic, dataset-driven initializer — sets synapse weights so the membrane variance is controlled near threshold from the very first batch
  • frontend_firing_init: Automatic, dataset-driven — finds frontend weights that achieve a target firing rate, preventing dead or always-firing frontend neurons
  • Custom init: Any torch.nn.init callable, applied per-layer via init_weights

For full parameter reference, verbose output format, and architecture constraints see Reference → Weight Initialisation → Automatic Initialization.

For the effect of initialization on training dynamics and accuracy, see Tutorial 5.

import torch
import torch.nn as nn
from nwavesdk.layers import H1v1Frontend, H1v1Synapse, H1v1Layer, prepare_net
from nwavesdk.init import fluct_init, frontend_firing_init
from nwavesdk.init.hardware import init_weights
from nwavesdk.data import NWaveDataGen, NWaveDataloaderConfig

Model and dataset used in this page

We use a small H1v1 network throughout — one frontend stage (Frontend → Layer) plus two downstream synapse-layer pairs.

DEVICE = "cpu"
N_CHANNELS = 13   # matches the yes/no audio dataset
HIDDEN = 32
DT = 8e-3         # 8 ms

class SmallH1v1Net(nn.Module):
    def __init__(self):
        super().__init__()
        self.frontend = H1v1Frontend(N_CHANNELS, device=DEVICE)
        self.l1 = H1v1Layer(N_CHANNELS, taus=10e-3, dt=DT, device=DEVICE)
        self.s2 = H1v1Synapse(N_CHANNELS, HIDDEN, device=DEVICE)
        self.l2 = H1v1Layer(HIDDEN, taus=10e-3, dt=DT, device=DEVICE)
        self.s3 = H1v1Synapse(HIDDEN, 4, device=DEVICE)
        self.l3 = H1v1Layer(4, taus=10e-3, dt=DT, device=DEVICE)

    def forward(self, x):
        prepare_net(self)
        mem1_list, mem2_list, mem3_list = [], [], []
        spk1_list, spk2_list, spk3_list = [], [], []

        for t in range(x.shape[1]):
            q0 = self.frontend(x[:, t, :])
            s1, m1 = self.l1(q0)
            q2 = self.s2(s1)
            s2, m2 = self.l2(q2)
            q3 = self.s3(s2)
            s3, m3 = self.l3(q3)
            spk1_list.append(s1); mem1_list.append(m1)
            spk2_list.append(s2); mem2_list.append(m2)
            spk3_list.append(s3); mem3_list.append(m3)

        spikes   = [torch.stack(spk1_list, 1), torch.stack(spk2_list, 1), torch.stack(spk3_list, 1)]
        membranes = [torch.stack(mem1_list, 1), torch.stack(mem2_list, 1), torch.stack(mem3_list, 1)]
        return spikes, membranes

print("SmallH1v1Net defined.")

SmallH1v1Net defined.

Dataset-driven initializers

fluct_init and frontend_firing_init are automatic — they read a few batches from your training DataLoader to calibrate weights. You do not need to compute statistics by hand.

Load the yes/no dataset first:

DATA_ROOT = "../data_for_nwave_commands"

data_config = NWaveDataloaderConfig(
    batch_size=16,
    val_split=0.2,
    test_split=0.0,
    shuffle_train=True,
    random_state=42,
    num_workers=2,
)

dm = NWaveDataGen(
    data_parent="../data_for_nwave_commands",  # one subfolder per class
    sample_rate=16000,
    recording_duration_s=1.0,    # pad/trim all clips to 1 second
    sim_time_s=8e-3,             # 8 ms time bins → 125 timesteps for a 1-second clip
    dataloader_config=data_config,
    task="classification",
    return_filename=False,
)

loaders = dm.dataloaders()
train_loader = loaders["train"]
val_loader   = loaders["val"]

print("DataLoaders ready.")

2026-05-20 09:22:03,553 - root - WARNING - Using 13 valid freqs out of 16 for sr=16000Hz (Nyquist=8000.0Hz).



Classes (loading wavs):   0%|          | 0/2 [00:00<?, ?it/s]



Loading no:   0%|          | 0/3536 [00:00<?, ?it/s]



Loading yes:   0%|          | 0/3625 [00:00<?, ?it/s]



Filtering no:   0%|          | 0/3536 [00:00<?, ?it/s]



Filtering yes:   0%|          | 0/3625 [00:00<?, ?it/s]


DataLoaders ready.

frontend_firing_init

Binary-searches frontend weights until the frontend layer fires at a target rate. target_fr=0.3 means 30 % of timesteps should produce a spike.

Both fluct_init and frontend_firing_init are quantization-aware: they work correctly when layers are built with quantization_bit set. Always apply frontend_firing_init before fluct_init so the frontend operating point is fixed before fluct_init propagates firing rates through the dense layers.

frontend_firing_init(model, train_loader, target_fr=0.3, n_batches=4)

model_auto = SmallH1v1Net()

frontend_firing_init(
    model_auto,
    train_loader,
    target_fr=0.3,
    n_batches=4,
    verbose=True,
)

[frontend_firing_init] target_fr=30.0%  n_batches=4  epsilon=2.0%  n_filters=13  [H1V1]


/tmp/ipykernel_58437/2650779186.py:9: UserWarning: Frontend on chip uses 16 filters. Using a different amount of neurons 13 is allowed but not respecting the chip constraints.
  self.frontend = H1v1Frontend(N_CHANNELS, device=DEVICE)
/opt/conda/envs/PyTorch/lib/python3.10/site-packages/IPython/core/interactiveshell.py:3579: UserWarning: H1Layer: dt/taus ratio 0.800 exceeds the BPTT stability limit 0.412 (J at mem=0 = -2.89, need |J| < 1). With dt=8.0 ms the minimum safe taus is 19.4 ms. Gradients can overflow to NaN during BPTT. Increase taus or use a NaN guard (clip_grad_norm + skip step if grad_norm is not finite).
  exec(code_obj, self.user_global_ns, self.user_ns)


  neuron | w         fr(cont)
  neuron  0 | w=0.0773  fr=0.300  [OK  ]
  neuron  1 | w=0.0741  fr=0.300  [OK  ]
  neuron  2 | w=0.0726  fr=0.300  [OK  ]
  neuron  3 | w=0.0726  fr=0.300  [OK  ]
  neuron  4 | w=0.0726  fr=0.300  [OK  ]
  neuron  5 | w=0.0733  fr=0.300  [OK  ]
  neuron  6 | w=0.0749  fr=0.300  [OK  ]
  neuron  7 | w=0.0776  fr=0.300  [OK  ]
  neuron  8 | w=0.0794  fr=0.300  [OK  ]
  neuron  9 | w=0.0807  fr=0.300  [OK  ]
  neuron 10 | w=0.0809  fr=0.300  [OK  ]
  neuron 11 | w=0.0814  fr=0.300  [OK  ]
  neuron 12 | w=0.0813  fr=0.300  [OK  ]
[frontend_firing_init] done.

fluct_init

Sets dense synapse weights so sub-threshold membrane variance is controlled and the mean sits near threshold — maximising surrogate-gradient signal from the first batch.

xi_target controls the fluctuation-to-threshold ratio; alpha scales the mean.

Note: fluct_init skips the Frontend → Layer stage because frontend connectivity is 1-to-1 — weight variance cannot be tuned with a single input per neuron. It initialises only the downstream Synapse → Layer pairs.

Architecture constraint: No non-hardware modules (Dropout, BatchNorm, etc.) may sit before or between hardware stages at any level of the module tree. For parallel or non-sequential architectures, set model.layer_pairs explicitly as a list of (Synapse, Layer) tuples. See Reference → Weight Initialisation → Automatic Initialization for details.

fluct_init(
    model_auto,
    train_loader,
    xi_target=3.0,
    alpha=1.0,
    n_batches=4,
    verbose=True,
)

[fluct_init] ξ=3.0  α=1.0  dt=8.0ms  (stacked, adaptive µ)  [H1V1]
  Frontend stage skipped — nu_out=38.8Hz used as nu_in for layer 1
  Layer 1 | ν_in=38.8Hz  µ_W=0.0766  σ_FF=0.0444  µ_U=0.075
           → nu_2 = 26.0 Hz
  Layer 2 | ν_in=26.0Hz  µ_W=0.0238  σ_FF=0.0851  µ_U=0.038
[fluct_init] done.


/opt/conda/envs/PyTorch/lib/python3.10/site-packages/IPython/core/interactiveshell.py:3336: UserWarning: fluct_init layer 2: 2/4 neurons are dead after init. The fluctuation-driven regime (σ_FF > 0) requires µ_W ≤ 0.0585, but avoiding dead neurons needs µ_W > 0.0216. Consider a smaller ξ, lower α, or more input neurons (n_F=32).
  has_raised = await self.run_ast_nodes(code_ast.body, cell_name,

Custom initialization

Use init_weights to apply any torch.nn.init callable to a specific layer. Pass init=(fn, kwargs_dict) — the SDK maps the call through the hardware quantization chain so the final stored weights correspond correctly to the intended distribution.

model_custom = SmallH1v1Net()

# Xavier uniform on the first synapse
init_weights(model_custom.s2, init=(nn.init.xavier_uniform_, {"gain": 1.0}))

# Small normal distribution on the second synapse
init_weights(model_custom.s3, init=(nn.init.normal_, {"mean": 0.0, "std": 0.05}))

print("Custom init applied:")
print(f"  s2 weight — mean: {model_custom.s2.weight.mean():.4f}, std: {model_custom.s2.weight.std():.4f}")
print(f"  s3 weight — mean: {model_custom.s3.weight.mean():.4f}, std: {model_custom.s3.weight.std():.4f}")

Custom init applied:
  s2 weight — mean: 0.0082, std: 0.1944
  s3 weight — mean: -0.0021, std: 0.0459


/tmp/ipykernel_58437/2650779186.py:9: UserWarning: Frontend on chip uses 16 filters. Using a different amount of neurons 13 is allowed but not respecting the chip constraints.
  self.frontend = H1v1Frontend(N_CHANNELS, device=DEVICE)

Initializer at construction time

You can also pass init directly when constructing a Synapse or Frontend layer. This is equivalent to calling init_weights immediately after construction.

model_constructed = SmallH1v1Net.__new__(SmallH1v1Net)
nn.Module.__init__(model_constructed)
model_constructed.frontend = H1v1Frontend(N_CHANNELS, device=DEVICE)
model_constructed.l1 = H1v1Layer(N_CHANNELS, taus=10e-3, dt=DT, device=DEVICE)
model_constructed.s2 = H1v1Synapse(
    N_CHANNELS, HIDDEN, device=DEVICE,
    init=nn.init.xavier_normal_,           # passed at construction
)
model_constructed.l2 = H1v1Layer(HIDDEN, taus=10e-3, dt=DT, device=DEVICE)
model_constructed.s3 = H1v1Synapse(
    HIDDEN, 4, device=DEVICE,
    init=(nn.init.normal_, {"mean": 0.0, "std": 0.03}),  # fn + kwargs
)
model_constructed.l3 = H1v1Layer(4, taus=10e-3, dt=DT, device=DEVICE)

print("Model with construction-time init built.")
print(f"  s2 weight std: {model_constructed.s2.weight.std():.4f}")

Model with construction-time init built.
  s2 weight std: 0.2020


/tmp/ipykernel_58437/3547693083.py:3: UserWarning: Frontend on chip uses 16 filters. Using a different amount of neurons 13 is allowed but not respecting the chip constraints.
  model_constructed.frontend = H1v1Frontend(N_CHANNELS, device=DEVICE)
/opt/conda/envs/PyTorch/lib/python3.10/site-packages/IPython/core/interactiveshell.py:3519: UserWarning: H1Layer: dt/taus ratio 0.800 exceeds the BPTT stability limit 0.412 (J at mem=0 = -2.89, need |J| < 1). With dt=8.0 ms the minimum safe taus is 19.4 ms. Gradients can overflow to NaN during BPTT. Increase taus or use a NaN guard (clip_grad_norm + skip step if grad_norm is not finite).
  if await self.run_code(code, result, async_=asy):

Further reading