Qwen3-Next-80B-A3B is a hybrid: most of its layers replace softmax attention with Gated DeltaNet, a linear-attention block, and the rest use full attention with an output gate. Sparsity does the heavy lifting. The feed-forward path is a mixture of experts with 512 experts per layer, of which each token activates 10, which is how an 80B-parameter model runs with roughly 3B active parameters per token. I reimplemented the architecture in a single PyTorch file, no transformers import, no fused kernels, to see exactly what computes.
The config mirrors the official release: 48 layers, hidden size 2048, 16 attention heads over 2 KV heads, head dimension 256, vocabulary 151,936, context length 262,144, RoPE theta 10,000,000. Keeping the real numbers matters. Half the architecture lives in the config.
Zero-centered RMSNorm
Qwen3-Next normalizes with a zero-centered variant of RMSNorm: subtract the mean as LayerNorm would, but divide by the root mean square. It is used everywhere in the model, including on queries and keys before RoPE.
class ZeroCenteredRMSNorm(nn.Module):
def forward(self, x):
mean_x = torch.mean(x, dim=-1, keepdim=True) # B, T, 1
rms_x = torch.sqrt(torch.mean(x**2, dim=-1, keepdim=True)
+ self.config.rms_norm_eps)
x = ((x - mean_x) / rms_x) * self.gamma
return xGated attention, and grouped queries by broadcasting
The full-attention layers add a sigmoid gate computed from the input and applied to the attention output before the final projection. QK-norm runs before RoPE. For grouped-query attention I avoid repeat_interleave entirely: reshape queries to five dimensions so that each group of 8 query heads faces a single broadcast KV head.
g = torch.sigmoid(self.gate(x)) # B, T, C
q, k = self.norm_q(q), self.norm_k(k)
q, k = self.apply_rope(q, cos, sin), self.apply_rope(k, cos, sin)
repeat_factor = num_attention_heads // num_key_value_heads # 16 // 2 = 8
q = q.reshape(B, num_key_value_heads, repeat_factor, T, head_dim)
k = k.reshape(B, num_key_value_heads, 1, T, head_dim)
v = v.reshape(B, num_key_value_heads, 1, T, head_dim)
scores = q @ k.transpose(-1, -2) / math.sqrt(head_dim) # broadcast over dim 2
scores = scores.masked_fill(mask == 0, float('-inf'))
ctx = F.softmax(scores, dim=-1) @ v
ctx = ctx * g
x = self.proj_o(ctx)Condensed from GatedAttention.forward
Gated DeltaNet
The linear-attention blocks project q, k, v through short per-head convolutions (kernel size 4), apply SiLU, and L2-normalize queries and keys. Two learned projections, alpha and beta, gate a delta update computed from the value and the query-key difference; the result is normalized and gated again before the output projection.
q, k = F.silu(q), F.silu(k)
q = F.normalize(q, p=2, dim=-1, eps=1e-6)
k = F.normalize(k, p=2, dim=-1, eps=1e-6)
v = F.silu(v)
alpha = self.proj_alpha(x) # per-token gates
beta = self.proj_beta(x)
delta = alpha * v + beta * (q - k) # the delta update
x = self.norm_delta(delta)
x = g * x
x = self.proj_o(x)Condensed from GatedDeltaNet.forward
This is the static form of the delta rule: the update is computed per position, with the recurrent state accumulation left out. Writing it this way first makes the role of each projection legible before the recurrence complicates the picture.
512 experts, 10 active
The router produces logits over 512 experts, keeps the top 10 per token, and normalizes with a softmax over only those 10, matching the release. Dispatch is a loop over experts with index_add_ scattering weighted expert outputs back to token positions. A vectorized dispatch would be faster; the loop is the version you can read.
gate_logits, gate_idxs = router_logits.topk(num_experts_per_tok, dim=-1)
gate_probs = F.softmax(gate_logits, dim=-1) # over the selected 10 only
y_flat = torch.zeros_like(x_flat)
for expert_id in range(num_experts):
mask = (gate_idxs_flat == expert_id)
if not mask.any():
continue
expert_tokens = token_indices[mask]
expert_weights = gate_probs_flat[mask]
x_selected = x_flat[expert_tokens]
act = F.silu(self.fc1[expert_id](x_selected)) * self.fc2[expert_id](x_selected)
h = self.fc3[expert_id](act)
y_flat.index_add_(0, expert_tokens, expert_weights.unsqueeze(1) * h)Condensed from the MoE forward
What this study version simplifies
The released model interleaves the two block types, roughly three DeltaNet layers for each attention layer; my file runs the 24 DeltaNet blocks and then the 24 gated-attention blocks as two stacks, keeping full_attention_interval in the config for the faithful wiring. The config also carries the shared-expert size, the auxiliary router loss coefficient, and a partial-rotary factor of 0.25 from the release; this implementation applies RoPE to the full head dimension and trains without the auxiliary loss. Each omission is visible as a config field the code does not read, which is a reasonable map of what a minimal reimplementation keeps and what the production model adds.