Ali Janati.
Research in deep learning.
I study how neural networks learn, organize knowledge, and generalize.
My research spans model architecture, optimization, training dynamics, and interpretability. I am especially interested in how a model’s internal structure gives rise to its capabilities, how those capabilities evolve during training, and how understanding these relationships can lead to better architectures and training methods, and to safer and better-aligned models.
I am a Senior Machine Learning Researcher at a startup in San Francisco, where I work on world models. I am an alumnus of Columbia and Mines Paris.
On this website, I share my papers, implementations, experimental studies, and articles, along with thoughts and ideas across my research and broader interests.
Recent
- Oct 2026 New Faynt paper and article on training compact Transformer policies to play all 26 characters in competitive Melee, from human replays through distillation and reinforcement learning.
- Oct 2026 Our paper on router sensitivity for expert pruning was accepted to NeurIPS 2026 - AXIOM Workshop. The revised paper is now available, with experiments on lightweight adaptation and the accuracy-compression tradeoff.
- Sep 2026 New article on a report I worked on: decoder-only Transformer optimization 101, from 163,645 to 3,217,492 tokens/s on one RTX 6000 Pro to teach a policy to play a game.
- Aug 2026 New articles: from-scratch reimplementations of Qwen3-Next-80B-A3B, gpt-oss-20B, Gemma 3 270M, and Llama 3.1 8B, plus speculative decoding across tokenizers.
- Aug 2026 New paper on arXiv: router sensitivity under lightweight fine-tuning identifies prunable experts in mixture-of-experts models.
- Aug 2026 New paper on arXiv: post-grokking collapse at the representation-readout interface in Muon-trained transformers.
- Feb 2026 New paper on arXiv: uncertainty-aware multimodal emotion recognition through Dirichlet parameterization.
- Jul 2024 Released Medical-Whisper-Large-v3 on Hugging Face: 35,000+ downloads.
§ 01
Papers
Papers.
-
Faynt: Scaling and Optimizing Policies for Competitive Melee
arXiv · Oct 2026
A family of 10M- and 75M-parameter Transformer policies that play all 26 characters in Super Smash Bros. Melee. We study scaling, optimization, curricula, distillation, and reinforcement learning, with open model weights, benchmark suites, and tournament tools.
-
Router Sensitivity Under Lightweight Fine-Tuning Identifies Prunable Experts in Mixture-of-Experts Models
NeurIPS 2026 · AXIOM WorkshopAccepted · Revised Sep 2026
Brief router-only fine-tuning reveals which experts to prune in mixture-of-experts models. We study how adapter placement shapes this signal and the resulting tradeoffs in accuracy, memory, and inference speed.
-
Post-Grokking Collapse at the Representation-Readout Interface in Muon-Trained Transformers
arXiv · Aug 2026
Muon reaches grokking faster than AdamW, but generalization later collapses at the interface between representations and readout. Fourier analysis shows the task-solving circuit itself remains accurate.
-
Uncertainty-Aware Multimodal Emotion Recognition through Dirichlet Parameterization
arXiv · Feb 2026
A lightweight, privacy-preserving emotion recognition framework for edge devices, fusing speech, text, and facial signals with Dirichlet-parameterized uncertainty. Validated on five benchmarks.
-
Improving Allele-Specific Epigenomic Signal Coverage by 10-Fold Using Hidden Markov Modeling and Machine Learning
bioRxiv · May 2024
A variational hidden Markov model that extends allele-specific methylation calling to ten times more of the genome than standard short-read methods.
Earlier research, with Prof. Pierre Gentine’s group at Columbia: fine-tuned the IBM-NASA geospatial vision transformer to segment wind-damaged forest in satellite imagery.
§ 02
Articles
Articles.
Write-ups on deep learning: architectures from scratch, training methods, and inference optimization, among other things.
-
Oct 2026
Faynt: learning competitive Melee
A walk through our Faynt research: learning from human replays, refining a policy with winning demonstrations and distillation, improving it through self-play, and fitting inference inside a game frame.
-
Sep 2026
Decoder-only Transformer optimization 101: 164K to 3.2M tokens/s on one GPU
A walk through a report I worked on: GEMM tiling and split-K, fusion and FlashAttention, buffer reuse, CUDA-Oxide kernels on two streams, and a smaller model that still beats the level-9 CPU.
-
Aug 2026
Qwen3-Next-80B-A3B from scratch: DeltaNet, gated attention, and 512 experts
The full hybrid stack in one PyTorch file: linear-attention DeltaNet blocks, gated attention with QK-norm, zero-centered RMSNorm, and a 512-expert MoE with top-10 routing.
-
Aug 2026
gpt-oss-20B: the attention pattern and the MoE, reimplemented
Alternating sliding-window and full attention as two masks, RoPE in the rotation-matrix convention, and a 32-expert MoE with gpt-oss's clamped SwiGLU.
-
Aug 2026
Gemma 3 270M, line by line
Dual RoPE frequencies, QK-norm, sandwich normalization, GeGLU, and fp32 attention accumulation in a model small enough to read whole.
-
Aug 2026
Llama 3.1 8B with a hand-written KV cache
Grouped-query attention by broadcasting instead of repeat_kv, RoPE with a position offset, and the twenty lines of caching behind fast decoding.
-
Aug 2026
Speculative decoding from scratch, including across tokenizers
The accept-reject rule and batched prefix acceptance, then Universal Assisted Generation between models that do not share a tokenizer.
§ 03
Open source
Open source.
-
From-scratch PyTorch reimplementations of modern architectures: Llama 3.1 8B, GPT-2, gpt-oss-20B, Qwen3-80B-A3B, Gemma 270M, and Magistral Small, plus speculative decoding and quantization-aware training.
-
Whisper large-v3 fine-tuned for medical conversation: 14% lower word-error rate on validation, released as open weights. 35,000+ downloads on Hugging Face.
-
Released Mixtral checkpoints from our expert-pruning experiments, including router adaptation, all-module adaptation, and three pruning families. The collection contains 14 checkpoints plus the original Mixtral model. See the configurations below and the router-sensitivity paper for the experimental results.
Checkpoint configurations
Browse 14 checkpoints grouped by adaptation and pruning setup. Each link gives the number of experts retained per layer; all checkpoints route each token to two experts.
Family Released configuration Checkpoints Gate-only LoRA Rank 4; 5,000 training steps 8 experts All-module LoRA Rank 4; 10,000 training steps 8 experts Random pruning 1 to 4 experts removed per layer 7 experts · 6 experts · 5 experts · 4 experts L2 pruning after gate LoRA Repository label: L2 norm after gate adaptation 7 experts · 6 experts · 5 experts · 4 experts Pruning after all-module LoRA Repository label: exhaustive LoRA + pruning 7 experts · 6 experts · 5 experts · 4 experts The table lists the released checkpoint configurations from their model cards and repository labels. The paper's main router-only experiment uses rank 8 and 500 steps.
-
The experiment suite behind the post-grokking-collapse paper: Muon and AdamW sweeps on modular arithmetic, freezing and unembedding-rate interventions at the representation-readout interface, and a generality suite across moduli, widths, and training configurations.
-
A CycleGAN that transfers Claude Monet’s painting style to photographs.