Ali Janati.
Research in deep learning.
Efficient training and inference for large models, real-time speech-to-speech systems, and interpretability: measuring which parts of a model are doing the work, and using that to make models smaller and faster.
I am a senior machine learning scientist at a startup in San Francisco, where I work on world models and speech. I studied at Columbia and Mines Paris.
Recent
- Aug 2026 New articles: from-scratch reimplementations of Qwen3-Next-80B-A3B, gpt-oss-20B, Gemma 3 270M, and Llama 3.1 8B, plus speculative decoding across tokenizers.
- Aug 2026 New paper on arXiv: router sensitivity under lightweight fine-tuning identifies prunable experts in mixture-of-experts models.
- Aug 2026 New paper on arXiv: post-grokking collapse at the representation-readout interface in Muon-trained transformers.
- Feb 2026 New paper on arXiv: uncertainty-aware multimodal emotion recognition through Dirichlet parameterization.
- Jul 2024 Released Medical-Whisper-Large-v3 on Hugging Face: 35,000+ downloads.
§ 01
Papers
Papers.
Most of my work asks which parts of a large model are actually doing the work. Recent papers span expert pruning in mixture-of-experts models, training dynamics after grokking, uncertainty in multimodal recognition, and machine learning for epigenomics.
-
Router Sensitivity Under Lightweight Fine-Tuning Identifies Prunable Experts in Mixture-of-Experts Models
arXiv · Aug 2026
Routers that barely move under light fine-tuning point to experts that can be pruned with minimal loss in quality.
-
Post-Grokking Collapse at the Representation-Readout Interface in Muon-Trained Transformers
arXiv · Aug 2026
Muon reaches grokking faster than AdamW, but generalization later collapses at the interface between representations and readout. Fourier analysis shows the task-solving circuit itself remains accurate.
-
Uncertainty-Aware Multimodal Emotion Recognition through Dirichlet Parameterization
arXiv · Feb 2026
A lightweight, privacy-preserving emotion recognition framework for edge devices, fusing speech, text, and facial signals with Dirichlet-parameterized uncertainty. Validated on five benchmarks.
-
Improving Allele-Specific Epigenomic Signal Coverage by 10-Fold Using Hidden Markov Modeling and Machine Learning
bioRxiv · May 2024
A variational hidden Markov model that extends allele-specific methylation calling to ten times more of the genome than standard short-read methods.
Earlier research, with Prof. Pierre Gentine’s group at Columbia: fine-tuned the IBM-NASA geospatial vision transformer to segment wind-damaged forest in satellite imagery.
§ 02
Articles
Articles.
Write-ups of the architectures I reimplement from scratch in PyTorch-Playground: what is new in each model, how I implemented it, and the code that matters.
-
Aug 2026
Qwen3-Next-80B-A3B from scratch: DeltaNet, gated attention, and 512 experts
The full hybrid stack in one PyTorch file: linear-attention DeltaNet blocks, gated attention with QK-norm, zero-centered RMSNorm, and a 512-expert MoE with top-10 routing.
-
Aug 2026
gpt-oss-20B: the attention pattern and the MoE, reimplemented
Alternating sliding-window and full attention as two masks, RoPE in the rotation-matrix convention, and a 32-expert MoE with gpt-oss's clamped SwiGLU.
-
Aug 2026
Gemma 3 270M, line by line
Dual RoPE frequencies, QK-norm, sandwich normalization, GeGLU, and fp32 attention accumulation in a model small enough to read whole.
-
Aug 2026
Llama 3.1 8B with a hand-written KV cache
Grouped-query attention by broadcasting instead of repeat_kv, RoPE with a position offset, and the twenty lines of caching behind fast decoding.
-
Aug 2026
Speculative decoding from scratch, including across tokenizers
The accept-reject rule and batched prefix acceptance, then Universal Assisted Generation between models that do not share a tokenizer.
§ 03
Open source
Open source.
-
PyTorch-Playground
From-scratch PyTorch reimplementations of modern architectures: Llama 3.1 8B, GPT-2, gpt-oss-20B, Qwen3-80B-A3B, Gemma 270M, and Magistral Small, plus speculative decoding and quantization-aware training.
-
Medical-Whisper-Large-v3
Whisper large-v3 fine-tuned for medical conversation: 14% lower word-error rate on validation, released as open weights. 35,000+ downloads on Hugging Face.
-
MoE pruning models
Open artifacts behind the router-sensitivity paper: Mixtral-8x7B and Qwen1.5-MoE variants with experts pruned and ablated, along with the SFT datasets used to recover performance. The Mixtral series is curated as a collection; the full set lives on the profile.
-
muon-grokking
The experiment suite behind the post-grokking-collapse paper: Muon and AdamW sweeps on modular arithmetic, freezing and unembedding-rate interventions at the representation-readout interface, and a generality suite across moduli, widths, and training configurations.
-
GAN_Monet_Style
A CycleGAN that transfers Claude Monet’s painting style to photographs.