Jiaqi Leng | Ph.D. Student, UC San Diego

Hi there! I am Jiaqi Leng, a first-year Ph.D. student at the Halıcıoğlu Data Science Institute, UC San Diego, fortunately supervised by Hao Zhang in the Hao AI Lab. Previously, I earned my B.S. in Computer Science from Fudan University.

My research focuses on building efficient and scalable models through machine learning systems and model architecture design. I am particularly interested in the algorithm side of machine learning systems, including model architectures. I have previously worked on sparse attention.

I welcome opportunities for academic collaboration and discussion. Feel free to drop me an email!

News

Nov 2026	We released Looping Deeper Without Growing the KV Cache, which loops causal encoder–decoders for deeper computation at no extra KV cache cost.
Sep 2026	I launched Awesome RSI Work, a curated reading desk for recursive self-improvement, specialized inference systems, kernels, and research agents. More on the Projects page.
Apr 2026	I presented our work on length-generalizable sparse attention at ICLR 2026. [Poster]

Selected Publications

Looping Deeper Without Growing the KV Cache: Recurrent Depth for Causal Encoder-Decoders

Jiaqi Leng, Minshen Zhang, Zhaofeng Sun, Yu-Yang Qian, Peng Zhao, and Hao Zhang

arXiv preprint, 2026

Abs arXiv
We design a looped causal encoder–decoder (CED) that gains recurrent depth with zero round-wise KV cache growth.

Key findings:

recurrence belongs in the encoder, iteratively refining global KV rather than re-reading static decoder projections

training under stochastic recurrence depth yields a fixed point where every round losslessly shares the last round’s KV cache

Result: matches an unlooped CED’s minimal memory footprint and recovers 86–93% of a looped Transformer’s gain with 1/6 of its global KV

At scale: looping the 552B-parameter DeepSeek-V4.1-Flash for 8 rounds on one 4×GB200 node holds 36.8× less KV per token, fits 38 128K-token requests, and serves 3.6× higher throughput on 8K-token prompts.
Understanding and Improving Length Generalization in Hierarchical Sparse Attention Models

Jiaqi Leng^*, Xiang Hu^*, Junxiong Wang, Jianguo Li, Wei Wu, and Yucheng Lu

In Proceedings of the 14th International Conference on Learning Representations (ICLR), 2026

Abs arXiv Code Poster
We analyze chunk-based sparse attention for long-context generalization and identify its essential ingredients.

Key principles:

non-linear Chunk Encoder with a dedicated CLS token for retrieval

Bypassing Residual Path to integrate global information

enforced selection sparsity in pre-training to close the train–test gap

Evidence: theoretical motivation and unified ablations

Result: state-of-the-art training-free length extrapolation, generalizing 4K-trained models to 32M tokens on RULER and BABILong.
Hardware-aligned Hierarchical Sparse Attention for Efficient Long-term Memory Access

Xiang Hu, Jiaqi Leng, Jun Zhao, Kewei Tu, and Wei Wu

In Proceedings of the 39th Conference on Neural Information Processing Systems (NeurIPS), 2025

Abs arXiv Code
We introduce Hierarchical Sparse Attention (HSA) to give RNNs efficient long-range random access without losing linear complexity.

Method: chunk inputs, select top-k chunks, and hierarchically aggregate using token-level relevance; a hardware-aligned kernel keeps it efficient

System: Mamba + HSA (RAMba)

Result: perfect passkey retrieval up to 64M tokens from 4K pre-training, plus strong downstream gains with near-constant memory.