Test-time scaling (TTS)—the dynamic allocation of compute during inference—is a promising direction for improving reasoning in large language models (LLMs). However, a systematic comparison of well-known TTS strategies under identical conditions is missing, and the influence of model type and problem difficulty on performance remains unclear. To address these gaps, we conduct the first large-scale study of TTS, spanning over thirty billion tokens generated using eight open-source LLMs (7B to 235B parameters), across four reasoning datasets. We observe consistent trends: no single TTS strategy universally dominates; reasoning models exhibit distinct trace-quality patterns forming short-horizon and long-horizon categories; and optimal performance scales monotonically with compute budget. Based on these insights, we provide a practical recipe for selecting the best TTS strategy, considering problem difficulty, model type, and compute budget.


Aradhye Agarwal

Ayan Sengupta

Tanmoy Chakraborty

The Art of Scaling Test-Time Compute for Large Language Models

CurDKV is a value-guided KV cache compression method for autoregressive LLMs based on approximated CUR decomposition and leverage scores. Instead of ranking tokens purely by attention scores, CurDKV computes importance from both keys and values to better preserve the attention output softmax(QKᵀ)V, directly targeting eviction loss. Using fast Gaussian random projections to approximate leverage scores, CurDKV remains compatible with FlashAttention and Grouped Query Attention (GQA). On LongBench and Ruler, it achieves up to 9.6% higher accuracy than SnapKV/ChunkKV under aggressive compression ratios (e.g., 90% cache reduction), while reducing generation latency by up to 40% at high compression, offering a practical speed–accuracy tradeoff.


Siddhant Chaudhary

Value-Guided KV Compression for LLMs via Approximated CUR Decomposition

Knowledge distillation (KD) is a key technique for compressing large language models into smaller ones while preserving performance. Despite the recent traction of KD research, its effectiveness for smaller language models (LMs) and the mechanisms driving knowledge transfer remain underexplored. In this work, we present the first large-scale empirical and statistical analysis of KD across models ranging from 0.5B to 7B parameters on 14 complex reasoning tasks in a zero-shot setting. Our findings reveal that KD can improve the average performance of smaller models by up to 10%, with a peak task specific gain of 22%, while providing only marginal benefits (∼ 1.3%) for larger models. Surprisingly, teacher performance has a minimal impact on student outcomes, while teacher task expertise impacts KD effectiveness. A correlation study indicates that smaller LMs benefit more from KD, whereas larger LMs show diminished gains. Additionally, we uncover a misalignment between improvements in student performance and reasoning fidelity, suggesting that while KD enhances accuracy, it does not always maintain the structured decision-making processes of the teacher. Our ablation study further highlights the importance of teacher signals and logit smoothing in influencing students’ performance after distillation. Overall, our study offers a comprehensive empirical and statistical assessment of KD, highlighting both its benefits and trade-offs when distilling knowledge from larger to smaller LMs.

Suhas Kamasetty Ramesh

On the Generalization vs Fidelity Paradox in Knowledge Distillation

MonteCLoRA is a Bayesian variant of LoRA for parameter-efficient fine-tuning of LLMs. It models low-rank adapters as mixtures of Gaussians with Wishart and Dirichlet hyperpriors, and uses Monte Carlo estimation to obtain low-variance, unbiased updates of LoRA parameters. This reduces sensitivity to hyperparameters and stabilizes fine-tuning, improving both robustness and accuracy on NLU and NLG benchmarks with only O(r) extra parameters for rank r.


Vaibhav Seth

Arinjay Pathak

Aastha Verma

Natraj Raman

Sriram Gopalakrishnan

Niladri Chatterjee

Robust and Efficient Fine-tuning of LLMs with Bayesian Reparameterization of Low-Rank Adaptation

Neural scaling laws have revolutionized the design and optimization of large-scale AI models by revealing predictable relationships between model size, dataset volume, and computational resources. Early research established power-law relationships in model performance, leading to compute-optimal scaling strategies. However, recent studies highlighted their limitations across architectures, modalities, and deployment contexts. Sparse models, mixture-of-experts, retrieval-augmented learning, and multimodal models often deviate from traditional scaling patterns. Moreover, scaling behaviors vary across domains such as vision, reinforcement learning, and fine-tuning, underscoring the need for more nuanced approaches. In this survey, we synthesize insights from over 50 studies, examining the theoretical foundations, empirical findings, and practical implications of scaling laws. We also explore key challenges, including data efficiency, inference scaling, and architecture-specific constraints, advocating for adaptive scaling strategies tailored to real-world applications. We suggest that while scaling laws provide a useful guide, they do not always generalize across all architectures and training strategies.


Yash Goel

How to Upscale Neural Networks with Scaling Law? A Survey and Practical Guidelines

This work introduces PruneNet, a calibration-free structured pruning method for large language models that treats pruning as policy learning over intrinsic model properties. A lightweight policy network learns which FFN dimensions to keep by minimizing a spectral distribution shift between original and compressed weight matrices via a KS-distance penalty. PruneNet compresses LLaMA-2-7B in ~15 minutes, preserves 80–95% of zero-shot performance at 20–30% compression, and outperforms SliceGPT and other structured baselines both in accuracy and throughput—without relying on calibration datasets and with minimal benefit from recovery finetuning.

You Only Prune Once: Designing Calibration-Free Model Compression with Policy Learning

This position paper argues that the current obsession with neural scaling laws and ever-larger LLMs is unsustainable and increasingly inefficient. The authors advocate for a systematic theory of *downscaling laws*—principled ways to shrink models, data, and compute while retaining performance. They connect classical scaling laws, carbon footprint models, pruning laws, domain-continual pretraining, and ensemble scaling, and propose a concrete pipeline where a large model is decomposed into specialized small language models whose ensemble can outperform the original model at comparable compute.


Enough of Scaling LLMs! Let’s Focus on Downscaling

Fine-tuning large language models is expensive, and selective PEFT methods try to reduce cost by updating only a small subset of parameters. However, most existing selective approaches rely on a static mask chosen once via a heuristic, which can lock in bad choices and limit performance. This work proposes ID3, a dynamic selective PEFT framework that repeatedly recomputes parameter importance, gradually unmasks parameters over training, and balances exploration and exploitation of the parameter space. ID3 introduces: (i) increment-S, an incremental masking schedule that provably halves the number of effective gradient updates compared to static masking; and (ii) D3, a magnitude-and-gradient-based importance metric that subsumes prior Fisher- and magnitude-based heuristics as special cases. Across 16 tasks spanning GLUE, NER, summarization, and math reasoning, ID3 consistently outperforms static selective PEFT baselines and can even match or surpass full fine-tuning while updating as little as 0.17% of parameters. The method is architecture- and PEFT-agnostic, integrating cleanly with adapters and LoRA, and is released as an open-source selective PEFT toolkit.

Step-by-Step Unmasking for Parameter-Efficient Fine-tuning of Large Language Models

This work proposes MPDistil, a meta-policy knowledge distillation framework that makes teacher–student learning collaborative and competitive instead of one-way. MPDistil introduces a lightweight meta-teacher that optimizes a joint utility over teacher and student, and a curriculum-learning policy that selects tasks to help the student eventually surpass the teacher. On GLUE and SuperGLUE, a 6-layer BERT student distilled with MPDistil can outperform its 12-layer teacher on several tasks, and the method also narrows the performance gap when distilling from very large models like DeBERTa-v2-xxlarge to smaller students.

Shantanu Dixit

Md. Shad Akhtar

A Good Learner can Teach Better: Teacher-Student Collaborative Knowledge Distillation

This work proposes TransJect, an encoder-only Transformer with enforced injectivity and Lipschitz continuity. By combining orthogonal attention, injective residual connections, and mixture-of-experts, TransJect preserves pairwise distances between token representations across layers, achieves lower entropy, and improves accuracy on both short- and long-sequence benchmarks while also reducing perplexity for language modeling.

Publications

ArXiv Preprint

The Art of Scaling Test-Time Compute for Large Language Models

NeurIPS 2025

Value-Guided KV Compression for LLMs via Approximated CUR Decomposition

ACL 2025

On the Generalization vs Fidelity Paradox in Knowledge Distillation

TMLR 2025

Robust and Efficient Fine-tuning of LLMs with Bayesian Reparameterization of Low-Rank Adaptation

TMLR 2025

How to Upscale Neural Networks with Scaling Law? A Survey and Practical Guidelines

ICLR 2025

You Only Prune Once: Designing Calibration-Free Model Compression with Policy Learning

ICML 2025

Enough of Scaling LLMs! Let’s Focus on Downscaling

TACL 2025

Step-by-Step Unmasking for Parameter-Efficient Fine-tuning of Large Language Models

ICLR 2024

A Good Learner can Teach Better: Teacher-Student Collaborative Knowledge Distillation

EMNLP 2023

Manifold-Preserving Transformers are Effective for Short-Long Range Encoding