Does Math Reasoning Improve General LLM Capabilities?

Abstract

Math reasoning has become the poster child of progress in large language models, with new models rapidly surpassing human-level performance on benchmarks like MATH and AIME. But as math leaderboards improve week by week, it is worth asking: do these gains reflect broader problem-solving ability, or just narrow overfitting? We evaluate over 20 open-weight reasoning-tuned models across a broad suite of tasks, including math, scientific QA, agent planning, coding, and standard instruction-following. We surprisingly find that most models that succeed in math fail to transfer their gains to other domains. Through controlled experiments and detailed ablations, we identify on-policy fine-tuning as the key mechanism underlying cross-domain transfer, regardless of whether the training signal comes from RL or supervised learning. Latent-space and token-distribution analyses reveal that off-policy SFT induces substantial representation and output drift, while on-policy RL preserves general-domain structure. Our results suggest a need to rethink post-training recipes, particularly the reliance on off-policy SFT-distilled data to advance reasoning models.

1 Do math gains transfer?

Background. Reasoning-tuned LLMs now surpass human-level scores on MATH and AIME, and math is widely treated as a proxy for reasoning at large.

Question. Do math gains transfer to broader capabilities — or are they narrow overfitting?

Approach. We audit 20+ open-weight reasoning models on 12 benchmarks in 3 task groups (math reasoning: MATH500, AIME24/25, OlympiadBench; other reasoning: GPQA-Diamond, LiveCodeBench, ACPBench, HeadQA; non-reasoning: CoQA, IFEval, HaluEval, MC-TACO), and quantify transfer with a Transferability Index (TI), built up in three steps:

① Per-benchmark gain, z-normalized within each group \[ \Delta R_b \;=\; R^{\text{model}}_b - R^{\text{base}}_b, \qquad \sigma_g \;=\; \operatorname{Std}\!\big\{\Delta R_b : b \in \mathcal{B}_g\big\}, \qquad \delta_b \;=\; \frac{\Delta R_b}{\sigma_g}. \] ② Robust, difficulty-weighted domain index \[ s_b \;=\; \operatorname{sign}(\delta_b)\,|\delta_b|^{1/2}, \qquad \hat{w}_b \;=\; \frac{100 - R^{\text{base}}_b}{\sum_{u \in \mathcal{B}_g} w_u}, \qquad \mathrm{DI}_g \;=\; \sum_{b \in \mathcal{B}_g} \hat{w}_b\, s_b. \] ③ Transfer relative to math \[ \mathrm{TI}_g(\%) \;=\; \frac{\mathrm{DI}_g}{\mathrm{DI}_{\text{math}}}\times 100, \qquad g \in \{\text{other},\ \text{non}\}. \]

A signed square root tempers extreme benchmarks and harder tasks are up-weighted. \(\mathrm{TI}_g \gt 0\) indicates positive transfer; \(\mathrm{TI}_g \lt 0\) indicates forgetting.

Transferability Index across 20+ open reasoning models — **Figure 1.** Transferability of math reasoning across 20+ open models, grouped by base model. Dashed lines are group means.

RL-tuned transfers · SFT-tuned often forgets

Result Transferability splits by training paradigm, not by size or family: RL-tuned models transfer their math gains; SFT-tuned models often show negative TI — catastrophic forgetting of general skills.

2 Controlled study: UniReason

Design. Hold everything fixed except the fine-tuning paradigm:

Base & data: Qwen3-14B-Base; identical math queries (MATH + DeepScaleR, ~47K examples).
SFT (off-policy): imitate CoT traces distilled from Qwen3-32B; rejection sampling keeps only traces with correct answers.
RL (on-policy): GRPO with an answer-correctness reward computed from the same labels.

The resulting models are named UniReason.

Table 1: per-benchmark results of UniReason models — **Table 1.** Per-benchmark results. RL (highlighted rows) beats both SFT variants on all three task groups; SFT collapses on non-reasoning tasks (e.g., HaluEval: 2.3 for SFT-think vs 40.7 for RL; base 35.7).

+24.0

non-reasoning average: RL over the best SFT model

−24.6 vs +7.5

non-reasoning change vs base: SFT (think) forgets, RL improves

+52.2

TI(non) of the RL model, vs −104.1 / −278.9 for SFT

Result Trained on identical math data, only the on-policy RL model preserves — and improves — general capabilities. The same pattern holds for Qwen3-4B and Llama3.1-8B, with more diverse training data, and on additional evaluation benchmarks (Appendix A.1).

3 Why? Latent & token-space diagnostics

Hypothesis. Transfer depends on how much fine-tuning perturbs the base model. We measure this drift in two spaces, on identical inputs before vs after tuning: the latent space (PCA of hidden states across layers) and the output space (KL divergence and token-rank shifts).

Latent drift — distance between representation centers \[ z^{(*)} \;=\; \frac{1}{L}\sum_{i=1}^{L} z^{(*)}_i, \qquad z^{(*)}_i \;=\; \big(\Delta m^{(*)}_{i,1},\; m^{(*)}_{i,2}\big), \qquad d^{(*)} \;=\; \big\lVert z^{(*)} - z^{(\text{orig})} \big\rVert_2. \] Output drift — token-level KL divergence to the backbone \[ \mathrm{KL}\!\big(\pi_\theta \,\Vert\, \pi_{\text{base}}\big) \;=\; \sum_{v \in \mathcal{V}} \pi_\theta(v \mid x_{<t})\, \log\frac{\pi_\theta(v \mid x_{<t})}{\pi_{\text{base}}(v \mid x_{<t})}. \]

\(m^{(*)}_{i,1}, m^{(*)}_{i,2}\) are the PC1/PC2 mean projections of layer-\(i\) hidden states; smaller \(d^{(*)}\) and lower KL both mean the tuned model stays closer to the base.

PCA shift of hidden states across models and tasks — **Figure 2.** Latent space: PCA shift of hidden states (base → tuned). RL (bottom row) shows the smallest centroid drift \(d^{(*)}\) on every task type. Mean shift on non-reasoning inputs: 36.9 (RL) vs 113.7 (SFT no-think).

KL divergence of RL and SFT models to the backbone — **Figure 3.** Output space: KL divergence to the backbone. RL stays close to the base distribution (0.019 vs 0.283 on IFEval; 0.084 vs 0.372 on MATH-500).

Word clouds of rank-shifted tokens for RL and SFT models — **Figure 4.** Rank-shifted tokens: RL (left) moves a few logical-structure tokens (*But*, *Wait*, So); SFT (right) shifts many irrelevant ones. Mean token-rank shift: 0.98 (RL) vs 10.6 (SFT).

4 On-policy updates drive transfer

Approach. Write SFT and RL as the same objective and ablate each lever. Standard supervised fine-tuning maximizes the likelihood of fixed reference completions \(y^\star\):

Supervised fine-tuning \[ \mathcal{L}_{\text{SFT}}(\theta) \;=\; -\,\mathbb{E}_{(x,\,y^\star)\sim\mathcal{D}}\big[\log \pi_\theta(y^\star \mid x)\big]. \] Reinforcement learning — advantage-weighted, on its own samples \[ \mathcal{L}_{\text{RL}}(\theta) \;=\; -\,\mathbb{E}_{x\sim\mathcal{D}}\,\mathbb{E}_{y\sim\pi_\theta(\cdot\mid x)} \big[A(x,y)\,\log \pi_\theta(y \mid x)\big]. \] Unified surrogate — four levers \[ \mathcal{L}_{q,w,\beta}(\theta) \;=\; -\,\mathbb{E}_{x\sim\mathcal{D}}\,\mathbb{E}_{y\sim q(\cdot\mid x)} \big[\,w(x,y)\,\log \pi_\theta(y \mid x)\,\big] \;+\; \beta\,\mathbb{E}_{x\sim\mathcal{D}} \big[\mathrm{KL}\!\big(\pi_\theta(\cdot\mid x)\,\Vert\,\pi_{\text{ref}}(\cdot\mid x)\big)\big]. \]

\(q\): sampling distribution (off- vs on-policy) · \(w\): credit assignment (uniform vs advantage \(A\)) · \(\beta\): KL regularization · plus negative gradients from failed rollouts.

Ablation — Qwen3-8B-Base, same math queries (Table 4)

Setting	Math	Other	Non	TI-Other	TI-Non
Base	27.6	23.6	33.6	–	–
Off-policy SFT	41.9	34.4	26.6	18.3	−40.5
On-policy SFT	33.7	35.7	35.0	68.6	30.2
Off-policy RL	45.5	35.9	31.7	36.4	4.5
On-policy RL (no KL)	37.1	38.2	35.8	65.6	39.3
On-policy RL	38.6	39.9	35.0	63.7	32.4

Key finding The sampling distribution — not the loss family — is what matters. On-policy SFT transfers as well as on-policy RL (TI-Other 68.6 vs 63.7); merely switching SFT from off- to on-policy flips TI-Non from −40.5 to +30.2.

Advantage weighting and negative gradients concentrate updates on task-salient tokens; KL regularization plays only a subtle role.
On-policy methods also show steadier gradient norms — no abrupt early-step spikes (Figure 5).

Takeaways

On-policy fine-tuning — whether RL or SFT — is the key mechanism behind cross-domain transfer.
Math gains ≠ general gains: off-policy SFT-distilled models catastrophically forget non-reasoning skills.
The fine-tuning paradigm — not model size or architecture — predicts transferability.
Post-training recipes should rethink their reliance on massive off-policy SFT-distilled data.

BibTeX

@inproceedings{huan2026transferability,
  title     = {Does Math Reasoning Improve General {LLM} Capabilities?
               Understanding Transferability of {LLM} Reasoning},
  author    = {Huan, Maggie Ziyu and Li, Yuetai and Zheng, Tianyu and Xu, Xiaoyu
               and Kim, Seungone and Du, Minxin and Poovendran, Radha
               and Neubig, Graham and Yue, Xiang},
  booktitle = {Proceedings of the 43rd International Conference on Machine Learning (ICML)},
  year      = {2026}
}