SEAL: Steerable Reasoning Calibration of Large Language Models for Free
TL;DR Highlight
Fixing reasoning models like DeepSeek-R1 that waste tokens on unnecessary 'double-checking' and 'direction switching' by steering their latent space — boosting accuracy while cutting tokens.
Who Should Read
ML engineers deploying reasoning models like DeepSeek-R1 or QwQ in production who want to reduce token costs and response latency. Developers seeking inference quality improvements without model fine-tuning.
Core Mechanics
- CoT reasoning processes classified into 3 types: 'execution' (step-by-step problem solving), 'reflection' (reviewing previous steps), and 'transition' (switching directions)
- Wrong answers have overwhelmingly more reflection/transition tokens — correct samples average 1,534 tokens vs. incorrect samples averaging much more
- Steering vectors extracted from the latent space can suppress unproductive reflection/transition while preserving useful execution steps
- This improves both accuracy AND reduces token count simultaneously — a rare win-win
Evidence
- Math-500 with DeepSeek-R1-Distill-Qwen-1.5B: accuracy 67.0% → 78.0% (+11%), tokens 4,526 → 3,154 (-30.3%)
- Math-500 Hard with DeepSeek-R1-Distill-Qwen-1.5B: accuracy 54.2% → 68.3% (+14.1%), tokens -28.8%
- Consistent improvements across LiveCodeBench and other benchmarks as well
How to Apply
- If using DeepSeek-R1-Distill or QwQ-32B-Preview: grab the code from VITA-Group/SEAL on GitHub, extract steering vectors offline using 1,000 Math-500 training samples, then add them to hidden states at layer 20 (1.5B/7B) or 55 (32B) with α=1.0 during inference.
- No fine-tuning required — this is a pure inference-time intervention that can be applied to any compatible reasoning model.
Code Example
# SEAL Core Logic Summary (pseudo-code)
# Step 1: Offline - extract steering vector
def extract_steering_vector(model, train_samples, target_layer=20):
exec_reps, refl_trans_reps = [], []
for sample in train_samples: # ~1000 samples
output = model.generate(sample)
thoughts = output.split('\n\n') # split thoughts by double newline
for thought in thoughts:
hidden = model.get_hidden_state(thought, layer=target_layer) # representation of '\n\n' token
# keyword-based classification
if thought.startswith('Wait') or 'let me check' in thought.lower():
refl_trans_reps.append(hidden) # reflection
elif thought.startswith('Alternatively') or 'another approach' in thought.lower():
refl_trans_reps.append(hidden) # transition
else:
exec_reps.append(hidden) # execution
H_exec = mean(exec_reps)
H_refl_trans = mean(refl_trans_reps)
steering_vector = H_exec - H_refl_trans # core formula
return steering_vector
# Step 2: Online - intervene during inference
def decode_with_seal(model, question, steering_vector, alpha=1.0, layer=20):
# modify hidden state at the '\n\n' token position at the end of each thought
# H_new = H_original + alpha * steering_vector
# subsequent token generation proceeds based on the modified H_new
return model.generate_with_hook(
question,
hook_layer=layer,
hook_fn=lambda h: h + alpha * steering_vector
)Terminology
Related Resources
Original Abstract (Expand)
Large Language Models (LLMs), such as OpenAI's o1-series have demonstrated compelling capabilities for complex reasoning tasks via the extended chain-of-thought (CoT) reasoning mechanism. However, recent studies reveal substantial redundancy in the CoT reasoning traces, which not only increases inference latency but also negatively impacts model performance by diverting attention to unnecessary reasoning paths. To address this issue, we investigate the internal reasoning structures of LLMs and categorize them into three primary thought types: execution, reflection, and transition thoughts. Moreover, our analysis reveals that excessive reflection and transition thoughts are strongly correlated with failure cases and these thought categories exhibit clear separation in the latent space. Based on these, we introduce SEAL (Steerable reasoning calibration), a training-free approach that seamlessly calibrates the CoT process, improving accuracy while demonstrating significant efficiency gains. SEAL consists of an offline stage for extracting the reasoning steering vector in the latent space, followed by an on-the-fly calibration of the reasoning trace through representation intervention using the steering vector. Notably, the steering vector exhibits strong transferability across various tasks. Extensive experiments across multiple models (DeepSeek-R1-Distill and QwQ-32B-Preview) and benchmarks (Math500, GSM8K, LiveCodeBench) validate the effectiveness of SEAL, up to a 11% improvement in accuracy while reducing reasoning tokens by 11.8% to 50.4%. Our code is publicly available at https://github.com/VITA-Group/SEAL.