Knowledge Distillation for Reasoning
Distilling chain-of-thought reasoning into smaller models
Performed knowledge distillation of large language models into smaller models while preserving chain-of-thought and multi-step reasoning capabilities. Analyzed generalization, calibration, and faithfulness of reasoning traces across model scales.
Technologies: PyTorch, Hugging Face, LLaMA 3
Duration: Nov 2025 - Dec 2025
Objectives
- Transfer complex reasoning abilities from large models to more efficient smaller models
- Preserve the interpretability and correctness of reasoning chains
- Analyze trade-offs between model size and reasoning quality
Findings
- Successfully distilled reasoning capabilities with minimal performance degradation
- Identified key factors affecting reasoning faithfulness across model scales
- Developed metrics for evaluating reasoning trace quality
This research contributes to making advanced AI reasoning more accessible and deployable on resource-constrained devices.