This Ai Paper Introduces Grpo-based Open-rs: A Low-cost Reinforcement Learning Framework To Enhance Reasoning In Small Language Models

One peculiar attraction connected ample connection models has been improving their logical reasoning and problem-solving skills. Reinforcement learning (RL) is progressively utilized successful this abstraction for monolithic models and compact versions that tin execute good successful restricted computing environments. One awesome situation successful this section is improving a model’s reasoning capacity without relying connected highly ample infrastructure aliases excessive training time. Leading models require costly hardware and proprietary information pipelines, putting them retired of scope for smaller labs aliases companies. This raises nan mobility of whether smaller models tin beryllium enhanced utilizing cost-efficient approaches and execute capacity comparable to their larger counterparts connected challenging tasks for illustration mathematics reasoning.

Several methods person been explored to reside this. Chain-of-thought prompting helps guideline models done problem steps. Search algorithms specified arsenic Beam Search and Monte Carlo Tree Search are besides utilized to amended nan logical travel of answers. Reinforcement learning itself has been tested successful aggregate settings. However, galore of these approaches are still bound by nan aforesaid issues: they dangle connected monolithic datasets aliases lead to unstable capacity successful small-scale setups. Furthermore, nan results often neglect to lucifer those of proprietary models for illustration OpenAI’s o1-preview.

Research introduced by a squad from Knovel Engineering Lab successful Singapore and VNU University of Science successful Vietnam focused connected overcoming these problems. The researchers utilized a 1.5-billion-parameter exemplary named DeepSeek-R1-Distill-Qwen-1.5B. They adopted nan Group Relative Policy Optimization (GRPO) algorithm for their setup, training nan exemplary utilizing 4 NVIDIA A40 GPUs pinch 48 GB VRAM each, each wrong a strict 24-hour limit. Their cardinal nonsubjective was to heighten nan model’s reasoning without ample financial aliases computational investment. Their training consumed only $42 successful computing costs, a drastic simplification compared to baselines that require thousands of dollars.

The squad assembled a dataset of 39,659 mathematics-specific questions to execute this by refining 2 existing datasets—open-s1 and open-deep scale. The filtering process progressive removing trivial aliases noisy questions utilizing different models specified arsenic Qwen2.5-7B-Instruct and DeepSeek-R1-Distill-Qwen-1.5B. The reward strategy was rule-based and focused connected 3 components: correctness of answers (using boxed notation), structural formatting (enforced pinch tags), and output magnitude (rewarded pinch a cosine usability to beforehand concise reasoning). The GRPO algorithm was utilized to sample group responses and use score-based optimization, avoiding nan request for a captious exemplary and frankincense reducing computational demands further.

The capacity of this attack was tested crossed 5 benchmark datasets: AMC23, AIME24, MATH-500, OlympiadBench, and Minerva. In 1 experiment, utilizing conscionable nan open-s1 dataset, nan model’s AMC23 accuracy improved from 63% to 70% wrong nan first 100 world steps but later declined. In different proceedings that mixed 7,000 samples of mixed difficulty, nan accuracy connected AMC23 roseate to 80%, and AIME24 reached 46.7%. The exemplary named Open-RS2, trained successful that setup, besides showed competitory scores connected OlympiadBench (52.4%) and MATH-500 (85%). In nan last experiment, nan cosine reward helped modulate output magnitude to a scope of 1000–3500 tokens, and nan exemplary maintained 72.5% accuracy connected AMC23 and 84.4% connected MATH-500.

This investigation showed that effective reasoning successful mini connection models is achievable moreover pinch constricted resources. The problem of training mini models without important hardware finance was addressed pinch a low-cost and businesslike training strategy. The projected method utilized reinforcement learning and curated information to present amazingly beardown results. With continued improvements successful reward creation and optimization stability, mini models whitethorn soon rival their larger counterparts successful applicable reasoning tasks.

Check out the Paper and GitHub Page. All in installments for this investigation goes to nan researchers of this project. Also, feel free to travel america on Twitter and don’t hide to subordinate our 85k+ ML SubReddit.