This Ai Paper Introduces Fastcurl: A Curriculum Reinforcement Learning Framework With Context Extension For Efficient Training Of R1-like Reasoning Models

Large connection models person transformed really machines comprehend and make text, particularly successful analyzable problem-solving areas for illustration mathematical reasoning. These systems, known arsenic R1-like models, are designed to emulate slow and deliberate thought processes. Their cardinal spot is handling intricate tasks requiring step-by-step reasoning crossed agelong sequences. These capabilities make them valuable for applications specified arsenic solving Olympiad-level mathematics problems aliases logical reasoning tasks, wherever extent and coherence of reasoning are essential.

A important situation successful training these models is nan extended computation for reinforcement learning utilizing agelong discourse windows. Tasks that require multi-step logic unit models to nutrient agelong outputs which consumes much resources and slows down learning. Further, not each agelong responses lend meaningfully to accuracy; galore see redundant reasoning. These inefficiencies successful consequence procreation and precocious GPU usage make it difficult to efficaciously standard training, peculiarly erstwhile moving pinch models pinch 1.5 cardinal parameters.

Previous attempts to reside this rumor see models for illustration DeepScaleR, which uses a staged discourse magnitude hold strategy during training. DeepScaleR starts pinch an 8K discourse model and expands gradually to 24K complete 3 training phases. Although this attack helps guideline nan exemplary to negociate longer reasoning chains efficiently, it still demands astir 70,000 A100 GPU hours. DeepScaleR reduces that to 3,800 hours done a progressive strategy but still requires sizeable hardware, including setups pinch up to 32 GPUs successful immoderate stages. This shows that while improvements are possible, nan solution remains costly and complex.

Researchers astatine Tencent introduced a method called FASTCURL to flooded nan inefficiencies of accepted reinforcement learning training. This method presents a curriculum-based strategy aligned pinch discourse model description . FASTCURL splits nan dataset based connected input punctual magnitude into short, long, and mixed categories. The training progresses successful 4 stages, each utilizing a different dataset and discourse model setting. This attack ensures nan exemplary learns elemental reasoning earlier advancing to longer, much analyzable reasoning steps. The researchers stress that nan full training process runs connected a azygous node pinch conscionable 8 GPUs, reducing setup complexity.

The attack involves a deliberate segmentation of information by input length, driven by nan presumption that longer prompts usually lead to longer and much analyzable outputs. The exemplary first learns utilizing short prompts nether an 8K window. As training proceeds, nan exemplary transitions to a mixed dataset pinch 16K model length, past to nan agelong dataset pinch nan aforesaid model size, and yet reviews nan mixed information again. Each shape is trained for 1 iteration, and FASTCURL requires astir 860 training steps. This is businesslike compared to DeepScaleR’s 1,750 steps, representing a 50% simplification successful training clip and assets usage while maintaining effectiveness.

In capacity evaluations, FASTCURL-1.5B-Preview showed improvements complete different models crossed 5 benchmarks. It scored 88.0 connected MATH 500, 43.1 connected AIME 2024, 74.2 connected AMC 2023, 31.6 connected Minerva Math, and 50.4 connected OlympiadBench, pinch an mean PASS@1 people of 57.5. Compared to DeepScaleR-1.5B-Preview, which scored an mean of 57.0, FASTCURL performed amended successful 4 of 5 datasets. These results item that FASTCURL tin outperform existing techniques while consuming importantly less resources. The exemplary besides showed amended generalization, peculiarly connected datasets for illustration AMC 2023 and Minerva Math, indicating robustness.

The investigation intelligibly outlines a computational problem successful training R1-like reasoning models and offers an innovative program strategy arsenic a solution. The method provides an businesslike and applicable training model by combining input-based information segmentation pinch discourse description . FASTCURL delivers beardown capacity utilizing less steps and constricted hardware, proving that strategical training creation tin beryllium arsenic powerful arsenic earthy computational scale.

Check out the Paper. All in installments for this investigation goes to nan researchers of this project. Also, feel free to travel america on Twitter and don’t hide to subordinate our 85k+ ML SubReddit.

🔥 [Register Now] miniCON Virtual Conference connected OPEN SOURCE AI: FREE REGISTRATION + Certificate of Attendance + 3 Hour Short Event (April 12, 9 am- 12 p.m. PST) + Hands connected Workshop [Sponsored]