This paper proposes a more efficient version of the SOAP optimizer, called Factored One-sided Adam-Moment (FOAM), which retains the benefits of SOAP while using less memory and running faster. Practitioners caring about optimizing large language models may benefit from this more efficient optimizer.
Firehose
Filtered to Papers, tagged “optimizers” · clear filters
Browse: People · Companies · Papers · Podcasts · Hacker News · Deep dives