“`html
When scholars are constructing extensive language models (ELMs), their objective is to enhance performance within a specific computational and financial framework. Given that training a model can total millions of dollars, creators must be prudent with financially influential choices regarding, for example, the model design, optimizers, and training datasets prior to committing to a model. To predict the quality and precision of an extensive model’s forecasts, practitioners frequently rely on scaling laws: employing smaller, more economical models to attempt to simulate the performance of a substantially larger target model. The difficulty, however, lies in the multitude of methods to formulate a scaling law.
New findings from researchers at MIT and the MIT-IBM Watson AI Lab tackle this by gathering and distributing a compilation of hundreds of models and metrics related to training and performance to estimate more than a thousand scaling laws. From this, the team devised a meta-analysis and guideline for how to opt for smaller models and assess scaling laws across various ELM model categories, ensuring that the budget is utilized optimally for generating dependable performance forecasts.
“The idea that one might want to construct mathematical models of the training process is a few years old, but what’s novel here is that most previous efforts focused on saying, ‘can we make post-hoc statements about what occurred when we trained all of these models, so that while we’re determining how to train a new large-scale model, we can make the best choices regarding our compute budget?’” states Jacob Andreas, associate professor in the Department of Electrical Engineering and Computer Science and principal investigator with the MIT-IBM Watson AI Lab.
The study was recently presented at the International Conference on Machine Learning by Andreas, alongside researchers Leshem Choshen and Yang Zhang from IBM Research.
Inferring performance
Regardless of the perspective, creating ELMs is a costly venture: from decision-making about the number of parameters and tokens, data selection and size, to training methods and determining output accuracy adjusted for target applications and tasks. Scaling laws provide a means to predict model behavior by correlating a large model’s loss with the performance of smaller, less expensive models from the same lineage, eliminating the necessity to fully train every candidate. The primary distinctions between the smaller models are the number of parameters and token training size. According to Choshen, clarifying scaling laws not only enhances pre-training decisions but also democratizes the domain by enabling researchers without extensive resources to comprehend and construct effective scaling laws.
The structural form of scaling laws is relatively straightforward, incorporating elements from the smaller models that capture the number of parameters and their scaling impact, the number of training tokens and their scaling impact, and the baseline performance for the model category in question. Collectively, they assist researchers in estimating a target large model’s performance loss; the smaller the loss, the more favorable the target model’s outputs are anticipated to be.
These laws empower research teams to balance trade-offs effectively and to experiment with the best allocation of limited resources. They are specifically advantageous for evaluating the scaling of particular variables, such as token count, and for A/B testing various pre-training configurations.
In general, scaling laws are not novel; however, within the field of AI, they surfaced as models expanded and expenses soared. “It’s as though scaling laws simply emerged at some stage in the field,” remarks Choshen. “They began to gain attention, but no one truly examined their efficacy or what needs to be done to form a solid scaling law.” Moreover, scaling laws themselves were a bit of a black box. “Whenever individuals have devised scaling laws in the past, it has usually been a single model, or one model family, and one dataset, along with one developer,” states Andreas. “There hadn’t been much comprehensive meta-analysis, as everyone was independently training their own scaling laws. So, [we were curious about] are there overarching trends that can be observed across these efforts?”
Creating improvement
To explore this, Choshen, Andreas, and Zhang assembled a vast dataset. They gathered ELMs from 40 model families, including Pythia, OPT, OLMO, LLaMA, Bloom, T5-Pile, ModuleFormer mixture-of-experts, GPT, and several others. This collection included 485 unique, pre-trained models, along with available data about their training checkpoints, computational expenditure (FLOPs), training epochs, and the seed, alongside 1.9 million performance metrics of loss and downstream tasks. The models varied in their structures, weights, and more. Utilizing these models, the researchers fitted over 1,000 scaling laws and compared their accuracy across architectures, model sizes, and training configurations, as well as testing how the quantity of models, inclusion of intermediate training checkpoints, and partial training influenced the predictive strength of scaling laws relating to target models. They employed measures of absolute relative error (ARE); this signifies the gap between the scaling law’s forecast and the actual loss of a large, trained model. Consequently, the team assessed the scaling laws, and following analysis, distilled actionable recommendations for AI professionals concerning what constitutes effective scaling laws.
Their shared guidelines guide the developer through steps and options to consider and expectations. Initially, it’s vital to establish a compute budget and desired model accuracy. The team discovered that a 4 percent ARE is an ideal accuracy one might anticipate due to random seed fluctuations, but even up to 20 percent ARE remains useful for decision-making. The researchers pinpointed several elements that enhance predictions, such as including intermediate training checkpoints, rather than depending solely on final losses; this made scaling laws more trustworthy. Nonetheless, very early training data prior to 10 billion tokens tends to be unreliable, lessen accuracy, and should be excluded. They recommend focusing on training multiple models across a range of sizes to bolster the robustness of the scaling law’s prediction, not merely larger models; selecting five models serves as a strong starting point.
Generally, incorporating larger models enhances prediction, but funds can be conserved by partially training the target model to about 30 percent of its dataset and utilizing that for extrapolation. If the budget is highly restricted, developers should contemplate training one smaller model within the target model family and borrowing scaling law parameters from a model family with a similar structure; however, this may prove ineffective for encoder–decoder models. Lastly, the MIT-IBM research group discovered that when scaling laws were analyzed across model families, there was a robust correlation between two sets of hyperparameters, indicating that three of the five hyperparameters accounted for nearly all of the variation and could likely encapsulate the model dynamics. All together, these guidelines offer a systematic methodology for enhancing the efficiency, reliability, and accessibility of scaling law estimation for AI researchers operating under various budget limitations.
Several unexpected findings emerged during this investigation: partially trained small models remain highly predictive, and additionally, the intermediate training phases from a fully trained model can be utilized (as if they are standalone models) for predicting another target model’s performance. “Essentially, you incur no cost in the training since you’ve already trained the complete model; thus, the half-trained model, for example, is merely a byproduct of your previous work,” states Choshen. Another aspect highlighted by Andreas was that, when compiled, the variability across model families and various experiments became markedly apparent and noisier than anticipated. Surprisingly, the researchers discovered that it’s feasible to apply the scaling laws on large models to predict performance down to smaller models. Previous research in the field has speculated that smaller models represented a “different beast” compared to larger ones; however, Choshen contends otherwise. “If they are entirely different, they should exhibit completely different behavior, yet they do not.”
While this research concentrated on model training duration, the researchers intend to expand their examination to model inference. Andreas states it’s not, “how does my model improve as I add more training data or more parameters, but rather as I allow it to reflect for longer, draw more samples. I believe there are certainly insights to be gained regarding how to also construct predictive models of how much contemplation is necessary at runtime.” He argues that the theory of inference time scaling laws may become increasingly critical because, “it’s not as though I’m going to train one model and then be finished. [Instead,] every time a user approaches me, they will have a new inquiry, and I need to discern how deeply [my model needs] to think to formulate the best response. Thus, constructing those sort of predictive models, like we are doing in this study, becomes even more vital.”
This investigation was partially supported by the MIT-IBM Watson AI Lab and a Sloan Research Fellowship.
“`