Should you take your umbrella before stepping out the door? Examining the weather predictions in advance will be beneficial only if those predictions are precise.

Spatial forecasting challenges, such as weather prediction or estimating air pollution, deal with forecasting the value of a variable in a fresh location by leveraging known values from other areas. Researchers commonly use established validation techniques to ascertain how much to rely on these forecasts.

However, MIT researchers have demonstrated that these commonly used validation techniques can perform poorly in spatial forecasting tasks. This could lead an individual to mistakenly believe that a prediction is accurate or that a novel forecasting approach is reliable, when, in fact, the opposite may be true.

The team developed a methodology to evaluate prediction-validation techniques and employed it to demonstrate that two traditional methods can be significantly inaccurate in spatial contexts. They then investigated why these approaches can falter and formulated a new technique aimed at managing the types of data used in spatial predictions.

In trials using both real and simulated data, their innovative method yielded more precise validations than the two most widely utilized techniques. The team scrutinized each method through realistic spatial scenarios, including estimating wind speed at Chicago’s O’Hare Airport and predicting air temperature in five metropolitan areas across the U.S.

Their validation approach could be implemented across a variety of challenges, assisting climate scientists in forecasting sea surface temperatures and helping epidemiologists in assessing the effects of air pollution on specific health conditions.

“We hope that this will lead to more dependable evaluations when researchers develop new predictive techniques and provide a deeper understanding of how well these methods perform,” states Tamara Broderick, an associate professor in MIT’s Electrical Engineering and Computer Science (EECS) department, a member of the Laboratory for Information and Decision Systems, and affiliated with the Computer Science and Artificial Intelligence Laboratory (CSAIL).

Broderick co-authors the paper alongside lead author and MIT postdoctoral researcher David R. Burt and EECS graduate student Yunyi Shen. Their findings will be presented at the International Conference on Artificial Intelligence and Statistics.

Assessing validations

Broderick’s team has recently partnered with oceanographers and atmospheric scientists to develop machine-learning forecasting models that address challenges with a pronounced spatial aspect.

During this collaboration, they observed that conventional validation techniques can be flawed in spatial contexts. These traditional methods withhold a small portion of the training data, referred to as validation data, to evaluate the accuracy of the forecasting model.

To investigate the underlying issue, they conducted a meticulous analysis and discovered that conventional techniques rely on assumptions unsuitable for spatial data. Evaluation techniques depend on assumptions regarding how validation data and the data intended for prediction, known as test data, are correlated.

Traditional approaches presume that validation data and test data are independent and identically distributed, which suggests that the value of any data point is independent of the others. However, in a spatial context, this assumption often does not hold.

For example, a researcher might utilize validation data from EPA air pollution sensors to evaluate the accuracy of a model that predicts air quality in conservation areas. Nevertheless, the EPA sensors are not independent since their locations are determined based on the positioning of other sensors.

Additionally, it is possible that the validation data emanates from EPA sensors located near urban centers while the conservation sites are situated in rural regions. Given that these datasets are sourced from different geographical areas, they likely exhibit dissimilar statistical characteristics and are therefore not identically distributed.

“Our experiments revealed that one can derive erroneous outcomes in spatial situations when the assumptions made by the validation method fail,” Broderick notes.

The researchers needed to devise a new assumption.

Specifically spatial

Focusing specifically on a spatial framework, where data is collected from diverse locations, they created a method that assumes validation data and test data change gradually across space.

For instance, air pollution levels are unlikely to fluctuate drastically between two neighboring residences.

“This regularity assumption is fitting for numerous spatial processes and enables us to formulate a means to evaluate spatial predictors in the spatial framework. To our knowledge, no previous work has implemented a systematic theoretical assessment of the shortcomings to develop an improved method,” states Broderick.

To employ their evaluation methodology, users input their predictor, the target locations for predictions, and their validation data, after which the system autonomously handles the remaining steps. Ultimately, it predicts how accurate the forecaster’s estimate will be for the specified location. However, successfully evaluating their validation method posed a challenge.

“We are evaluating an evaluation technique rather than a method, so we had to take a step back, think carefully, and explore creative ways to conduct the appropriate experiments,” Broderick clarifies.

Initially, they devised several tests using simulated data, which, while unrealistic, allowed them to carefully control critical parameters. Subsequently, they produced more authentic, semi-simulated data by altering real datasets. Finally, they executed several experiments using actual data.

The utilization of three distinctive datasets from realistic problems, such as estimating property values in England based on location and predicting wind speeds, enabled them to perform a thorough evaluation. In the majority of cases, their methodology outperformed both conventional methods utilized for comparison.

Looking forward, the researchers intend to apply these techniques to enhance uncertainty quantification in spatial frameworks. They are also interested in exploring additional areas where the regularity assumption could boost predictor performance, such as in time-series data.

This study is partially funded by the National Science Foundation and the Office of Naval Research.