A domestic robot programmed to undertake chores in an industrial setting may struggle to efficiently clean the sink or dispose of trash when utilized in a homeowner’s kitchen, as this unfamiliar setting contrasts with its training environment.
To prevent such issues, engineers typically strive to make the simulated training settings as similar as possible to the real-world conditions where the agent will operate.
Nevertheless, researchers from MIT and other institutions have recently discovered that, contrary to popular belief, sometimes training in a entirely distinct environment produces a more proficient artificial intelligence agent.
Their findings suggest that, in certain scenarios, preparing a simulated AI agent in an environment with reduced unpredictability, or “noise,” allowed it to outperform a rival AI agent trained in the same, noise-filled world used for testing both agents.
The researchers refer to this surprising occurrence as the indoor training effect.
“If we practice tennis in an indoor setting devoid of noise, we may find it simpler to master various shots. Subsequently, when we transition to a noisier venue, like a breezy tennis court, we could have a greater chance of excelling at tennis compared to if we began our practice in the windy setting,” explains Serena Bono, a research assistant at the MIT Media Lab and lead author of a study on the indoor training effect.
The scientists investigated this effect by training AI agents to compete in Atari games, which they altered by introducing some unpredictability. They were taken aback to observe that the indoor training effect consistently manifested across different Atari games and variations.
They aspire that these findings inspire further research aimed at enhancing training strategies for AI agents.
“This introduces a completely new dimension to consider. Instead of merely trying to align the training and testing surroundings, we might be able to create simulated setups where an AI agent achieves even superior learning outcomes,” adds co-author Spandan Madan, a graduate student at Harvard University.
Bono and Madan collaborated on the paper with Ishaan Grover, an MIT graduate student; Mao Yasueda, a graduate student from Yale University; Cynthia Breazeal, a professor of media arts and sciences and head of the Personal Robotics Group at the MIT Media Lab; Hanspeter Pfister, the An Wang Professor of Computer Science at Harvard; and Gabriel Kreiman, a professor at Harvard Medical School. The research is set to be showcased at the Association for the Advancement of Artificial Intelligence Conference.
Challenges in Training
The researchers aimed to understand the reasons behind the dismal performance of reinforcement learning agents when evaluated in environments different from the ones they were trained in.
Reinforcement learning employs a trial-and-error approach in which the agent navigates a training space and learns to undertake actions that maximize its rewards.
The team devised a method to intentionally introduce a certain degree of noise into a component of the reinforcement learning challenge known as the transition function. The transition function outlines the likelihood that an agent will shift from one state to another, grounded in the action it selects.
If the agent is engaged in a game of Pac-Man, a transition function might indicate the probability that the ghosts on the game board will move up, down, left, or right. In standard reinforcement learning, the AI would both train and be evaluated using the same transition function.
The researchers incorporated noise into the transition function through this conventional method and, as anticipated, it negatively impacted the agent’s performance in Pac-Man.
However, when the researchers prepared the agent using a noise-free Pac-Man game, and then tested it in an environment where noise was introduced into the transition function, it outperformed an agent that had trained on the noisy game.
“The typical guideline suggests capturing the transition function of the deployment conditions as accurately as possible during training to maximize outcomes. We rigorously assessed this notion because we found it hard to believe ourselves,” Madan states.
By injecting various degrees of noise into the transition function, the researchers could assess multiple environments, yet it did not create authentic games. The increased noise in Pac-Man made it more probable for ghosts to randomly teleport to different squares.
To determine if the indoor training effect was present in standard Pac-Man games, they altered fundamental probabilities so that ghosts moved in a more typical manner but were more inclined to move up and down rather than left and right. AI agents trained in noise-free conditions continued to perform better in these realistic games.
“It wasn’t solely a result of how we introduced noise to fabricate makeshift environments. This appears to be an inherent characteristic of the reinforcement learning issue, which was even more astonishing to observe,” Bono comments.
Insights on Exploration
Upon deeper investigation, the researchers noticed certain correlations in the exploration behavior of the AI agents within the training space.
When both AI agents primarily explore similar regions, the agent trained in the non-noisy environment tends to excel, possibly because it can more easily grasp the game’s rules without the disruption of noise.
If their exploration behaviors diverge, the agent trained in the noisy environment often achieves superior results. This may happen because the agent must comprehend patterns that it could not learn in the noise-free setting.
“If I only learn to play tennis using my forehand in a non-noisy space, but then in a noisy one, I must also incorporate my backhand, I won’t perform as well in the non-noisy environment,” Bono explains.
In the future, the researchers aspire to investigate how the indoor training effect may arise in more intricate reinforcement learning environments or with different methodologies such as computer vision and natural language processing. They also aim to construct training environments specifically designed to exploit the indoor training effect, which could enhance AI agents’ performance in unpredictable settings.