“`html
Studies have indicated that large language models (LLMs) often place excessive importance on information located at the start and conclusion of a document or discussion, while somewhat overlooking the middle content.
This “position bias” implies that if a lawyer is utilizing an LLM-integrated virtual assistant to locate a specific phrase in a 30-page affidavit, the LLM is more inclined to discover the relevant text if it resides on the first or last pages.
Researchers at MIT have uncovered the mechanism driving this occurrence.
They developed a theoretical framework to investigate how information navigates through the machine-learning architecture that underpins LLMs. They discovered that particular design choices that dictate how the model processes input data can contribute to position bias.
Their experiments highlighted that model architectures, especially those impacting how information disperses among input words within the model, can generate or augment position bias, with training data also exacerbating the issue.
Alongside identifying the roots of position bias, their framework can be employed to identify and rectify it in future model configurations.
This could result in more dependable chatbots that remain focused throughout extended dialogues, medical AI systems that provide more equitable reasoning when processing extensive patient data, and code assistants that give greater attention to all sections of a program.
“These models function as black boxes, so as a user of an LLM, you likely remain unaware that position bias could lead your model to produce inconsistent results. You simply input your documents in any order and anticipate them to function correctly. However, by gaining a deeper understanding of the underlying mechanisms of these black-box models, we can enhance them by addressing these shortcomings,” states Xinyi Wu, a graduate student affiliated with the MIT Institute for Data, Systems, and Society (IDSS) and the Laboratory for Information and Decision Systems (LIDS), and principal author of a paper detailing this research.
Her co-authors consist of Yifei Wang, a postdoctoral researcher at MIT; and senior authors Stefanie Jegelka, an associate professor in electrical engineering and computer science (EECS) as well as a member of IDSS and the Computer Science and Artificial Intelligence Laboratory (CSAIL); and Ali Jadbabaie, professor and chair of the Department of Civil and Environmental Engineering, a core faculty member of IDSS, and a principal investigator at LIDS. The research will be presented at the International Conference on Machine Learning.
Examining Attention
LLMs such as Claude, Llama, and GPT-4 are powered by a specific neural network architecture called transformers. Transformers are designed to process sequential data by encoding sentences into segments referred to as tokens, subsequently learning the interrelations among tokens to predict subsequent words.
These models have become highly proficient at this due to the attention mechanism, which employs interlinked layers of data processing nodes to interpret context by permitting tokens to selectively focus on, or attend to, related tokens.
However, if every token can attend to all others in a 30-page document, it quickly becomes computationally impractical. Therefore, when engineers construct transformer models, they frequently implement attention masking techniques that restrict the words a token can attend to.
For example, a causal mask permits words to only attend to the preceding ones.
Engineers also utilize positional encodings to assist the model in comprehending the position of each word in a sentence, thereby enhancing performance.
The MIT researchers devised a graph-based theoretical framework to investigate how these modeling decisions, including attention masks and positional encodings, might influence position bias.
“Everything is interconnected within the attention mechanism, making it very challenging to analyze. Graphs provide a flexible language to describe the dependencies among words within the attention mechanism and trace these relationships across various layers,” Wu explains.
Their theoretical investigation indicated that causal masking introduces an inherent bias towards the beginning of an input, even when such bias is absent in the data.
If the initial words hold relatively little significance for a sentence’s meaning, causal masking may still cause the transformer to pay more heed to its start.
“Although it’s often the case that earlier and later words in a sentence carry greater weight, if an LLM is applied to a task that deviates from natural language generation, such as ranking or information retrieval, these biases can prove to be extremely detrimental,” Wu notes.
As a model evolves, incorporating additional layers of the attention mechanism amplifies this bias, as earlier segments of the input are utilized more frequently in the model’s reasoning process.
They also discovered that employing positional encodings to strengthen the connection of words to nearby words can alleviate position bias. This technique redirects the model’s attention appropriately, although its efficacy may diminish in models featuring more attention layers.
Additionally, these design choices are merely one source of position bias; some may arise from the training data the model employs to learn how to prioritize words in a sequence.
“If you are aware that your data is biased in a certain manner, then you should also fine-tune your model in conjunction with adjusting your modeling choices,” Wu suggests.
Lost in the Middle
Once they had established the theoretical framework, the researchers conducted experiments where they systematically altered the position of the correct answer within text sequences for an information retrieval task.
The experiments demonstrated a “lost-in-the-middle” phenomenon, where retrieval accuracy exhibited a U-shaped pattern. Models achieved optimal performance when the correct answer was found at the start of the sequence. Performance dipped as the answer approached the middle before slightly recovering if the correct answer was positioned near the end.
Ultimately, their findings propose that employing an alternative masking technique, removing excess layers from the attention mechanism, or strategically using positional encodings might minimize position bias and enhance a model’s accuracy.
“By integrating theory with experimentation, we have been able to examine the repercussions of model design decisions that were elusive at the time. If you aim to employ a model in high-stakes situations, it is essential to comprehend when it will function effectively, when it won’t, and the underlying reasons,” Jadbabaie remarks.
In the future, the researchers plan to further investigate the implications of positional encodings and examine how position bias could be strategically leveraged in specific applications.
“These researchers provide a rare theoretical perspective on the attention mechanism central to the transformer model. They offer a persuasive analysis that clarifies persistent peculiarities in transformer behavior, demonstrating that attention mechanisms, particularly with causal masks, inherently bias models towards the beginning of sequences. The paper achieves an excellent balance—mathematical clarity combined with insights that delve deep into practical systems,” remarks Amin Saberi, professor and director of the Stanford University Center for Computational Market Design, who was not part of this work.
This research is partially funded by the U.S. Office of Naval Research, the National Science Foundation, and an Alexander von Humboldt Professorship.
“`