“`html
Is this film critique a praise or a criticism? Does this news article discuss commerce or technology? Is this virtual chatbot dialogue drifting into providing financial counsel? Is this online health information platform disseminating false data?
Such types of automated dialogues, whether involving the pursuit of a film or dining review, or acquiring data about your financial accounts or health records, are becoming more widespread. Now more than ever, these assessments are being conducted by advanced algorithms, referred to as text classifiers, rather than by humans. But how can we ascertain the accuracy of these classifications?
Currently, a group at MIT’s Laboratory for Information and Decision Systems (LIDS) has devised a groundbreaking method to not only assess how effectively these classifiers are performing their roles but also to further enhance their precision.
The new evaluation and corrective software was created by Kalyan Veeramachaneni, a lead research scientist at LIDS, along with his students Lei Xu and Sarah Alnegheimish, and two additional contributors. This software package is being made available for free download to anyone interested in utilizing it.
A conventional technique for testing these classification systems involves generating what are known as synthetic examples — sentences that closely mirror those that have previously been classified. For instance, researchers might take a sentence already identified by a classifier as a positive review and see if modifying a word or two while keeping the original meaning might trick the classifier into labeling it as negative. Alternatively, a statement previously assessed as misinformation could be miscategorized as accurate. This ability to deceive the classifiers creates adversarial examples.
Individuals have attempted various methods to uncover the weaknesses in these classifiers, Veeramachaneni notes. However, existing techniques for identifying these vulnerabilities struggle with this task and overlook many instances they should capture, he points out.
Increasingly, businesses are attempting to use such evaluation tools in real-time, overseeing the output of chatbots utilized for various functions to ensure they do not generate inappropriate responses. For example, a bank might deploy a chatbot to address routine customer inquiries like checking account balances or applying for a credit card, but it must ensure that its replies can never be misconstrued as financial guidance, which would expose the company to liability. “Before presenting the chatbot’s answer to the end-user, they want to use the text classifier to determine if it’s providing financial advice or not,” Veeramachaneni explains. However, it is vital to test the classifier to gauge how dependable its evaluations are.
“These chatbots, or summarization tools, are being implemented across the board,” he states, to interact with external clients as well as internally, for instance, offering information about HR matters. It is crucial to integrate these text classifiers to identify content they should not disclose and filter those out before the output reaches the user.
This is where adversarial examples come into play — those statements already classified but that yield a different result when they are slightly altered while preserving the same meaning. How can individuals verify that the meaning remains unchanged? By employing another large language model (LLM) that interprets and compares meanings. Therefore, if the LLM indicates that the two sentences convey the same meaning, but the classifier categorizes them differently, “that represents an adversarial sentence — it can deceive the classifier,” Veeramachaneni explains. When the researchers analyzed these adversarial statements, “we discovered that most of the time, this involved merely a one-word adjustment,” although those utilizing LLMs to create these alternative sentences often didn’t recognize this.
Further exploration, employing LLMs to scrutinize countless examples, revealed that certain specific words had a disproportionate effect on altering the classifications, allowing the assessment of a classifier’s accuracy to target this small subset of words that appear to have the greatest impact. They found that just one-tenth of 1 percent of all 30,000 words within the system’s vocabulary could account for nearly half of all these classification reversals in some specific contexts.
Lei Xu PhD ’23, a recent LIDS graduate who performed much of the analysis as part of his thesis work, “utilized numerous intriguing estimation techniques to determine the most powerful words that can alter the overall classification and mislead the classifier,” Veeramachaneni notes. The objective is to facilitate much more focused searches, rather than sifting through all possible word replacements, thus making the computational task of generating adversarial examples significantly more manageable. “He’s leveraging large language models, interestingly, to comprehend the influence of a single word.”
Additionally, using LLMs, he explores other words closely related to these powerful terms, and so forth, enabling an overall ranking of words based on their impact on the outcomes. Once these adversarial sentences are identified, they can be employed to retrain the classifier, accounting for them and thereby enhancing the classifier’s resistance against such errors.
Enhancing the accuracy of classifiers may not seem significant if it merely involves categorizing news articles or determining whether reviews of films or restaurants are favorable or unfavorable. However, classifiers are increasingly being applied in scenarios where the results truly matter, whether in preventing the unintended disclosure of sensitive medical, financial, or security information, guiding critical research—such as in the properties of chemical compounds or the folding of proteins for biomedical purposes, or in detecting and blocking hate speech or known misinformation.
As a result of this investigation, the team introduced a new metric, which they term p, providing a measurement of how resilient a given classifier is against single-word assaults. Given the significance of such misclassifications, the research group has made its products accessible as open-source for public use. The package comprises two components: SP-Attack, which generates adversarial sentences to evaluate classifiers in any particular application, and SP-Defense, which aims to bolster the classifier’s robustness by generating and utilizing adversarial sentences for model retraining.
In some tests, where competing methodologies for evaluating classifier outputs yielded a 66 percent success rate for adversarial assaults, this team’s system halved that success rate, bringing it down to 33.7 percent. In other scenarios, the improvement was as minimal as a 2 percent difference, but even that can be quite significant, Veeramachaneni asserts, since these systems are employed for billions of interactions, and even a small percentage can influence millions of transactions.
The findings of the team were published on July 7 in the journal Expert Systems in a paper authored by Xu, Veeramachaneni, and Alnegheimish of LIDS, alongside Laure Berti-Equille at IRD in Marseille, France, and Alfredo Cuesta-Infante at the Universidad Rey Juan Carlos in Spain.
“`