“`html
Travel specialists assist in arranging comprehensive logistics — such as transportation, accommodation, dining, and housing — for corporate travelers, tourists, and everyone in between. For those intending to handle their own bookings, extensive language models (ELMs) appear to be a promising resource for this endeavor, due to their capability for iterative interaction using natural language, providing basic common sense reasoning, gathering information, and utilizing additional tools to aid in the task at hand. Nonetheless, recent studies have revealed that cutting-edge ELMs encounter difficulties with intricate logistical and mathematical reasoning, and face challenges with multiple constraints, such as trip planning, where they have been observed to offer satisfactory solutions only 4 percent of the time or less, even when augmented with additional tools and application programming interfaces (APIs).
Consequently, a research team from MIT and the MIT-IBM Watson AI Lab redefined the challenge to determine if they could enhance the success rate of ELM solutions for sophisticated problems. “We believe many of these planning issues are inherently combinatorial optimization problems,” where one must fulfill several constraints in a verifiable manner, asserts Chuchu Fan, associate professor in the MIT Department of Aeronautics and Astronautics (AeroAstro) as well as the Laboratory for Information and Decision Systems (LIDS). She is also a researcher at the MIT-IBM Watson AI Lab. Her group employs machine learning, control theory, and formal methods to create safe and certifiable control systems for robotics, autonomous frameworks, controllers, and human-machine collaborations.
Recognizing the applicable nature of their research for travel organization, the team aimed to construct an accessible framework that acts as an AI travel intermediary to assist in formulating realistic, rational, and comprehensive travel itineraries. To realize this goal, the researchers integrated common ELMs with algorithms and a complete satisfiability solver. Solvers are mathematical instruments that meticulously verify if criteria can be fulfilled and how, yet they necessitate complex programming for application. This makes them ideal partners for ELMs in scenarios where users seek assistance with planning in an efficient manner, without needing programming expertise or extensive research into travel options. Moreover, if a user’s constraint cannot be accommodated, the innovative approach can pinpoint and articulate the issue, suggesting alternative options to the user, who can then decide to accept, decline, or adjust them until a valid itinerary is devised, if feasible.
“Different complexities of travel planning are challenges everyone will confront eventually. There are various demands, prerequisites, constraints, and tangible data that can be gathered,” notes Fan. “Our intention is not to request ELMs to create a travel itinerary. Instead, an ELM here serves as a translator, converting this natural language description of the problem into a format that a solver can manage [and then provide that to the user],” Fan elaborates.
Co-authoring a paper on the research with Fan are Yang Zhang of MIT-IBM Watson AI Lab, AeroAstro graduate student Yilun Hao, and graduate student Yongchao Chen of MIT LIDS and Harvard University. This research was recently presented at the Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics.
Analyzing the solver
Mathematics often tends to be domain-specific. For instance, in natural language processing, ELMs execute regressions to predict the next token, or “word,” in a sequence to analyze or generate a document. This approach works effectively for generalizing various human inputs. Independently, however, ELMs would not suffice for formal verification applications, like those in aerospace or cybersecurity, where circuit connections and constrained tasks must be complete and verifiable; otherwise, vulnerabilities can arise and lead to critical safety concerns. In this context, solvers excel, yet they require fixed input formats and struggle with unsatisfiable queries. However, a hybrid approach offers an opportunity to devise solutions for complex issues, such as trip planning, in an intuitive manner for everyday individuals.
“The solver is genuinely pivotal here because when we create these algorithms, we precisely understand how the problem is tackled as an optimization challenge,” explains Fan. Specifically, the research team utilized a solver called satisfiability modulo theories (SMT), which assesses whether a formula can be satisfied. “With this specific solver, it doesn’t just perform optimization. It’s reasoning over various different algorithms to ascertain whether the planning problem can be resolved. That’s crucial in travel planning. It’s not a traditional mathematical optimization problem because people introduce numerous limitations, constraints, and restrictions,” Fan points out.
Translation in motion
The “travel agent” functions in a cycle of four steps that can be repeated as necessary. The researchers implemented GPT-4, Claude-3, or Mistral-Large as the LLM for the method. Initially, the LLM breaks down a user’s requested travel plan prompt into planning steps, identifying preferences concerning budget, accommodation, transportation, destinations, attractions, dining, and trip duration in days, alongside any other user instructions. These steps are subsequently converted into executable Python code (with natural language annotations for each constraint), which accesses APIs like CitySearch, FlightSearch, and so on to gather data, and invokes the SMT solver to initiate execution of the outlined constraints satisfaction problem. If a valid and comprehensive solution is available, the solver relays the result to the LLM, which then generates a coherent itinerary for the user.
If one or more constraints cannot be satisfied, the framework starts searching for alternatives. The solver provides code that identifies the conflicting constraints (along with relevant annotations) that the LLM subsequently conveys to the user with a possible solution. The user then decides how to proceed until a resolution (or the maximum allowed iterations) is achieved.
Generalizable and resilient planning
The researchers evaluated their methodology employing the previously mentioned ELMs against other baselines: GPT-4 alone, OpenAI o1-preview independently, GPT-4 with a tool for data gathering, and a search algorithm that optimizes for total expense. Utilizing the TravelPlanner dataset, which includes data for feasible plans, the team examined numerous performance metrics: how often a method could deliver a solution, whether the solution adhered to commonsense standards like avoiding visits to two cities in a single day, the method’s capacity to satisfy one or more constraints, and a final pass rate indicating the ability to meet all constraints. The new approach generally achieved over a 90 percent pass rate, compared to 10 percent or lower for the baselines. The team also investigated the addition of a JSON representation during the query stage, further facilitating the method’s ability to deliver solutions with 84.4-98.9 percent pass rates.
The MIT-IBM team posed additional challenges to their methodology. They assessed how critical each aspect of their solution was — such as omitting human feedback or the solver — and how that influenced plan modifications to unsatisfiable queries within 10 or 20 iterations using a newly created dataset called UnsatChristmas, which includes unseen constraints, alongside a modified version of TravelPlanner. On average, the MIT-IBM framework attained 78.6 and 85 percent success rates, which improve to 81.6 and 91.7 percent with further rounds of plan adjustments. The researchers evaluated its performance with newly introduced, unseen constraints, and paraphrased query-step and step-code prompts. In both instances, it excelled, particularly with an 86.7 percent pass rate for the paraphrasing evaluation.
Ultimately, the MIT-IBM researchers applied their framework to other fields with tasks involving block selection, task assignment, the traveling salesman problem, and warehouses. Here, the method must select numbered, colored blocks and maximize its score; optimize robot task assignments for various scenarios; plan trips with minimized travel distances; and complete and optimize robot tasks.
“I believe this is a highly effective and innovative framework that can save considerable time for humans, and it represents a very novel combination of the LLM and the solver,” states Hao.
This research was partially funded by the Office of Naval Research and the MIT-IBM Watson AI Lab.
“`