General practitioners aided by AI may perform diagnoses, conduct and analyze tests, and execute procedures akin to specialists

Vehicles that are outfitted somewhere between a doctor’s clinic and a hospital, featuring an AI assistant to support medical generalists through unfamiliar diagnoses and treatments, have the potential to enhance healthcare accessibility in rural regions.

The Advanced Research Projects Agency for Health, or ARPA-H, is funding the creation of such a clinic, with leadership from the University of Michigan spearheading one of two major teams responsible for designing and developing the AI aspect.

Even prior to rural hospitals scaling back services and closing their doors, securing reliable medical care posed difficulties for individuals residing in isolated areas. The new initiative aimed at increasing accessibility evokes images of Knight Rider merged with Northern Exposure, in a scenario where rural healthcare providers amplify their expertise on-the-go with assistance from their mobile unit.

“Our goal is to transport the hospital to homes, or to church parking lots—whether it be in Michigan’s Upper Peninsula or amidst Indiana—where the closest medical facility that provides necessary care may be a two-hour drive away,” stated Jason Corso, the director of the AI project and a professor at U-M specializing in robotics and electrical engineering and computer science.

A mobile clinic would decrease healthcare expenses by minimizing the necessity for permanent structures in its service region and enable physician assistants and nurses to carry out more complex procedures with guidance from the AI assistant, Corso explained. Additionally, the AI assistant would learn about the needs of clinic operators and patients, eliminating the need for manual software adjustments.

Supported with up to $25 million, the U-M-directed AI team unites a diverse group of hospital experts, rural clinicians, and engineers, divided into three subteams. Collectively, they represent eight universities along with the research and development firm RTX BBN Technologies. The AI assistant will form one element of ARPA-H’s five-part initiative aimed at prototyping the equipment necessary for this mobile healthcare clinic.

Other components seek to connect various data sources within the clinic, those worn by the patient, and in the patient’s medical records; create a compact CT scanner for mobile 3D imaging; and develop a prototype mobile clinic. Eventually, the team will evaluate the AI assistant within the mobile clinic, anticipated in the third year. Until then, a stationary clinic, designed akin to the proposed van, will be utilized to determine how effectively the assistant addresses the needs of patients and clinicians.

The clinic AI evolves from prior work led by Corso, where AI agents were developed to offer intelligent guidance across diverse situations. He noted that cooking serves as an excellent testing ground as it involves a combination of raw ingredients and tools, a series of actions, and proficient techniques—all in a relatively low-stakes environment. His team subsequently expanded on the strategies necessary to enhance a person’s culinary skills, guiding soldiers through critical battlefield medical procedures. In the current endeavor, the team envisions the AI assistant collaborating with a family doctor or nurse practitioner—someone well-versed in foundational knowledge but lacking the specialized training or expertise.

The team segments the work into numerous aspects. The technical team, proficient in computer science, will create models capable of representing medical tasks, activities occurring within the van and with the patient, and the status of both the patient and the general practitioner. The aim is to develop an AI assistant that can not merely observe the generalist’s actions and guide them through unfamiliar tasks but also detect when something unexpected arises and adapt accordingly.

Part of this capability would include recognizing the emotional state of the individuals involved, such as the generalist experiencing stress if the patient’s condition deteriorates significantly. Collaborators in nursing will contribute their expertise in interpreting human behavior and adjusting the guidance provided. With their insights, the AI assistant may learn to modify its communication style in high-pressure situations.

The medical team and system integration team will compile the dataset required to train the models powering the AI. This task involves evaluating biases present in the data that could result in inaccurate diagnoses and treatments. The medical team will also supply comprehensive guidance on executing medical tasks in areas such as cardiac and trauma care.

The systems integration and technical teams will construct the prototype AI assistant, known as VIGIL for Vectors of Intelligent Guidance in Long-Reach Rural Healthcare. Following this, the systems integration and medical teams will iteratively refine it through testing in clinical environments, addressing pain points identified by healthcare professionals and patients.

The technical team comprises:

• U-M: Corso (PI; physical AI), Anhong Guo (interactive systems), Andrew Owens (multimodal AI), Mike Oelke (project management)
• Colorado State University: Nathaniel Blanchard (computer vision), Nikhil Krishnaswamy (AI language), Sarath Sreedharan (AI physical systems), Bruce Draper (computer vision)
• Stevens Institute of Technology: Enrique Dunn (3D environment modeling)
• Northeastern University: Ehsan Elhamifar (task and recovery monitoring)
• University of Pennsylvania: Alison Marie Pouch (medical image analysis)
• Purdue University: Jeffrey Siskind (AI physical systems)
• University of Rochester: Chenliang Xu (multimodal AI)

The medical team includes:

• U-M: Prashant Mahajan and Kevin Ward, Milisa Manojlovich (nursing), Donald Likosky (outcomes assessment), Francis Pagani (cardiac surgery)
• University of Pennsylvania: Pouch (medical image analysis), Emily Mackay (cardiac anesthesia)
• Central Michigan University: Sethu Reddy (endocrinology), Alison Arnold (rural medicine), Steve Vance (rural medicine)

The systems team consists of:

• RTX BBN: Brian van Voorst (lead scientist and research fellow), Muntaha Samad (principal research engineer)

Guo serves as an assistant professor of computer science and engineering; Owens holds the position of assistant professor in electrical and computer engineering; Ward and Mahajan are professors of emergency medicine; Manojlovich is a professor in systems, populations, and leadership; Likosky and Pagani are professors of cardiac surgery.