Crucial for numerous sectors from Hollywood’s CGI to product creation, 3D modeling software frequently employs textual or visual prompts to influence various features of aesthetics, including hue and shape. While this serves as a sensible initial interface, these technologies still fall short in realism due to their disregard for an essential facet of human sensation: touch.
Central to the individuality of tangible items are their tactile characteristics, such as texture, irregularities, or the sensation of materials like timber or stone. Current modeling techniques usually necessitate sophisticated CAD skills and seldom provide touch feedback that crucially influences how we perceive and engage with the physical environment.
With this consideration, scholars at MIT’s Computer Science and Artificial Intelligence Laboratory (CSAIL) have developed an innovative system for stylizing 3D models using image prompts, effectively duplicating both visual aesthetics and tactile characteristics.
The “TactStyle” instrument created by the CSAIL team enables artists to stylize 3D models based on visual references while simultaneously integrating the anticipated tactile traits of textures. TactStyle distinguishes between visual and geometric stylization, allowing for the reproduction of both visual and tactile attributes from a solitary image input.
PhD candidate Faraz Faruqi, the primary author of a recent study on the initiative, notes that TactStyle could possess extensive applications, ranging from home decoration and personal items to tools for tactile education. TactStyle gives users the ability to download a baseline design — like a headphone stand from Thingiverse — and personalize it with the styles and textures of their choice. In the realm of education, learners can examine various textures from across the globe without leaving their classrooms, while in product development, rapid prototyping is simplified as designers can quickly print several iterations to enhance tactile features.
“One could envision utilizing such a system for everyday items, like phone holders and earbud cases, to facilitate more intricate textures and amplify tactile feedback in diverse manners,” Faruqi mentions, who co-authored the paper with MIT Associate Professor Stefanie Mueller, the head of the Human-Computer Interaction (HCI) Engineering Group at CSAIL. “It’s possible to craft tactile educational instruments that illustrate a multitude of concepts in disciplines like biology, geometry, and topography.”
Conventional techniques for texture replication generally rely on specialized tactile sensors — for example, GelSight, designed at MIT — which physically contact an object to discern its surface microgeometry as a “heightfield.” However, this necessitates having either a physical object or its recorded surface for duplication. TactStyle enables users to reproduce the surface microgeometry by utilizing generative AI to directly create a heightfield from an image of the texture.
Moreover, on platforms such as the 3D printing repository Thingiverse, it is challenging to take specific designs and tailor them. Indeed, if a user lacks sufficient technical know-how, manually altering a design risks “breaking” it, rendering it unprintable. All these factors led Faruqi to contemplate developing a tool that allows for high-level customization of downloadable models while maintaining functionality.
In trials, TactStyle exhibited notable advancements over traditional stylization techniques by generating precise links between a texture’s visual representation and its heightfield. This permits the direct replication of tactile features from an image. A particular psychophysical study revealed that users perceive the textures produced by TactStyle as congruent with both the anticipated tactile attributes from visual input and the tactile aspects of the original texture, creating a cohesive tactile and visual experience.
TactStyle builds upon an existing approach, known as “Style2Fab,” to adjust the model’s color channels to align with the visual style of the input image. Users initially provide an image of the desired texture, followed by a finely-tuned variational autoencoder that translates the input image into an associated heightfield. This heightfield is then applied to alter the model’s geometry to establish the tactile characteristics.
The color and geometry stylization modules function collaboratively, modifying both the visual and tactile properties of the 3D model from a sole image input. Faruqi asserts that the key innovation resides in the geometry stylization module, which employs a refined diffusion model to generate heightfields from texture images — a capability that previous stylization frameworks have failed to replicate accurately.
Looking forward, Faruqi mentions that the team intends to enhance TactStyle to create novel 3D models using generative AI with integrated textures. This advancement entails exploring the specific pipeline necessary to replicate both shape and function of the 3D models being produced. They also plan to explore “visuo-haptic mismatches” to craft unique experiences with materials that contradict typical expectations, such as an item that appears to be made of marble but feels like wood.
Faruqi and Mueller collaborated on the new paper alongside PhD candidates Maxine Perroni-Scharf and Yunyi Zhu, visiting undergraduate student Jaskaran Singh Walia, visiting master’s student Shuyue Feng, and assistant professor Donald Degraen from the Human Interface Technology (HIT) Lab NZ in New Zealand.