Tasty Bits and Bytes

Title:Tasty Bits and Bytes
Institute:University of Twente (HMI)
Place:Enschede The Netherlands
Type:Capita selecta and Research Topics
End date:not present
HMI ContactGijs HuismanMerijn Bruijnes


The current popularity of ICTs that offer augmented or virtual reality experiences, such as Oculus Rift, Google Glass, and Microsoft Hololens, suggests that these technologies will become increasingly more commonplace in our daily lives. With this the question arises of how these mixed reality technologies will be of benefit to us in our day-to-day activities. One such activity that could take advantage of mixed reality technologies is the consumption of food and beverages. Considering the fact that the perception of food is highly multisensory, being not only governed by taste and smell, but to a strong degree by our visual, auditory and tactile senses, mixed reality technologies could be used to enhance our dining experiences. In the Tasty Bits and Bytes project we will explore the use of mixed reality technology to digitally (using visual, auditory, olfactory, and tactile stimuli) enhance the experience of consuming food and beverages.

The setting for these challenges and projects is a mixed reality restaurant table that hosts a variety of technologies to provide a novel food and beverage experience.

Actuated Plate
Imagine eating peas but the peas try to avoid your fork. Build a system that can tilt a plate to move the food around (e.g. Arduino). Build a tracking system to track the location of the food and the fork (e.g. Kinect). Implement the logic necessary to control the plate based on the location of the food and the fork.

Background subtraction + projection mapping
Imagine a yellow cake, “Yummy citron is my favorite”, as you pick it up it turns red, “well, I like strawberry too”. Background subtraction is a CV method to easily detect changes in a scene and with projection mapping you can project on parts of the scene (e.g. the piece of cake). Combining these you can keep projecting on the object while moving it around. The challenge is to project on (parts of) a scene without ‘triggering’ the change detection by background subtraction and what happens if the object changes shape (e.g. because you ate half).

Force Feedback Fun
Imagine scooping at your ice-cream, halfway through you feel resistance, “Ah, the cookie, my favorite”, but the cookie is a lie. Create force feedback cutlery (e.g. use Phantom Premium) and create a compelling illusion of texture in food. Might require tracking of food (e.g. Kinect, webcam).

Parallax 3D mapping
Imagine a tiny chef preparing a meal on your table. Project a compelling 3D-animated scene on a table (e.g. Unity3D) and track the head of the user (e.g. Kinect) to correct the viewpoint to create the illusion of 3D.

Sensing Taste
Imagine a restaurant where the waiter comes after your first byte to season the food to exactly your taste. Use whatever sensors you prefer to build a system that can sense how a person likes a taste (e.g. Kinect, Leap, EEG) and come up with appropriate automatic responses (e.g. change color of lights, automatic salt dispenser, whip the customer).