Emoticon-based video search

3 November 2015

Researchers Spencer Cappallo, Thomas Mensink and Cees Snoek of the University of Amsterdam and Qualcomm Research Netherlands developed a search engine that allows a search through YouTube videos with the use of emoticons. Instead of words, users enter one or more emoticons to start their search. This is not ‘just funny’, according to Snoek, but also the most practical way when using a small screen, such as a smart phone or watch.

Visitors to of the site Emoji2Video can make a selection out of 385 emoticons. These small-sized pictograms of things, people, animals and emotions, are a standard feature on many phones, tablets and computers nowadays. Selecting the emoticon of a dancer and a piano will return a list of videos in which both are featured. The search engine runs its search intuitively, regardless which language one speaks.

Deep learning techniques

To make connect emoticons to videos, the researchers applied deep learning techniques. Every fifty frames the images depicted in the video were analysed, and this information was translated into fitting emoticons. Cappallo: ‘The translation of image to emoticon is comprised of two steps. Firstly, the image is translated into known visual concepts. Secondly, we use textual information to calculate the measure in which these concepts are relevant for a specific emoticon. This way, we can translate a video to emoji without the need for specific videos for every emoji.’

The search engine is still firmly in a development phase. Currently, only 45,000 videos are available; a fraction of the videos on YouTube. Nonetheless, the tool has a clear use for facilitating video searches on small devices, as well as by those who cannot (yet) read or write well.

Published by  Faculty of Science