Research

Multimedia Analytics Lab Amsterdam (MultiX) is a research group within the Informatics Institute at the University of Amsterdam. As the amount of multimedia data on the internet is growing, it is getting more difficult to get access to all that information. MultiX develops artificial intelligence (AI) techniques that help people understand large collections of multimedia data. Multimedia data can be imagery, text, video, graphs, but also other informational context like geocoordinates. The predominant question the group tries to answer: How can you bring together all this information in a way that users get a better understanding of it?

The researchers work with three main themes. Multimedia integration is the first one. Images, texts, videos; they all provide a different type of information. How do you combine them in a proper way? The second theme revolves around the interaction of users and how to make use of that interaction. How can you improve machine intelligence by learning from the user? The last theme is multimedia visualization. By developing effective interfaces researchers present information in a way the user can interact with. A great example is the ArtSight demo, a large scale dataset where the user can browse multiple art collections around the world, navigating by colour, genre, type and objects. Or the Multimedia Pivot Table, a generic multimedia analytics tool for analyzing large collections of images.

Applications can be found within the fields of health, forensics, law enforcement, cultural heritage, urban liveability, and social media analysis. For each partner they work with, MultiX tries to extract the richest information possible from large collections of data. In collaboration with the Netherlands Forensic Institute for example, the group works on the analysis of deep fake videos. And for the City of Amsterdam they conduct research into the automatic detection of waste in the streets.

Facts & figures

While other research groups traditionally separate the work on computer vision, text and visualization, MultiX brings the research on multimedia together in a unique way. This distinguishes them in the Netherlands and beyond. About a dozen PhD students and post-docs work in the group and it plans to grow in the coming years.

The group received several European funds in recent years like the Interreg programme funding and the Marie Curie Fellowship. Other prominent funds come from the two ICAI labs that are a part of the research group. ICAI (Innovation Center for Artificial Intelligence) is a national network that stimulates AI technology and talent development between academia, industry and government.

Partnership & collaborations 

Industry partner of the AI for Medical Imaging (AIM) Lab, one of the ICAI labs MultiX participates in, is the Inception Institute of Artificial Intelligence (IIAI) from the United Arab Emirates. This lab looks at medical image recognition using AI, but also at the combination of images and electronic health records. In the second ICAI lab, the Police Lab AI, the group collaborates with the Dutch Police and Utrecht University. The researchers develop state-of-the-art AI techniques to improve the safety in the Netherlands in a socially, legally and ethically responsible way.

Within ASGARD, a large European project on law enforcement, the group worked on new techniques like hypergraphs. Hypergraphs allow you to model group relations instead of only binary relations. This technology, which the group will continue to develop, has a great advantage in gaining insight into such diverse applications as social network analysis or fraud detection.

Future mission 

Due to the AI revival ten years ago, automated techniques have become incredibly good in the last few years. At some point researchers will start to see the limits of what automatic AI techniques can do. Breakthroughs will then occur into integrating information rather than looking at individual data streams. Interactions with humans will also start to play a more important role. How can you combine machine intelligence with humans to create synergy? That way AI can start learning from the user.

MultiX positions itself in the research theme AI with clear links to Data Science and People, Society & Technology.