Research

Multimedia Analytics Lab Amsterdam (MultiX) is a research group within the Informatics Institute at the University of Amsterdam. As the amount of multimedia data on the internet is growing, it is getting more difficult to get access to all that information. MultiX develops artificial intelligence (AI) techniques that help people understand large collections of multimedia data. Multimedia data can be imagery, text, video, graphs, but also other informational context like geocoordinates. The predominant question the group tries to answer: How can you bring together all this information in a way that users get a better understanding of it?

The researchers work with three main themes. Multimedia integration is the first one. Images, texts, videos; they all provide a different type of information. How do you combine them in a proper way? The second theme revolves around the interaction of users and how to make use of that interaction. How can you improve machine intelligence by learning from the user? The last theme is multimedia visualization. By developing effective interfaces researchers present information in a way the user can interact with. A great example is the ArtSight demo, a large scale dataset where the user can browse multiple art collections around the world, navigating by colour, genre, type and objects. Or the Multimedia Pivot Table, a generic multimedia analytics tool for analyzing large collections of images.

Applications can be found within the fields of health, forensics, law enforcement, cultural heritage, urban liveability, and social media analysis. For each partner they work with, MultiX tries to extract the richest information possible from large collections of data. In collaboration with the Netherlands Forensic Institute for example, the group works on the analysis of deep fake videos. And for the City of Amsterdam they conduct research into the automatic detection of waste in the streets.

Facts & figures

While other research groups traditionally separate the work on computer vision, text and visualization, MultiX brings the research on multimedia together in a unique way.  This distinguishes them in the Netherlands and beyond.  About a dozen PhD students and post-docs work in the group and it plans to grow in the coming years.

The group received several European funds in recent years like the Marie Curie Fellowship on new form of digital journalism and the ASGARD project, a large project to  develop tools supporting law enforcement.

Other prominent funds are from the labs in ICAI (Innovation Center for Artificial Intelligence), a national network that stimulates AI technology and talent development between academia, industry and government, namely the recently finished AIMLab and the Police Lab AI and the current  AI4Forensics lab. Other current projects are the ClickNL AI4Film project developing novel AI techniques that are tailored to film by learning from analysis, production practice, and theory, the UvA funded AI4Fintech program in which we study anti-money laundering and the NWO project AI4Intelligence (led by MultiX), COMCRIM, and SCRIPT all three targeting tools for supporting law enforcement.

Partnership & collaborations

Industry partner of the AI for Medical Imaging (AIM) Lab, one of the ICAI labs MultiX participates in, is CORE42 from the United Arab Emirates. This lab looked at medical image recognition using AI, but also at the combination of images and electronic health records. In the Police Lab AI, the group collaborated with the Dutch Police and Utrecht University. The researchers have developed state-of-the-art AI techniques to improve the safety in the Netherlands in a socially, legally and ethically responsible way. The AI4Forensis lab is a collaboration with the Netherlands Forensic Institute developing tools for forensic investigation of  various types of data. MultiX is also part of AI4Fintech, the UvA wide program involving FNWI, the Faculty of Law, and the Faculty of Economics and Business.

Future mission 

Due to the AI revival ten years ago, automated techniques have become incredibly good in the last few years. At some point researchers will start to see the limits of what automatic AI techniques can do. Breakthroughs will then occur into integrating information rather than looking at individual data streams. Interactions with humans will also start to play a more important role. How can you combine machine intelligence with humans to create synergy? That way AI can start learning from the user.

MultiX positions itself in the research theme AI with clear links to Data Science and People, Society & Technology.