Joëlle Pineau, the new CEO of Facebook AI Research, talks about his rise within the Facebook labs and decodes the latest advances in deep learning models.
JDN. Why did Meta set up a fundamental research laboratory on artificial intelligence?
Gioelle Pineau. The FIERA laboratory (for Facebook AI Research, ed) of Meta’s mission is to push the state of the art of artificial intelligence ever further. Having a fundamental research laboratory, Meta is able to face the challenges of tomorrow in terms of AI. Faced with the dazzling progress in this technological field in the last decade, FAIR relies on an open and transparent research model, also in terms of models and source codes. The goal is to stimulate feedback and exchange of ideas about our work with our various academic and industrial partners in order to move our projects forward quickly.
Pytorch whose development we have entrusted to a foundation (the PyTorch Foundation, ed)which has partly redesigned the way we develop AI, is one of our main achievements.
You just took the helm of FAIR. What have your roles been since you joined Meta in 2017?
I was initially hired to set up the FAIR laboratory in Montreal. That took about two years. In the meantime, I have been appointed co-director of FAIR along with Antoine Bordes, who lives in Paris. My role was to bring our mission to our research teams, the company and the scientific community, the challenge was to make the strategy clear and align the vision to achieve our goals. I had to make sure that the teams had the necessary resources to carry out their projects in terms of budget, calculation capacity … but also define and apply a data strategy so that the use we have of data as part of our scientific approach is in line with the legislation. Finally, my goal was to bring the best researchers in the cutting-edge fields of AI to our teams.
“We have extended in particular to reinforcement learning, robotics or even responsible AI”
In five years we have evolved a lot. Historically, FAIR has primarily focused on deep learning. At the time, we only had about fifty researchers. It is therefore in this area that we had to position ourselves. Over the years, our membership has grown. This allows us today to cover a much wider range of areas. We have especially extended to reinforcement learning, robotics, and even responsible AI.
Did you continue to closely follow some research topics?
I continued to conduct research on interactive dialogue systems and speech processing through language models and reinforcement learning. This is the theoretical core of my historical research areas. I also work a lot on AI in the healthcare sector. This is a process I started long before joining Meta, when I was a professor at McGill University in Montreal, and it continues today. I am still supporting two post-docs on this topic.
You also worked on Meta’s BlenderBot conversational AI. How did you intervene?
A whole team of FAIR researchers carried out the scientific aspect of the project. We have decided to implement BlenderBot on the Internet in the United States. I worked a lot on this last step. The main goal was to make sure the bot behaved sensibly. A strategy has been defined to evaluate the performance of dialogue quality and security in order to obtain reliable AI.
Ultimately, this implementation aimed to provide access to this technology, both to research communities and the general public. It’s a way to popularize our work, but also to understand user behavior. This experience allows us to collect usage data that obviously we do not have in the laboratory and that allow us to improve our research.
Microsoft has developed a large language model with 530 billion parameters. What do you think of the race for these giant AIs in which Meta is also participating?
The community has been wondering about the relevance of these models for ten years. We all tell ourselves that their size must decrease. There are three ingredients to the success of these AIs. The volume of the parameter is one of them. The second refers to the number of GPUs available to train them. We’ve gone from a few dozen to a few hundred and now to a few thousand GPUs running for several months. Third, you need to ingest enough data. If we don’t have a good balance between data volume, computational power and model size, we don’t want sauce.
“We have enough memory to absorb even larger models”
It is not yet known which of these parameters will reach its limit first. We have enough memory to accommodate even larger models. On the side of the training data, however, the room for maneuver is shrinking. This is explained by the strengthening of copyright and privacy rules, as well as by the difficulty of accessing massive amounts of information representative of a population.
What will the escape be?
The future could pass through the emergence of multimodal transformers capable of feeding on heterogeneous data: images, video, language, sounds, medical data … The other great field of research concerns generative models. In this area we have just announced the Make-A-Video technology that produces short videos from texts that describe a scene in motion, in the same logic as Dall-E in terms of still images.
Generative AI will soon be able to produce different data formats, for example 3D virtual reality with intelligent avatars, sensory sound, tactile and olfactory experiences, especially in the context of the metaverse …
Joëlle Pineau was appointed Managing Director of Facebook AI Research (FAIR) on 20 October 2022. She previously co-directed FAIR with Antoine Bordes, who will now focus on leading FAIR EMEA laboratories. Joëlle Pineau is an associate professor at McGill University’s School of Computer Science.