Meta has unveiled Chameleon, its new multimodal artificial intelligence designed to tackle the growing competition in the field of generative AI. Chameleon stands out for being natively multimodal, seamlessly integrating components from different modalities such as images, text, and code.
According to the paper published by the research team, Chameleon’s architecture allows for outstanding performance in tasks that require a deep understanding of both visual and textual information. Among Chameleon’s notable capabilities are image captioning and visual question answering (VQA), as well as its competitiveness in exclusively textual tasks.
Traditionally, multimodal models are created through a process known as “late fusion”, where the AI system processes different modalities separately and then merges the encodings for inference. However, this approach limits the ability of models to seamlessly integrate information across different modalities.
Chameleon has adopted an architecture of “early fusion based on mixed tokens”, which means that it has been designed from scratch to learn from an interleaved mixture of images, text, and other modalities. This methodology transforms images into discrete tokens, similar to how language models handle words, and uses a unified vocabulary of text, code, and image tokens.
Compared to similar models like Google Gemini, Chameleon offers a more cohesive integration of modalities during content generation, as it does not require specific components for each modality.
The Chameleon training was carried out in two stages, using a vast dataset that includes 4.4 trillion text tokens, image-text pairs, and interleaved text and image sequences. The Chameleon models, with 7,000 and 34,000 billion parameters, were trained for over 5 million hours on 80 GB Nvidia A100 GPUs.
The experiments showed that Chameleon can perform a wide range of text and multimodal tasks with market-leading performance. In VQA and image captioning tests, Chameleon-34B outperformed models like Flamingo, IDEFICS, and Llava-1.5. Additionally, it matched the performance of other models with fewer training examples in context and with smaller models.
Despite the complexity of multimodality, Chameleon remains competitive in text-only tasks, comparable to models like Mixtral 8x7B and Gemini-Pro in logical reasoning and reading comprehension tests. Researchers highlight that Chameleon unlocks new multimodal reasoning and generation capabilities, offering user-preferred results in documents that combine text and images in an interleaved manner.
