-
Background Large language models (LLMs) have demonstrated substantial proficiency in comprehending and generating human language. However, their capabilities are currently limited to text processing. The real world is inherently multimodal, with the exchange of information through various channels such as vision, language, sound, and touch. There is a vital objective in advancing multimodal systems to enrich LLMs with the ability to handle multimodal perception.
-
Existing Work The conventional approach involves integrating multimodal encoders into the LLM, enabling the processing of information across various modalities and leveraging sophisticated text-processing abilities to produce coherent responses. However, this method poses challenges, including the difficulty of generating multimodal data, the need for large amounts of data to accurately represent high-definition images and high-fidelity audio, and the exponentially increasing computational complexity for processing long sequences.
- Introduced AnyGPT, a token-based any-to-any multimodal language model
-
Challenge 1: Generating and representing multimodal data AnyGPT employs multimodal tokenizers to compress raw multimodal data such as images and audio into discrete semantic tokens, allowing the core LLM to unify tasks like perception, understanding, reasoning, and generation autoregressively at the semantic level. Detokenizers then translate back the discrete representations into the original modal representations at the perceptual level. This discrete representation technique, which filters out high-frequency, modality-specific perceptual information while retaining essential low-frequency semantic information, facilitates stable model training without altering the existing LLM architecture or training paradigms.
-
Challenge 2: Balancing performance and efficiency AnyGPT adopts a two-stage framework for high-fidelity generation encompassing semantic information modeling and perceptual information modeling. Initially, the LLM generates content that has undergone fusion and alignment at the semantic level, and then non-autoregressive models convert the multimodal semantic tokens into high-fidelity multimodal content at the perceptual level. This method strikes a balance between performance and efficiency, enabling AnyGPT to replicate any speaker's voice using just a 3-second speech prompt while significantly reducing the voice sequence length for LLM processing.
-
AnyGPT uses semantic-level SEED tokens decoded into high-quality images by a Diffusion Model for visual language modeling; SoundStorm is employed for acoustic token generation from semantic tokens using a non-autoregressive Masked Language Model trained on the Multilingual LibriSpeech (MLS) dataset, then transformed into raw audio data; for music, Encodec tokens are used to filter out high-frequency details beyond human perception before being reconstructed into high-fidelity audio data. This approach expands the traditional LLM capability to handle various modal interactions. Experimental results reveal that AnyGPT can facilitate any-to-any multimodal conversation and proves that discrete representations effectively and conveniently unify multiple modalities within a language model.
AnyGPT is a multimodal language model architecture that achieves seamless conversion and unified processing across modalities through discrete sequence modeling, delivering the ability to generate from any modality to any other without needing alterations to the current LLM architecture or training paradigms. It efficiently processes and generates high-quality multimodal content, with performance comparable to specialized models.