kyegomez/MultiModalCrossAttn
The open source implementation of the cross attention mechanism from the paper: "JOINTLY TRAINING LARGE AUTOREGRESSIVE MULTIMODAL MODELS"
This project offers a pre-built component for developers working on advanced AI models that combine different types of data, like text and images. It provides the core mechanism to allow these distinct data streams to 'understand' each other. Developers can integrate this to create AI that processes information more holistically.
No commits in the last 6 months.
Use this if you are an AI/ML engineer or researcher building large autoregressive models that need to process and integrate information from multiple modalities, such as text and images, to generate unified outputs.
Not ideal if you are looking for an end-user application or a complete multimodal AI system, as this is a foundational building block for developers.
Stars
37
Forks
1
Language
Python
License
MIT
Category
Last pushed
Mar 11, 2024
Commits (30d)
0
Get this data via API
curl "https://pt-edge.onrender.com/api/v1/quality/transformers/kyegomez/MultiModalCrossAttn"
Open to everyone — 100 requests/day, no key needed. Get a free key for 1,000/day.
Higher-rated alternatives
kyegomez/RT-X
Pytorch implementation of the models RT-1-X and RT-2-X from the paper: "Open X-Embodiment:...
kyegomez/PALI3
Implementation of PALI3 from the paper PALI-3 VISION LANGUAGE MODELS: SMALLER, FASTER, STRONGER"
chuanyangjin/MMToM-QA
[🏆Outstanding Paper Award at ACL 2024] MMToM-QA: Multimodal Theory of Mind Question Answering
lyuchenyang/Macaw-LLM
Macaw-LLM: Multi-Modal Language Modeling with Image, Video, Audio, and Text Integration
Muennighoff/vilio
🥶Vilio: State-of-the-art VL models in PyTorch & PaddlePaddle