HKUNLP/efficient-attention
[EVA ICLR'23; LARA ICML'22] Efficient attention mechanisms via control variates, random features, and importance sampling
This project provides advanced attention mechanisms (LARA and EVA) to make image classification and machine translation models run faster and more efficiently. It takes in standard image datasets like ImageNet or text data for translation tasks, and outputs more optimized models. This is for machine learning engineers and researchers who are building and training advanced AI models for vision and language applications.
No commits in the last 6 months.
Use this if you are a machine learning engineer working with vision transformers or language models and need to improve the efficiency and speed of your attention mechanisms during model training and inference.
Not ideal if you are an end-user without a strong background in deep learning model development or if you need a plug-and-play solution for basic image or text processing without delving into model architecture.
Stars
87
Forks
4
Language
Python
License
Apache-2.0
Category
Last pushed
Mar 07, 2023
Commits (30d)
0
Get this data via API
curl "https://pt-edge.onrender.com/api/v1/quality/transformers/HKUNLP/efficient-attention"
Open to everyone — 100 requests/day, no key needed. Get a free key for 1,000/day.
Higher-rated alternatives
microsoft/LoRA
Code for loralib, an implementation of "LoRA: Low-Rank Adaptation of Large Language Models"
jadore801120/attention-is-all-you-need-pytorch
A PyTorch implementation of the Transformer model in "Attention is All You Need".
bhavnicksm/vanilla-transformer-jax
JAX/Flax implimentation of 'Attention Is All You Need' by Vaswani et al....
kyegomez/SparseAttention
Pytorch Implementation of the sparse attention from the paper: "Generating Long Sequences with...
AbdelStark/attnres
Rust implementation of Attention Residuals from MoonshotAI/Kimi