saillab/alpaca-turkish-cleaned
Viewer
β’
Updated
β’
52k
β’
35
β’
1
Selim Γzten PRO
comarproject
AI & ML interests
None yet
Recent Activity
liked
a dataset
about 2 hours ago
saillab/alpaca-turkish-cleaned
liked
a model
16 days ago
answerdotai/ModernBERT-base
Organizations
comarproject's activity
liked
a
dataset
about 2 hours ago
liked
a
model
16 days ago
reacted to
m-ric's
post with π₯π
16 days ago
Post
1966
After 6 years, BERT, the workhorse of encoder models, finally gets a replacement: πͺπ²πΉπ°πΌπΊπ² π πΌπ±π²πΏπ»πππ₯π§! π€
We talk a lot about β¨Generative AIβ¨, meaning "Decoder version of the Transformers architecture", but this is only one of the ways to build LLMs: encoder models, that turn a sentence in a vector, are maybe even more widely used in industry than generative models.
The workhorse for this category has been BERT since its release in 2018 (that's prehistory for LLMs).
It's not a fancy 100B parameters supermodel (just a few hundred millions), but it's an excellent workhorse, kind of a Honda Civic for LLMs.
Many applications use BERT-family models - the top models in this category cumulate millions of downloads on the Hub.
β‘οΈ Now a collaboration between Answer.AI and LightOn just introduced BERT's replacement: ModernBERT.
π§π;ππ₯:
ποΈ Architecture changes:
β First, standard modernizations:
- Rotary positional embeddings (RoPE)
- Replace GeLU with GeGLU,
- Use Flash Attention 2
β¨ The team also introduced innovative techniques like alternating attention instead of full attention, and sequence packing to get rid of padding overhead.
π₯ As a result, the model tops the game of encoder models:
It beats previous standard DeBERTaV3 for 1/5th the memory footprint, and runs 4x faster!
Read the blog post π https://huggingface.co/blog/modernbert
We talk a lot about β¨Generative AIβ¨, meaning "Decoder version of the Transformers architecture", but this is only one of the ways to build LLMs: encoder models, that turn a sentence in a vector, are maybe even more widely used in industry than generative models.
The workhorse for this category has been BERT since its release in 2018 (that's prehistory for LLMs).
It's not a fancy 100B parameters supermodel (just a few hundred millions), but it's an excellent workhorse, kind of a Honda Civic for LLMs.
Many applications use BERT-family models - the top models in this category cumulate millions of downloads on the Hub.
β‘οΈ Now a collaboration between Answer.AI and LightOn just introduced BERT's replacement: ModernBERT.
π§π;ππ₯:
ποΈ Architecture changes:
β First, standard modernizations:
- Rotary positional embeddings (RoPE)
- Replace GeLU with GeGLU,
- Use Flash Attention 2
β¨ The team also introduced innovative techniques like alternating attention instead of full attention, and sequence packing to get rid of padding overhead.
π₯ As a result, the model tops the game of encoder models:
It beats previous standard DeBERTaV3 for 1/5th the memory footprint, and runs 4x faster!
Read the blog post π https://huggingface.co/blog/modernbert
- 1 reply
liked
a
model
18 days ago
liked
a
model
19 days ago
liked
a
Space
21 days ago
liked
a
model
26 days ago
liked
a
Space
27 days ago
liked
a
dataset
28 days ago
liked
3
models
28 days ago
liked
a
Space
28 days ago
liked
a
dataset
29 days ago
liked
a
Space
about 1 month ago
liked
4
models
about 1 month ago
