Towards Best Practices for Open Datasets for LLM Training
Paper
•
2501.08365
•
Published
•
44
AI & ML interests
None defined yet.
Recent Activity
View all activity
yandex's activity
mryab
authored
a
paper
5 days ago
mryab
authored
a
paper
2 months ago
puhsu
authored
a
paper
7 months ago
artnitolog
posted
an
update
7 months ago
Post
2541
Recently, we open-sourced YaFSDP, Yandex’s tool for efficient distributed training of LLMs.
Here are some of the key ideas used in YaFSDP to provide speedup and memory savings over FSDP:
• Allocate and utilize just two buffers throughout the transformer for all collected weights to circumvent the torch memory allocator;
• Gather small normalization layers at the beginning of the iteration and average the gradients only at the end;
• Move gradient division to the very end of the backward pass.
To learn more about how YaFSDP works, check out our latest blog post: https://medium.com/yandex/yafsdp-a-tool-for-faster-llm-training-and-optimized-gpu-utilization-is-no-632b7539f5b3
Here are some of the key ideas used in YaFSDP to provide speedup and memory savings over FSDP:
• Allocate and utilize just two buffers throughout the transformer for all collected weights to circumvent the torch memory allocator;
• Gather small normalization layers at the beginning of the iteration and average the gradients only at the end;
• Move gradient division to the very end of the backward pass.
To learn more about how YaFSDP works, check out our latest blog post: https://medium.com/yandex/yafsdp-a-tool-for-faster-llm-training-and-optimized-gpu-utilization-is-no-632b7539f5b3
mryab
authored
a
paper
7 months ago
artnitolog
posted
an
update
7 months ago
Post
2082
Today we are introducing YaFSDP, Yandex’s tool for efficient distributed LLM training. YaFSDP can be used in conjunction with huggingface workflows and is up to 25% faster compared to FSDP.
Learn more here: https://github.com/yandex/YaFSDP
Learn more here: https://github.com/yandex/YaFSDP
mryab
authored
5
papers
9 months ago
BLOOM: A 176B-Parameter Open-Access Multilingual Language Model
Paper
•
2211.05100
•
Published
•
28
RuCoLA: Russian Corpus of Linguistic Acceptability
Paper
•
2210.12814
•
Published
•
1
Petals: Collaborative Inference and Fine-tuning of Large Models
Paper
•
2209.01188
•
Published
•
2
Sequoia: Scalable, Robust, and Hardware-aware Speculative Decoding
Paper
•
2402.12374
•
Published
•
3
The Hallucinations Leaderboard -- An Open Effort to Measure Hallucinations in Large Language Models
Paper
•
2404.05904
•
Published
•
8
mryab
authored
a
paper
11 months ago
mryab
authored
2
papers
about 1 year ago
mryab
authored
a
paper
over 1 year ago
puhsu
authored
2
papers
over 1 year ago
mryab
authored
3
papers
over 1 year ago
FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPU
Paper
•
2303.06865
•
Published
•
1
SWARM Parallelism: Training Large Models Can Be Surprisingly Communication-Efficient
Paper
•
2301.11913
•
Published
•
1
Distributed Deep Learning in Open Collaborations
Paper
•
2106.10207
•
Published
•
2