GRID-6X : Layout for Seamless Image Assembly
โข
7
Prithiv Sakthi
prithivMLmods
AI & ML interests
computer vision, multimodality, realism engine adapters @starngerzonehf
Recent Activity
new activity
about 13 hours ago
prithivMLmods/Deepthink-Reasoning:Since 1 is greater than 0, 9.11 is greater than 9.9.
updated
a model
about 18 hours ago
prithivMLmods/Qwen2-VL-OCR-2B-Instruct
Articles
Organizations
prithivMLmods's activity
reacted to
clem's
post with ๐ค
about 15 hours ago
Post
1024
Cool to see
@ylecun
joining the top 10 of most followed on HF!
(and leaderboard by @mvaloatto is here: mvaloatto/TCTF)
(and leaderboard by @mvaloatto is here: mvaloatto/TCTF)
- 1 reply
reacted to
Xenova's
post with ๐ฅ
2 days ago
Post
4159
First project of 2025: Vision Transformer Explorer
I built a web app to interactively explore the self-attention maps produced by ViTs. This explains what the model is focusing on when making predictions, and provides insights into its inner workings! ๐คฏ
Try it out yourself! ๐
webml-community/attention-visualization
Source code: https://github.com/huggingface/transformers.js-examples/tree/main/attention-visualization
I built a web app to interactively explore the self-attention maps produced by ViTs. This explains what the model is focusing on when making predictions, and provides insights into its inner workings! ๐คฏ
Try it out yourself! ๐
webml-community/attention-visualization
Source code: https://github.com/huggingface/transformers.js-examples/tree/main/attention-visualization
posted
an
update
3 days ago
Post
2770
Triangulum Catalogued ๐ฅ๐ซ
๐ฏTriangulum is a collection of pretrained and instruction-tuned generative models, designed for multilingual applications. These models are trained using synthetic datasets based on long chains of thought, enabling them to perform complex reasoning tasks effectively.
+ Triangulum-10B : prithivMLmods/Triangulum-10B
+ Quants : prithivMLmods/Triangulum-10B-GGUF
+ Triangulum-5B : prithivMLmods/Triangulum-5B
+ Quants : prithivMLmods/Triangulum-5B-GGUF
+ Triangulum-1B : prithivMLmods/Triangulum-1B
+ Quants : prithivMLmods/Triangulum-1B-GGUF
๐ฏTriangulum is a collection of pretrained and instruction-tuned generative models, designed for multilingual applications. These models are trained using synthetic datasets based on long chains of thought, enabling them to perform complex reasoning tasks effectively.
+ Triangulum-10B : prithivMLmods/Triangulum-10B
+ Quants : prithivMLmods/Triangulum-10B-GGUF
+ Triangulum-5B : prithivMLmods/Triangulum-5B
+ Quants : prithivMLmods/Triangulum-5B-GGUF
+ Triangulum-1B : prithivMLmods/Triangulum-1B
+ Quants : prithivMLmods/Triangulum-1B-GGUF
- 1 reply
reacted to
merve's
post with ๐
3 days ago
Post
3722
supercharge your LLM apps with smolagents ๐ฅ
however cool your LLM is, without being agentic it can only go so far
enter smolagents: a new agent library by Hugging Face to make the LLM write code, do analysis and automate boring stuff!
Here's our blog for you to get started https://huggingface.co/blog/smolagents
however cool your LLM is, without being agentic it can only go so far
enter smolagents: a new agent library by Hugging Face to make the LLM write code, do analysis and automate boring stuff!
Here's our blog for you to get started https://huggingface.co/blog/smolagents
reacted to
davidberenstein1957's
post with ๐ง
5 days ago
Post
1838
Fine-tuning ModernBERT for text classification using synthetic data generation
From prompt to model in 3 steps.
1 dataset description
20 minutes of generating
60 minutes of fine-tuning on my Macbook Pro
Tutorial: https://nbsanity.com/static/552eb50cbd91bedb4e5b73fddca2664a/fine-tune-modernbert-classifier.html
From prompt to model in 3 steps.
1 dataset description
20 minutes of generating
60 minutes of fine-tuning on my Macbook Pro
Tutorial: https://nbsanity.com/static/552eb50cbd91bedb4e5b73fddca2664a/fine-tune-modernbert-classifier.html
reacted to
nyuuzyou's
post with ๐
6 days ago
Post
2120
๐จ KLING AI Dataset -
nyuuzyou/klingai
A collection of 12,782 AI-generated media items featuring:
- High-quality image and video generations at various resolutions
- Complete metadata including user IDs, prompts, and generation parameters
- Content generated using text-to-image, text-to-video, and image-to-video modalities
- Full generation settings and technical parameters
A collection of 12,782 AI-generated media items featuring:
- High-quality image and video generations at various resolutions
- Complete metadata including user IDs, prompts, and generation parameters
- Content generated using text-to-image, text-to-video, and image-to-video modalities
- Full generation settings and technical parameters
reacted to
nicolay-r's
post with โค๏ธ
7 days ago
Post
2066
๐ข Deligted to share the most recent milestone on quick deployment of Named Entity Recognition (NER) in Gen-AI powered systems.
Releasing the bulk-ner 0.25.0 which represent a tiny framework that would save you time for deploing NER with any model.
๐ Why is this important? In the era of GenAI the handling out textual output might be challenging. Instead, recognizing named-entities via domain-oriented systems for your donwstream LLM would be preferable option.
๐ฆ: https://pypi.org/project/bulk-ner/0.25.0/
๐: https://github.com/nicolay-r/bulk-ner
I noticed that the direct adaptaion of the LM for NER would result in spending signifcant amount of time on formatting your texts according to the NER-model needs.
In particular:
1. Processing CONLL format with B-I-O tags from model outputs
2. Input trimming: long input content might not be completely fitted
To cope with these problems, in version 0.25.0 I made a huge steps forward by providing:
โ ๐ Python API support: see screenshot below for a quick deployment (see screenshot below ๐ธ)
โ ๐ชถ No-string: dependencies are now clear, so it is purely Python implementation for API calls.
โ ๐ Simplified output formatting: we use lists to represent texts with inner lists that refer to annotated objects (see screenshot below ๐ธ)
๐ We have a colab for a quick start here (or screenshot for bash / Python API ๐ธ)
https://colab.research.google.com/github/nicolay-r/ner-service/blob/main/NER_annotation_service.ipynb
๐ The code for pipeline deployment is taken from the AREkit project:
https://github.com/nicolay-r/AREkit
Releasing the bulk-ner 0.25.0 which represent a tiny framework that would save you time for deploing NER with any model.
๐ Why is this important? In the era of GenAI the handling out textual output might be challenging. Instead, recognizing named-entities via domain-oriented systems for your donwstream LLM would be preferable option.
๐ฆ: https://pypi.org/project/bulk-ner/0.25.0/
๐: https://github.com/nicolay-r/bulk-ner
I noticed that the direct adaptaion of the LM for NER would result in spending signifcant amount of time on formatting your texts according to the NER-model needs.
In particular:
1. Processing CONLL format with B-I-O tags from model outputs
2. Input trimming: long input content might not be completely fitted
To cope with these problems, in version 0.25.0 I made a huge steps forward by providing:
โ ๐ Python API support: see screenshot below for a quick deployment (see screenshot below ๐ธ)
โ ๐ชถ No-string: dependencies are now clear, so it is purely Python implementation for API calls.
โ ๐ Simplified output formatting: we use lists to represent texts with inner lists that refer to annotated objects (see screenshot below ๐ธ)
๐ We have a colab for a quick start here (or screenshot for bash / Python API ๐ธ)
https://colab.research.google.com/github/nicolay-r/ner-service/blob/main/NER_annotation_service.ipynb
๐ The code for pipeline deployment is taken from the AREkit project:
https://github.com/nicolay-r/AREkit
reacted to
AdinaY's
post with ๐
8 days ago
Post
3472
The Chinese community is shipping ๐ข
DeepSeek V3 (685 B MoE) has quietly released on the hub!
Base: deepseek-ai/DeepSeek-V3-Base
Instruct: deepseek-ai/DeepSeek-V3
Canโt wait to see whatโs next!
DeepSeek V3 (685 B MoE) has quietly released on the hub!
Base: deepseek-ai/DeepSeek-V3-Base
Instruct: deepseek-ai/DeepSeek-V3
Canโt wait to see whatโs next!
-
- 1 reply
reacted to
AdinaY's
post with ๐
9 days ago
Post
2883
QvQ-72B-Preview๐ an open weight model for visual reasoning just released by Alibaba_Qwen team
Qwen/qvq-676448c820912236342b9888
โจ Combines visual understanding & language reasoning.
โจ Scores 70.3 on MMMU
โจ Outperforms Qwen2-VL-72B-Instruct in complex problem-solving
Qwen/qvq-676448c820912236342b9888
โจ Combines visual understanding & language reasoning.
โจ Scores 70.3 on MMMU
โจ Outperforms Qwen2-VL-72B-Instruct in complex problem-solving
reacted to
merve's
post with ๐ฅ
10 days ago
Post
4152
QwQ can see ๐ฅ
Qwen team released QvQ, a large vision LM with reasoning ๐ฑ
it outperforms proprietary VLMs on several benchmarks, comes with open weights and a demo!
Check them out โฌ๏ธ
Demo Qwen/QVQ-72B-preview
Model Qwen/QVQ-72B-Preview
Read more https://qwenlm.github.io/blog/qvq-72b-preview/
Congratulations @JustinLin610 and team!
Qwen team released QvQ, a large vision LM with reasoning ๐ฑ
it outperforms proprietary VLMs on several benchmarks, comes with open weights and a demo!
Check them out โฌ๏ธ
Demo Qwen/QVQ-72B-preview
Model Qwen/QVQ-72B-Preview
Read more https://qwenlm.github.io/blog/qvq-72b-preview/
Congratulations @JustinLin610 and team!
-
- 2 replies
reacted to
sayakpaul's
post with ๐ฅ
11 days ago
Post
3745
Commits speak louder than words ๐คช
* 4 new video models
* Multiple image models, including SANA & Flux Control
* New quantizers -> GGUF & TorchAO
* New training scripts
Enjoy this holiday-special Diffusers release ๐ค
Notes: https://github.com/huggingface/diffusers/releases/tag/v0.32.0
* 4 new video models
* Multiple image models, including SANA & Flux Control
* New quantizers -> GGUF & TorchAO
* New training scripts
Enjoy this holiday-special Diffusers release ๐ค
Notes: https://github.com/huggingface/diffusers/releases/tag/v0.32.0
reacted to
nicolay-r's
post with ๐ง
12 days ago
Post
2250
๐ข So far I noticed that ๐ง reasoning with llm ๐ค in English is tend to be more accurate than in other languages.
However, besides the GoogleTrans and other open transparent translators, I could not find one that could be easy to use solutions to avoid:
1.๐ด Third-party framework installation
2.๐ด Text chunking
3.๐ด support of meta-annotation like spans / objects / etc.
๐ To cope problem of IR from non-english texts, I am happy to share the bulk-translate 0.25.0. ๐
โญ https://github.com/nicolay-r/bulk-translate
bulk-translate is a tiny Python ๐ no-string framework that allows translate series of texts with the pre-annotated fixed-spans that are invariant for translator.
It supports ๐จโ๐ป API for quick data translation with (optionaly) annotated objects in texts (see figure below) in Python ๐
I make it accessible as much as possible for RAG and / or LLM-powered app downstreams:
๐ https://github.com/nicolay-r/bulk-translate/wiki
All you have to do is to provide iterator of texts, where each text:
1. โ String object
2. โ List of strings and nested lists that represent spans (value + any ID data).
๐ค By default I provide a wrapper over googletrans which you can override with your own ๐ฅ
https://github.com/nicolay-r/bulk-translate/blob/master/models/googletrans_310a.py
However, besides the GoogleTrans and other open transparent translators, I could not find one that could be easy to use solutions to avoid:
1.๐ด Third-party framework installation
2.๐ด Text chunking
3.๐ด support of meta-annotation like spans / objects / etc.
๐ To cope problem of IR from non-english texts, I am happy to share the bulk-translate 0.25.0. ๐
โญ https://github.com/nicolay-r/bulk-translate
bulk-translate is a tiny Python ๐ no-string framework that allows translate series of texts with the pre-annotated fixed-spans that are invariant for translator.
It supports ๐จโ๐ป API for quick data translation with (optionaly) annotated objects in texts (see figure below) in Python ๐
I make it accessible as much as possible for RAG and / or LLM-powered app downstreams:
๐ https://github.com/nicolay-r/bulk-translate/wiki
All you have to do is to provide iterator of texts, where each text:
1. โ String object
2. โ List of strings and nested lists that represent spans (value + any ID data).
๐ค By default I provide a wrapper over googletrans which you can override with your own ๐ฅ
https://github.com/nicolay-r/bulk-translate/blob/master/models/googletrans_310a.py
posted
an
update
13 days ago
reacted to
Jaward's
post with ๐ฅ
14 days ago
Post
1772
Implements from first-principle a discrete flow matching model for code generation- trained a small sized 2D dfm model on two variations of code for binary search. The result was amazing, code in comment:
Code: https://github.com/Jaykef/ai-algorithms/blob/main/dfm.ipynb
Code: https://github.com/Jaykef/ai-algorithms/blob/main/dfm.ipynb
- 1 reply
reacted to
suayptalha's
post with ๐
15 days ago
Post
1619
๐ FastLlama Series is Live!
๐ฆพ Experience faster, lighter, and smarter language models! The new FastLlama makes Meta's LLaMA models work with smaller file sizes, lower system requirements, and higher performance. The model supports 8 languages, including English, German, and Spanish.
๐ค Built on the LLaMA 3.2-1B-Instruct model, fine-tuned with Hugging Face's SmolTalk and MetaMathQA-50k datasets, and powered by LoRA (Low-Rank Adaptation) for groundbreaking mathematical reasoning.
๐ป Its compact size makes it versatile for a wide range of applications!
๐ฌ Chat with the model:
๐ Chat Link: suayptalha/Chat-with-FastLlama
๐ Model Link: suayptalha/FastLlama-3.2-1B-Instruct
๐ฆพ Experience faster, lighter, and smarter language models! The new FastLlama makes Meta's LLaMA models work with smaller file sizes, lower system requirements, and higher performance. The model supports 8 languages, including English, German, and Spanish.
๐ค Built on the LLaMA 3.2-1B-Instruct model, fine-tuned with Hugging Face's SmolTalk and MetaMathQA-50k datasets, and powered by LoRA (Low-Rank Adaptation) for groundbreaking mathematical reasoning.
๐ป Its compact size makes it versatile for a wide range of applications!
๐ฌ Chat with the model:
๐ Chat Link: suayptalha/Chat-with-FastLlama
๐ Model Link: suayptalha/FastLlama-3.2-1B-Instruct
reacted to
m-ric's
post with ๐ฅ
15 days ago
Post
1895
After 6 years, BERT, the workhorse of encoder models, finally gets a replacement: ๐ช๐ฒ๐น๐ฐ๐ผ๐บ๐ฒ ๐ ๐ผ๐ฑ๐ฒ๐ฟ๐ป๐๐๐ฅ๐ง! ๐ค
We talk a lot about โจGenerative AIโจ, meaning "Decoder version of the Transformers architecture", but this is only one of the ways to build LLMs: encoder models, that turn a sentence in a vector, are maybe even more widely used in industry than generative models.
The workhorse for this category has been BERT since its release in 2018 (that's prehistory for LLMs).
It's not a fancy 100B parameters supermodel (just a few hundred millions), but it's an excellent workhorse, kind of a Honda Civic for LLMs.
Many applications use BERT-family models - the top models in this category cumulate millions of downloads on the Hub.
โก๏ธ Now a collaboration between Answer.AI and LightOn just introduced BERT's replacement: ModernBERT.
๐ง๐;๐๐ฅ:
๐๏ธ Architecture changes:
โ First, standard modernizations:
- Rotary positional embeddings (RoPE)
- Replace GeLU with GeGLU,
- Use Flash Attention 2
โจ The team also introduced innovative techniques like alternating attention instead of full attention, and sequence packing to get rid of padding overhead.
๐ฅ As a result, the model tops the game of encoder models:
It beats previous standard DeBERTaV3 for 1/5th the memory footprint, and runs 4x faster!
Read the blog post ๐ https://huggingface.co/blog/modernbert
We talk a lot about โจGenerative AIโจ, meaning "Decoder version of the Transformers architecture", but this is only one of the ways to build LLMs: encoder models, that turn a sentence in a vector, are maybe even more widely used in industry than generative models.
The workhorse for this category has been BERT since its release in 2018 (that's prehistory for LLMs).
It's not a fancy 100B parameters supermodel (just a few hundred millions), but it's an excellent workhorse, kind of a Honda Civic for LLMs.
Many applications use BERT-family models - the top models in this category cumulate millions of downloads on the Hub.
โก๏ธ Now a collaboration between Answer.AI and LightOn just introduced BERT's replacement: ModernBERT.
๐ง๐;๐๐ฅ:
๐๏ธ Architecture changes:
โ First, standard modernizations:
- Rotary positional embeddings (RoPE)
- Replace GeLU with GeGLU,
- Use Flash Attention 2
โจ The team also introduced innovative techniques like alternating attention instead of full attention, and sequence packing to get rid of padding overhead.
๐ฅ As a result, the model tops the game of encoder models:
It beats previous standard DeBERTaV3 for 1/5th the memory footprint, and runs 4x faster!
Read the blog post ๐ https://huggingface.co/blog/modernbert
- 1 reply
reacted to
anton-l's
post with ๐
15 days ago
Post
2103
Introducing ๐๐
๐ข๐ง๐๐๐๐ญ๐ก: the best public math pre-training dataset with 50B+ tokens!
HuggingFaceTB/finemath
Math remains challenging for LLMs and by training on FineMath we see considerable gains over other math datasets, especially on GSM8K and MATH.
We build the dataset by:
๐ ๏ธ carefully extracting math data from Common Crawl;
๐ iteratively filtering and recalling high quality math pages using a classifier trained on synthetic annotations to identify math reasoning and deduction.
We conducted a series of ablations comparing the performance of Llama-3.2-3B-Base after continued pre-training on FineMath and observe notable gains compared to the baseline model and other public math datasets.
We hope this helps advance the performance of LLMs on math and reasoning! ๐
Weโre also releasing all the ablation models as well as the evaluation code.
HuggingFaceTB/finemath-6763fb8f71b6439b653482c2
HuggingFaceTB/finemath
Math remains challenging for LLMs and by training on FineMath we see considerable gains over other math datasets, especially on GSM8K and MATH.
We build the dataset by:
๐ ๏ธ carefully extracting math data from Common Crawl;
๐ iteratively filtering and recalling high quality math pages using a classifier trained on synthetic annotations to identify math reasoning and deduction.
We conducted a series of ablations comparing the performance of Llama-3.2-3B-Base after continued pre-training on FineMath and observe notable gains compared to the baseline model and other public math datasets.
We hope this helps advance the performance of LLMs on math and reasoning! ๐
Weโre also releasing all the ablation models as well as the evaluation code.
HuggingFaceTB/finemath-6763fb8f71b6439b653482c2
posted
an
update
15 days ago
Post
2464
Qwen2VL Models: Vision and Language Processing ๐
๐FT; [ Latex OCR, Math Parsing, Text Analogy OCRTest ]
Colab Demo: prithivMLmods/Qwen2-VL-OCR-2B-Instruct
โ๏ธDemo : prithivMLmods/Qwen2-VL-2B . The demo includes the Qwen2VL 2B Base Model.
๐ฏThe space handles documenting content from the input image along with standardized plain text. It includes adjustment tools with over 30 font styles, file formatting support for PDF and DOCX, textual alignments, font size adjustments, and line spacing modifications.
๐PDFs are rendered using the ReportLab software library toolkit.
๐งตModels :
+ prithivMLmods/Qwen2-VL-OCR-2B-Instruct
+ prithivMLmods/Qwen2-VL-Ocrtest-2B-Instruct
+ prithivMLmods/Qwen2-VL-Math-Prase-2B-Instruct
๐Sample Document :
+ https://drive.google.com/file/d/1Hfqqzq4Xc-3eTjbz-jcQY84V5E1YM71E/view?usp=sharing
๐ฆCollection :
+ prithivMLmods/vision-language-models-67639f790e806e1f9799979f
.
.
.
@prithivMLmods ๐ค
๐FT; [ Latex OCR, Math Parsing, Text Analogy OCRTest ]
Colab Demo: prithivMLmods/Qwen2-VL-OCR-2B-Instruct
โ๏ธDemo : prithivMLmods/Qwen2-VL-2B . The demo includes the Qwen2VL 2B Base Model.
๐ฏThe space handles documenting content from the input image along with standardized plain text. It includes adjustment tools with over 30 font styles, file formatting support for PDF and DOCX, textual alignments, font size adjustments, and line spacing modifications.
๐PDFs are rendered using the ReportLab software library toolkit.
๐งตModels :
+ prithivMLmods/Qwen2-VL-OCR-2B-Instruct
+ prithivMLmods/Qwen2-VL-Ocrtest-2B-Instruct
+ prithivMLmods/Qwen2-VL-Math-Prase-2B-Instruct
๐Sample Document :
+ https://drive.google.com/file/d/1Hfqqzq4Xc-3eTjbz-jcQY84V5E1YM71E/view?usp=sharing
๐ฆCollection :
+ prithivMLmods/vision-language-models-67639f790e806e1f9799979f
.
.
.
@prithivMLmods ๐ค
- 1 reply
reacted to
m-ric's
post with ๐ง
15 days ago
Post
2280
๐๐ฎ๐ ๐ ๐ข๐ง๐ ๐
๐๐๐ ๐ซ๐๐ฅ๐๐๐ฌ๐๐ฌ ๐๐ข๐๐จ๐ญ๐ซ๐จ๐ง, ๐ ๐ฆ๐ข๐๐ซ๐จ๐ฌ๐๐จ๐ฉ๐ข๐ ๐ฅ๐ข๐ ๐ญ๐ก๐๐ญ ๐ฌ๐จ๐ฅ๐ฏ๐๐ฌ ๐๐๐ ๐ญ๐ซ๐๐ข๐ง๐ข๐ง๐ ๐๐ ๐ฉ๐๐ซ๐๐ฅ๐ฅ๐๐ฅ๐ข๐ณ๐๐ญ๐ข๐จ๐ง ๐ฅณ
๐ฐ๏ธ Llama-3.1-405B took 39 million GPU-hours to train, i.e. about 4.5 thousand years.
๐ด๐ป If they had needed all this time, we would have GPU stories from the time of Pharaoh ๐: "Alas, Lord of Two Lands, the shipment of counting-stones arriving from Cathay was lost to pirates, this shall delay the building of your computing temple by many moons "
๐ ๏ธ But instead, they just parallelized the training on 24k H100s, which made it take just a few months.
This required parallelizing across 4 dimensions: data, tensor, context, pipeline.
And it is infamously hard to do, making for bloated code repos that hold together only by magic.
๐ค ๐๐๐ ๐ป๐ผ๐ ๐๐ฒ ๐ฑ๐ผ๐ป'๐ ๐ป๐ฒ๐ฒ๐ฑ ๐ต๐๐ด๐ฒ ๐ฟ๐ฒ๐ฝ๐ผ๐ ๐ฎ๐ป๐๐บ๐ผ๐ฟ๐ฒ! Instead of building mega-training codes, Hugging Face colleagues cooked in the other direction, towards tiny 4D parallelism libs. A team has built Nanotron, already widely used in industry.
And now a team releases Picotron, a radical approach to code 4D Parallelism in just a few hundred lines of code, a real engineering prowess, making it much easier to understand what's actually happening!
โก ๐๐'๐ ๐๐ถ๐ป๐, ๐๐ฒ๐ ๐ฝ๐ผ๐๐ฒ๐ฟ๐ณ๐๐น:
Counting in MFU (Model FLOPs Utilization, how much the model actually uses all the compute potential), this lib reaches ~50% on SmolLM-1.7B model with 8 H100 GPUs, which is really close to what huge libs would reach. (Caution: the team is leading further benchmarks to verify this)
Go take a look ๐ https://github.com/huggingface/picotron/tree/main/picotron
๐ฐ๏ธ Llama-3.1-405B took 39 million GPU-hours to train, i.e. about 4.5 thousand years.
๐ด๐ป If they had needed all this time, we would have GPU stories from the time of Pharaoh ๐: "Alas, Lord of Two Lands, the shipment of counting-stones arriving from Cathay was lost to pirates, this shall delay the building of your computing temple by many moons "
๐ ๏ธ But instead, they just parallelized the training on 24k H100s, which made it take just a few months.
This required parallelizing across 4 dimensions: data, tensor, context, pipeline.
And it is infamously hard to do, making for bloated code repos that hold together only by magic.
๐ค ๐๐๐ ๐ป๐ผ๐ ๐๐ฒ ๐ฑ๐ผ๐ป'๐ ๐ป๐ฒ๐ฒ๐ฑ ๐ต๐๐ด๐ฒ ๐ฟ๐ฒ๐ฝ๐ผ๐ ๐ฎ๐ป๐๐บ๐ผ๐ฟ๐ฒ! Instead of building mega-training codes, Hugging Face colleagues cooked in the other direction, towards tiny 4D parallelism libs. A team has built Nanotron, already widely used in industry.
And now a team releases Picotron, a radical approach to code 4D Parallelism in just a few hundred lines of code, a real engineering prowess, making it much easier to understand what's actually happening!
โก ๐๐'๐ ๐๐ถ๐ป๐, ๐๐ฒ๐ ๐ฝ๐ผ๐๐ฒ๐ฟ๐ณ๐๐น:
Counting in MFU (Model FLOPs Utilization, how much the model actually uses all the compute potential), this lib reaches ~50% on SmolLM-1.7B model with 8 H100 GPUs, which is really close to what huge libs would reach. (Caution: the team is leading further benchmarks to verify this)
Go take a look ๐ https://github.com/huggingface/picotron/tree/main/picotron
- 1 reply
reacted to
burtenshaw's
post with ๐
15 days ago
Post
2620
People are flexing their end of year stats, so I made this app to show hub stats in a tidy design!
Thanks @Ameeeee and @jfcalvo for the feature from Argilla!
burtenshaw/recap
Thanks @Ameeeee and @jfcalvo for the feature from Argilla!
burtenshaw/recap
- 1 reply