Add model card.

README.md

@@ -1,3 +1,153 @@
 ---
-license: apache-2.0
+language: 
+  - en
+  - ja
+  - pt
+  - es
+  - ko
+  - ar
+  - tr
+  - th
+  - fr
+  - id
+  - ru
+  - de
+  - fa
+  - it
+  - zh
+  - pl
+  - hi
+  - ur
+  - nl
+  - el
+  - ms
+  - ca
+  - sr
+  - sv
+  - uk
+  - he
+  - fi
+  - cs
+  - ta
+  - ne
+  - vi
+  - hu
+  - eo
+  - bn
+  - mr
+  - ml
+  - hr
+  - no
+  - sw
+  - sl
+  - te
+  - az
+  - da
+  - ro
+  - gl
+  - gu
+  - ps
+  - mk
+  - kn
+  - bg
+  - lv
+  - eu
+  - pa
+  - et
+  - mn
+  - sq
+  - si
+  - sd
+  - la
+  - is
+  - jv
+  - lt
+  - ku
+  - am
+  - bs
+  - hy
+  - or
+  - sk
+  - uz
+  - cy
+  - my
+  - su
+  - br
+  - as
+  - af
+  - be
+  - fy
+  - kk
+  - ga
+  - lo
+  - ka
+  - km
+  - sa
+  - mg
+  - so
+  - ug
+  - ky
+  - gd
+  - yi
+tags:
+  - Twitter
+  - Multilingual
+license: "apache-2.0"
 ---
+
+# TwHIN-BERT: A Socially-Enriched Pre-trained Language Model for Multilingual Tweet Representations
+[![PRs Welcome](https://img.shields.io/badge/PRs-welcome-green.svg?style=flat-square)](http://makeapullrequest.com)
+[![arXiv](https://img.shields.io/badge/arXiv-2203.15827-b31b1b.svg)](https://arxiv.org/abs/2209.07562)
+
+
+This repo contains models, code and pointers to datasets from our paper: [TwHIN-BERT: A Socially-Enriched Pre-trained Language Model for Multilingual Tweet Representations](https://arxiv.org/abs/2209.07562).
+[[PDF]](https://arxiv.org/pdf/2209.07562.pdf)
+[[HuggingFace Models]](https://huggingface.co/Twitter)
+
+### Overview
+TwHIN-BERT is a new multi-lingual Tweet language model that is trained on 7 billion Tweets from over 100 distinct languages. TwHIN-BERT differs from prior pre-trained language models as it is trained with not only text-based self-supervision (e.g., MLM), but also with a social objective based on the rich social engagements within a Twitter Heterogeneous Information Network (TwHIN).
+
+TwHIN-BERT can be used as a drop-in replacement for BERT in a variety of NLP and recommendation tasks. It not only outperforms similar models semantic understanding tasks such text classification), but also **social recommendation **tasks such as predicting user to Tweet engagement.
+
+## 1. Pretrained Models
+
+We initially release two pretrained TwHIN-BERT models (base and large) that are compatible wit the [HuggingFace BERT models](https://github.com/huggingface/transformers).
+
+
+| Model | Size | Download Link (🤗 HuggingFace) |
+| ------------- | ------------- | --------- |
+| TwHIN-BERT-base   | 280M parameters | [Twitter/TwHIN-BERT-base](https://huggingface.co/Twitter/twhin-bert-base) |
+| TwHIN-BERT-large  | 550M parameters | [Twitter/TwHIN-BERT-large](https://huggingface.co/Twitter/twhin-bert-large) |
+
+
+To use these models in 🤗 Transformers:
+```python
+from transformers import AutoTokenizer, AutoModel
+tokenizer = AutoTokenizer.from_pretrained('Twitter/twhin-bert-large')
+model = AutoModel.from_pretrained('Twitter/twhin-bert-large')
+inputs = tokenizer("I'm using TwHIN-BERT! #TwHIN-BERT #NLP", return_tensors="pt")
+outputs = model(**inputs)
+```
+
+
+
+<!-- ## 2. Set up environment and data
+### Environment
+TBD
+
+
+## 3. Fine-tune TwHIN-BERT
+
+TBD -->
+
+
+## Citation
+If you use TwHIN-BERT or out datasets in your work, please cite, please cite the following:
+```bib
+@article{zhang2022twhin,
+  title={TwHIN-BERT: A Socially-Enriched Pre-trained Language Model for Multilingual Tweet Representations},
+  author={Zhang, Xinyang and Malkov, Yury and Florez, Omar and Park, Serim and McWilliams, Brian and Han, Jiawei and El-Kishky, Ahmed},
+  journal={arXiv preprint arXiv:2209.07562},
+  year={2022}
+}
+```