update readme

README.md

@@ -1,3 +1,220 @@
 ---
+tags:
+- zero-shot-image-classification
+- clip
+library_tag: open_clip
 license: apache-2.0
+library_name: open_clip
+pipeline_tag: zero-shot-image-classification
 ---
+# Model card for CLIP-UniRepLKNet_large-laion5B-s11B-b75K
+
+#  Table of Contents
+1. [Quick Start](#model-details)
+2. [Model Details](#model-details)
+3. [Uses](#uses)
+4. [Training Details](#training-details)
+5. [Evaluation](#evaluation)
+6. [Citation](#citation)
+
+# Quick Start
+```python
+model_large = unireplknet_l()
+print(model_large)
+ckpt = torch.load("UniRepLKNet-L-b75k_s10B_CLIP-in1k_75.72.pt")
+model_large.load_state_dict(ckpt,strict=False) 
+# Since we do not need cls heads in CLIP pretraining.
+print("Loaded CLIP Pretrained Models")
+```
+# Model Details
+
+## Model Description
+
+A series of CLIP UniRepLKNet models trained on LAION-2B (english), a subset of [LAION-5B](https://arxiv.org/abs/2210.08402), using [OpenCLIP](https://github.com/mlfoundations/open_clip).
+
+| Model | Dataset | Resolution | Top-1 ImageNet Zero-Shot (%) |
+| ----- | ------- | ---------- | ------------ |
+| [UniRepLKNet_large.laion5b_s11b_b75k](https://huggingface.co/Yiyuan/CLIP-UniRepLKNet-L-laion5B-s10B-b75k) | LAION-5B | 224x224 | 75.7 |
+
+
+The core training run was performed in pieces over a period of ~ 2 months. The global batch size for the core run was 76800. The last ~25% of training was re-done at a 320x320 image resolution. See more details in [Training Details](#training-details).
+
+Goals:
+  * Push the size of largest convolutional CLIP image tower into the performance range of ViT-g to ViT-G w/ improved image size scaling for downstream use.
+
+Firsts:
+  * Largest released ConvNeXt model pretrained (847M params w/ 198 GMAC and 125 MActs @ 256x256 for image)
+  * A non-ViT image tower CLIP model (with no previous image tower pretrain) achieving > 79% ImageNet top-1 zero-shot
+
+The models utilize:
+  * the UniRepLKNet model (`unireplknet_l`) as the image tower
+  * a standard projection at end of image tower
+  * a text tower with same size (with 1024, heads 16, depth 24) as ViT-H-14 and ViT-g-14 models
+
+
+The models are trained at 224x224 image resolution at the beginning 192 epochs, then they're rewinded to 320x320 image resolution in the following 64 epochs. The size of the combined image + text CLIP model is 1.2B params w/ 222 GMAC and 146 MActs. UniRepLKNet excels in the both training and inference efficiency.
+
+
+Model training done by Ross Wightman across both the [stability.ai](https://stability.ai/) cluster and the [JUWELS Booster](https://apps.fz-juelich.de/jsc/hps/juwels/booster-overview.html) supercomputer. See acknowledgements below.
+
+# Uses
+
+As per the original [OpenAI CLIP model card](https://github.com/openai/CLIP/blob/d50d76daa670286dd6cacf3bcd80b5e4823fc8e1/model-card.md), this model is intended as a research output for research communities. We hope that this model will enable researchers to better understand and explore zero-shot, arbitrary image classification. We also hope it can be used for interdisciplinary studies of the potential impact of such model. 
+
+The OpenAI CLIP paper includes a discussion of potential downstream impacts to provide an example for this sort of analysis. Additionally, the LAION-5B blog (https://laion.ai/blog/laion-5b/) and upcoming paper include additional discussion as it relates specifically to the training dataset. 
+
+## Direct Use
+
+Zero-shot image classification, image and text retrieval, among others.
+
+## Downstream Use
+
+Image classification and other image task fine-tuning, linear probe image classification, image generation guiding and conditioning, among others.
+
+## Out-of-Scope Use
+
+As per the OpenAI models,
+
+**Any** deployed use case of the model - whether commercial or not - is currently out of scope. Non-deployed use cases such as image search in a constrained environment, are also not recommended unless there is thorough in-domain testing of the model with a specific, fixed class taxonomy. This is because our safety assessment demonstrated a high need for task specific testing especially given the variability of CLIP’s performance with different class taxonomies. This makes untested and unconstrained deployment of the model in any use case currently potentially harmful. 
+
+Certain use cases which would fall under the domain of surveillance and facial recognition are always out-of-scope regardless of performance of the model. This is because the use of artificial intelligence for tasks such as these can be premature currently given the lack of testing norms and checks to ensure its fair use.
+
+Since the model has not been purposefully trained in or evaluated on any languages other than English, its use should be limited to English language use cases.
+
+Further the above notice, the LAION-5B dataset used in training of these models has additional considerations, see below.
+
+# Training Details
+
+## Training Data
+
+This model was trained with LAION-2B -- A 2 billion sample English subset of LAION-5B (https://laion.ai/blog/laion-5b/).
+
+## Training Procedure
+
+The main training run was done at global batch size of 76800 for 256 checkpoint intervals of 45M samples for a total of ~11.5B samples seen over training.
+
+
+A slurm srun command line below for a 128 8-GPU (40GB A100) configuration:
+
+```
+srun --cpu_bind=v --accel-bind=gn python -m training.main \
+    --save-frequency 1 \
+    --name "large-5b-76800-bf16" \
+    --resume "latest" \
+    --logs "/runs" \
+    --log-every-n-steps 100 \
+    --train-data="pipe:aws s3 cp s3://laion5b/laion2B-data/{000000..231349}.tar -" \
+    --train-num-samples 45646078 \
+    --dataset-type webdataset \
+    --warmup 10000 \
+    --batch-size=600 \
+    --epochs=256 \
+    --dataset-resampled \
+    --precision amp_bfloat16 \
+    --grad-clip-norm 5.0 \
+    --lr 5e-4 \
+    --workers=6 \
+    --beta2 0.99 \
+    --model "unireplknet_l" \
+    --seed 0 \
+    --ddp-static-graph \
+    --local-loss \
+    --gather-with-grad \
+    --grad-checkpointing \
+    --report-to "tensorboard"
+```
+
+
+# Evaluation
+
+Evaluation done with code in the [LAION CLIP Benchmark suite](https://github.com/LAION-AI/CLIP_benchmark).
+
+
+## Results
+
+These models achieve between 75.7 and 76.3 top-1 zero-shot accuracy on ImageNet-1k.
+
+
+# Citation
+
+**BibTeX:**
+
+UniRepLKNet
+
+```bibtex
+@inproceedings{ding2024unireplknet,
+  title={UniRepLKNet: A Universal Perception Large-Kernel ConvNet for Audio Video Point Cloud Time-Series and Image Recognition},
+  author={Ding, Xiaohan and Zhang, Yiyuan and Ge, Yixiao and Zhao, Sijie and Song, Lin and Yue, Xiangyu and Shan, Ying},
+  booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
+  pages={5513--5524},
+  year={2024}
+}
+```
+
+LAION-5B
+```bibtex
+@inproceedings{schuhmann2022laionb,
+  title={{LAION}-5B: An open large-scale dataset for training next generation image-text models},
+  author={Christoph Schuhmann and
+          Romain Beaumont and
+          Richard Vencu and
+          Cade W Gordon and
+          Ross Wightman and
+          Mehdi Cherti and
+          Theo Coombes and
+          Aarush Katta and
+          Clayton Mullis and
+          Mitchell Wortsman and
+          Patrick Schramowski and
+          Srivatsa R Kundurthy and
+          Katherine Crowson and
+          Ludwig Schmidt and
+          Robert Kaczmarczyk and
+          Jenia Jitsev},
+  booktitle={Thirty-sixth Conference on Neural Information Processing Systems Datasets and Benchmarks Track},
+  year={2022},
+  url={https://openreview.net/forum?id=M3Y74vmsMcY}
+}
+```
+
+OpenCLIP software
+```bibtex
+@software{ilharco_gabriel_2021_5143773,
+  author       = {Ilharco, Gabriel and
+                  Wortsman, Mitchell and
+                  Wightman, Ross and
+                  Gordon, Cade and
+                  Carlini, Nicholas and
+                  Taori, Rohan and
+                  Dave, Achal and
+                  Shankar, Vaishaal and
+                  Namkoong, Hongseok and
+                  Miller, John and
+                  Hajishirzi, Hannaneh and
+                  Farhadi, Ali and
+                  Schmidt, Ludwig},
+  title        = {OpenCLIP},
+  month        = jul,
+  year         = 2021,
+  note         = {If you use this software, please cite it as below.},
+  publisher    = {Zenodo},
+  version      = {0.1},
+  doi          = {10.5281/zenodo.5143773},
+  url          = {https://doi.org/10.5281/zenodo.5143773}
+}
+```
+
+OpenAI CLIP paper
+
+
+```bibtex
+@misc{rw2019timm,
+  author = {Ross Wightman},
+  title = {PyTorch Image Models},
+  year = {2019},
+  publisher = {GitHub},
+  journal = {GitHub repository},
+  doi = {10.5281/zenodo.4414861},
+  howpublished = {\url{https://github.com/rwightman/pytorch-image-models}}
+}
+```