First stab at readme

README.md

@@ -0,0 +1,74 @@
+---
+license: cc-by-sa-3.0
+datasets:
+- databricks/databricks-dolly-15k
+language:
+- en
+---
+
+# Open-Instruct Dolly 7B
+
+This model is a 7B LLaMa model finetuned on the Dolly dataset. *please note this is a model diff - see below for usage instructions*.
+
+This was trained as part of the paper [How Far Can Camels Go? Exploring the State of Instruction Tuning on Open Resources](arxiv.org/abs/xxxx).
+The codebase used to train and evaluate this model can be found at [https://github.com/allenai/open-instruct](https://github.com/allenai/open-instruct).
+
+## Usage
+
+We assume you have access to a LLaMa model in HF format already. You can find details on getting access and converting the model here:
+[https://huggingface.co/docs/transformers/main/model_doc/llama](https://huggingface.co/docs/transformers/main/model_doc/llama)
+
+Clone [https://github.com/allenai/open-instruct](https://github.com/allenai/open-instruct) and install the required dependencies, or just copy `scripts/weight_diff.py`
+and install the minimal requirements listed in `weight-diff-requirements.txt`. Then download or clone this model diff to the same machine.
+
+Then, run:
+```bash
+python scripts/weight_diff.py recover --path_raw ${hf_llama_path} --path_tuned ${output_path} --path_diff ${diff_location}
+```
+
+And you will have a recovered model! Note this takes up a decent amount of RAM, especially for the larger models.
+
+## Input Format
+
+The model is trained to use the following format:
+```
+<|user|>
+Your message here!
+<|assistant|>
+```
+
+For best results, format all inputs in this manner.
+
+## Performance
+
+Here is the performance of this model across benchmarks explored in our paper [How Far Can Camels Go? Exploring the State of Instruction Tuning on Open Resources](arxiv.org/abs/xxxx):
+
+| MMLU 0-shot | MMLU 5-shot | GSM Direct | GSM CoT | BBH Direct | BBH CoT | TydiQA Gold-Passage | TydiQA Closed-book | Codex-Eval Pass@1 | Codex-Eval Pass@10 | AlpacaFarm vs Davinci-003 | Average |
+|:-----------:|:-----------:|:----------:|:-------:|:----------:|:-------:|:-------------------:|:------------------:|:-----------------:|:------------------:|:-------------------------:|---------|
+|    0.380    |    0.358    |    0.050   |  0.070  |    0.272   |  0.244  |        43.569       |        8.718       |       0.111       |        0.221       |           12.67           | 20.7    |
+
+
+If you use this model, please cite our work and the original dataset:
+
+```
+@article{camelevaluation,
+  title={How Far Can Camels Go? Exploring the State of Instruction Tuning on Open Resources},
+  author={Yizhong Wang, Hamish Ivison, Pradeep Dasigi, Jack Hessel, Tushar Khot, Khyathi Raghavi Chandu, David Wadden, Kelsey MacMillan, Noah A. Smith, Iz Beltagy, Hannaneh Hajishirzi},
+  year={2023}
+}
+```
+
+```
+@misc{dolly,
+  author = {Databricks},
+  title = {Free Dolly: Introducing the World's First Truly Open Instruction-Tuned LLM},
+  year = {2023},
+  publisher = {GitHub},
+  journal = {GitHub repository},
+  howpublished = {Blog post},
+  url = {https://www.databricks.com/blog/2023/04/12/dolly-first-open-commercially-viable-instruction-tuned-llm}
+}
+```
+
+
+