| license: other | |
| library_name: peft | |
| tags: | |
| - llama2 | |
| base_model: NousResearch/Llama-2-13b-hf | |
| # Model Card for PIPPA ShareGPT Subset QLora 13b | |
| It is an experimental Lora focused on Roleplay that uses a subset of [PIPPA ShareGPT](https://huggingface.co/datasets/kingbri/PIPPA-shareGPT) | |
| ## Usage | |
| Custom | |
| ``` | |
| SYSTEM: Do thing | |
| USER: {prompt} | |
| CHARACTER: | |
| ``` | |
| ## Bias, Risks, and Limitations | |
| This Lora is not intended for supplying factual information or advice in any form | |
| ## Training Details | |
| ### Training Data | |
| 1k of conversation from [PIPPA ShareGPT](https://huggingface.co/datasets/kingbri/PIPPA-shareGPT) | |
| ### Training Procedure | |
| The version of this Lora uploaded on this repository was trained using a 8x RTX A6000 cluster in 8-bit with regular LoRA adapters and 32-bit AdamW optimizer. | |
| #### Training Hyperparameters | |
| Training using a fork of [Axolotl](https://github.com/OpenAccess-AI-Collective/axolotl) with two paths [Patch 1](https://github.com/bdashore3/axolotl/commit/995557bdf3c6c8b3e839b224ef9513fc2b097f30) [Patch 2](https://github.com/bdashore3/axolotl/commit/8970280de2ea01e41c044406051922715f4086cb) | |
| - load_in_4bit: true | |
| - lora_r: 16 | |
| - lora_alpha: 16 | |
| - lora_dropout: 0.01 | |
| - gradient_accumulation_steps: 8 | |
| - micro_batch_size: 4 | |
| - num_epochs: 3 | |
| - learning_rate: 0.000065 | |
| ## Environmental Impact | |
| Finetuning this model on 8xNVIDIA A6000 48GB in parallel takes about 30 minutes (13B) |