I need a help to overcome Cuda out of memory
Hello, I am trying to fine tune JAIS 13B Chat version model using instruction tuning to perform a classification task on Arabic dataset , I applied quantization and PEFT, but I am getting cuda of memory when I reach trainer.train() step, can you help me in this matter ? I am using 3*RTX A6000 resource
Hello
@HanaRasheed
, Tnx for opening this discussion.
Well, it would have been more practical to open it in the model you are facing the issue with (which 13B variant ?) but to answer your message, generally, it is hard to point out where could be went wrong ! Can you provide more context, which quantization format, code you are running, format of the dataset in case you can't provide the dataset. Also, i would suggest to try out LLMTools and see of it works.
Anyway, I would be glad to help figure it out ounce you provide more context of the error.
On a seperate note, i would suggest to use inceptionai/jais-family-13b or inceptionai/jais-adapted-13b as they are more recent and better in terms of benchmarks results.