|
--- |
|
language: |
|
- en |
|
tags: |
|
- 7B |
|
- mistralAI |
|
- v0.1 |
|
- mistral |
|
- gguf |
|
--- |
|
|
|
## Steps to run model |
|
|
|
1. **install requirements** |
|
~~~ |
|
pip install gguf |
|
pip install sentencepiece |
|
pip install typing |
|
pip install torch |
|
~~~ |
|
|
|
2. **For linux: use utorrent to download the model weights from magnet link** |
|
|
|
_Commands_ |
|
|
|
~~~ |
|
sudo snap install utorrent |
|
utorrent |
|
~~~ |
|
This opens utorrent UI, click "Add Torrent from URL" and paste |
|
|
|
**`magnet:?xt=urn:btih:208b101a0f51514ecf285885a8b0f6fb1a1e4d7d&dn=mistral-7B-v0.1&tr=udp%3A%2F%http://2Ftracker.opentrackr.org%3A1337%2Fannounce&tr=https%3A%2F%http://2Ftracker1.520.jp%3A443%2Fannounce`** |
|
|
|
|
|
|
|
3. **use `convert.py` script from https://github.com/ggerganov/llama.cpp/blob/master/convert.py for** |
|
|
|
`python3 convert.py --outtype f16 "/path/to/mistral-7B-v0.1"` |
|
|
|
4. **install ollama (link: https://github.com/jmorganca/ollama)** |
|
~~~ |
|
curl https://ollama.ai/install.sh | sh |
|
~~~ |
|
|
|
(Hint: if there are errors with the above command, try uninstalling curl from snap and install with apt) |
|
|
|
~~~ |
|
sudo snap remove curl |
|
sudo apt install curl |
|
~~~ |
|
|
|
### modelfile for ollama |
|
|
|
~~~ |
|
echo "FROM /path/to/mistral-7B-v0.1/ggml-model-f16.gguf" > modelfile |
|
~~~ |
|
|
|
|
|
### run ollama |
|
|
|
~~~ |
|
ollama create mistral -f modelfile |
|
~~~ |
|
|
|
### make the folder that contains modelfile and downloaded mistral model weights traversable with a+x |
|
|
|
~~~ |
|
sudo chmod -R a+x /path/to/mistral-7B-v0.1 |
|
ollama create mistral -f /path/to/mistral-7B-v0.1/modelfile |
|
|
|
ollama run mistral |
|
~~~ |
|
|
|
|
|
**Hardware specs used** |
|
|
|
~~~ |
|
Architecture: x86_64 |
|
CPU op-mode(s): 32-bit, 64-bit |
|
Address sizes: 39 bits physical, 48 bits virtual |
|
Byte Order: Little Endian |
|
CPU(s): 20 |
|
On-line CPU(s) list: 0-19 |
|
Vendor ID: GenuineIntel |
|
Model name: 12th Gen Intel(R) Core(TM) i7-12700H |
|
CPU family: 6 |
|
Model: 154 |
|
Thread(s) per core: 2 |
|
Core(s) per socket: 14 |
|
Socket(s): 1 |
|
Stepping: 3 |
|
CPU max MHz: 4700.0000 |
|
CPU min MHz: 400.0000 |
|
BogoMIPS: 5376.00 |
|
~~~ |
|
|
|
|
