|
--- |
|
license: creativeml-openrail-m |
|
datasets: |
|
- amphora/QwQ-LongCoT-130K |
|
language: |
|
- en |
|
base_model: prithivMLmods/QwQ-LCoT-7B-Instruct |
|
pipeline_tag: text-generation |
|
library_name: transformers |
|
tags: |
|
- Long-CoT |
|
- Qwen2.5 |
|
- 7B |
|
- safetensors |
|
- text-generation-inference |
|
- QwQ |
|
- SFT |
|
- Math |
|
- Qwen with Questions |
|
- llama-cpp |
|
- gguf-my-repo |
|
--- |
|
|
|
# Triangle104/QwQ-LCoT-7B-Instruct-Q5_K_S-GGUF |
|
This model was converted to GGUF format from [`prithivMLmods/QwQ-LCoT-7B-Instruct`](https://huggingface.co/prithivMLmods/QwQ-LCoT-7B-Instruct) using llama.cpp via the ggml.ai's [GGUF-my-repo](https://huggingface.co/spaces/ggml-org/gguf-my-repo) space. |
|
Refer to the [original model card](https://huggingface.co/prithivMLmods/QwQ-LCoT-7B-Instruct) for more details on the model. |
|
|
|
--- |
|
Model details: |
|
- |
|
The QwQ-LCoT-7B-Instruct is a fine-tuned language model designed for advanced reasoning and instruction-following tasks. It leverages the Qwen2.5-7B base model and has been fine-tuned on the amphora/QwQ-LongCoT-130K dataset, focusing on chain-of-thought (CoT) reasoning. |
|
|
|
Key Features: |
|
|
|
Model Size: |
|
|
|
7.62B parameters (FP16 precision). |
|
|
|
Model Sharding: |
|
|
|
The model weights are split into 4 shards (safetensors) for efficient storage and download: |
|
model-00001-of-00004.safetensors (4.88 GB) |
|
model-00002-of-00004.safetensors (4.93 GB) |
|
model-00003-of-00004.safetensors (4.33 GB) |
|
model-00004-of-00004.safetensors (1.09 GB) |
|
|
|
Tokenizer: |
|
|
|
Byte-pair encoding (BPE) based. |
|
Files included: |
|
vocab.json (2.78 MB) |
|
merges.txt (1.82 MB) |
|
tokenizer.json (11.4 MB) |
|
|
|
Special tokens mapped in special_tokens_map.json (e.g., <pad>, <eos>). |
|
|
|
Configuration Files: |
|
|
|
config.json: Defines model architecture and hyperparameters. |
|
generation_config.json: Settings for inference and text generation tasks. |
|
|
|
Training Dataset: |
|
|
|
Dataset Name: amphora/QwQ-LongCoT-130K |
|
Size: 133k examples. |
|
Focus: Chain-of-Thought reasoning for complex tasks. |
|
|
|
Use Cases: |
|
|
|
Instruction Following: |
|
Handle user instructions effectively, even for multi-step tasks. |
|
|
|
Reasoning Tasks: |
|
|
|
Perform logical reasoning and generate detailed step-by-step solutions. |
|
|
|
Text Generation: |
|
|
|
Generate coherent, context-aware responses. |
|
|
|
--- |
|
## Use with llama.cpp |
|
Install llama.cpp through brew (works on Mac and Linux) |
|
|
|
```bash |
|
brew install llama.cpp |
|
|
|
``` |
|
Invoke the llama.cpp server or the CLI. |
|
|
|
### CLI: |
|
```bash |
|
llama-cli --hf-repo Triangle104/QwQ-LCoT-7B-Instruct-Q5_K_S-GGUF --hf-file qwq-lcot-7b-instruct-q5_k_s.gguf -p "The meaning to life and the universe is" |
|
``` |
|
|
|
### Server: |
|
```bash |
|
llama-server --hf-repo Triangle104/QwQ-LCoT-7B-Instruct-Q5_K_S-GGUF --hf-file qwq-lcot-7b-instruct-q5_k_s.gguf -c 2048 |
|
``` |
|
|
|
Note: You can also use this checkpoint directly through the [usage steps](https://github.com/ggerganov/llama.cpp?tab=readme-ov-file#usage) listed in the Llama.cpp repo as well. |
|
|
|
Step 1: Clone llama.cpp from GitHub. |
|
``` |
|
git clone https://github.com/ggerganov/llama.cpp |
|
``` |
|
|
|
Step 2: Move into the llama.cpp folder and build it with `LLAMA_CURL=1` flag along with other hardware-specific flags (for ex: LLAMA_CUDA=1 for Nvidia GPUs on Linux). |
|
``` |
|
cd llama.cpp && LLAMA_CURL=1 make |
|
``` |
|
|
|
Step 3: Run inference through the main binary. |
|
``` |
|
./llama-cli --hf-repo Triangle104/QwQ-LCoT-7B-Instruct-Q5_K_S-GGUF --hf-file qwq-lcot-7b-instruct-q5_k_s.gguf -p "The meaning to life and the universe is" |
|
``` |
|
or |
|
``` |
|
./llama-server --hf-repo Triangle104/QwQ-LCoT-7B-Instruct-Q5_K_S-GGUF --hf-file qwq-lcot-7b-instruct-q5_k_s.gguf -c 2048 |
|
``` |
|
|
