|
--- |
|
license: apache-2.0 |
|
tags: |
|
- control vectors |
|
- exllamav2 |
|
- creative writing |
|
- text generation |
|
- inference |
|
- model integration |
|
--- |
|
# Creative Writing Control Vectors Integration for ExLlamaV2 |
|
|
|
This project provides a wrapper to integrate [jukofyork's creative writing control vectors](https://huggingface.co/jukofyork/creative-writing-control-vectors-v3.0) with [ExLlamaV2](https://github.com/turboderp/exllamav2/). While ExLlamaV2 does not natively support control vectors, this wrapper enables loading and injecting GGUF control vectors into the model for dynamic text generation control. |
|
|
|
## Overview |
|
|
|
- Wrapper for using control vectors with ExLlamaV2 |
|
- Supports loading control vectors from GGUF format |
|
- Injects vectors directly into ExLlamaV2 inference |
|
- Enables dynamic text generation control |
|
|
|
## Usage |
|
|
|
1. Download model in ExLlamaV2 format |
|
2. Create a "-vectors" directory next to model directory |
|
3. Download the control vectors from [jukofyork's repository](https://huggingface.co/jukofyork/creative-writing-control-vectors-v3.0) and place them in the "-vectors" directory. |
|
4. Run inference with the `--control_vectors` (`-vc`) parameter. |
|
|
|
Example command: |
|
```bash |
|
python test_inference.py -m Meta-Llama-3-70B-Instruct-8bpw \ |
|
-p "<prompt>" \ |
|
--control_vectors language:simple:0.5,optimism:optimism:0.5 |
|
``` |
|
|
|
## Directory Structure |
|
|
|
Ensure your directory structure follows this format to correctly load the control vectors: |
|
|
|
``` |
|
models/ |
|
├── Meta-Llama-3-70B-Instruct-8bpw/ |
|
│ └── model files... |
|
└── Meta-Llama-3-70B-Instruct-8bpw-vectors/ |
|
├── llama-3:70b-language__debias.gguf |
|
├── llama-3:70b-language__simple.gguf |
|
├── llama-3:70b-language__ornate.gguf |
|
└── ... |
|
``` |
|
|
|
## Limitations |
|
|
|
- Proof of concept implementation |
|
- May impact model performance |
|
- Limited testing with different vector combinations |
|
- No guarantee of exact equivalence to llama.cpp behavior |
|
|
|
## Acknowledgments |
|
|
|
- Control vectors from [jukofyork's creative-writing-control-vectors-v3.0](https://huggingface.co/jukofyork/creative-writing-control-vectors-v3.0) |
|
- [ExLlamaV2 by turboderp](https://github.com/turboderp/exllamav2/) |
