|
--- |
|
license: llama2 |
|
language: |
|
- en |
|
pipeline_tag: text-generation |
|
--- |
|
# InvestLM |
|
This is the repo for a new financial domain large language model, InvestLM, tuned on [Mixtral-8x7B-v0.1](https://huggingface.co/mistralai/Mixtral-8x7B-v0.1), using a carefully curated instruction dataset related to financial investment. We provide guidance on how to use InvestLM for inference. |
|
|
|
Github Link: [InvestLM](https://github.com/AbaciNLP/InvestLM) |
|
|
|
<font color="#0000FF">Test only, not for sharing.</font> |
|
|
|
# About AWQ |
|
[AWQ](https://github.com/casper-hansen/AutoAWQ) is an efficient, accurate, and blazing-fast low-bit weight quantization method, currently supporting 4-bit quantization. Compared to GPTQ, it offers faster Transformers-based inference. |
|
|
|
# Inference |
|
Please use the following command to log in hugging face first. |
|
``` |
|
huggingface-cli login |
|
``` |
|
## Prompt template |
|
|
|
``` |
|
[INST] {prompt} [/INST] |
|
``` |
|
|
|
## How to use this AWQ model from Python code |
|
``` |
|
pip3 install --upgrade "autoawq>=0.1.6" "transformers>=4.35.0" |
|
``` |
|
|
|
``` |
|
from transformers import AutoTokenizer, TextStreamer |
|
|
|
quant_path = "yixuantt/InvestLM-Mistral-AWQ" |
|
|
|
# Load model |
|
model = AutoModelForCausalLM.from_pretrained( |
|
quant_path, |
|
low_cpu_mem_usage=True, |
|
device_map="cuda:0" |
|
) |
|
tokenizer = AutoTokenizer.from_pretrained(quant_path) |
|
|
|
# Convert prompt to tokens |
|
prompt_template = "[INST] {prompt} [/INST]" |
|
prompt = "What is finance?" |
|
|
|
tokens = tokenizer( |
|
prompt_template.format(prompt=prompt), |
|
return_tensors='pt' |
|
).input_ids.cuda() |
|
|
|
# Generate output |
|
generation_output = model.generate( |
|
tokens, |
|
max_new_tokens = 512 |
|
) |
|
|
|
print("Output: ", tokenizer.decode(generation_output[0])) |
|
|
|
``` |
