README.md

# TransGPT

**发布中文TransGPT（7B）模型**

test case:
|input_text|predict|
|:-- |:--- |
|我想了解如何申请和更新驾驶证？|xx|




## Usage

First, you pass your input through the transformer model, then you get the generated sentence.

Install package:
```
pip install sentencepiece
pip install transformers>=4.28.0
```

```python
import torch
import transformers
from transformers import LlamaTokenizer, LlamaForCausalLM

def generate_prompt(text):
    return f"""Below is an instruction that describes a task. Write a response that appropriately completes the request.

### Instruction:
{text}

### Response:"""

checkpoint="DUOMO-Lab/TransGPT-v0"
tokenizer = LlamaTokenizer.from_pretrained(checkpoint)
model = LlamaForCausalLM.from_pretrained(checkpoint).half().cuda()
model.eval()

text = '我想了解如何申请和更新驾驶证？'
prompt = generate_prompt(text)
input_ids = tokenizer.encode(prompt, return_tensors='pt').to('cuda')


with torch.no_grad():
    output_ids = model.generate(
        input_ids=input_ids,
        max_new_tokens=1024,
        temperature=1,
        top_k=20,
        top_p=0.9,
        repetition_penalty=1.15
    ).cuda()
output = tokenizer.decode(output_ids[0], skip_special_tokens=True)
print(output.replace(text, '').strip())
```


output:
```shell
我想了解如何申请和更新驾驶证？

```

## 模型来源
release合并后的模型权重。

HuggingFace版本权重（.bin文件）可用于：
- 使用Transformers进行训练和推理
- 使用text-generation-webui搭建界面

PyTorch版本权重（.pth文件）可用于：
- 使用llama.cpp工具进行量化和部署



模型文件组成：
```
TransGPT
    config.json
    generation_config.json
    pytorch_model-00001-of-00002.bin
    pytorch_model-00002-of-00002.bin
    pytorch_model.bin.index.json
    special_tokens_map.json
    tokenizer.json
    tokenizer.model
    tokenizer_config.json
```

硬件要求：14G显存

### 微调数据集

1. ～xx万条文本数据集（用于领域内预训练）：[DUOMO-Lab/TransGPT-pt](https://huggingface.co/datasets/DUOMO-Lab/TransGPT-pt)
2. ～5.6万条对话数据（用于微调）:[finetune_data](https://huggingface.co/data/finetune)


如果需要训练LLaMA模型，请参考[https://github.com/DUOMO/TransGPT](https://github.com/DUOMO/TransGPT)


## Citation

```latex
@software{TransGPT,
  author = {Wang Peng},
  title = {DUOMO/TransGPT},
  year = {2023},
  url = {https://github.com/DUOMO/TransGPT},
}
```


## Reference
- https://github.com/shibing624/textgen