metadata
license: apache-2.0
RWKV-x070-2B9-CJE-Instruct Model Card
Model Overview
- Model Name: RWKV-x070-2B9-CJE-Instruct
- Description: An instruction-tuned model specialized for Japanese, Chinese, and English languages
- Base Model: rwkv-x070-2b9-world-v3-40%trained-20250113-ctx4k.pth
- Architecture: RWKV x070 "Goose"
- Parameters: 2.9B
- Model Dimension: 2560
- Number of Layers: 32
Fine-tuning Details
Training Configuration
- Trainer: RWKV-LM-RLHF (https://github.com/OpenMOSE/RWKV-LM-RLHF)
- PEFT Mode: Hybrid Training combining frozen embeddings and Bone (Block Affine Transformation) + full parameter training
- SFT Method: SmoothingLoss SFT
- Context Window: 5120 (trained with 1024 token overlap)
- Compute Power: AMD Instinct MI100 x 2 60hrs (100％ solar energy)
Dataset Specifications
- Size: 800k pairs
- Content:
- Mixed data in Japanese, Chinese, and English
- Conversations
- Programming code
- Translation tasks
- Chain-of-Thought reasoning tasks
How to use
- Install latest RWKV-Infer (Linux,WSL) (https://github.com/OpenMOSE/RWKV-Infer)
- make folder 'models'
- move rwkv-x070-2b9-cje-instruct-1.pth to models folder
curl http://127.0.0.1:9000/loadmodel -X POST -H "Content-Type: application/json" -d '{"model_filename":"models/rwkv-x070-2b9-cje-instruct-1.pth","model_viewname":"RWKV x070 2B9 CJE Instruct-1","model_strategy":"fp16","endtoken":"\\n\\n\\x17"}'
- Enjoy with openai compatible api http://127.0.0.1:9000/v1 :)
Important Note
- Set the end token as '\n\n\x17'
User: who are you?\n\n\x17
Assistant: gooday i'm rwkv\n\n\x17
Limitations and Considerations
- This is an experimental model; inference stability is not fully guaranteed
- Unexpected behaviors may occur
- Continuous improvements are being made; feedback is welcome
License
Apache License 2.0
Acknowledgments
We express our gratitude to the RWKV base model and the RWKV community for their support in developing this model.