|
--- |
|
license: cc-by-nc-4.0 |
|
language: |
|
- en |
|
--- |
|
|
|
# MistralPirate-7b-v0.3 |
|
|
|
## Model Card |
|
|
|
### Description |
|
MistralPirate-7b-v0.3 is a sophisticated language model fine-tuned for generating intricate and authentic pirate-themed content. This version, correcting our version control from v2 to v0.3, builds upon MistralPirate-7b-v2 and leverages advancements from Mistral Instruct v0.2. It shows improved performance in pirate dialect accuracy and perplexity scores. |
|
|
|
- **Developed by**: phanerozoic |
|
- **License**: cc-by-nc-4.0 |
|
- **Finetuned from**: Mistral Instruct v0.2 |
|
|
|
### Version Control Correction |
|
Correcting version control to v0.3 to reflect developmental progression and enhancements over the previous version. |
|
|
|
### Comparative Analysis with Previous Model |
|
MistralPirate-7b-v0.3 demonstrates notable improvements over its predecessor in several key areas: |
|
- **Pirate Dialect**: The new model uses richer and more immersive pirate vernacular, enhancing the thematic experience. |
|
- **Technical Accuracy**: It shows a deeper understanding of complex sailing scenarios, providing detailed and practical advice in response to intricate questions. |
|
- **Language Coherence**: The model maintains a consistent tone and style, effectively blending pirate jargon with technical expertise. |
|
|
|
### Direct Use |
|
Ideal for interactive storytelling, gaming, advanced educational content, and conversational AI in pirate-themed settings. |
|
|
|
### Downstream Use |
|
Suitable for tasks requiring detailed language generation and domain-specific knowledge, like advanced thematic content creation or immersive language learning. |
|
|
|
### Out-of-Scope Use |
|
Not intended for general-purpose language modeling or non-pirate-themed contexts. Usage outside its specialization may result in suboptimal performance. |
|
|
|
### Bias, Risks, and Limitations |
|
Limited by its training data, may inherit biases. Best used where pirate-themed language is appropriate, not for serious or sensitive communication. |
|
|
|
### Recommendations |
|
Recommended for thematic contexts, with an understanding of its specialized focus. Not for accurate information outside pirate dialect specialization. |
|
|
|
### Custom Stopping Strings Usage |
|
Custom stopping strings employed for output quality: |
|
|
|
- "}," |
|
- "User:" |
|
- "You:" |
|
- "\nUser" |
|
- "\nUser:" |
|
|
|
### Training Data |
|
Trained on a vast dataset in ChatML format, ensuring diverse and rich inputs. |
|
|
|
### Preprocessing |
|
Advanced preprocessing into ChatML format. |
|
|
|
### Training Hyperparameters and Fine-Tuning Details |
|
- Training Regime: FP32 |
|
- Warmup Steps: 1 |
|
- Per Device Train Batch Size: 2 |
|
- Gradient Accumulation Steps: 64 |
|
- Max Steps: 1500 |
|
- Learning Rate: 0.00015 |
|
- Logging Steps: 1 |
|
- Save Steps: 1 |
|
- Lora Alpha: 32 |
|
- Dimension Count: 16 |
|
- Specific Lora Fine-Tuning Run: |
|
- Step: 26 |
|
- Loss: 1.4906 |
|
- Learning Rate: 0.00019814951887490748 |
|
- Epoch: 0.01 |
|
|
|
### Speeds, Sizes, Times |
|
Approximately 12 minutes training time on RTX 6000 Ada GPU. |
|
|
|
### Testing Data |
|
Achieved a perplexity score of 5.17 against the Wikitext database. |
|
|
|
### Factors |
|
Focus on language coherence, pirate dialect adherence, and technical accuracy. |
|
|
|
### Metrics |
|
Primary metric: Perplexity. Qualitative assessments of dialect authenticity and technical content. |
|
|
|
### Results |
|
Marked improvement in sophisticated output with authentic pirate tone. Lower perplexity score demonstrates enhanced language modeling. |
|
|
|
### Summary |
|
Represents a significant advancement in domain-specific language modeling, excelling in complex, authentic pirate-themed content. |
|
|
|
### Model Architecture and Objective |
|
Based on Mistral Instruct v0.2, fine-tuned for high coherence and technical accuracy in pirate-themed content. |
|
|
|
### Compute Infrastructure |
|
Trained on RTX 6000 Ada GPU for efficient training and improved perplexity scores. |
|
|
|
### Hardware |
|
- Type: RTX 6000 Ada |
|
- Utilization: Approx. 12 minutes for training. |
|
|
|
### Acknowledgments |
|
Gratitude to Mistral and Mistral Instruct v0.2 teams. Appreciation to the language modeling community for support in domain-specific model enhancement. |
