Hugging Face's logo

EmbeddedLLM
/
mistral-7b-instruct-v0.3-onnx

Text Generation
ONNX
English
ONNX
DML
DirectML
ONNXRuntime
mistral
conversational
custom_code
Model card Files Community
mistral-7b-instruct-v0.3-onnx / README.md
pstan's picture
pstan
Create README.md
84a2633 verified
preview code
|
raw
history blame
2.65 kB
metadata
license: apache-2.0
pipeline_tag: text-generation
tags:
  - ONNX
  - DML
  - DirectML
  - ONNXRuntime
  - mistral
  - conversational
  - custom_code
inference: false

Mistral-7B-Instruct-v0.3 ONNX

Model Summary

The Mistral-7B-Instruct-v0.3 is an optimized version of the Mistral-7B model, fine-tuned for instruction-based tasks. This model is available in ONNX format to accelerate inference using ONNX Runtime, specifically optimized for CPU and DirectML. DirectML is a high-performance, hardware-accelerated DirectX 12 library for machine learning, providing GPU acceleration for a wide range of supported hardware and drivers, including AMD, Intel, NVIDIA, and Qualcomm GPUs.

Model Description

  • Developed by: Mistral AI
  • Model type: ONNX
  • Language(s) (NLP): Python, C, C++
  • License: Apache License Version 2.0
  • Model Description: This model is a conversion of the Mistral-7B-Instruct-v0.3 for ONNX Runtime inference, optimized for CPU and DirectML.

Usage

Installation and Setup

To use the Mistral-7B-Instruct-v0.3 ONNX model on Windows with DirectML, follow these steps:

  1. Create and activate a Conda environment:
conda create -n onnx python=3.10
conda activate onnx
  1. Install Git LFS:
winget install -e --id GitHub.GitLFS
  1. Install Hugging Face CLI:
pip install huggingface-hub[cli]
  1. Download the model:
huggingface-cli download EmbeddedLLM/mistral-7b-instruct-v0.3-int4-onnx-directml --include directml/* --local-dir .\mistral-7b-instruct-v0.3
  1. Install necessary Python packages:
pip install numpy
pip install onnxruntime-directml
pip install --pre onnxruntime-genai-directml
  1. Install Visual Studio 2015 runtime:
conda install conda-forge::vs2015_runtime
  1. Download the example script:
Invoke-WebRequest -Uri "https://raw.githubusercontent.com/microsoft/onnxruntime-genai/main/examples/python/phi3-qa.py" -OutFile "phi3-qa.py"
  1. Run the example script:
python phi3-qa.py -m .\mistral-7b-instruct-v0.3

Hardware Requirements

  • Minimum Configuration:

  • Windows: DirectX 12-capable GPU (AMD/Nvidia)

  • CPU: x86_64 / ARM64

  • Tested Configurations:

  • GPU: AMD Ryzen 8000 Series iGPU (DirectML)

  • CPU: AMD Ryzen CPU

Optimized Configurations

The following optimized configurations are available:

  1. ONNX model for int4 DML: Optimized for AMD, Intel, and NVIDIA GPUs on Windows, quantized to int4.
  2. ONNX model for int4 CPU: Optimized for CPU, using int4 quantization.