Hugging Face's logo

qualcomm
/
Whisper-Tiny-En

Automatic Speech Recognition
PyTorch
TF Lite
foundation
android
Model card Files Community
qaihm-bot commited on
Commit
41b6ea9
·
verified ·
1 Parent(s): 0f5ad97

Upload README.md with huggingface_hub

Browse files
Files changed (1) hide show
  1. README.md +62 -28
README.md CHANGED
@@ -15,7 +15,7 @@ tags:
15
 
16
  OpenAI’s Whisper ASR (Automatic Speech Recognition) model is a state-of-the-art system designed for transcribing spoken language into written text. It exhibits robust performance in realistic, noisy environments, making it highly reliable for real-world applications. Specifically, it excels in long-form transcription, capable of accurately transcribing audio clips up to 30 seconds long. Time to the first token is the encoder's latency, while time to each additional token is decoder's latency, where we assume a mean decoded length specified below.
17
 
18
- This model is an implementation of Whisper-Tiny-En found [here](https://github.com/openai/whisper/tree/main).
19
  This repository provides scripts to run Whisper-Tiny-En on Qualcomm® devices.
20
  More details on model performance across various devices, can be found
21
  [here](https://aihub.qualcomm.com/models/whisper_tiny_en).
@@ -33,17 +33,45 @@ More details on model performance across various devices, can be found
33
  - Number of parameters (WhisperDecoder): 28.2M
34
  - Model size (WhisperDecoder): 108 MB
35
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
36
 
37
 
38
 
39
- | Device | Chipset | Target Runtime | Inference Time (ms) | Peak Memory Range (MB) | Precision | Primary Compute Unit | Target Model
40
- | ---|---|---|---|---|---|---|---|
41
- | Samsung Galaxy S23 Ultra (Android 13) | Snapdragon® 8 Gen 2 | TFLite | 98.932 ms | 14 - 54 MB | FP16 | GPU | [WhisperEncoder.tflite](https://huggingface.co/qualcomm/Whisper-Tiny-En/blob/main/WhisperEncoder.tflite)
42
- | Samsung Galaxy S23 Ultra (Android 13) | Snapdragon® 8 Gen 2 | TFLite | 3.793 ms | 7 - 9 MB | FP16 | NPU | [WhisperDecoder.tflite](https://huggingface.co/qualcomm/Whisper-Tiny-En/blob/main/WhisperDecoder.tflite)
43
- | Samsung Galaxy S23 Ultra (Android 13) | Snapdragon® 8 Gen 2 | QNN Model Library | 185.163 ms | 0 - 50 MB | FP16 | NPU | [WhisperEncoder.so](https://huggingface.co/qualcomm/Whisper-Tiny-En/blob/main/WhisperEncoder.so)
44
- | Samsung Galaxy S23 Ultra (Android 13) | Snapdragon® 8 Gen 2 | QNN Model Library | 2.387 ms | 0 - 142 MB | FP16 | NPU | [WhisperDecoder.so](https://huggingface.co/qualcomm/Whisper-Tiny-En/blob/main/WhisperDecoder.so)
45
-
46
-
47
 
48
  ## Installation
49
 
@@ -99,23 +127,25 @@ device. This script does the following:
99
  ```bash
100
  python -m qai_hub_models.models.whisper_tiny_en.export
101
  ```
102
-
103
  ```
104
- Profile Job summary of WhisperEncoder
105
- --------------------------------------------------
106
- Device: Snapdragon X Elite CRD (11)
107
- Estimated Inference Time: 148.68 ms
108
- Estimated Peak Memory Range: 0.47-0.47 MB
109
- Compute Units: NPU (313) | Total (313)
110
-
111
- Profile Job summary of WhisperDecoder
112
- --------------------------------------------------
113
- Device: Snapdragon X Elite CRD (11)
114
- Estimated Inference Time: 2.06 ms
115
- Estimated Peak Memory Range: 10.14-10.14 MB
116
- Compute Units: NPU (447) | Total (447)
117
-
118
-
 
 
 
119
  ```
120
 
121
 
@@ -243,15 +273,19 @@ provides instructions on how to use the `.so` shared library in an Android appl
243
  Get more details on Whisper-Tiny-En's performance across various devices [here](https://aihub.qualcomm.com/models/whisper_tiny_en).
244
  Explore all available models on [Qualcomm® AI Hub](https://aihub.qualcomm.com/)
245
 
 
246
  ## License
247
- - The license for the original implementation of Whisper-Tiny-En can be found
248
- [here](https://github.com/openai/whisper/blob/main/LICENSE).
249
- - The license for the compiled assets for on-device deployment can be found [here](https://qaihub-public-assets.s3.us-west-2.amazonaws.com/qai-hub-models/Qualcomm+AI+Hub+Proprietary+License.pdf)
 
250
 
251
  ## References
252
  * [Robust Speech Recognition via Large-Scale Weak Supervision](https://cdn.openai.com/papers/whisper.pdf)
253
  * [Source Model Implementation](https://github.com/openai/whisper/tree/main)
254
 
 
 
255
  ## Community
256
  * Join [our AI Hub Slack community](https://aihub.qualcomm.com/community/slack) to collaborate, post questions and learn more about on-device AI.
257
  * For questions or feedback please [reach out to us](mailto:[email protected]).
 
15
 
16
  OpenAI’s Whisper ASR (Automatic Speech Recognition) model is a state-of-the-art system designed for transcribing spoken language into written text. It exhibits robust performance in realistic, noisy environments, making it highly reliable for real-world applications. Specifically, it excels in long-form transcription, capable of accurately transcribing audio clips up to 30 seconds long. Time to the first token is the encoder's latency, while time to each additional token is decoder's latency, where we assume a mean decoded length specified below.
17
 
18
+ This model is an implementation of Whisper-Tiny-En found [here]({source_repo}).
19
  This repository provides scripts to run Whisper-Tiny-En on Qualcomm® devices.
20
  More details on model performance across various devices, can be found
21
  [here](https://aihub.qualcomm.com/models/whisper_tiny_en).
 
33
  - Number of parameters (WhisperDecoder): 28.2M
34
  - Model size (WhisperDecoder): 108 MB
35
 
36
+ | Model | Device | Chipset | Target Runtime | Inference Time (ms) | Peak Memory Range (MB) | Precision | Primary Compute Unit | Target Model
37
+ |---|---|---|---|---|---|---|---|---|
38
+ | WhisperEncoder | Samsung Galaxy S23 | Snapdragon® 8 Gen 2 | TFLITE | 103.909 ms | 20 - 87 MB | FP16 | GPU | [Whisper-Tiny-En.tflite](https://huggingface.co/qualcomm/Whisper-Tiny-En/blob/main/WhisperEncoder.tflite) |
39
+ | WhisperEncoder | Samsung Galaxy S23 | Snapdragon® 8 Gen 2 | QNN | 135.518 ms | 0 - 54 MB | FP16 | NPU | [Whisper-Tiny-En.so](https://huggingface.co/qualcomm/Whisper-Tiny-En/blob/main/WhisperEncoder.so) |
40
+ | WhisperEncoder | Samsung Galaxy S24 | Snapdragon® 8 Gen 3 | TFLITE | 84.725 ms | 22 - 50 MB | FP16 | GPU | [Whisper-Tiny-En.tflite](https://huggingface.co/qualcomm/Whisper-Tiny-En/blob/main/WhisperEncoder.tflite) |
41
+ | WhisperEncoder | Samsung Galaxy S24 | Snapdragon® 8 Gen 3 | QNN | 113.054 ms | 0 - 187 MB | FP16 | NPU | [Whisper-Tiny-En.so](https://huggingface.co/qualcomm/Whisper-Tiny-En/blob/main/WhisperEncoder.so) |
42
+ | WhisperEncoder | QCS8550 (Proxy) | QCS8550 Proxy | TFLITE | 98.695 ms | 20 - 99 MB | FP16 | GPU | [Whisper-Tiny-En.tflite](https://huggingface.co/qualcomm/Whisper-Tiny-En/blob/main/WhisperEncoder.tflite) |
43
+ | WhisperEncoder | QCS8550 (Proxy) | QCS8550 Proxy | QNN | 102.692 ms | 0 - 6 MB | FP16 | NPU | Use Export Script |
44
+ | WhisperEncoder | SA8255 (Proxy) | SA8255P Proxy | TFLITE | 100.209 ms | 18 - 62 MB | FP16 | GPU | [Whisper-Tiny-En.tflite](https://huggingface.co/qualcomm/Whisper-Tiny-En/blob/main/WhisperEncoder.tflite) |
45
+ | WhisperEncoder | SA8255 (Proxy) | SA8255P Proxy | QNN | 105.009 ms | 0 - 6 MB | FP16 | NPU | Use Export Script |
46
+ | WhisperEncoder | SA8775 (Proxy) | SA8775P Proxy | TFLITE | 105.883 ms | 7 - 51 MB | FP16 | GPU | [Whisper-Tiny-En.tflite](https://huggingface.co/qualcomm/Whisper-Tiny-En/blob/main/WhisperEncoder.tflite) |
47
+ | WhisperEncoder | SA8775 (Proxy) | SA8775P Proxy | QNN | 103.64 ms | 1 - 2 MB | FP16 | NPU | Use Export Script |
48
+ | WhisperEncoder | SA8650 (Proxy) | SA8650P Proxy | TFLITE | 101.593 ms | 19 - 114 MB | FP16 | GPU | [Whisper-Tiny-En.tflite](https://huggingface.co/qualcomm/Whisper-Tiny-En/blob/main/WhisperEncoder.tflite) |
49
+ | WhisperEncoder | SA8650 (Proxy) | SA8650P Proxy | QNN | 104.738 ms | 0 - 10 MB | FP16 | NPU | Use Export Script |
50
+ | WhisperEncoder | QCS8450 (Proxy) | QCS8450 Proxy | TFLITE | 150.469 ms | 20 - 53 MB | FP16 | GPU | [Whisper-Tiny-En.tflite](https://huggingface.co/qualcomm/Whisper-Tiny-En/blob/main/WhisperEncoder.tflite) |
51
+ | WhisperEncoder | QCS8450 (Proxy) | QCS8450 Proxy | QNN | 180.709 ms | 0 - 195 MB | FP16 | NPU | Use Export Script |
52
+ | WhisperEncoder | Snapdragon 8 Elite QRD | Snapdragon® 8 Elite | TFLITE | 77.729 ms | 20 - 40 MB | FP16 | GPU | [Whisper-Tiny-En.tflite](https://huggingface.co/qualcomm/Whisper-Tiny-En/blob/main/WhisperEncoder.tflite) |
53
+ | WhisperEncoder | Snapdragon 8 Elite QRD | Snapdragon® 8 Elite | QNN | 100.989 ms | 0 - 195 MB | FP16 | NPU | Use Export Script |
54
+ | WhisperEncoder | Snapdragon X Elite CRD | Snapdragon® X Elite | QNN | 95.58 ms | 0 - 0 MB | FP16 | NPU | Use Export Script |
55
+ | WhisperDecoder | Samsung Galaxy S23 | Snapdragon® 8 Gen 2 | TFLITE | 3.76 ms | 3 - 5 MB | FP16 | NPU | [Whisper-Tiny-En.tflite](https://huggingface.co/qualcomm/Whisper-Tiny-En/blob/main/WhisperDecoder.tflite) |
56
+ | WhisperDecoder | Samsung Galaxy S23 | Snapdragon® 8 Gen 2 | QNN | 2.356 ms | 3 - 133 MB | FP16 | NPU | [Whisper-Tiny-En.so](https://huggingface.co/qualcomm/Whisper-Tiny-En/blob/main/WhisperDecoder.so) |
57
+ | WhisperDecoder | Samsung Galaxy S24 | Snapdragon® 8 Gen 3 | TFLITE | 2.891 ms | 3 - 221 MB | FP16 | NPU | [Whisper-Tiny-En.tflite](https://huggingface.co/qualcomm/Whisper-Tiny-En/blob/main/WhisperDecoder.tflite) |
58
+ | WhisperDecoder | Samsung Galaxy S24 | Snapdragon® 8 Gen 3 | QNN | 1.618 ms | 4 - 27 MB | FP16 | NPU | [Whisper-Tiny-En.so](https://huggingface.co/qualcomm/Whisper-Tiny-En/blob/main/WhisperDecoder.so) |
59
+ | WhisperDecoder | QCS8550 (Proxy) | QCS8550 Proxy | TFLITE | 3.718 ms | 3 - 5 MB | FP16 | NPU | [Whisper-Tiny-En.tflite](https://huggingface.co/qualcomm/Whisper-Tiny-En/blob/main/WhisperDecoder.tflite) |
60
+ | WhisperDecoder | QCS8550 (Proxy) | QCS8550 Proxy | QNN | 2.284 ms | 10 - 11 MB | FP16 | NPU | Use Export Script |
61
+ | WhisperDecoder | SA8255 (Proxy) | SA8255P Proxy | TFLITE | 3.651 ms | 3 - 5 MB | FP16 | NPU | [Whisper-Tiny-En.tflite](https://huggingface.co/qualcomm/Whisper-Tiny-En/blob/main/WhisperDecoder.tflite) |
62
+ | WhisperDecoder | SA8255 (Proxy) | SA8255P Proxy | QNN | 2.233 ms | 10 - 12 MB | FP16 | NPU | Use Export Script |
63
+ | WhisperDecoder | SA8775 (Proxy) | SA8775P Proxy | TFLITE | 3.786 ms | 3 - 5 MB | FP16 | NPU | [Whisper-Tiny-En.tflite](https://huggingface.co/qualcomm/Whisper-Tiny-En/blob/main/WhisperDecoder.tflite) |
64
+ | WhisperDecoder | SA8775 (Proxy) | SA8775P Proxy | QNN | 2.226 ms | 10 - 11 MB | FP16 | NPU | Use Export Script |
65
+ | WhisperDecoder | SA8650 (Proxy) | SA8650P Proxy | TFLITE | 3.644 ms | 3 - 5 MB | FP16 | NPU | [Whisper-Tiny-En.tflite](https://huggingface.co/qualcomm/Whisper-Tiny-En/blob/main/WhisperDecoder.tflite) |
66
+ | WhisperDecoder | SA8650 (Proxy) | SA8650P Proxy | QNN | 2.297 ms | 10 - 12 MB | FP16 | NPU | Use Export Script |
67
+ | WhisperDecoder | QCS8450 (Proxy) | QCS8450 Proxy | TFLITE | 4.266 ms | 3 - 218 MB | FP16 | NPU | [Whisper-Tiny-En.tflite](https://huggingface.co/qualcomm/Whisper-Tiny-En/blob/main/WhisperDecoder.tflite) |
68
+ | WhisperDecoder | QCS8450 (Proxy) | QCS8450 Proxy | QNN | 2.741 ms | 10 - 36 MB | FP16 | NPU | Use Export Script |
69
+ | WhisperDecoder | Snapdragon 8 Elite QRD | Snapdragon® 8 Elite | TFLITE | 2.429 ms | 1 - 31 MB | FP16 | NPU | [Whisper-Tiny-En.tflite](https://huggingface.co/qualcomm/Whisper-Tiny-En/blob/main/WhisperDecoder.tflite) |
70
+ | WhisperDecoder | Snapdragon 8 Elite QRD | Snapdragon® 8 Elite | QNN | 1.394 ms | 10 - 34 MB | FP16 | NPU | Use Export Script |
71
+ | WhisperDecoder | Snapdragon X Elite CRD | Snapdragon® X Elite | QNN | 2.056 ms | 10 - 10 MB | FP16 | NPU | Use Export Script |
72
 
73
 
74
 
 
 
 
 
 
 
 
 
75
 
76
  ## Installation
77
 
 
127
  ```bash
128
  python -m qai_hub_models.models.whisper_tiny_en.export
129
  ```
 
130
  ```
131
+ Profiling Results
132
+ ------------------------------------------------------------
133
+ WhisperEncoder
134
+ Device : Samsung Galaxy S23 (13)
135
+ Runtime : TFLITE
136
+ Estimated inference time (ms) : 103.9
137
+ Estimated peak memory usage (MB): [20, 87]
138
+ Total # Ops : 271
139
+ Compute Unit(s) : GPU (260 ops) CPU (11 ops)
140
+
141
+ ------------------------------------------------------------
142
+ WhisperDecoder
143
+ Device : Samsung Galaxy S23 (13)
144
+ Runtime : TFLITE
145
+ Estimated inference time (ms) : 3.8
146
+ Estimated peak memory usage (MB): [3, 5]
147
+ Total # Ops : 557
148
+ Compute Unit(s) : NPU (557 ops)
149
  ```
150
 
151
 
 
273
  Get more details on Whisper-Tiny-En's performance across various devices [here](https://aihub.qualcomm.com/models/whisper_tiny_en).
274
  Explore all available models on [Qualcomm® AI Hub](https://aihub.qualcomm.com/)
275
 
276
+
277
  ## License
278
+ * The license for the original implementation of Whisper-Tiny-En can be found [here](https://github.com/openai/whisper/blob/main/LICENSE).
279
+ * The license for the compiled assets for on-device deployment can be found [here](https://qaihub-public-assets.s3.us-west-2.amazonaws.com/qai-hub-models/Qualcomm+AI+Hub+Proprietary+License.pdf)
280
+
281
+
282
 
283
  ## References
284
  * [Robust Speech Recognition via Large-Scale Weak Supervision](https://cdn.openai.com/papers/whisper.pdf)
285
  * [Source Model Implementation](https://github.com/openai/whisper/tree/main)
286
 
287
+
288
+
289
  ## Community
290
  * Join [our AI Hub Slack community](https://aihub.qualcomm.com/community/slack) to collaborate, post questions and learn more about on-device AI.
291
  * For questions or feedback please [reach out to us](mailto:[email protected]).