Text Generation
Transformers
Safetensors
English
stripedhyena
custom_code
Zymrael commited on
Commit
00e8992
·
1 Parent(s): cabde32

chore: update readme

Browse files
Files changed (1) hide show
  1. README.md +5 -2
README.md CHANGED
@@ -14,7 +14,9 @@ language:
14
 
15
  One of the focus areas at Together Research is new architectures for long context, improved training, and inference performance over the Transformer architecture. Spinning out of a research program from our team and academic collaborators, with roots in **signal processing-inspired sequence models**, we are excited to introduce the **StripedHyena** models. StripedHyena is the **first alternative model competitive with the best open-source Transformers** of similar sizes in short and long-context evaluations.
16
 
17
- - Read more here in [our blog](https://www.together.ai/blog/stripedhyena-7b)
 
 
18
  - Play with the model on our playground!
19
  - Dive into the details of our [standalone implementation](https://github.com/togethercomputer/stripedhyena), and our related research: [1](https://arxiv.org/abs/2302.10866), [2](https://arxiv.org/abs/2310.18780), [3](https://arxiv.org/abs/2311.05908).
20
 
@@ -23,5 +25,6 @@ One of the focus areas at Together Research is new architectures for long contex
23
  StripedHyena is a hybrid architecture composed of multi-head, grouped-query attention and gated convolutions arranged in [Hyena](https://arxiv.org/abs/2302.10866) blocks, different from traditional decoder-only Transformers.
24
  - Costant memory decoding in Hyena blocks via representation of convolutions as state-space models (modal or canonical form), or as truncated filters.
25
  - Low latency, faster decoding and higher throughput than Transformers.
26
- - Improvement to training and inference-optimal scaling laws, compared to optimized Transformer architectures such as Llama.
27
  - Trained on sequences of up to 32k, allowing it to process longer prompts.
 
 
14
 
15
  One of the focus areas at Together Research is new architectures for long context, improved training, and inference performance over the Transformer architecture. Spinning out of a research program from our team and academic collaborators, with roots in **signal processing-inspired sequence models**, we are excited to introduce the **StripedHyena** models. StripedHyena is the **first alternative model competitive with the best open-source Transformers** of similar sizes in short and long-context evaluations.
16
 
17
+ **StripedHyena-Hessian-7B (SH 7B)** is our **base model** for this release.
18
+
19
+ - Read more here in [our blog](https://www.together.ai/blog/stripedhyena-7b).
20
  - Play with the model on our playground!
21
  - Dive into the details of our [standalone implementation](https://github.com/togethercomputer/stripedhyena), and our related research: [1](https://arxiv.org/abs/2302.10866), [2](https://arxiv.org/abs/2310.18780), [3](https://arxiv.org/abs/2311.05908).
22
 
 
25
  StripedHyena is a hybrid architecture composed of multi-head, grouped-query attention and gated convolutions arranged in [Hyena](https://arxiv.org/abs/2302.10866) blocks, different from traditional decoder-only Transformers.
26
  - Costant memory decoding in Hyena blocks via representation of convolutions as state-space models (modal or canonical form), or as truncated filters.
27
  - Low latency, faster decoding and higher throughput than Transformers.
28
+ - Improvement to training and inference-optimal scaling laws, compared to optimized Transformer architectures such as Llama-2.
29
  - Trained on sequences of up to 32k, allowing it to process longer prompts.
30
+