Upload GPT

README.md

@@ -0,0 +1,199 @@
+---
+library_name: transformers
+tags: []
+---
+
+# Model Card for Model ID
+
+<!-- Provide a quick summary of what the model is/does. -->
+
+
+
+## Model Details
+
+### Model Description
+
+<!-- Provide a longer summary of what this model is. -->
+
+This is the model card of a 🤗 transformers model that has been pushed on the Hub. This model card has been automatically generated.
+
+- **Developed by:** [More Information Needed]
+- **Funded by [optional]:** [More Information Needed]
+- **Shared by [optional]:** [More Information Needed]
+- **Model type:** [More Information Needed]
+- **Language(s) (NLP):** [More Information Needed]
+- **License:** [More Information Needed]
+- **Finetuned from model [optional]:** [More Information Needed]
+
+### Model Sources [optional]
+
+<!-- Provide the basic links for the model. -->
+
+- **Repository:** [More Information Needed]
+- **Paper [optional]:** [More Information Needed]
+- **Demo [optional]:** [More Information Needed]
+
+## Uses
+
+<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
+
+### Direct Use
+
+<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
+
+[More Information Needed]
+
+### Downstream Use [optional]
+
+<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
+
+[More Information Needed]
+
+### Out-of-Scope Use
+
+<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
+
+[More Information Needed]
+
+## Bias, Risks, and Limitations
+
+<!-- This section is meant to convey both technical and sociotechnical limitations. -->
+
+[More Information Needed]
+
+### Recommendations
+
+<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
+
+Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
+
+## How to Get Started with the Model
+
+Use the code below to get started with the model.
+
+[More Information Needed]
+
+## Training Details
+
+### Training Data
+
+<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
+
+[More Information Needed]
+
+### Training Procedure
+
+<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
+
+#### Preprocessing [optional]
+
+[More Information Needed]
+
+
+#### Training Hyperparameters
+
+- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
+
+#### Speeds, Sizes, Times [optional]
+
+<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
+
+[More Information Needed]
+
+## Evaluation
+
+<!-- This section describes the evaluation protocols and provides the results. -->
+
+### Testing Data, Factors & Metrics
+
+#### Testing Data
+
+<!-- This should link to a Dataset Card if possible. -->
+
+[More Information Needed]
+
+#### Factors
+
+<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
+
+[More Information Needed]
+
+#### Metrics
+
+<!-- These are the evaluation metrics being used, ideally with a description of why. -->
+
+[More Information Needed]
+
+### Results
+
+[More Information Needed]
+
+#### Summary
+
+
+
+## Model Examination [optional]
+
+<!-- Relevant interpretability work for the model goes here -->
+
+[More Information Needed]
+
+## Environmental Impact
+
+<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
+
+Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
+
+- **Hardware Type:** [More Information Needed]
+- **Hours used:** [More Information Needed]
+- **Cloud Provider:** [More Information Needed]
+- **Compute Region:** [More Information Needed]
+- **Carbon Emitted:** [More Information Needed]
+
+## Technical Specifications [optional]
+
+### Model Architecture and Objective
+
+[More Information Needed]
+
+### Compute Infrastructure
+
+[More Information Needed]
+
+#### Hardware
+
+[More Information Needed]
+
+#### Software
+
+[More Information Needed]
+
+## Citation [optional]
+
+<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
+
+**BibTeX:**
+
+[More Information Needed]
+
+**APA:**
+
+[More Information Needed]
+
+## Glossary [optional]
+
+<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
+
+[More Information Needed]
+
+## More Information [optional]
+
+[More Information Needed]
+
+## Model Card Authors [optional]
+
+[More Information Needed]
+
+## Model Card Contact
+
+[More Information Needed]

adapter_v2.py

@@ -0,0 +1,418 @@
+# Copyright Lightning AI. Licensed under the Apache License 2.0, see LICENSE file.
+
+"""Implementation of the paper:
+
+LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Model
+https://arxiv.org/abs/2304.15010
+
+Port for LitGPT
+"""
+
+from dataclasses import dataclass
+from typing import Any, Dict, List, Optional, Tuple, Type, Union
+
+import torch
+import torch.nn as nn
+from typing_extensions import Self
+
+import litgpt
+from litgpt.adapter import GPT as BaseModel
+from litgpt.adapter import Block as BaseBlock
+from litgpt.adapter import CausalSelfAttention as BaseCausalSelfAttention
+from litgpt.adapter import Config as BaseConfig
+from litgpt.model import KVCache
+from litgpt.utils import map_old_state_dict_weights
+from litgpt.model import KVCache, apply_rope
+from litgpt.smoe import AdapterV2SMoE
+
+from transformers import PreTrainedModel
+
+@dataclass
+class Config(BaseConfig):
+    @property
+    def mlp_class(self) -> Type:
+        return getattr(litgpt.adapter_v2, self.mlp_class_name)
+
+@dataclass
+class ConfigSMOE(BaseConfig):
+    use_smoe: bool=False
+    num_experts: int=4
+    top_k: int=1
+    alpha: int=0
+    model_type: str = "gpt"
+    def __init__(self, *args, **kwargs):
+        super(ConfigSMOE, self).__init__(*args, **kwargs)
+    
+    @property
+    def mlp_class(self) -> Type:
+        return getattr(litgpt.adapter_v2, self.mlp_class_name)
+    def load_extra(self, extra_config):
+        for k in list(extra_config.keys()):
+            setattr(self, k, extra_config[k])
+
+def adapter_filter(key: str, value: Any) -> bool:
+    
+    adapter_substrings = (
+        # regular adapter v1 parameters
+        "adapter_wte",
+        "gating_factor",
+        # adapter v2: new bias and scale used in Linear
+        "adapter_scale",
+        "adapter_bias",
+        # adapter v2: Norm parameters are now trainable
+        "norm_1",
+        "norm_2",
+        "ln_f",
+        # smoe: gating mechanism
+        "gate",
+        )
+    return any(s in key for s in adapter_substrings)
+
+
+class AdapterV2Linear(torch.nn.Module):
+    def __init__(self, in_features: int, out_features: int, **kwargs) -> None:
+        super().__init__()
+        self.linear = torch.nn.Linear(in_features, out_features, **kwargs)
+        self.adapter_bias = torch.nn.Parameter(torch.zeros(out_features), requires_grad=False)
+        self.adapter_scale = torch.nn.Parameter(torch.ones(out_features), requires_grad=False)
+
+    def forward(self, x: torch.Tensor) -> torch.Tensor:
+        # breakpoint()
+        return self.adapter_scale * (self.linear(x) + self.adapter_bias)
+
+    def reset_parameters(self) -> None:
+        nn.init.zeros_(self.adapter_bias)
+        nn.init.ones_(self.adapter_scale)
+
+class GPT(BaseModel, PreTrainedModel):
+    config_class=ConfigSMOE
+
+    def __init__(self, config: ConfigSMOE) -> None:
+        # Skip the parent class __init__ altogether and replace it to avoid useless allocations
+        nn.Module.__init__(self)
+        # super().__init__(config)
+        assert config.padded_vocab_size is not None
+        self.config = config
+        if config.use_smoe:
+            print("🐙 Run AdapterV2SMoE")
+            self.lm_head = AdapterV2SMoE(
+                in_features=config.n_embd,
+                out_features=config.padded_vocab_size,
+                num_experts=config.num_experts,
+                top_k=config.top_k,
+                bias=config.lm_head_bias
+            )
+            self.transformer = nn.ModuleDict(
+                dict(
+                    wte=nn.Embedding(config.padded_vocab_size, config.n_embd),
+                    h=nn.ModuleList(BlockSMoE(config, i) for i in range(config.n_layer)),
+                    ln_f=config.norm_class(config.n_embd, eps=config.norm_eps),
+                )
+            )
+        else:
+            print("🐙 Run AdapterV2Linear")
+            self.lm_head = AdapterV2Linear(config.n_embd, config.padded_vocab_size, bias=config.lm_head_bias)
+            self.transformer = nn.ModuleDict(
+                dict(
+                    wte=nn.Embedding(config.padded_vocab_size, config.n_embd),
+                    h=nn.ModuleList(Block(config, i) for i in range(config.n_layer)),
+                    ln_f=config.norm_class(config.n_embd, eps=config.norm_eps),
+                )
+            )
+        self.max_seq_length = self.config.block_size
+        self.mask_cache: Optional[torch.Tensor] = None
+
+    def forward(
+        self, idx: torch.Tensor, input_pos: Optional[torch.Tensor] = None, lm_head_chunk_size: int = 0
+    ) -> Union[torch.Tensor, List[torch.Tensor]]:
+        T = idx.size(1)
+        if self.max_seq_length < T:
+            raise ValueError(f"Cannot forward sequence of length {T}, max seq length is only {self.max_seq_length}.")
+
+        if input_pos is not None:  # use the kv cache
+            cos = self.cos.index_select(0, input_pos)
+            sin = self.sin.index_select(0, input_pos)
+            if self.mask_cache is None:
+                raise TypeError("You need to call `gpt.set_kv_cache()`")
+            mask = self.mask_cache.index_select(2, input_pos)
+        else:
+            cos = self.cos[:T]
+            sin = self.sin[:T]
+            mask = None
+
+        x = self.transformer.wte(idx)  # token embeddings of shape (b, t, n_embd)
+        if self.config.scale_embeddings:
+            x = x * (self.config.n_embd**0.5)
+        for block in self.transformer.h:
+            x = block(x, cos, sin, mask, input_pos)
+        x = self.transformer.ln_f(x)
+        if self.config.use_smoe:
+            if lm_head_chunk_size > 0:
+                outputs = []
+                routers = []
+                for x_i in x.split(lm_head_chunk_size, dim = 1):
+                    output, router = self.lm_head(x_i)
+                    outputs.append(output)
+                    routers.append(router)
+                return outputs, routers
+            output, router = self.lm_head(x)
+            return output, router #(b, t, vocab_size)
+        else:
+            if lm_head_chunk_size > 0:
+                # chunk the lm head logits to reduce the peak memory used by autograd
+                return [self.lm_head(x_i) for x_i in x.split(lm_head_chunk_size, dim=1)]
+            return self.lm_head(x)  # (b, t, vocab_size)
+
+    @classmethod
+    def from_name(cls, name: str, **kwargs: Any) -> Self:
+        return cls(Config.from_name(name, **kwargs))
+
+    def _init_weights(self, module: nn.Module) -> None:
+        """Meant to be used with `gpt.apply(gpt._init_weights)`. Unused method left for completeness."""
+        super()._init_weights(module)
+        if isinstance(module, AdapterV2Linear):
+            module.reset_parameters()
+
+    def _load_from_state_dict(self, state_dict: Dict, prefix: str, *args: Any, **kwargs: Any) -> None:
+        """For compatibility with base checkpoints."""
+        mapping = {"lm_head.weight": "lm_head.linear.weight", "lm_head.bias": "lm_head.linear.bias"}
+        state_dict = map_old_state_dict_weights(state_dict, mapping, prefix)
+        super()._load_from_state_dict(state_dict, prefix, *args, **kwargs)
+
+
+class Block(BaseBlock):
+    """The implementation is identical to `litgpt.model.Block` with the exception that
+    we replace the attention layer where adaption is implemented."""
+
+    def __init__(self, config: Config, block_idx: int) -> None:
+        # Skip the parent class __init__ altogether and replace it to avoid useless allocations
+        nn.Module.__init__(self)
+        self.norm_1 = config.norm_class(config.n_embd, eps=config.norm_eps)
+        if config.use_smoe:
+            self.attn = CausalSelfAttentionSMoE(config, block_idx)
+        else:
+            self.attn = CausalSelfAttention(config, block_idx)
+        if not config.shared_attention_norm:
+            self.norm_2 = config.norm_class(config.n_embd, eps=config.norm_eps)
+        self.mlp = config.mlp_class(config)
+
+        self.config = config
+
+class BlockSMoE(Block):
+    def forward(
+        self,
+        x: torch.Tensor,
+        cos: torch.Tensor,
+        sin: torch.Tensor,
+        mask: Optional[torch.Tensor] = None,
+        input_pos: Optional[torch.Tensor] = None,
+    ) -> torch.Tensor:
+        x_normed = self.norm_1(x)
+        attention_output, _ = self.attn(x_normed, cos, sin, mask, input_pos)
+        if self.config.parallel_residual:
+            x_normed = x_normed if self.config.shared_attention_norm else self.norm_2(x)
+            x = self.mlp(x_normed) + attention_output + x
+        else:
+            x = attention_output + x
+            x = self.mlp(self.norm_2(x)) + x
+        return x
+
+
+class CausalSelfAttention(BaseCausalSelfAttention):
+    """A modification of `litgpt.adapter.CausalSelfAttention` that uses the Adapter V2 Linear class"""
+
+    def __init__(self, config: Config, block_idx: int) -> None:
+        # Skip the parent class __init__ altogether and replace it to avoid useless allocations
+        nn.Module.__init__(self)
+        shape = (config.n_head + 2 * config.n_query_groups) * config.head_size
+        # key, query, value projections for all heads, but in a batch
+        if config.use_smoe:
+            self.attn = AdapterV2SMoE(
+                in_features=config.n_embd,
+                out_features=shape,
+                num_experts=config.num_experts,
+                top_k=config.top_k,
+                bias=config.bias
+            )
+            # output projection
+            # if `head_size` is explicitly specified in the config, `n_emd` might not be equal to `head_size * n_head`
+            self.proj = AdapterV2SMoE(
+                in_features=config.head_size * config.n_head,
+                out_features=config.n_embd,
+                num_experts=config.num_experts,
+                top_k=config.top_k,
+                bias=config.bias
+            )
+            # disabled by default
+        else:
+            self.attn = AdapterV2Linear(in_features=config.n_embd, out_features=shape, bias=config.bias)
+            # output projection
+            # if `head_size` is explicitly specified in the config, `n_emd` might not be equal to `head_size * n_head`
+            self.proj = AdapterV2Linear(config.head_size * config.n_head, config.n_embd, bias=config.bias)
+            # disabled by default
+        self.kv_cache: Optional[KVCache] = None
+
+        if block_idx >= config.adapter_start_layer:
+            # adapter embedding layer
+            self.adapter_wte = nn.Embedding(config.adapter_prompt_length, config.n_embd)
+            # gate for adaption
+            self.gating_factor = torch.nn.Parameter(torch.zeros(1, 1, config.n_head, 1))
+            # kv cache for inference
+            self.adapter_kv_cache: Optional[Tuple[torch.Tensor, torch.Tensor]] = None
+        self.block_idx = block_idx
+
+        self.config = config
+
+    def _load_from_state_dict(self, state_dict: Dict, prefix: str, *args: Any, **kwargs: Any) -> None:
+        """For compatibility with base checkpoints."""
+        mapping = {
+            "attn.weight": "attn.linear.weight",
+            "attn.bias": "attn.linear.bias",
+            "proj.weight": "proj.linear.weight",
+            "proj.bias": "proj.linear.bias",
+        }
+        state_dict = map_old_state_dict_weights(state_dict, mapping, prefix)
+        # For compatibility with older checkpoints
+        if (key := prefix + "gating_factor") in state_dict and state_dict[key].size(1) == self.config.n_head:
+            state_dict[key] = state_dict[key].permute(0, 2, 1, 3)
+        super()._load_from_state_dict(state_dict, prefix, *args, **kwargs)
+
+class CausalSelfAttentionSMoE(CausalSelfAttention):
+    def forward(
+        self,
+        x: torch.Tensor,
+        cos: torch.Tensor,
+        sin: torch.Tensor,
+        mask: Optional[torch.Tensor] = None,
+        input_pos: Optional[torch.Tensor] = None,
+    ) -> torch.Tensor:
+        B, T, C = x.size()  # batch size, sequence length, embedding dimensionality (n_embd)
+        
+        # breakpoint()
+        qkv, _ = self.attn(x)
+
+        # assemble into a number of query groups to support MHA, MQA and GQA together (see `config.n_query_groups`)
+        q_per_kv = self.config.n_head // self.config.n_query_groups
+        total_qkv = q_per_kv + 2  # each group has 1+ queries, 1 key, and 1 value
+        qkv = qkv.view(B, T, self.config.n_query_groups, total_qkv, self.config.head_size)
+        qkv = qkv.permute(0, 2, 3, 1, 4)  # (B, n_query_groups, total_qkv, T, hs)
+
+        # split batched computation into three
+        q, k, v = qkv.split((q_per_kv, 1, 1), dim=2)
+
+        # maybe repeat k and v if for the non multi-head attention cases
+        # training: flash attention requires it
+        # inference: multi-query would require a full kv cache so avoid it to limit its memory usage
+        if self.config.n_query_groups != self.config.n_head and (input_pos is None or self.config.n_query_groups != 1):
+            k = k.expand(B, self.config.n_query_groups, q_per_kv, T, self.config.head_size)
+            v = v.expand(B, self.config.n_query_groups, q_per_kv, T, self.config.head_size)
+
+        q = q.reshape(B, -1, T, self.config.head_size)  # (B, nh_q, T, hs)
+        k = k.reshape(B, -1, T, self.config.head_size)  # (B, nh_k, T, hs)
+        v = v.reshape(B, -1, T, self.config.head_size)  # (B, nh_v, T, hs)
+
+        q_roped = apply_rope(q[..., : self.config.rope_n_elem], cos, sin)
+        k_roped = apply_rope(k[..., : self.config.rope_n_elem], cos, sin)
+        q = torch.cat((q_roped, q[..., self.config.rope_n_elem :]), dim=-1)
+        k = torch.cat((k_roped, k[..., self.config.rope_n_elem :]), dim=-1)
+
+        if input_pos is not None:
+            if not isinstance(self.kv_cache, KVCache):
+                raise TypeError("You need to call `gpt.set_kv_cache()`")
+            k, v = self.kv_cache(input_pos, k, v)
+
+        y = self.scaled_dot_product_attention(q, k, v, mask)
+
+        y = y.reshape(B, T, self.config.head_size * self.config.n_head)  # re-assemble all head outputs side by side
+
+        # output projection
+        return self.proj(y)
+
+class GptNeoxMLP(litgpt.model.GptNeoxMLP):
+    def __init__(self, config: Config) -> None:
+        nn.Module.__init__(self)
+        if config.use_smoe:
+            self.fc = AdapterV2SMoE(
+                in_features=config.n_embd,
+                out_features=config.intermediate_size,
+                num_experts=config.num_experts,
+                top_k=config.top_k,
+                bias=config.bias
+            )
+            # output projection
+            # if `head_size` is explicitly specified in the config, `n_emd` might not be equal to `head_size * n_head`
+            self.proj = AdapterV2SMoE(
+                in_features=config.intermediate_size,
+                out_features=config.n_embd,
+                num_experts=config.num_experts,
+                top_k=config.top_k,
+                bias=config.bias
+            )
+        else:
+            self.fc = AdapterV2Linear(config.n_embd, config.intermediate_size, bias=config.bias)
+            self.proj = AdapterV2Linear(config.intermediate_size, config.n_embd, bias=config.bias)
+
+        self.config = config
+
+    def _load_from_state_dict(self, state_dict: Dict, prefix: str, *args: Any, **kwargs: Any) -> None:
+        """For compatibility with base checkpoints."""
+        mapping = {
+            "fc.weight": "fc.linear.weight",
+            "fc.bias": "fc.linear.bias",
+            "proj.weight": "proj.linear.weight",
+            "proj.bias": "proj.linear.bias",
+        }
+        state_dict = map_old_state_dict_weights(state_dict, mapping, prefix)
+        super()._load_from_state_dict(state_dict, prefix, *args, **kwargs)
+
+
+class LLaMAMLP(litgpt.model.LLaMAMLP):
+    def __init__(self, config: Config) -> None:
+        nn.Module.__init__(self)
+        self.fc_1 = AdapterV2Linear(config.n_embd, config.intermediate_size, bias=config.bias)
+        self.fc_2 = AdapterV2Linear(config.n_embd, config.intermediate_size, bias=config.bias)
+        self.proj = AdapterV2Linear(config.intermediate_size, config.n_embd, bias=config.bias)
+
+        self.config = config
+
+    def _load_from_state_dict(self, state_dict: Dict, prefix: str, *args: Any, **kwargs: Any) -> None:
+        """For compatibility with base checkpoints."""
+        mapping = {
+            "fc_1.weight": "fc_1.linear.weight",
+            "fc_1.bias": "fc_1.linear.bias",
+            "fc_2.weight": "fc_2.linear.weight",
+            "fc_2.bias": "fc_2.linear.bias",
+            "proj.weight": "proj.linear.weight",
+            "proj.bias": "proj.linear.bias",
+        }
+        state_dict = map_old_state_dict_weights(state_dict, mapping, prefix)
+        super()._load_from_state_dict(state_dict, prefix, *args, **kwargs)
+
+
+class GemmaMLP(LLaMAMLP):
+    def forward(self, x: torch.Tensor) -> torch.Tensor:
+        x_fc_1 = self.fc_1(x)
+        x_fc_2 = self.fc_2(x)
+        x = torch.nn.functional.gelu(x_fc_1, approximate=self.config.gelu_approximate) * x_fc_2
+        return self.proj(x)
+
+
+class LLaMAMoE(litgpt.model.LLaMAMoE):
+    def __init__(self, config: Config) -> None:
+        nn.Module.__init__(self)
+        self.gate = AdapterV2Linear(config.n_embd, config.n_expert, bias=False)
+        self.experts = nn.ModuleList(LLaMAMLP(config) for _ in range(config.n_expert))
+
+        self.config = config
+
+    def _load_from_state_dict(self, state_dict: Dict, prefix: str, *args: Any, **kwargs: Any) -> None:
+        """For compatibility with base checkpoints."""
+        mapping = {"gate.weight": "gate.linear.weight"}
+        state_dict = map_old_state_dict_weights(state_dict, mapping, prefix)
+        super()._load_from_state_dict(state_dict, prefix, *args, **kwargs)
+
+
+def mark_only_adapter_v2_as_trainable(model: GPT) -> None:
+    """Sets requires_grad=False for all non-adapter weights"""
+    for name, param in model.named_parameters():
+        param.requires_grad = adapter_filter(name, param)

config.json

@@ -0,0 +1,46 @@
+{
+  "alpha": 0,
+  "architectures": [
+    "GPT"
+  ],
+  "auto_map": {
+    "AutoConfig": "adapter_v2.ConfigSMOE",
+    "AutoModelForCausalLM": "adapter_v2.GPT"
+  },
+  "bias": true,
+  "block_size": 2048,
+  "gelu_approximate": "tanh",
+  "head_size": 64,
+  "hf_config": {
+    "name": "phi-1_5",
+    "org": "microsoft"
+  },
+  "intermediate_size": 8192,
+  "lm_head_bias": true,
+  "mlp_class_name": "GptNeoxMLP",
+  "model_type": "gpt",
+  "n_embd": 2048,
+  "n_expert": 0,
+  "n_expert_per_token": 0,
+  "n_head": 32,
+  "n_layer": 24,
+  "n_query_groups": 32,
+  "name": "phi-1_5",
+  "norm_class_name": "LayerNorm",
+  "norm_eps": 1e-05,
+  "num_experts": 4,
+  "padded_vocab_size": 51200,
+  "padding_multiple": 512,
+  "parallel_residual": true,
+  "rope_base": 10000,
+  "rope_condense_ratio": 1,
+  "rope_n_elem": 32,
+  "rotary_percentage": 0.5,
+  "scale_embeddings": false,
+  "shared_attention_norm": true,
+  "top_k": 1,
+  "torch_dtype": "float32",
+  "transformers_version": "4.41.2",
+  "use_smoe": false,
+  "vocab_size": 50257
+}

model-00001-of-00006.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:9257c7fe5285b1efee2affb45f72e2a23f667e13dc557b17aed91a2780d80a22
+size 973924520

model-00002-of-00006.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:3b4cbfc9c16bf5fea33a09ae60edce34fae39afd0e2dfbc5a60a680b699bce63
+size 941001200

model-00003-of-00006.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:7429e9ddd1b770fc96bda8c5b77ae66765934912ad2e8517cff870052fb08954
+size 991308496

model-00004-of-00006.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:dddccb48c26c9ff3113064fd8b8744921960f77b517cc0a39dda23a163f02712
+size 957868952

model-00005-of-00006.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:050a1695ccfe9fe40fadfad1306b47219b76177732e0d7d920083625e01b5f5f
+size 941058528

model-00006-of-00006.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:6a623b66694dc4ee1075e5903c27e03dbeef8b507470b0f8a14c1177f783aac4
+size 873727464

model.safetensors.index.json

@@ -0,0 +1,490 @@
+{
+  "metadata": {
+    "total_size": 5678836480
+  },
+  "weight_map": {
+    "lm_head.adapter_bias": "model-00001-of-00006.safetensors",
+    "lm_head.adapter_scale": "model-00001-of-00006.safetensors",
+    "lm_head.linear.bias": "model-00001-of-00006.safetensors",
+    "lm_head.linear.weight": "model-00001-of-00006.safetensors",
+    "transformer.h.0.attn.attn.adapter_bias": "model-00001-of-00006.safetensors",
+    "transformer.h.0.attn.attn.adapter_scale": "model-00001-of-00006.safetensors",
+    "transformer.h.0.attn.attn.linear.bias": "model-00001-of-00006.safetensors",
+    "transformer.h.0.attn.attn.linear.weight": "model-00001-of-00006.safetensors",
+    "transformer.h.0.attn.proj.adapter_bias": "model-00001-of-00006.safetensors",
+    "transformer.h.0.attn.proj.adapter_scale": "model-00001-of-00006.safetensors",
+    "transformer.h.0.attn.proj.linear.bias": "model-00001-of-00006.safetensors",
+    "transformer.h.0.attn.proj.linear.weight": "model-00001-of-00006.safetensors",
+    "transformer.h.0.mlp.fc.adapter_bias": "model-00001-of-00006.safetensors",
+    "transformer.h.0.mlp.fc.adapter_scale": "model-00001-of-00006.safetensors",
+    "transformer.h.0.mlp.fc.linear.bias": "model-00001-of-00006.safetensors",
+    "transformer.h.0.mlp.fc.linear.weight": "model-00001-of-00006.safetensors",
+    "transformer.h.0.mlp.proj.adapter_bias": "model-00001-of-00006.safetensors",
+    "transformer.h.0.mlp.proj.adapter_scale": "model-00001-of-00006.safetensors",
+    "transformer.h.0.mlp.proj.linear.bias": "model-00002-of-00006.safetensors",
+    "transformer.h.0.mlp.proj.linear.weight": "model-00002-of-00006.safetensors",
+    "transformer.h.0.norm_1.bias": "model-00001-of-00006.safetensors",
+    "transformer.h.0.norm_1.weight": "model-00001-of-00006.safetensors",
+    "transformer.h.1.attn.attn.adapter_bias": "model-00002-of-00006.safetensors",
+    "transformer.h.1.attn.attn.adapter_scale": "model-00002-of-00006.safetensors",
+    "transformer.h.1.attn.attn.linear.bias": "model-00002-of-00006.safetensors",
+    "transformer.h.1.attn.attn.linear.weight": "model-00002-of-00006.safetensors",
+    "transformer.h.1.attn.proj.adapter_bias": "model-00002-of-00006.safetensors",
+    "transformer.h.1.attn.proj.adapter_scale": "model-00002-of-00006.safetensors",
+    "transformer.h.1.attn.proj.linear.bias": "model-00002-of-00006.safetensors",
+    "transformer.h.1.attn.proj.linear.weight": "model-00002-of-00006.safetensors",
+    "transformer.h.1.mlp.fc.adapter_bias": "model-00002-of-00006.safetensors",
+    "transformer.h.1.mlp.fc.adapter_scale": "model-00002-of-00006.safetensors",
+    "transformer.h.1.mlp.fc.linear.bias": "model-00002-of-00006.safetensors",
+    "transformer.h.1.mlp.fc.linear.weight": "model-00002-of-00006.safetensors",
+    "transformer.h.1.mlp.proj.adapter_bias": "model-00002-of-00006.safetensors",
+    "transformer.h.1.mlp.proj.adapter_scale": "model-00002-of-00006.safetensors",
+    "transformer.h.1.mlp.proj.linear.bias": "model-00002-of-00006.safetensors",
+    "transformer.h.1.mlp.proj.linear.weight": "model-00002-of-00006.safetensors",
+    "transformer.h.1.norm_1.bias": "model-00002-of-00006.safetensors",
+    "transformer.h.1.norm_1.weight": "model-00002-of-00006.safetensors",
+    "transformer.h.10.attn.adapter_wte.weight": "model-00004-of-00006.safetensors",
+    "transformer.h.10.attn.attn.adapter_bias": "model-00003-of-00006.safetensors",
+    "transformer.h.10.attn.attn.adapter_scale": "model-00003-of-00006.safetensors",
+    "transformer.h.10.attn.attn.linear.bias": "model-00003-of-00006.safetensors",
+    "transformer.h.10.attn.attn.linear.weight": "model-00003-of-00006.safetensors",
+    "transformer.h.10.attn.gating_factor": "model-00003-of-00006.safetensors",
+    "transformer.h.10.attn.proj.adapter_bias": "model-00003-of-00006.safetensors",
+    "transformer.h.10.attn.proj.adapter_scale": "model-00003-of-00006.safetensors",
+    "transformer.h.10.attn.proj.linear.bias": "model-00004-of-00006.safetensors",
+    "transformer.h.10.attn.proj.linear.weight": "model-00004-of-00006.safetensors",
+    "transformer.h.10.mlp.fc.adapter_bias": "model-00004-of-00006.safetensors",
+    "transformer.h.10.mlp.fc.adapter_scale": "model-00004-of-00006.safetensors",
+    "transformer.h.10.mlp.fc.linear.bias": "model-00004-of-00006.safetensors",
+    "transformer.h.10.mlp.fc.linear.weight": "model-00004-of-00006.safetensors",
+    "transformer.h.10.mlp.proj.adapter_bias": "model-00004-of-00006.safetensors",
+    "transformer.h.10.mlp.proj.adapter_scale": "model-00004-of-00006.safetensors",
+    "transformer.h.10.mlp.proj.linear.bias": "model-00004-of-00006.safetensors",
+    "transformer.h.10.mlp.proj.linear.weight": "model-00004-of-00006.safetensors",
+    "transformer.h.10.norm_1.bias": "model-00003-of-00006.safetensors",
+    "transformer.h.10.norm_1.weight": "model-00003-of-00006.safetensors",
+    "transformer.h.11.attn.adapter_wte.weight": "model-00004-of-00006.safetensors",
+    "transformer.h.11.attn.attn.adapter_bias": "model-00004-of-00006.safetensors",
+    "transformer.h.11.attn.attn.adapter_scale": "model-00004-of-00006.safetensors",
+    "transformer.h.11.attn.attn.linear.bias": "model-00004-of-00006.safetensors",
+    "transformer.h.11.attn.attn.linear.weight": "model-00004-of-00006.safetensors",
+    "transformer.h.11.attn.gating_factor": "model-00004-of-00006.safetensors",
+    "transformer.h.11.attn.proj.adapter_bias": "model-00004-of-00006.safetensors",
+    "transformer.h.11.attn.proj.adapter_scale": "model-00004-of-00006.safetensors",
+    "transformer.h.11.attn.proj.linear.bias": "model-00004-of-00006.safetensors",
+    "transformer.h.11.attn.proj.linear.weight": "model-00004-of-00006.safetensors",
+    "transformer.h.11.mlp.fc.adapter_bias": "model-00004-of-00006.safetensors",
+    "transformer.h.11.mlp.fc.adapter_scale": "model-00004-of-00006.safetensors",
+    "transformer.h.11.mlp.fc.linear.bias": "model-00004-of-00006.safetensors",
+    "transformer.h.11.mlp.fc.linear.weight": "model-00004-of-00006.safetensors",
+    "transformer.h.11.mlp.proj.adapter_bias": "model-00004-of-00006.safetensors",
+    "transformer.h.11.mlp.proj.adapter_scale": "model-00004-of-00006.safetensors",
+    "transformer.h.11.mlp.proj.linear.bias": "model-00004-of-00006.safetensors",
+    "transformer.h.11.mlp.proj.linear.weight": "model-00004-of-00006.safetensors",
+    "transformer.h.11.norm_1.bias": "model-00004-of-00006.safetensors",
+    "transformer.h.11.norm_1.weight": "model-00004-of-00006.safetensors",
+    "transformer.h.12.attn.adapter_wte.weight": "model-00004-of-00006.safetensors",
+    "transformer.h.12.attn.attn.adapter_bias": "model-00004-of-00006.safetensors",
+    "transformer.h.12.attn.attn.adapter_scale": "model-00004-of-00006.safetensors",
+    "transformer.h.12.attn.attn.linear.bias": "model-00004-of-00006.safetensors",
+    "transformer.h.12.attn.attn.linear.weight": "model-00004-of-00006.safetensors",
+    "transformer.h.12.attn.gating_factor": "model-00004-of-00006.safetensors",
+    "transformer.h.12.attn.proj.adapter_bias": "model-00004-of-00006.safetensors",
+    "transformer.h.12.attn.proj.adapter_scale": "model-00004-of-00006.safetensors",
+    "transformer.h.12.attn.proj.linear.bias": "model-00004-of-00006.safetensors",
+    "transformer.h.12.attn.proj.linear.weight": "model-00004-of-00006.safetensors",
+    "transformer.h.12.mlp.fc.adapter_bias": "model-00004-of-00006.safetensors",
+    "transformer.h.12.mlp.fc.adapter_scale": "model-00004-of-00006.safetensors",
+    "transformer.h.12.mlp.fc.linear.bias": "model-00004-of-00006.safetensors",
+    "transformer.h.12.mlp.fc.linear.weight": "model-00004-of-00006.safetensors",
+    "transformer.h.12.mlp.proj.adapter_bias": "model-00004-of-00006.safetensors",
+    "transformer.h.12.mlp.proj.adapter_scale": "model-00004-of-00006.safetensors",
+    "transformer.h.12.mlp.proj.linear.bias": "model-00004-of-00006.safetensors",
+    "transformer.h.12.mlp.proj.linear.weight": "model-00004-of-00006.safetensors",
+    "transformer.h.12.norm_1.bias": "model-00004-of-00006.safetensors",
+    "transformer.h.12.norm_1.weight": "model-00004-of-00006.safetensors",
+    "transformer.h.13.attn.adapter_wte.weight": "model-00004-of-00006.safetensors",
+    "transformer.h.13.attn.attn.adapter_bias": "model-00004-of-00006.safetensors",
+    "transformer.h.13.attn.attn.adapter_scale": "model-00004-of-00006.safetensors",
+    "transformer.h.13.attn.attn.linear.bias": "model-00004-of-00006.safetensors",
+    "transformer.h.13.attn.attn.linear.weight": "model-00004-of-00006.safetensors",
+    "transformer.h.13.attn.gating_factor": "model-00004-of-00006.safetensors",
+    "transformer.h.13.attn.proj.adapter_bias": "model-00004-of-00006.safetensors",
+    "transformer.h.13.attn.proj.adapter_scale": "model-00004-of-00006.safetensors",
+    "transformer.h.13.attn.proj.linear.bias": "model-00004-of-00006.safetensors",
+    "transformer.h.13.attn.proj.linear.weight": "model-00004-of-00006.safetensors",
+    "transformer.h.13.mlp.fc.adapter_bias": "model-00004-of-00006.safetensors",
+    "transformer.h.13.mlp.fc.adapter_scale": "model-00004-of-00006.safetensors",
+    "transformer.h.13.mlp.fc.linear.bias": "model-00004-of-00006.safetensors",
+    "transformer.h.13.mlp.fc.linear.weight": "model-00004-of-00006.safetensors",
+    "transformer.h.13.mlp.proj.adapter_bias": "model-00004-of-00006.safetensors",
+    "transformer.h.13.mlp.proj.adapter_scale": "model-00004-of-00006.safetensors",
+    "transformer.h.13.mlp.proj.linear.bias": "model-00004-of-00006.safetensors",
+    "transformer.h.13.mlp.proj.linear.weight": "model-00004-of-00006.safetensors",
+    "transformer.h.13.norm_1.bias": "model-00004-of-00006.safetensors",
+    "transformer.h.13.norm_1.weight": "model-00004-of-00006.safetensors",
+    "transformer.h.14.attn.adapter_wte.weight": "model-00004-of-00006.safetensors",
+    "transformer.h.14.attn.attn.adapter_bias": "model-00004-of-00006.safetensors",
+    "transformer.h.14.attn.attn.adapter_scale": "model-00004-of-00006.safetensors",
+    "transformer.h.14.attn.attn.linear.bias": "model-00004-of-00006.safetensors",
+    "transformer.h.14.attn.attn.linear.weight": "model-00004-of-00006.safetensors",
+    "transformer.h.14.attn.gating_factor": "model-00004-of-00006.safetensors",
+    "transformer.h.14.attn.proj.adapter_bias": "model-00004-of-00006.safetensors",
+    "transformer.h.14.attn.proj.adapter_scale": "model-00004-of-00006.safetensors",
+    "transformer.h.14.attn.proj.linear.bias": "model-00004-of-00006.safetensors",
+    "transformer.h.14.attn.proj.linear.weight": "model-00004-of-00006.safetensors",
+    "transformer.h.14.mlp.fc.adapter_bias": "model-00004-of-00006.safetensors",
+    "transformer.h.14.mlp.fc.adapter_scale": "model-00004-of-00006.safetensors",
+    "transformer.h.14.mlp.fc.linear.bias": "model-00004-of-00006.safetensors",
+    "transformer.h.14.mlp.fc.linear.weight": "model-00004-of-00006.safetensors",
+    "transformer.h.14.mlp.proj.adapter_bias": "model-00004-of-00006.safetensors",
+    "transformer.h.14.mlp.proj.adapter_scale": "model-00004-of-00006.safetensors",
+    "transformer.h.14.mlp.proj.linear.bias": "model-00004-of-00006.safetensors",
+    "transformer.h.14.mlp.proj.linear.weight": "model-00004-of-00006.safetensors",
+    "transformer.h.14.norm_1.bias": "model-00004-of-00006.safetensors",
+    "transformer.h.14.norm_1.weight": "model-00004-of-00006.safetensors",
+    "transformer.h.15.attn.adapter_wte.weight": "model-00005-of-00006.safetensors",
+    "transformer.h.15.attn.attn.adapter_bias": "model-00004-of-00006.safetensors",
+    "transformer.h.15.attn.attn.adapter_scale": "model-00004-of-00006.safetensors",
+    "transformer.h.15.attn.attn.linear.bias": "model-00005-of-00006.safetensors",
+    "transformer.h.15.attn.attn.linear.weight": "model-00005-of-00006.safetensors",
+    "transformer.h.15.attn.gating_factor": "model-00004-of-00006.safetensors",
+    "transformer.h.15.attn.proj.adapter_bias": "model-00005-of-00006.safetensors",
+    "transformer.h.15.attn.proj.adapter_scale": "model-00005-of-00006.safetensors",
+    "transformer.h.15.attn.proj.linear.bias": "model-00005-of-00006.safetensors",
+    "transformer.h.15.attn.proj.linear.weight": "model-00005-of-00006.safetensors",
+    "transformer.h.15.mlp.fc.adapter_bias": "model-00005-of-00006.safetensors",
+    "transformer.h.15.mlp.fc.adapter_scale": "model-00005-of-00006.safetensors",
+    "transformer.h.15.mlp.fc.linear.bias": "model-00005-of-00006.safetensors",
+    "transformer.h.15.mlp.fc.linear.weight": "model-00005-of-00006.safetensors",
+    "transformer.h.15.mlp.proj.adapter_bias": "model-00005-of-00006.safetensors",
+    "transformer.h.15.mlp.proj.adapter_scale": "model-00005-of-00006.safetensors",
+    "transformer.h.15.mlp.proj.linear.bias": "model-00005-of-00006.safetensors",
+    "transformer.h.15.mlp.proj.linear.weight": "model-00005-of-00006.safetensors",
+    "transformer.h.15.norm_1.bias": "model-00004-of-00006.safetensors",
+    "transformer.h.15.norm_1.weight": "model-00004-of-00006.safetensors",
+    "transformer.h.16.attn.adapter_wte.weight": "model-00005-of-00006.safetensors",
+    "transformer.h.16.attn.attn.adapter_bias": "model-00005-of-00006.safetensors",
+    "transformer.h.16.attn.attn.adapter_scale": "model-00005-of-00006.safetensors",
+    "transformer.h.16.attn.attn.linear.bias": "model-00005-of-00006.safetensors",
+    "transformer.h.16.attn.attn.linear.weight": "model-00005-of-00006.safetensors",
+    "transformer.h.16.attn.gating_factor": "model-00005-of-00006.safetensors",
+    "transformer.h.16.attn.proj.adapter_bias": "model-00005-of-00006.safetensors",
+    "transformer.h.16.attn.proj.adapter_scale": "model-00005-of-00006.safetensors",
+    "transformer.h.16.attn.proj.linear.bias": "model-00005-of-00006.safetensors",
+    "transformer.h.16.attn.proj.linear.weight": "model-00005-of-00006.safetensors",
+    "transformer.h.16.mlp.fc.adapter_bias": "model-00005-of-00006.safetensors",
+    "transformer.h.16.mlp.fc.adapter_scale": "model-00005-of-00006.safetensors",
+    "transformer.h.16.mlp.fc.linear.bias": "model-00005-of-00006.safetensors",
+    "transformer.h.16.mlp.fc.linear.weight": "model-00005-of-00006.safetensors",
+    "transformer.h.16.mlp.proj.adapter_bias": "model-00005-of-00006.safetensors",
+    "transformer.h.16.mlp.proj.adapter_scale": "model-00005-of-00006.safetensors",
+    "transformer.h.16.mlp.proj.linear.bias": "model-00005-of-00006.safetensors",
+    "transformer.h.16.mlp.proj.linear.weight": "model-00005-of-00006.safetensors",
+    "transformer.h.16.norm_1.bias": "model-00005-of-00006.safetensors",
+    "transformer.h.16.norm_1.weight": "model-00005-of-00006.safetensors",
+    "transformer.h.17.attn.adapter_wte.weight": "model-00005-of-00006.safetensors",
+    "transformer.h.17.attn.attn.adapter_bias": "model-00005-of-00006.safetensors",
+    "transformer.h.17.attn.attn.adapter_scale": "model-00005-of-00006.safetensors",
+    "transformer.h.17.attn.attn.linear.bias": "model-00005-of-00006.safetensors",
+    "transformer.h.17.attn.attn.linear.weight": "model-00005-of-00006.safetensors",
+    "transformer.h.17.attn.gating_factor": "model-00005-of-00006.safetensors",
+    "transformer.h.17.attn.proj.adapter_bias": "model-00005-of-00006.safetensors",
+    "transformer.h.17.attn.proj.adapter_scale": "model-00005-of-00006.safetensors",
+    "transformer.h.17.attn.proj.linear.bias": "model-00005-of-00006.safetensors",
+    "transformer.h.17.attn.proj.linear.weight": "model-00005-of-00006.safetensors",
+    "transformer.h.17.mlp.fc.adapter_bias": "model-00005-of-00006.safetensors",
+    "transformer.h.17.mlp.fc.adapter_scale": "model-00005-of-00006.safetensors",
+    "transformer.h.17.mlp.fc.linear.bias": "model-00005-of-00006.safetensors",
+    "transformer.h.17.mlp.fc.linear.weight": "model-00005-of-00006.safetensors",
+    "transformer.h.17.mlp.proj.adapter_bias": "model-00005-of-00006.safetensors",
+    "transformer.h.17.mlp.proj.adapter_scale": "model-00005-of-00006.safetensors",
+    "transformer.h.17.mlp.proj.linear.bias": "model-00005-of-00006.safetensors",
+    "transformer.h.17.mlp.proj.linear.weight": "model-00005-of-00006.safetensors",
+    "transformer.h.17.norm_1.bias": "model-00005-of-00006.safetensors",
+    "transformer.h.17.norm_1.weight": "model-00005-of-00006.safetensors",
+    "transformer.h.18.attn.adapter_wte.weight": "model-00005-of-00006.safetensors",
+    "transformer.h.18.attn.attn.adapter_bias": "model-00005-of-00006.safetensors",
+    "transformer.h.18.attn.attn.adapter_scale": "model-00005-of-00006.safetensors",
+    "transformer.h.18.attn.attn.linear.bias": "model-00005-of-00006.safetensors",
+    "transformer.h.18.attn.attn.linear.weight": "model-00005-of-00006.safetensors",
+    "transformer.h.18.attn.gating_factor": "model-00005-of-00006.safetensors",
+    "transformer.h.18.attn.proj.adapter_bias": "model-00005-of-00006.safetensors",
+    "transformer.h.18.attn.proj.adapter_scale": "model-00005-of-00006.safetensors",
+    "transformer.h.18.attn.proj.linear.bias": "model-00005-of-00006.safetensors",
+    "transformer.h.18.attn.proj.linear.weight": "model-00005-of-00006.safetensors",
+    "transformer.h.18.mlp.fc.adapter_bias": "model-00005-of-00006.safetensors",
+    "transformer.h.18.mlp.fc.adapter_scale": "model-00005-of-00006.safetensors",
+    "transformer.h.18.mlp.fc.linear.bias": "model-00005-of-00006.safetensors",
+    "transformer.h.18.mlp.fc.linear.weight": "model-00005-of-00006.safetensors",
+    "transformer.h.18.mlp.proj.adapter_bias": "model-00005-of-00006.safetensors",
+    "transformer.h.18.mlp.proj.adapter_scale": "model-00005-of-00006.safetensors",
+    "transformer.h.18.mlp.proj.linear.bias": "model-00005-of-00006.safetensors",
+    "transformer.h.18.mlp.proj.linear.weight": "model-00005-of-00006.safetensors",
+    "transformer.h.18.norm_1.bias": "model-00005-of-00006.safetensors",
+    "transformer.h.18.norm_1.weight": "model-00005-of-00006.safetensors",
+    "transformer.h.19.attn.adapter_wte.weight": "model-00005-of-00006.safetensors",
+    "transformer.h.19.attn.attn.adapter_bias": "model-00005-of-00006.safetensors",
+    "transformer.h.19.attn.attn.adapter_scale": "model-00005-of-00006.safetensors",
+    "transformer.h.19.attn.attn.linear.bias": "model-00005-of-00006.safetensors",
+    "transformer.h.19.attn.attn.linear.weight": "model-00005-of-00006.safetensors",
+    "transformer.h.19.attn.gating_factor": "model-00005-of-00006.safetensors",
+    "transformer.h.19.attn.proj.adapter_bias": "model-00005-of-00006.safetensors",
+    "transformer.h.19.attn.proj.adapter_scale": "model-00005-of-00006.safetensors",
+    "transformer.h.19.attn.proj.linear.bias": "model-00005-of-00006.safetensors",
+    "transformer.h.19.attn.proj.linear.weight": "model-00005-of-00006.safetensors",
+    "transformer.h.19.mlp.fc.adapter_bias": "model-00005-of-00006.safetensors",
+    "transformer.h.19.mlp.fc.adapter_scale": "model-00005-of-00006.safetensors",
+    "transformer.h.19.mlp.fc.linear.bias": "model-00005-of-00006.safetensors",
+    "transformer.h.19.mlp.fc.linear.weight": "model-00005-of-00006.safetensors",
+    "transformer.h.19.mlp.proj.adapter_bias": "model-00005-of-00006.safetensors",
+    "transformer.h.19.mlp.proj.adapter_scale": "model-00005-of-00006.safetensors",
+    "transformer.h.19.mlp.proj.linear.bias": "model-00006-of-00006.safetensors",
+    "transformer.h.19.mlp.proj.linear.weight": "model-00006-of-00006.safetensors",
+    "transformer.h.19.norm_1.bias": "model-00005-of-00006.safetensors",
+    "transformer.h.19.norm_1.weight": "model-00005-of-00006.safetensors",
+    "transformer.h.2.attn.adapter_wte.weight": "model-00002-of-00006.safetensors",
+    "transformer.h.2.attn.attn.adapter_bias": "model-00002-of-00006.safetensors",
+    "transformer.h.2.attn.attn.adapter_scale": "model-00002-of-00006.safetensors",
+    "transformer.h.2.attn.attn.linear.bias": "model-00002-of-00006.safetensors",
+    "transformer.h.2.attn.attn.linear.weight": "model-00002-of-00006.safetensors",
+    "transformer.h.2.attn.gating_factor": "model-00002-of-00006.safetensors",
+    "transformer.h.2.attn.proj.adapter_bias": "model-00002-of-00006.safetensors",
+    "transformer.h.2.attn.proj.adapter_scale": "model-00002-of-00006.safetensors",
+    "transformer.h.2.attn.proj.linear.bias": "model-00002-of-00006.safetensors",
+    "transformer.h.2.attn.proj.linear.weight": "model-00002-of-00006.safetensors",
+    "transformer.h.2.mlp.fc.adapter_bias": "model-00002-of-00006.safetensors",
+    "transformer.h.2.mlp.fc.adapter_scale": "model-00002-of-00006.safetensors",
+    "transformer.h.2.mlp.fc.linear.bias": "model-00002-of-00006.safetensors",
+    "transformer.h.2.mlp.fc.linear.weight": "model-00002-of-00006.safetensors",
+    "transformer.h.2.mlp.proj.adapter_bias": "model-00002-of-00006.safetensors",
+    "transformer.h.2.mlp.proj.adapter_scale": "model-00002-of-00006.safetensors",
+    "transformer.h.2.mlp.proj.linear.bias": "model-00002-of-00006.safetensors",
+    "transformer.h.2.mlp.proj.linear.weight": "model-00002-of-00006.safetensors",
+    "transformer.h.2.norm_1.bias": "model-00002-of-00006.safetensors",
+    "transformer.h.2.norm_1.weight": "model-00002-of-00006.safetensors",
+    "transformer.h.20.attn.adapter_wte.weight": "model-00006-of-00006.safetensors",
+    "transformer.h.20.attn.attn.adapter_bias": "model-00006-of-00006.safetensors",
+    "transformer.h.20.attn.attn.adapter_scale": "model-00006-of-00006.safetensors",
+    "transformer.h.20.attn.attn.linear.bias": "model-00006-of-00006.safetensors",
+    "transformer.h.20.attn.attn.linear.weight": "model-00006-of-00006.safetensors",
+    "transformer.h.20.attn.gating_factor": "model-00006-of-00006.safetensors",
+    "transformer.h.20.attn.proj.adapter_bias": "model-00006-of-00006.safetensors",
+    "transformer.h.20.attn.proj.adapter_scale": "model-00006-of-00006.safetensors",
+    "transformer.h.20.attn.proj.linear.bias": "model-00006-of-00006.safetensors",
+    "transformer.h.20.attn.proj.linear.weight": "model-00006-of-00006.safetensors",
+    "transformer.h.20.mlp.fc.adapter_bias": "model-00006-of-00006.safetensors",
+    "transformer.h.20.mlp.fc.adapter_scale": "model-00006-of-00006.safetensors",
+    "transformer.h.20.mlp.fc.linear.bias": "model-00006-of-00006.safetensors",
+    "transformer.h.20.mlp.fc.linear.weight": "model-00006-of-00006.safetensors",
+    "transformer.h.20.mlp.proj.adapter_bias": "model-00006-of-00006.safetensors",
+    "transformer.h.20.mlp.proj.adapter_scale": "model-00006-of-00006.safetensors",
+    "transformer.h.20.mlp.proj.linear.bias": "model-00006-of-00006.safetensors",
+    "transformer.h.20.mlp.proj.linear.weight": "model-00006-of-00006.safetensors",
+    "transformer.h.20.norm_1.bias": "model-00006-of-00006.safetensors",
+    "transformer.h.20.norm_1.weight": "model-00006-of-00006.safetensors",
+    "transformer.h.21.attn.adapter_wte.weight": "model-00006-of-00006.safetensors",
+    "transformer.h.21.attn.attn.adapter_bias": "model-00006-of-00006.safetensors",
+    "transformer.h.21.attn.attn.adapter_scale": "model-00006-of-00006.safetensors",
+    "transformer.h.21.attn.attn.linear.bias": "model-00006-of-00006.safetensors",
+    "transformer.h.21.attn.attn.linear.weight": "model-00006-of-00006.safetensors",
+    "transformer.h.21.attn.gating_factor": "model-00006-of-00006.safetensors",
+    "transformer.h.21.attn.proj.adapter_bias": "model-00006-of-00006.safetensors",
+    "transformer.h.21.attn.proj.adapter_scale": "model-00006-of-00006.safetensors",
+    "transformer.h.21.attn.proj.linear.bias": "model-00006-of-00006.safetensors",
+    "transformer.h.21.attn.proj.linear.weight": "model-00006-of-00006.safetensors",
+    "transformer.h.21.mlp.fc.adapter_bias": "model-00006-of-00006.safetensors",
+    "transformer.h.21.mlp.fc.adapter_scale": "model-00006-of-00006.safetensors",
+    "transformer.h.21.mlp.fc.linear.bias": "model-00006-of-00006.safetensors",
+    "transformer.h.21.mlp.fc.linear.weight": "model-00006-of-00006.safetensors",
+    "transformer.h.21.mlp.proj.adapter_bias": "model-00006-of-00006.safetensors",
+    "transformer.h.21.mlp.proj.adapter_scale": "model-00006-of-00006.safetensors",
+    "transformer.h.21.mlp.proj.linear.bias": "model-00006-of-00006.safetensors",
+    "transformer.h.21.mlp.proj.linear.weight": "model-00006-of-00006.safetensors",
+    "transformer.h.21.norm_1.bias": "model-00006-of-00006.safetensors",
+    "transformer.h.21.norm_1.weight": "model-00006-of-00006.safetensors",
+    "transformer.h.22.attn.adapter_wte.weight": "model-00006-of-00006.safetensors",
+    "transformer.h.22.attn.attn.adapter_bias": "model-00006-of-00006.safetensors",
+    "transformer.h.22.attn.attn.adapter_scale": "model-00006-of-00006.safetensors",
+    "transformer.h.22.attn.attn.linear.bias": "model-00006-of-00006.safetensors",
+    "transformer.h.22.attn.attn.linear.weight": "model-00006-of-00006.safetensors",
+    "transformer.h.22.attn.gating_factor": "model-00006-of-00006.safetensors",
+    "transformer.h.22.attn.proj.adapter_bias": "model-00006-of-00006.safetensors",
+    "transformer.h.22.attn.proj.adapter_scale": "model-00006-of-00006.safetensors",
+    "transformer.h.22.attn.proj.linear.bias": "model-00006-of-00006.safetensors",
+    "transformer.h.22.attn.proj.linear.weight": "model-00006-of-00006.safetensors",
+    "transformer.h.22.mlp.fc.adapter_bias": "model-00006-of-00006.safetensors",
+    "transformer.h.22.mlp.fc.adapter_scale": "model-00006-of-00006.safetensors",
+    "transformer.h.22.mlp.fc.linear.bias": "model-00006-of-00006.safetensors",
+    "transformer.h.22.mlp.fc.linear.weight": "model-00006-of-00006.safetensors",
+    "transformer.h.22.mlp.proj.adapter_bias": "model-00006-of-00006.safetensors",
+    "transformer.h.22.mlp.proj.adapter_scale": "model-00006-of-00006.safetensors",
+    "transformer.h.22.mlp.proj.linear.bias": "model-00006-of-00006.safetensors",
+    "transformer.h.22.mlp.proj.linear.weight": "model-00006-of-00006.safetensors",
+    "transformer.h.22.norm_1.bias": "model-00006-of-00006.safetensors",
+    "transformer.h.22.norm_1.weight": "model-00006-of-00006.safetensors",
+    "transformer.h.23.attn.adapter_wte.weight": "model-00006-of-00006.safetensors",
+    "transformer.h.23.attn.attn.adapter_bias": "model-00006-of-00006.safetensors",
+    "transformer.h.23.attn.attn.adapter_scale": "model-00006-of-00006.safetensors",
+    "transformer.h.23.attn.attn.linear.bias": "model-00006-of-00006.safetensors",
+    "transformer.h.23.attn.attn.linear.weight": "model-00006-of-00006.safetensors",
+    "transformer.h.23.attn.gating_factor": "model-00006-of-00006.safetensors",
+    "transformer.h.23.attn.proj.adapter_bias": "model-00006-of-00006.safetensors",
+    "transformer.h.23.attn.proj.adapter_scale": "model-00006-of-00006.safetensors",
+    "transformer.h.23.attn.proj.linear.bias": "model-00006-of-00006.safetensors",
+    "transformer.h.23.attn.proj.linear.weight": "model-00006-of-00006.safetensors",
+    "transformer.h.23.mlp.fc.adapter_bias": "model-00006-of-00006.safetensors",
+    "transformer.h.23.mlp.fc.adapter_scale": "model-00006-of-00006.safetensors",
+    "transformer.h.23.mlp.fc.linear.bias": "model-00006-of-00006.safetensors",
+    "transformer.h.23.mlp.fc.linear.weight": "model-00006-of-00006.safetensors",
+    "transformer.h.23.mlp.proj.adapter_bias": "model-00006-of-00006.safetensors",
+    "transformer.h.23.mlp.proj.adapter_scale": "model-00006-of-00006.safetensors",
+    "transformer.h.23.mlp.proj.linear.bias": "model-00006-of-00006.safetensors",
+    "transformer.h.23.mlp.proj.linear.weight": "model-00006-of-00006.safetensors",
+    "transformer.h.23.norm_1.bias": "model-00006-of-00006.safetensors",
+    "transformer.h.23.norm_1.weight": "model-00006-of-00006.safetensors",
+    "transformer.h.3.attn.adapter_wte.weight": "model-00002-of-00006.safetensors",
+    "transformer.h.3.attn.attn.adapter_bias": "model-00002-of-00006.safetensors",
+    "transformer.h.3.attn.attn.adapter_scale": "model-00002-of-00006.safetensors",
+    "transformer.h.3.attn.attn.linear.bias": "model-00002-of-00006.safetensors",
+    "transformer.h.3.attn.attn.linear.weight": "model-00002-of-00006.safetensors",
+    "transformer.h.3.attn.gating_factor": "model-00002-of-00006.safetensors",
+    "transformer.h.3.attn.proj.adapter_bias": "model-00002-of-00006.safetensors",
+    "transformer.h.3.attn.proj.adapter_scale": "model-00002-of-00006.safetensors",
+    "transformer.h.3.attn.proj.linear.bias": "model-00002-of-00006.safetensors",
+    "transformer.h.3.attn.proj.linear.weight": "model-00002-of-00006.safetensors",
+    "transformer.h.3.mlp.fc.adapter_bias": "model-00002-of-00006.safetensors",
+    "transformer.h.3.mlp.fc.adapter_scale": "model-00002-of-00006.safetensors",
+    "transformer.h.3.mlp.fc.linear.bias": "model-00002-of-00006.safetensors",
+    "transformer.h.3.mlp.fc.linear.weight": "model-00002-of-00006.safetensors",
+    "transformer.h.3.mlp.proj.adapter_bias": "model-00002-of-00006.safetensors",
+    "transformer.h.3.mlp.proj.adapter_scale": "model-00002-of-00006.safetensors",
+    "transformer.h.3.mlp.proj.linear.bias": "model-00002-of-00006.safetensors",
+    "transformer.h.3.mlp.proj.linear.weight": "model-00002-of-00006.safetensors",
+    "transformer.h.3.norm_1.bias": "model-00002-of-00006.safetensors",
+    "transformer.h.3.norm_1.weight": "model-00002-of-00006.safetensors",
+    "transformer.h.4.attn.adapter_wte.weight": "model-00002-of-00006.safetensors",
+    "transformer.h.4.attn.attn.adapter_bias": "model-00002-of-00006.safetensors",
+    "transformer.h.4.attn.attn.adapter_scale": "model-00002-of-00006.safetensors",
+    "transformer.h.4.attn.attn.linear.bias": "model-00002-of-00006.safetensors",
+    "transformer.h.4.attn.attn.linear.weight": "model-00002-of-00006.safetensors",
+    "transformer.h.4.attn.gating_factor": "model-00002-of-00006.safetensors",
+    "transformer.h.4.attn.proj.adapter_bias": "model-00002-of-00006.safetensors",
+    "transformer.h.4.attn.proj.adapter_scale": "model-00002-of-00006.safetensors",
+    "transformer.h.4.attn.proj.linear.bias": "model-00002-of-00006.safetensors",
+    "transformer.h.4.attn.proj.linear.weight": "model-00002-of-00006.safetensors",
+    "transformer.h.4.mlp.fc.adapter_bias": "model-00002-of-00006.safetensors",
+    "transformer.h.4.mlp.fc.adapter_scale": "model-00002-of-00006.safetensors",
+    "transformer.h.4.mlp.fc.linear.bias": "model-00002-of-00006.safetensors",
+    "transformer.h.4.mlp.fc.linear.weight": "model-00002-of-00006.safetensors",
+    "transformer.h.4.mlp.proj.adapter_bias": "model-00002-of-00006.safetensors",
+    "transformer.h.4.mlp.proj.adapter_scale": "model-00002-of-00006.safetensors",
+    "transformer.h.4.mlp.proj.linear.bias": "model-00002-of-00006.safetensors",
+    "transformer.h.4.mlp.proj.linear.weight": "model-00002-of-00006.safetensors",
+    "transformer.h.4.norm_1.bias": "model-00002-of-00006.safetensors",
+    "transformer.h.4.norm_1.weight": "model-00002-of-00006.safetensors",
+    "transformer.h.5.attn.adapter_wte.weight": "model-00002-of-00006.safetensors",
+    "transformer.h.5.attn.attn.adapter_bias": "model-00002-of-00006.safetensors",
+    "transformer.h.5.attn.attn.adapter_scale": "model-00002-of-00006.safetensors",
+    "transformer.h.5.attn.attn.linear.bias": "model-00002-of-00006.safetensors",
+    "transformer.h.5.attn.attn.linear.weight": "model-00002-of-00006.safetensors",
+    "transformer.h.5.attn.gating_factor": "model-00002-of-00006.safetensors",
+    "transformer.h.5.attn.proj.adapter_bias": "model-00002-of-00006.safetensors",
+    "transformer.h.5.attn.proj.adapter_scale": "model-00002-of-00006.safetensors",
+    "transformer.h.5.attn.proj.linear.bias": "model-00002-of-00006.safetensors",
+    "transformer.h.5.attn.proj.linear.weight": "model-00002-of-00006.safetensors",
+    "transformer.h.5.mlp.fc.adapter_bias": "model-00002-of-00006.safetensors",
+    "transformer.h.5.mlp.fc.adapter_scale": "model-00002-of-00006.safetensors",
+    "transformer.h.5.mlp.fc.linear.bias": "model-00003-of-00006.safetensors",
+    "transformer.h.5.mlp.fc.linear.weight": "model-00003-of-00006.safetensors",
+    "transformer.h.5.mlp.proj.adapter_bias": "model-00003-of-00006.safetensors",
+    "transformer.h.5.mlp.proj.adapter_scale": "model-00003-of-00006.safetensors",
+    "transformer.h.5.mlp.proj.linear.bias": "model-00003-of-00006.safetensors",
+    "transformer.h.5.mlp.proj.linear.weight": "model-00003-of-00006.safetensors",
+    "transformer.h.5.norm_1.bias": "model-00002-of-00006.safetensors",
+    "transformer.h.5.norm_1.weight": "model-00002-of-00006.safetensors",
+    "transformer.h.6.attn.adapter_wte.weight": "model-00003-of-00006.safetensors",
+    "transformer.h.6.attn.attn.adapter_bias": "model-00003-of-00006.safetensors",
+    "transformer.h.6.attn.attn.adapter_scale": "model-00003-of-00006.safetensors",
+    "transformer.h.6.attn.attn.linear.bias": "model-00003-of-00006.safetensors",
+    "transformer.h.6.attn.attn.linear.weight": "model-00003-of-00006.safetensors",
+    "transformer.h.6.attn.gating_factor": "model-00003-of-00006.safetensors",
+    "transformer.h.6.attn.proj.adapter_bias": "model-00003-of-00006.safetensors",
+    "transformer.h.6.attn.proj.adapter_scale": "model-00003-of-00006.safetensors",
+    "transformer.h.6.attn.proj.linear.bias": "model-00003-of-00006.safetensors",
+    "transformer.h.6.attn.proj.linear.weight": "model-00003-of-00006.safetensors",
+    "transformer.h.6.mlp.fc.adapter_bias": "model-00003-of-00006.safetensors",
+    "transformer.h.6.mlp.fc.adapter_scale": "model-00003-of-00006.safetensors",
+    "transformer.h.6.mlp.fc.linear.bias": "model-00003-of-00006.safetensors",
+    "transformer.h.6.mlp.fc.linear.weight": "model-00003-of-00006.safetensors",
+    "transformer.h.6.mlp.proj.adapter_bias": "model-00003-of-00006.safetensors",
+    "transformer.h.6.mlp.proj.adapter_scale": "model-00003-of-00006.safetensors",
+    "transformer.h.6.mlp.proj.linear.bias": "model-00003-of-00006.safetensors",
+    "transformer.h.6.mlp.proj.linear.weight": "model-00003-of-00006.safetensors",
+    "transformer.h.6.norm_1.bias": "model-00003-of-00006.safetensors",
+    "transformer.h.6.norm_1.weight": "model-00003-of-00006.safetensors",
+    "transformer.h.7.attn.adapter_wte.weight": "model-00003-of-00006.safetensors",
+    "transformer.h.7.attn.attn.adapter_bias": "model-00003-of-00006.safetensors",
+    "transformer.h.7.attn.attn.adapter_scale": "model-00003-of-00006.safetensors",
+    "transformer.h.7.attn.attn.linear.bias": "model-00003-of-00006.safetensors",
+    "transformer.h.7.attn.attn.linear.weight": "model-00003-of-00006.safetensors",
+    "transformer.h.7.attn.gating_factor": "model-00003-of-00006.safetensors",
+    "transformer.h.7.attn.proj.adapter_bias": "model-00003-of-00006.safetensors",
+    "transformer.h.7.attn.proj.adapter_scale": "model-00003-of-00006.safetensors",
+    "transformer.h.7.attn.proj.linear.bias": "model-00003-of-00006.safetensors",
+    "transformer.h.7.attn.proj.linear.weight": "model-00003-of-00006.safetensors",
+    "transformer.h.7.mlp.fc.adapter_bias": "model-00003-of-00006.safetensors",
+    "transformer.h.7.mlp.fc.adapter_scale": "model-00003-of-00006.safetensors",
+    "transformer.h.7.mlp.fc.linear.bias": "model-00003-of-00006.safetensors",
+    "transformer.h.7.mlp.fc.linear.weight": "model-00003-of-00006.safetensors",
+    "transformer.h.7.mlp.proj.adapter_bias": "model-00003-of-00006.safetensors",
+    "transformer.h.7.mlp.proj.adapter_scale": "model-00003-of-00006.safetensors",
+    "transformer.h.7.mlp.proj.linear.bias": "model-00003-of-00006.safetensors",
+    "transformer.h.7.mlp.proj.linear.weight": "model-00003-of-00006.safetensors",
+    "transformer.h.7.norm_1.bias": "model-00003-of-00006.safetensors",
+    "transformer.h.7.norm_1.weight": "model-00003-of-00006.safetensors",
+    "transformer.h.8.attn.adapter_wte.weight": "model-00003-of-00006.safetensors",
+    "transformer.h.8.attn.attn.adapter_bias": "model-00003-of-00006.safetensors",
+    "transformer.h.8.attn.attn.adapter_scale": "model-00003-of-00006.safetensors",
+    "transformer.h.8.attn.attn.linear.bias": "model-00003-of-00006.safetensors",
+    "transformer.h.8.attn.attn.linear.weight": "model-00003-of-00006.safetensors",
+    "transformer.h.8.attn.gating_factor": "model-00003-of-00006.safetensors",
+    "transformer.h.8.attn.proj.adapter_bias": "model-00003-of-00006.safetensors",
+    "transformer.h.8.attn.proj.adapter_scale": "model-00003-of-00006.safetensors",
+    "transformer.h.8.attn.proj.linear.bias": "model-00003-of-00006.safetensors",
+    "transformer.h.8.attn.proj.linear.weight": "model-00003-of-00006.safetensors",
+    "transformer.h.8.mlp.fc.adapter_bias": "model-00003-of-00006.safetensors",
+    "transformer.h.8.mlp.fc.adapter_scale": "model-00003-of-00006.safetensors",
+    "transformer.h.8.mlp.fc.linear.bias": "model-00003-of-00006.safetensors",
+    "transformer.h.8.mlp.fc.linear.weight": "model-00003-of-00006.safetensors",
+    "transformer.h.8.mlp.proj.adapter_bias": "model-00003-of-00006.safetensors",
+    "transformer.h.8.mlp.proj.adapter_scale": "model-00003-of-00006.safetensors",
+    "transformer.h.8.mlp.proj.linear.bias": "model-00003-of-00006.safetensors",
+    "transformer.h.8.mlp.proj.linear.weight": "model-00003-of-00006.safetensors",
+    "transformer.h.8.norm_1.bias": "model-00003-of-00006.safetensors",
+    "transformer.h.8.norm_1.weight": "model-00003-of-00006.safetensors",
+    "transformer.h.9.attn.adapter_wte.weight": "model-00003-of-00006.safetensors",
+    "transformer.h.9.attn.attn.adapter_bias": "model-00003-of-00006.safetensors",
+    "transformer.h.9.attn.attn.adapter_scale": "model-00003-of-00006.safetensors",
+    "transformer.h.9.attn.attn.linear.bias": "model-00003-of-00006.safetensors",
+    "transformer.h.9.attn.attn.linear.weight": "model-00003-of-00006.safetensors",
+    "transformer.h.9.attn.gating_factor": "model-00003-of-00006.safetensors",
+    "transformer.h.9.attn.proj.adapter_bias": "model-00003-of-00006.safetensors",
+    "transformer.h.9.attn.proj.adapter_scale": "model-00003-of-00006.safetensors",
+    "transformer.h.9.attn.proj.linear.bias": "model-00003-of-00006.safetensors",
+    "transformer.h.9.attn.proj.linear.weight": "model-00003-of-00006.safetensors",
+    "transformer.h.9.mlp.fc.adapter_bias": "model-00003-of-00006.safetensors",
+    "transformer.h.9.mlp.fc.adapter_scale": "model-00003-of-00006.safetensors",
+    "transformer.h.9.mlp.fc.linear.bias": "model-00003-of-00006.safetensors",
+    "transformer.h.9.mlp.fc.linear.weight": "model-00003-of-00006.safetensors",
+    "transformer.h.9.mlp.proj.adapter_bias": "model-00003-of-00006.safetensors",
+    "transformer.h.9.mlp.proj.adapter_scale": "model-00003-of-00006.safetensors",
+    "transformer.h.9.mlp.proj.linear.bias": "model-00003-of-00006.safetensors",
+    "transformer.h.9.mlp.proj.linear.weight": "model-00003-of-00006.safetensors",
+    "transformer.h.9.norm_1.bias": "model-00003-of-00006.safetensors",
+    "transformer.h.9.norm_1.weight": "model-00003-of-00006.safetensors",
+    "transformer.ln_f.bias": "model-00006-of-00006.safetensors",
+    "transformer.ln_f.weight": "model-00006-of-00006.safetensors",
+    "transformer.wte.weight": "model-00001-of-00006.safetensors"
+  }
+}