| # Running Jina Embedding V3 on Text-Embedding-Inference | |
| * See branch: TEI-support | |
| * Changes Made to GTE styled architecture: | |
| 1. Removed the "roberta" prefix from all tensor names. | |
| 2. Renamed "mixer" to "attention" in encoder layers. | |
| 3. Converted "Wqkv" to "qkv_proj" for combined query, key, value projections. | |
| 4. Renamed "mlp.fc1" to "mlp.up_proj" and "mlp.fc2" to "mlp.down_proj". | |
| 5. Created "mlp.up_gate_proj" by duplicating and expanding "mlp.up_proj". | |
| 6. Renamed "norm1" to "attn_ln" and "norm2" to "mlp_ln" in encoder layers. | |
| 7. Changed "emb_ln" to "embeddings.LayerNorm". | |
| 8. Renamed "weight" to "gamma" and "bias" to "beta" for layer normalization layers. | |
| 9. Removed LoRA-related tensors. | |
| Features: | |
| 1. Structural Compatibility: The renamed model now closely matches the expected GTE architecture, allowing it to load without "tensor not found" errors. | |
| 2. Preservation of Core Weights: Most of the original model's weights are preserved, maintaining some of the learned features. | |
| 3. Adaptability: The script can handle various naming conventions and structures, making it somewhat flexible for future adjustments. | |
| 4. Transparency: The script provides a clear view of the tensor names and shapes after conversion, aiding in debugging. | |
| Limitations: | |
| 1. Approximated Architecture: The conversion is an approximation of the GTE architecture, not an exact match. This may affect model performance. | |
| 2. Loss of LoRA Adaptations: By removing LoRA-related tensors, we've lost the fine-tuning adaptations, potentially impacting the model's specialized capabilities. | |
| 3. Up-Gate Projection Approximation: The "up_gate_proj" is created by duplicating weights, which may not accurately represent the intended GTE architecture. | |
| 4. Potential Performance Impact: The structural changes, especially in the MLP layers, may affect the model's performance and output quality. | |
| 5. Lack of Positional Embeddings Handling: We haven't specifically addressed positional embeddings, which might be different between XLM-RoBERTa and GTE models. | |
| 6. Possible Missing Specialized Layers: There might be specialized layers or components in the GTE architecture that we haven't accounted for. | |
| 7. No Guarantee of Functional Equivalence: While the model now loads, there's no guarantee it will function identically to a true GTE model. | |
| 8. Config File Mismatch: We haven't addressed potential mismatches in the config.json file, which might cause issues during model initialization or inference. | |