Create README.md

README.md

@@ -0,0 +1,27 @@
+# Running Jina Embedding V3 on Text-Embedding-Inference
+Changes Made to GTE styled architecture:
+1. Removed the "roberta" prefix from all tensor names.
+2. Renamed "mixer" to "attention" in encoder layers.
+3. Converted "Wqkv" to "qkv_proj" for combined query, key, value projections.
+4. Renamed "mlp.fc1" to "mlp.up_proj" and "mlp.fc2" to "mlp.down_proj".
+5. Created "mlp.up_gate_proj" by duplicating and expanding "mlp.up_proj".
+6. Renamed "norm1" to "attn_ln" and "norm2" to "mlp_ln" in encoder layers.
+7. Changed "emb_ln" to "embeddings.LayerNorm".
+8. Renamed "weight" to "gamma" and "bias" to "beta" for layer normalization layers.
+9. Removed LoRA-related tensors.
+
+Features:
+1. Structural Compatibility: The renamed model now closely matches the expected GTE architecture, allowing it to load without "tensor not found" errors.
+2. Preservation of Core Weights: Most of the original model's weights are preserved, maintaining some of the learned features.
+3. Adaptability: The script can handle various naming conventions and structures, making it somewhat flexible for future adjustments.
+4. Transparency: The script provides a clear view of the tensor names and shapes after conversion, aiding in debugging.
+
+Limitations:
+1. Approximated Architecture: The conversion is an approximation of the GTE architecture, not an exact match. This may affect model performance.
+2. Loss of LoRA Adaptations: By removing LoRA-related tensors, we've lost the fine-tuning adaptations, potentially impacting the model's specialized capabilities.
+3. Up-Gate Projection Approximation: The "up_gate_proj" is created by duplicating weights, which may not accurately represent the intended GTE architecture.
+4. Potential Performance Impact: The structural changes, especially in the MLP layers, may affect the model's performance and output quality.
+5. Lack of Positional Embeddings Handling: We haven't specifically addressed positional embeddings, which might be different between XLM-RoBERTa and GTE models.
+6. Possible Missing Specialized Layers: There might be specialized layers or components in the GTE architecture that we haven't accounted for.
+7. No Guarantee of Functional Equivalence: While the model now loads, there's no guarantee it will function identically to a true GTE model.
+8. Config File Mismatch: We haven't addressed potential mismatches in the config.json file, which might cause issues during model initialization or inference.