--- tags: - setfit - sentence-transformers - text-classification - generated_from_setfit_trainer widget: - text: Ofgem protects Usio Energy customers' supplies and credit balances - text: Ofgem completes investigation into EDF Energy networks - finds no breach of obligations - text: 'Cyprus, the tech island: reflections on the Reflect Festival' - text: Ofgem appoints preferred bidder for Burbo Bank Extension offshore transmission assets - text: 'Tech Turmoil: Google Discontinues Google Play Music, Users Left in Limbo' metrics: - accuracy pipeline_tag: text-classification library_name: setfit inference: false base_model: mixedbread-ai/mxbai-embed-large-v1 model-index: - name: SetFit with mixedbread-ai/mxbai-embed-large-v1 results: - task: type: text-classification name: Text Classification dataset: name: Unknown type: unknown split: test metrics: - type: accuracy value: 0.6621621621621622 name: Accuracy --- # SetFit with mixedbread-ai/mxbai-embed-large-v1 This is a [SetFit](https://github.com/huggingface/setfit) model that can be used for Text Classification. This SetFit model uses [mixedbread-ai/mxbai-embed-large-v1](https://huggingface.co/mixedbread-ai/mxbai-embed-large-v1) as the Sentence Transformer embedding model. A OneVsRestClassifier instance is used for classification. The model has been trained using an efficient few-shot learning technique that involves: 1. Fine-tuning a [Sentence Transformer](https://www.sbert.net) with contrastive learning. 2. Training a classification head with features from the fine-tuned Sentence Transformer. ## Model Details ### Model Description - **Model Type:** SetFit - **Sentence Transformer body:** [mixedbread-ai/mxbai-embed-large-v1](https://huggingface.co/mixedbread-ai/mxbai-embed-large-v1) - **Classification head:** a OneVsRestClassifier instance - **Maximum Sequence Length:** 512 tokens ### Model Sources - **Repository:** [SetFit on GitHub](https://github.com/huggingface/setfit) - **Paper:** [Efficient Few-Shot Learning Without Prompts](https://arxiv.org/abs/2209.11055) - **Blogpost:** [SetFit: Efficient Few-Shot Learning Without Prompts](https://huggingface.co/blog/setfit) ## Evaluation ### Metrics | Label | Accuracy | |:--------|:---------| | **all** | 0.6622 | ## Uses ### Direct Use for Inference First install the SetFit library: ```bash pip install setfit ``` Then you can load this model and run inference. ```python from setfit import SetFitModel # Download from the 🤗 Hub model = SetFitModel.from_pretrained("amplyfi/all-labels") # Run inference preds = model("Cyprus, the tech island: reflections on the Reflect Festival") ``` ## Training Details ### Training Set Metrics | Training set | Min | Median | Max | |:-------------|:----|:--------|:----| | Word count | 4 | 10.0203 | 30 | ### Training Hyperparameters - batch_size: (16, 16) - num_epochs: (10, 10) - max_steps: -1 - sampling_strategy: oversampling - num_iterations: 10 - body_learning_rate: (2e-05, 2e-05) - head_learning_rate: 2e-05 - loss: CosineSimilarityLoss - distance_metric: cosine_distance - margin: 0.25 - end_to_end: False - use_amp: False - warmup_proportion: 0.1 - l2_weight: 0.01 - seed: 42 - eval_max_steps: -1 - load_best_model_at_end: False ### Training Results | Epoch | Step | Training Loss | Validation Loss | |:------:|:-----:|:-------------:|:---------------:| | 0.0009 | 1 | 0.2478 | - | | 0.0452 | 50 | 0.2177 | - | | 0.0903 | 100 | 0.2097 | - | | 0.1355 | 150 | 0.1949 | - | | 0.1807 | 200 | 0.1787 | - | | 0.2258 | 250 | 0.1494 | - | | 0.2710 | 300 | 0.1225 | - | | 0.3162 | 350 | 0.1146 | - | | 0.3613 | 400 | 0.0845 | - | | 0.4065 | 450 | 0.0726 | - | | 0.4517 | 500 | 0.076 | - | | 0.4968 | 550 | 0.0579 | - | | 0.5420 | 600 | 0.0688 | - | | 0.5872 | 650 | 0.0533 | - | | 0.6323 | 700 | 0.0547 | - | | 0.6775 | 750 | 0.0492 | - | | 0.7227 | 800 | 0.0498 | - | | 0.7678 | 850 | 0.037 | - | | 0.8130 | 900 | 0.0357 | - | | 0.8582 | 950 | 0.0398 | - | | 0.9033 | 1000 | 0.0428 | - | | 0.9485 | 1050 | 0.0494 | - | | 0.9937 | 1100 | 0.0334 | - | | 1.0388 | 1150 | 0.0327 | - | | 1.0840 | 1200 | 0.0265 | - | | 1.1292 | 1250 | 0.0283 | - | | 1.1743 | 1300 | 0.0347 | - | | 1.2195 | 1350 | 0.0299 | - | | 1.2647 | 1400 | 0.0235 | - | | 1.3098 | 1450 | 0.0223 | - | | 1.3550 | 1500 | 0.0254 | - | | 1.4002 | 1550 | 0.0249 | - | | 1.4453 | 1600 | 0.0209 | - | | 1.4905 | 1650 | 0.0239 | - | | 1.5357 | 1700 | 0.0229 | - | | 1.5808 | 1750 | 0.0193 | - | | 1.6260 | 1800 | 0.017 | - | | 1.6712 | 1850 | 0.0205 | - | | 1.7164 | 1900 | 0.0162 | - | | 1.7615 | 1950 | 0.0179 | - | | 1.8067 | 2000 | 0.0196 | - | | 1.8519 | 2050 | 0.0138 | - | | 1.8970 | 2100 | 0.0196 | - | | 1.9422 | 2150 | 0.0118 | - | | 1.9874 | 2200 | 0.0166 | - | | 2.0325 | 2250 | 0.0104 | - | | 2.0777 | 2300 | 0.0139 | - | | 2.1229 | 2350 | 0.0139 | - | | 2.1680 | 2400 | 0.0122 | - | | 2.2132 | 2450 | 0.0128 | - | | 2.2584 | 2500 | 0.0111 | - | | 2.3035 | 2550 | 0.0136 | - | | 2.3487 | 2600 | 0.0108 | - | | 2.3939 | 2650 | 0.0112 | - | | 2.4390 | 2700 | 0.0117 | - | | 2.4842 | 2750 | 0.0147 | - | | 2.5294 | 2800 | 0.0139 | - | | 2.5745 | 2850 | 0.0137 | - | | 2.6197 | 2900 | 0.0124 | - | | 2.6649 | 2950 | 0.012 | - | | 2.7100 | 3000 | 0.0118 | - | | 2.7552 | 3050 | 0.0124 | - | | 2.8004 | 3100 | 0.0109 | - | | 2.8455 | 3150 | 0.01 | - | | 2.8907 | 3200 | 0.0109 | - | | 2.9359 | 3250 | 0.0072 | - | | 2.9810 | 3300 | 0.0102 | - | | 3.0262 | 3350 | 0.0102 | - | | 3.0714 | 3400 | 0.0141 | - | | 3.1165 | 3450 | 0.0143 | - | | 3.1617 | 3500 | 0.0105 | - | | 3.2069 | 3550 | 0.0132 | - | | 3.2520 | 3600 | 0.011 | - | | 3.2972 | 3650 | 0.0104 | - | | 3.3424 | 3700 | 0.0104 | - | | 3.3875 | 3750 | 0.0078 | - | | 3.4327 | 3800 | 0.0095 | - | | 3.4779 | 3850 | 0.0118 | - | | 3.5230 | 3900 | 0.0076 | - | | 3.5682 | 3950 | 0.0087 | - | | 3.6134 | 4000 | 0.0098 | - | | 3.6585 | 4050 | 0.0114 | - | | 3.7037 | 4100 | 0.0086 | - | | 3.7489 | 4150 | 0.01 | - | | 3.7940 | 4200 | 0.0102 | - | | 3.8392 | 4250 | 0.0077 | - | | 3.8844 | 4300 | 0.0076 | - | | 3.9295 | 4350 | 0.0082 | - | | 3.9747 | 4400 | 0.0095 | - | | 4.0199 | 4450 | 0.0055 | - | | 4.0650 | 4500 | 0.009 | - | | 4.1102 | 4550 | 0.0086 | - | | 4.1554 | 4600 | 0.0086 | - | | 4.2005 | 4650 | 0.0075 | - | | 4.2457 | 4700 | 0.009 | - | | 4.2909 | 4750 | 0.0068 | - | | 4.3360 | 4800 | 0.0096 | - | | 4.3812 | 4850 | 0.008 | - | | 4.4264 | 4900 | 0.0075 | - | | 4.4715 | 4950 | 0.0069 | - | | 4.5167 | 5000 | 0.0076 | - | | 4.5619 | 5050 | 0.0058 | - | | 4.6070 | 5100 | 0.0077 | - | | 4.6522 | 5150 | 0.0073 | - | | 4.6974 | 5200 | 0.0083 | - | | 4.7425 | 5250 | 0.0059 | - | | 4.7877 | 5300 | 0.0066 | - | | 4.8329 | 5350 | 0.0065 | - | | 4.8780 | 5400 | 0.006 | - | | 4.9232 | 5450 | 0.008 | - | | 4.9684 | 5500 | 0.0073 | - | | 5.0136 | 5550 | 0.01 | - | | 5.0587 | 5600 | 0.0047 | - | | 5.1039 | 5650 | 0.0057 | - | | 5.1491 | 5700 | 0.0069 | - | | 5.1942 | 5750 | 0.0055 | - | | 5.2394 | 5800 | 0.0082 | - | | 5.2846 | 5850 | 0.0067 | - | | 5.3297 | 5900 | 0.0081 | - | | 5.3749 | 5950 | 0.0079 | - | | 5.4201 | 6000 | 0.0051 | - | | 5.4652 | 6050 | 0.0073 | - | | 5.5104 | 6100 | 0.007 | - | | 5.5556 | 6150 | 0.0069 | - | | 5.6007 | 6200 | 0.0066 | - | | 5.6459 | 6250 | 0.0073 | - | | 5.6911 | 6300 | 0.0063 | - | | 5.7362 | 6350 | 0.0049 | - | | 5.7814 | 6400 | 0.0042 | - | | 5.8266 | 6450 | 0.0076 | - | | 5.8717 | 6500 | 0.0077 | - | | 5.9169 | 6550 | 0.0071 | - | | 5.9621 | 6600 | 0.0079 | - | | 6.0072 | 6650 | 0.0073 | - | | 6.0524 | 6700 | 0.0069 | - | | 6.0976 | 6750 | 0.0049 | - | | 6.1427 | 6800 | 0.0065 | - | | 6.1879 | 6850 | 0.0046 | - | | 6.2331 | 6900 | 0.0063 | - | | 6.2782 | 6950 | 0.0061 | - | | 6.3234 | 7000 | 0.0066 | - | | 6.3686 | 7050 | 0.0049 | - | | 6.4137 | 7100 | 0.0048 | - | | 6.4589 | 7150 | 0.0062 | - | | 6.5041 | 7200 | 0.0067 | - | | 6.5492 | 7250 | 0.0059 | - | | 6.5944 | 7300 | 0.0078 | - | | 6.6396 | 7350 | 0.0074 | - | | 6.6847 | 7400 | 0.0058 | - | | 6.7299 | 7450 | 0.007 | - | | 6.7751 | 7500 | 0.0059 | - | | 6.8202 | 7550 | 0.0061 | - | | 6.8654 | 7600 | 0.0057 | - | | 6.9106 | 7650 | 0.0062 | - | | 6.9557 | 7700 | 0.0056 | - | | 7.0009 | 7750 | 0.0054 | - | | 7.0461 | 7800 | 0.0063 | - | | 7.0912 | 7850 | 0.0066 | - | | 7.1364 | 7900 | 0.0051 | - | | 7.1816 | 7950 | 0.0063 | - | | 7.2267 | 8000 | 0.0053 | - | | 7.2719 | 8050 | 0.0045 | - | | 7.3171 | 8100 | 0.0065 | - | | 7.3622 | 8150 | 0.0057 | - | | 7.4074 | 8200 | 0.0068 | - | | 7.4526 | 8250 | 0.0058 | - | | 7.4977 | 8300 | 0.0077 | - | | 7.5429 | 8350 | 0.0062 | - | | 7.5881 | 8400 | 0.0057 | - | | 7.6332 | 8450 | 0.0047 | - | | 7.6784 | 8500 | 0.0051 | - | | 7.7236 | 8550 | 0.0063 | - | | 7.7687 | 8600 | 0.0043 | - | | 7.8139 | 8650 | 0.0041 | - | | 7.8591 | 8700 | 0.0055 | - | | 7.9042 | 8750 | 0.0049 | - | | 7.9494 | 8800 | 0.0066 | - | | 7.9946 | 8850 | 0.007 | - | | 8.0397 | 8900 | 0.0057 | - | | 8.0849 | 8950 | 0.0049 | - | | 8.1301 | 9000 | 0.0043 | - | | 8.1752 | 9050 | 0.0054 | - | | 8.2204 | 9100 | 0.0045 | - | | 8.2656 | 9150 | 0.0043 | - | | 8.3107 | 9200 | 0.0054 | - | | 8.3559 | 9250 | 0.0048 | - | | 8.4011 | 9300 | 0.0046 | - | | 8.4463 | 9350 | 0.0039 | - | | 8.4914 | 9400 | 0.0073 | - | | 8.5366 | 9450 | 0.0071 | - | | 8.5818 | 9500 | 0.0068 | - | | 8.6269 | 9550 | 0.0055 | - | | 8.6721 | 9600 | 0.0062 | - | | 8.7173 | 9650 | 0.0055 | - | | 8.7624 | 9700 | 0.0068 | - | | 8.8076 | 9750 | 0.0052 | - | | 8.8528 | 9800 | 0.0049 | - | | 8.8979 | 9850 | 0.005 | - | | 8.9431 | 9900 | 0.0033 | - | | 8.9883 | 9950 | 0.0064 | - | | 9.0334 | 10000 | 0.0057 | - | | 9.0786 | 10050 | 0.0056 | - | | 9.1238 | 10100 | 0.0066 | - | | 9.1689 | 10150 | 0.0046 | - | | 9.2141 | 10200 | 0.0043 | - | | 9.2593 | 10250 | 0.0041 | - | | 9.3044 | 10300 | 0.0066 | - | | 9.3496 | 10350 | 0.0046 | - | | 9.3948 | 10400 | 0.0056 | - | | 9.4399 | 10450 | 0.0043 | - | | 9.4851 | 10500 | 0.0045 | - | | 9.5303 | 10550 | 0.0048 | - | | 9.5754 | 10600 | 0.0057 | - | | 9.6206 | 10650 | 0.0055 | - | | 9.6658 | 10700 | 0.0042 | - | | 9.7109 | 10750 | 0.0063 | - | | 9.7561 | 10800 | 0.0047 | - | | 9.8013 | 10850 | 0.0046 | - | | 9.8464 | 10900 | 0.0045 | - | | 9.8916 | 10950 | 0.0047 | - | | 9.9368 | 11000 | 0.0057 | - | | 9.9819 | 11050 | 0.0061 | - | ### Framework Versions - Python: 3.10.12 - SetFit: 1.1.0 - Sentence Transformers: 3.3.1 - Transformers: 4.42.2 - PyTorch: 2.5.1+cu124 - Datasets: 3.1.0 - Tokenizers: 0.19.1 ## Citation ### BibTeX ```bibtex @article{https://doi.org/10.48550/arxiv.2209.11055, doi = {10.48550/ARXIV.2209.11055}, url = {https://arxiv.org/abs/2209.11055}, author = {Tunstall, Lewis and Reimers, Nils and Jo, Unso Eun Seo and Bates, Luke and Korat, Daniel and Wasserblat, Moshe and Pereg, Oren}, keywords = {Computation and Language (cs.CL), FOS: Computer and information sciences, FOS: Computer and information sciences}, title = {Efficient Few-Shot Learning Without Prompts}, publisher = {arXiv}, year = {2022}, copyright = {Creative Commons Attribution 4.0 International} } ```