Maxwell Task Complexity Scorer-v1

Maxwell-v1 is a task/instruction complexity score annotator based on the ModernBERT-Large and trained on a large dataset of English instructions. It is designed to be fast, efficient and accurate for scoring/annotation related tasks.

Model Details

Model Description

Maxwell-v1 (MTCS-EXP-1) is an experimental task complexity scorer based on the new ModernBERT-Large backbone. It is trained over 66.5K diverse instruction-score pairs and performs similarly to other complexity scorers 34 times larger in size. For a given user instruction, the model predicts normalized scores between 0 and 1 across a single complexity dimension.

Maxwell-TCS can be used in a variety of downstreaming tasks such as prompt difficulty prediction, dataset annotation, dataset augmentation and more.

Developed by: thethinkmachine (Shreyan C))
Model type: Bidirectional Encoder Representations from Transformers, based on the ModernBERT-Large architecture.
Language(s) (NLP): English (en)
License: Apache License, Version 2.0
Finetuned from model: ModernBERT-Large

Applications

Prompt Complexity Scoring: Maxwell can be used to predict the complexity of a given instruction or prompt.
Dataset Annotation: Maxwell can be used to annotate the complexity of instructions in a dataset.
Reward Model: Maxwell can be used as a reward model for reinforcement learning tasks.

Bias, Risks, and Limitations

The dataset used for training has instruction-score pairs across a diversity of scores within the range [0,9]. However, it may be noted that the dataset is not perfectly balanced, with a higher concentration of scores around the mode score in the dataset, which is 5. Approximately 37.4% of the total instructions within the dataset have a complexity score of 5.

From a qualitative standpoint, the dataset is strongly biased towards general English instructions, which may not help the model generalize well to other languages. Furthermore, the dataset suffers from the scarcity of instructions from technical tasks such as coding, mathematics, etc. As such, the model is presently speculated to not perform well on instructions from these technical domains, further analysis is still required.

You are advised to use the model keeping these factors in mind.

Recommendations

To reduce the risk of biases, we normalized these scores to lie within the interval [0,1] using Min-Max Normalization before training the model. It is therefore recommended to use the model with normalized scores for better results.

NOTE: In case you wish to unnormalize the scores, simply multiply the normalized score by the class range (9).

Get Started

Use the code below to get started with the model.

from transformers import AutoTokenizer, AutoModelForSequenceClassification

tokenizer = AutoTokenizer.from_pretrained("budecosystem/Maxwell-Task-Complexity-Exp-v1")
model = AutoModelForSequenceClassification.from_pretrained("budecosystem/Maxwell-Task-Complexity-Exp-v1")

inputs = tokenizer("What is the capital of France?", return_tensors="pt")
outputs = model(**inputs)
logits = outputs.logits.item()
print(logits)
# 0.1020... (normalized score)
# 0.918... (unnormalized score = normalized score * 9)

Training Details

Training Data

We use BhabhaAI/DEITA-Complexity dataset for training the model. The dataset contains 66.5K diverse English instructions along with their complexity scores computed using the Evol-Complexity scoring scheme which ranks evolved instructions based on their complexity. The score labels range from 0 (least complex) to 9 (most complex), with the mode being 5.

Training Procedure

Training Hyperparameters

Base Model: ModernBERT-Large
Task: Sequence Classification
Training regime: FP32 Non-Mixed Precision
Batch size: 8
Max epochs: 3
Learning rate: 5e-5
Optimizer: AdamW
Warmup steps ratio: 0.1
Weight decay: 0.01
Max sequence length: 512

LoRA Hyperparameters

LoRA Target Modules: "attn.Wo", "attn.Wqkv", "mlp.Wi", "mlp.Wo"
LoRA Rank: 32
LoRA Alpha: 64
LoRA Dropout: 0.1
LoRA Initialization: PISSA

Environmental Impact

CO2 Emissions

Experiments were conducted using Google Cloud Platform in region asia-south1, which has a carbon efficiency of 0.92 kgCO2eq/kWh. A cumulative of 13.16 hours of computation was performed on hardware of type L4 (TDP of 72W).

Total emissions are estimated to be 0.87 kgCO2eq of which 100% was directly offset by the cloud provider.

Carbon emissions were estimated using the Machine Learning Impact calculator presented in Lacoste et al. (2019).

Hardware Type: 1xL4 GPU
Hours used: 16 hours
Cloud Provider: Google Cloud Platform
Compute Region: South Asia
Carbon Emitted: 0.87 kgCO2eq (fully offset by provider)

Author

Shreyan C @thethinkmachine/ @thinkingmachines

For any queries, suggestions or feedback, please contact Shreyan C at shreyan(at)bud(dot)studio / shreyan(dot)chaubey(at)gmail(dot)com.

budecosystem
/

Maxwell-Task-Complexity-Exp-v1