Upload README.md

README.md

@@ -1,199 +1,152 @@
 ---
 library_name: transformers
-tags: []
+license: apache-2.0
+language:
+- en
+base_model:
+- answerdotai/ModernBERT-large
+pipeline_tag: text-classification
+author: Shreyan C (@thethinkmachine)
 ---
 
-# Model Card for Model ID
-
-<!-- Provide a quick summary of what the model is/does. -->
-
-
+# Maxwell Task Complexity Scorer-v0.2
 
+Maxwell-TCS-v0.2 is a task/instruction complexity score annotator based on the ModernBERT-Large and trained on a large dataset of English instructions. It is designed to be fast, efficient and accurate for scoring/annotation related tasks.
 ## Model Details
 
 ### Model Description
 
-<!-- Provide a longer summary of what this model is. -->
-
-This is the model card of a 🤗 transformers model that has been pushed on the Hub. This model card has been automatically generated.
-
-- **Developed by:** [More Information Needed]
-- **Funded by [optional]:** [More Information Needed]
-- **Shared by [optional]:** [More Information Needed]
-- **Model type:** [More Information Needed]
-- **Language(s) (NLP):** [More Information Needed]
-- **License:** [More Information Needed]
-- **Finetuned from model [optional]:** [More Information Needed]
-
-### Model Sources [optional]
-
-<!-- Provide the basic links for the model. -->
-
-- **Repository:** [More Information Needed]
-- **Paper [optional]:** [More Information Needed]
-- **Demo [optional]:** [More Information Needed]
-
-## Uses
+Maxwell-TCS-v0.2 is an experimental SOTA **t**ask **c**omplexity **s**corer based on the state-of-the-art ModernBERT-Large backbone. It is trained under a data heavy setting over 66.5K diverse instruction-score pairs, and performs at par with other [complexity scorers](https://huggingface.co/hkust-nlp/deita-complexity-scorer) 34 times larger in size. For a given user instruction, the model predicts normalized scores between 0 and 1 across a single complexity dimension.
 
-<!-- Address questions around how the model is intended to be used, including the foreseeable users of the model and those affected by the model. -->
+Maxwell-TCS can be used in a variety of downstreaming tasks such as prompt difficulty prediction, dataset annotation, dataset augmentation and more.
 
-### Direct Use
+- **Developed by:** [Shreyan C]([thethinkmachine (Shreyan C)](https://huggingface.co/thethinkmachine))
+- **Model type:** Bidirectional Encoder Representations from Transformers, based on the ModernBERT-Large architecture.
+- **Language(s) (NLP):** English (en)
+- **License:** Apache License, Version 2.0
+- **Finetuned from model**: ModernBERT-Large
+## Applications
 
-<!-- This section is for the model use without fine-tuning or plugging into a larger ecosystem/app. -->
-
-[More Information Needed]
-
-### Downstream Use [optional]
-
-<!-- This section is for the model use when fine-tuned for a task, or when plugged into a larger ecosystem/app -->
-
-[More Information Needed]
-
-### Out-of-Scope Use
-
-<!-- This section addresses misuse, malicious use, and uses that the model will not work well for. -->
-
-[More Information Needed]
-
-## Bias, Risks, and Limitations
-
-<!-- This section is meant to convey both technical and sociotechnical limitations. -->
-
-[More Information Needed]
+- **Prompt Complexity Scoring:** Maxwell can be used to predict the complexity of a given instruction or prompt.
+- **Dataset Annotation:** Maxwell can be used to annotate the complexity of instructions in a dataset.
+- **Reward Model**: Maxwell can be used as a reward model for reinforcement learning tasks.
 
 ### Recommendations
 
-<!-- This section is meant to convey recommendations with respect to the bias, risk, and technical limitations. -->
+#### To reproduce Evol-Complexity scores:
+To reproduce the original Evol-Complexity scores, it is recommended to use the following transformation over the predicted scores,
+$$\text{S}_{predicted} \times (6 - 1) + 1$$
+This formula effectively converts normalized scores back to the continuous range [1,6]. Now all that remains is to ensure that the model's predictions correspond to the original discrete scoring scale, which can be achieved simply by rounding the transformed scores to the nearest integer.
+#### To use a different scaling factor:
+
+If you wish to use a different scaling factor, simply replace the values in the formula above with the desired range. For example, to scale the scores to the range [max,min], use the denormalization formula,
+$$\text{S}_{predicted} \times (\text{max} - \text{min}) + \text{min}$$
+**Note: Scaling factors are arbitrary and can be adjusted as needed. Ordering of the scores is preserved during min-max scaling, so the model should still be able to predict the relative complexity of instructions accurately regardless of the scaling factor used.**
 
-Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model. More information needed for further recommendations.
 
-## How to Get Started with the Model
+## Get Started
 
 Use the code below to get started with the model.
 
-[More Information Needed]
+```python
+import torch
+from transformers import AutoTokenizer, AutoModelForSequenceClassification
+
+model_name = "thethinkmachine/Maxwell-Task-Complexity-Exp-v1"
+tokenizer = AutoTokenizer.from_pretrained(model_name)
+model = AutoModelForSequenceClassification.from_pretrained(model_name)
+
+def get_deita_complexity_score(question: str) -> int:
+    inputs = tokenizer(question, return_tensors="pt")
+    with torch.no_grad():
+        outputs = model(**inputs)
+    normalized_pred = outputs.logits.squeeze()
+    final_score = normalized_pred * (6 - 1) + 1
+    final_score = torch.clamp(final_score, min=1.0, max=6.0)
+    final_score = torch.round(final_score)    
+    return final_score.item()
+
+query = "What is the capital of France?"
+final_score = get_deita_complexity_score(query)
+print(final_score)
+```
 
 ## Training Details
 
 ### Training Data
 
-<!-- This should link to a Dataset Card, perhaps with a short stub of information on what the training data is all about as well as documentation related to data pre-processing or additional filtering. -->
-
-[More Information Needed]
-
-### Training Procedure
-
-<!-- This relates heavily to the Technical Specifications. Content here should link to that section when it is relevant to the training procedure. -->
-
-#### Preprocessing [optional]
-
-[More Information Needed]
-
-
-#### Training Hyperparameters
-
-- **Training regime:** [More Information Needed] <!--fp32, fp16 mixed precision, bf16 mixed precision, bf16 non-mixed precision, fp16 non-mixed precision, fp8 mixed precision -->
-
-#### Speeds, Sizes, Times [optional]
-
-<!-- This section provides information about throughput, start/end time, checkpoint size if relevant, etc. -->
+We use [BhabhaAI/DEITA-Complexity](https://huggingface.co/datasets/BhabhaAI/DEITA-Complexity) 'deita'set for training the model. The dataset contains 66.5K diverse English instructions along with their complexity scores computed using the DEITA-Evol-Complexity scoring scheme which uses an LLM-judge to rank a sextuple containing 1 seed + 5 progressively complexified (*evolved*) instructions based on their contextual complexity & difficulty. The scheme assigns scores within [1, 6] range, with 1 being the least complex and 6 being the most complex.
 
-[More Information Needed]
+However, the training dataset used was observed to have instruction-score pairs across a diversity of scores within the range [0,9]. We suspect that this range includes scoring errors, as anomalous scores (0, 7, 8, 9) account for less than 1% of the total instructions.
 
-## Evaluation
+The distribution of scores within the dataset is as follows:
+| Score | Frequency | Relative Freq. |
+| ----- | --------- | -------------- |
+| 0     | 7         | < 0.1%         |
+| 1     | 8729      | 13.3%          |
+| 2     | 5399      | 8.2%           |
+| 3     | 10937     | 16.7%          |
+| 4     | 9801      | 15%            |
+| 5     | 24485     | 37.4%          |
+| 6     | 6123      | 9.3%           |
+| 7     | 6         | < 0.1%         |
+| 8     | 5         | < 0.1%         |
+| 9     | 5         | < 0.1%         |
 
-<!-- This section describes the evaluation protocols and provides the results. -->
+To mitigate the influence of these outliers, instructions with scores > 6 and < 1 were pruned from the dataset. It may also be observed that the dataset is not perfectly balanced, as approximately 37.4% of the instructions have a mode score of 5, indicating a higher concentration around this score. Min-Max scaling was applied to normalize the scores within the range [0,1] and preserve the ordering of complexity relationships.
 
-### Testing Data, Factors & Metrics
-
-#### Testing Data
-
-<!-- This should link to a Dataset Card if possible. -->
-
-[More Information Needed]
-
-#### Factors
-
-<!-- These are the things the evaluation is disaggregating by, e.g., subpopulations or domains. -->
-
-[More Information Needed]
-
-#### Metrics
-
-<!-- These are the evaluation metrics being used, ideally with a description of why. -->
-
-[More Information Needed]
-
-### Results
-
-[More Information Needed]
-
-#### Summary
+## Bias, Risks, and Limitations
 
+From a qualitative standpoint, the training dataset is strongly biased towards general English instructions, which may not help the model generalize well to other languages. Furthermore, the dataset suffers from the scarcity of instructions from technical tasks such as coding, mathematics, etc. As such, the model is presently speculated to not perform well on instructions from these technical domains, further analysis is still required.
 
+You are advised to use the model keeping these factors in mind.
 
-## Model Examination [optional]
+### Training Procedure
 
-<!-- Relevant interpretability work for the model goes here -->
+#### Training Hyperparameters
 
-[More Information Needed]
+- **Base Model:** ModernBERT-Large
+- **Task:** Sequence Classification
+- **Training regime:** FP32 Non-Mixed Precision
+- **Batch size:** 8
+- **Max epochs:** 3
+- **Learning rate:** 5e-5
+- **Optimizer:** AdamW
+- **Warmup steps ratio:** 0.1
+- **Weight decay:** 0.01
+- **Max sequence length:** 512
+### LoRA Hyperparameters
+
+- **LoRA Target Modules:** "attn.Wo", "attn.Wqkv", "mlp.Wi", "mlp.Wo"
+- **LoRA Rank:** 32
+- **LoRA Alpha:** 64
+- **LoRA Dropout:** 0.1
+- **LoRA Initialization:** PISSA
 
 ## Environmental Impact
 
-<!-- Total emissions (in grams of CO2eq) and additional considerations, such as electricity usage, go here. Edit the suggested text below accordingly -->
-
-Carbon emissions can be estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
-
-- **Hardware Type:** [More Information Needed]
-- **Hours used:** [More Information Needed]
-- **Cloud Provider:** [More Information Needed]
-- **Compute Region:** [More Information Needed]
-- **Carbon Emitted:** [More Information Needed]
-
-## Technical Specifications [optional]
-
-### Model Architecture and Objective
-
-[More Information Needed]
-
-### Compute Infrastructure
-
-[More Information Needed]
-
-#### Hardware
-
-[More Information Needed]
-
-#### Software
-
-[More Information Needed]
-
-## Citation [optional]
-
-<!-- If there is a paper or blog post introducing the model, the APA and Bibtex information for that should go in this section. -->
-
-**BibTeX:**
-
-[More Information Needed]
-
-**APA:**
-
-[More Information Needed]
-
-## Glossary [optional]
+### CO2 Emissions
 
-<!-- If relevant, include terms and calculations in this section that can help readers understand the model or model card. -->
+Experiments were conducted using Google Cloud Platform in region asia-south1, which has a carbon efficiency of 0.92 kgCO2eq/kWh. A cumulative of 13.16 hours of computation was performed on hardware of type L4 (TDP of 72W).
 
-[More Information Needed]
+Total emissions are estimated to be 0.87 kgCO2eq of which 100% was directly offset by the cloud provider.
 
-## More Information [optional]
+Carbon emissions were estimated using the [Machine Learning Impact calculator](https://mlco2.github.io/impact#compute) presented in [Lacoste et al. (2019)](https://arxiv.org/abs/1910.09700).
 
-[More Information Needed]
+- **Hardware Type:** 1xL4 GPU
+- **Hours used:** 16 hours
+- **Cloud Provider:** Google Cloud Platform
+- **Compute Region:** South Asia
+- **Carbon Emitted:** 0.87 kgCO2eq (fully offset by provider)
+## Author
 
-## Model Card Authors [optional]
+Shreyan C [@thethinkmachine](https://huggingface.co/thethinkmachine)/ [@thinkingmachines](https://huggingface.co/thinkingmachines)
 
-[More Information Needed]
+For any queries, suggestions or feedback, please contact Shreyan C at *shreyan(at)bud(dot)studio* / *shreyan(dot)chaubey(at)gmail(dot)com*.
 
-## Model Card Contact
+## References
 
-[More Information Needed]
+- [[2312.15685] What Makes Good Data for Alignment? A Comprehensive Study of Automatic Data Selection in Instruction Tuning](https://arxiv.org/abs/2312.15685)
+- [[2404.02948] PiSSA: Principal Singular Values and Singular Vectors Adaptation of Large Language Models](https://arxiv.org/abs/2404.02948)
+- [DEITA-Complexity](https://huggingface.co/datasets/BhabhaAI/DEITA-Complexity)
+- [ModernBERT-Large](https://huggingface.co/answerdotai/ModernBERT-large)