Inquiry About PEFT Compatibility Issue
Hello,
I wanted to bring to your attention an issue I’ve encountered when using the Geneformer code with the newest version of PEFT (v0.14, released in December). This version apparently introduces a stricter validation for the task_type parameter specified in LoraConfig.
The Geneformer code snippet setting this parameter in perturber_utils.py:
peft_config = LoraConfig(
lora_alpha=128,
lora_dropout=0.1,
r=64,
bias="none",
task_type="TokenClassification",
)
The new PEFT version checks whether the provided task_type is among the accepted TaskType values, and it seems that "TokenClassification" is not recognized as valid. This validation step was not enforced in previous versions, and the code ran without issues.
To address this, I attempted two approaches:
- Changing task_type to "TOKEN_CLS" (which I presumed to be the correct value). However, this resulted in issues during the model evaluation step. Specifically, the classifier_predict function fails as predict_logits contains NaN values.
- Setting task_type to None, which allowed the code to run to completion.
Could you please clarify the correct value for task_type? Additionally, do you have any insight into why using "TOKEN_CLS" causes the evaluation step to fail?
Thank you very much for your time and assistance