THEATLAS/FactCC-PENS

FactCC model for PENS dataset

The model has been fine-tuned on the PENS dataset to better adapt to the task of factuality assessment for news headlines.

Original paper: Evaluating the Factual Consistency of Abstractive Text Summarization

PENS paper: PENS: A Dataset and Generic Framework for Personalized News Headline Generation

Related paper: Fact-Preserved Personalized News Headline Generation

Example on how to calculate the FactCC score :

from transformers import BertForSequenceClassification, BertTokenizer
model_path = 'THEATLAS/FactCC-PENS'

tokenizer = BertTokenizer.from_pretrained(model_path)
model = BertForSequenceClassification.from_pretrained(model_path)

text='''The US has "passed the peak" on new coronavirus cases, the White House reported. They predict that some states would reopen this month.
The US has over 637,000 confirmed Covid-19 cases and over 30,826 deaths, the highest for any country in the world.'''
wrong_summary = '''The pandemic has almost not affected the US'''

input_dict = tokenizer(text, wrong_summary, max_length=512, padding='max_length', truncation='only_first', return_tensors='pt')
logits = model(**input_dict).logits

probs = torch.nn.functional.softmax(logits, dim=1)
fact_scores = probs[0][0].item()

print(f"fact_scores: {fact_scores}")

---

The following introduction is copied from the manueldeprada/FactCC repository.

This is a more modern implementation of the model and code from the original github repo

This model is trained to predict whether a summary is factual with respect to the original text. Basic usage:

from transformers import BertForSequenceClassification, BertTokenizer
model_path = 'THEATLAS/FactCC-PENS'

tokenizer = BertTokenizer.from_pretrained(model_path)
model = BertForSequenceClassification.from_pretrained(model_path)

text='''The US has "passed the peak" on new coronavirus cases, the White House reported. They predict that some states would reopen this month.
The US has over 637,000 confirmed Covid-19 cases and over 30,826 deaths, the highest for any country in the world.'''
wrong_summary = '''The pandemic has almost not affected the US'''

input_dict = tokenizer(text, wrong_summary, max_length=512, padding='max_length', truncation='only_first', return_tensors='pt')
logits = model(**input_dict).logits
pred = logits.argmax(dim=1)
model.config.id2label[pred.item()] # prints: INCORRECT

It can also be used with a pipeline. Beware that since pipelines are not thought to be used with pair of sentences, and you have to use this double-list hack:

>>> from transformers import pipeline

>>> pipe=pipeline(model="THEATLAS/FactCC-PENS")
>>> pipe([[[text1,summary1]],[[text2,summary2]]],truncation='only_first',padding='max_length')
# output [{'label': 'INCORRECT', 'score': 0.9979124665260315}, {'label': 'CORRECT', 'score': 0.879124665260315}]