Hugging Face's logo

daxa-ai
/
pebblo-classifier

Text Classification
Transformers
PyTorch
Joblib
English
distilbert
Inference Endpoints
Model card Files Community
13
nishan-dx Tihsrah-CD commited on
Commit
402f3c6
·
verified ·
1 Parent(s): 8fbe0dc

Model V9 Release (#12)

Browse files

- Update model to version 9: Improved performance metrics and evaluation results (5fbbe83dee7ef72c61a8173c4ccf27b19788fc2e)


Co-authored-by: Harshit <[email protected]>

Files changed (4) hide show
  1. README.md +80 -83
  2. config.json +6 -4
  3. label_encoder.joblib +2 -2
  4. pytorch_model.bin +2 -2
README.md CHANGED
@@ -1,12 +1,12 @@
1
  ---
2
  license: mit
3
  language:
4
- - en
5
  ---
6
 
7
  # Model Card for Model ID
8
 
9
- This model card outlines the Pebblo Classifier, a machine learning system specialized in text classification. Developed by DAXA.AI, this model is adept at categorizing various agreement documents within organizational structures, trained on 20 distinct labels.
10
 
11
  ## Model Details
12
 
@@ -88,102 +88,99 @@ print(decoded_label)
88
 
89
  ### Training Data
90
 
91
- The training dataset consists of 131,771 entries, with 20 unique labels. The labels span various document types, with instances distributed across three text sizes (128 ± x, 256 ± x, and 512 ± x words; x varies within 20).
92
  Here are the labels along with their respective counts in the dataset:
93
 
94
- | Agreement Type | Instances |
95
- | --------------------------------------- | --------- |
96
- | BOARD_MEETING_AGREEMENT | 4,225 |
97
- | CONSULTING_AGREEMENT | 2,965 |
98
- | CUSTOMER_LIST_AGREEMENT | 9,000 |
99
- | DISTRIBUTION_PARTNER_AGREEMENT | 5,162 |
100
- | EMPLOYEE_AGREEMENT | 3,921 |
101
- | ENTERPRISE_AGREEMENT | 4,217 |
102
- | ENTERPRISE_LICENSE_AGREEMENT | 9,000 |
103
- | EXECUTIVE_SEVERANCE_AGREEMENT | 9,000 |
104
- | FINANCIAL_REPORT_AGREEMENT | 8,381 |
105
- | HARMFUL_ADVICE | 2,025 |
106
- | INTERNAL_PRODUCT_ROADMAP_AGREEMENT | 7,037 |
107
- | LOAN_AND_SECURITY_AGREEMENT | 9,000 |
108
- | MEDICAL_ADVICE | 2,359 |
109
- | MERGER_AGREEMENT | 7,706 |
110
- | NDA_AGREEMENT | 5,229 |
111
- | NORMAL_TEXT | 9,000 |
112
- | PATENT_APPLICATION_FILLINGS_AGREEMENT | 9,000 |
113
- | PRICE_LIST_AGREEMENT | 9,000 |
114
- | SETTLEMENT_AGREEMENT | 3,754 |
115
- | SEXUAL_HARRASSMENT | 8,321 |
116
-
117
-
118
 
119
  ## Evaluation
120
 
121
  ### Testing Data & Metrics
122
 
123
  #### Testing Data
124
- Evaluation was performed on a dataset of 82,917 entries with a temperature range of 1-1.25 for randomness.
125
- Here are the labels along with their respective counts in the dataset:
126
-
127
- | Agreement Type | Instances |
128
- | --------------------------------------- | --------- |
129
- | BOARD_MEETING_AGREEMENT | 4,335 |
130
- | CONSULTING_AGREEMENT | 1,533 |
131
- | CUSTOMER_LIST_AGREEMENT | 4,995 |
132
- | DISTRIBUTION_PARTNER_AGREEMENT | 7,231 |
133
- | EMPLOYEE_AGREEMENT | 1,433 |
134
- | ENTERPRISE_AGREEMENT | 1,616 |
135
- | ENTERPRISE_LICENSE_AGREEMENT | 8,574 |
136
- | EXECUTIVE_SEVERANCE_AGREEMENT | 5,177 |
137
- | FINANCIAL_REPORT_AGREEMENT | 4,264 |
138
- | HARMFUL_ADVICE | 474 |
139
- | INTERNAL_PRODUCT_ROADMAP_AGREEMENT | 4,116 |
140
- | LOAN_AND_SECURITY_AGREEMENT | 6,354 |
141
- | MEDICAL_ADVICE | 289 |
142
- | MERGER_AGREEMENT | 7,079 |
143
- | NDA_AGREEMENT | 1,452 |
144
- | NORMAL_TEXT | 8,335 |
145
- | PATENT_APPLICATION_FILLINGS_AGREEMENT | 6,177 |
146
- | PRICE_LIST_AGREEMENT | 5,453 |
147
- | SETTLEMENT_AGREEMENT | 5,806 |
148
- | SEXUAL_HARRASSMENT | 4,750 |
149
 
 
 
150
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
151
 
152
  #### Metrics
153
 
154
- | Agreement Type | precision | recall | f1-score | support |
155
- | ------------------------------------------- | --------- | ------ | -------- | ------- |
156
- | BOARD_MEETING_AGREEMENT | 0.96 | 0.94 | 0.95 | 4335 |
157
- | CONSULTING_AGREEMENT | 0.77 | 0.89 | 0.83 | 1533 |
158
- | CUSTOMER_LIST_AGREEMENT | 0.84 | 0.87 | 0.85 | 4995 |
159
- | DISTRIBUTION_PARTNER_AGREEMENT | 0.71 | 0.64 | 0.67 | 7231 |
160
- | EMPLOYEE_AGREEMENT | 0.78 | 0.90 | 0.83 | 1433 |
161
- | ENTERPRISE_AGREEMENT | 0.19 | 0.72 | 0.30 | 1616 |
162
- | ENTERPRISE_LICENSE_AGREEMENT | 0.92 | 0.78 | 0.84 | 8574 |
163
- | EXECUTIVE_SEVERANCE_AGREEMENT | 0.96 | 0.85 | 0.90 | 5177 |
164
- | FINANCIAL_REPORT_AGREEMENT | 0.92 | 0.98 | 0.95 | 4264 |
165
- | HARMFUL_ADVICE | 0.82 | 0.92 | 0.87 | 474 |
166
- | INTERNAL_PRODUCT_ROADMAP_AGREEMENT | 0.94 | 0.97 | 0.96 | 4116 |
167
- | LOAN_AND_SECURITY_AGREEMENT | 0.92 | 0.96 | 0.94 | 6354 |
168
- | MEDICAL_ADVICE | 0.76 | 1.00 | 0.86 | 289 |
169
- | MERGER_AGREEMENT | 0.90 | 0.55 | 0.68 | 7079 |
170
- | NDA_AGREEMENT | 0.62 | 0.89 | 0.74 | 1452 |
171
- | NORMAL_TEXT | 0.99 | 0.99 | 0.99 | 6049 |
172
- | PATENT_APPLICATION_FILLINGS_AGREEMENT | 0.95 | 0.99 | 0.97 | 6177 |
173
- | PRICE_LIST_AGREEMENT | 0.81 | 0.75 | 0.78 | 5453 |
174
- | SETTLEMENT_AGREEMENT | 0.83 | 0.73 | 0.78 | 5806 |
175
- | SEXUAL_HARRASSMENT | 0.98 | 0.93 | 0.96 | 4750 |
176
- | | | | | |
177
- | accuracy | | | 0.84 | 87157 |
178
- | macro avg | 0.83 | 0.86 | 0.83 | 87157 |
179
- | weighted avg | 0.87 | 0.84 | 0.85 | 87157 |
180
-
181
 
182
  #### Results
183
 
184
- The model’s performance is summarized by precision, recall, and f1-score metrics, which are detailed across all 20 labels in the dataset. Based on the test data evaluation results, the model achieved an accuracy of 0.8376, a precision of 0.8744, and a recall of 0.8376. The F1-score, which is the harmonic mean of precision and recall, stands at 0.8478.
185
-
186
- The evaluation loss, which measures the discrepancy between the model’s predictions and the actual values, is 0.5616. Lower loss values indicate better model performance.
187
 
188
- The model was able to process approximately 101.886 samples per second during the evaluation, which took a total runtime of 855.4327 seconds. The model performed approximately 0.796 evaluation steps per second.
189
 
 
 
1
  ---
2
  license: mit
3
  language:
4
+ - en
5
  ---
6
 
7
  # Model Card for Model ID
8
 
9
+ This model card outlines the Pebblo Classifier, a machine learning system specialized in text classification. Developed by DAXA.AI, this model is adept at categorizing various agreement documents within organizational structures, trained on 21 distinct labels.
10
 
11
  ## Model Details
12
 
 
88
 
89
  ### Training Data
90
 
91
+ The training dataset consists of 141,055 entries, with 21 unique labels. The labels span various document types, with instances distributed across three text sizes (128 ± x, 256 ± x, and 512 ± x words; x varies within 20).
92
  Here are the labels along with their respective counts in the dataset:
93
 
94
+ | Agreement Type | Instances |
95
+ | ------------------------------------- | --------- |
96
+ | BOARD_MEETING_AGREEMENT | 4,206 |
97
+ | CONSULTING_AGREEMENT | 2,965 |
98
+ | CUSTOMER_LIST_AGREEMENT | 8,966 |
99
+ | DISTRIBUTION_PARTNER_AGREEMENT | 5,144 |
100
+ | EMPLOYEE_AGREEMENT | 3,876 |
101
+ | ENTERPRISE_AGREEMENT | 4,213 |
102
+ | ENTERPRISE_LICENSE_AGREEMENT | 8,999 |
103
+ | EXECUTIVE_SEVERANCE_AGREEMENT | 8,996 |
104
+ | FINANCIAL_REPORT_AGREEMENT | 11,384 |
105
+ | HARMFUL_ADVICE | 1,887 |
106
+ | INTERNAL_PRODUCT_ROADMAP_AGREEMENT | 6,982 |
107
+ | LOAN_AND_SECURITY_AGREEMENT | 8,957 |
108
+ | MEDICAL_ADVICE | 3,847 |
109
+ | MERGER_AGREEMENT | 7,704 |
110
+ | NDA_AGREEMENT | 5,221 |
111
+ | NORMAL_TEXT | 8,994 |
112
+ | PATENT_APPLICATION_FILLINGS_AGREEMENT | 8,802 |
113
+ | PRICE_LIST_AGREEMENT | 8,906 |
114
+ | SETTLEMENT_AGREEMENT | 3,737 |
115
+ | SEXUAL_CONTENT | 8,957 |
116
+ | SEXUAL_INCIDENT_REPORT | 8,321 |
 
117
 
118
  ## Evaluation
119
 
120
  ### Testing Data & Metrics
121
 
122
  #### Testing Data
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
 
123
 
124
+ Evaluation was performed on a dataset of 86,281 entries with a temperature range of 1-1.25 for randomness.
125
+ Here are the labels along with their respective counts in the dataset:
126
 
127
+ | Agreement Type | Instances |
128
+ | ------------------------------------- | --------- |
129
+ | BOARD_MEETING_AGREEMENT | 3,975 |
130
+ | CONSULTING_AGREEMENT | 1,430 |
131
+ | CUSTOMER_LIST_AGREEMENT | 4,488 |
132
+ | DISTRIBUTION_PARTNER_AGREEMENT | 6,696 |
133
+ | EMPLOYEE_AGREEMENT | 1,310 |
134
+ | ENTERPRISE_AGREEMENT | 1,501 |
135
+ | ENTERPRISE_LICENSE_AGREEMENT | 7,967 |
136
+ | EXECUTIVE_SEVERANCE_AGREEMENT | 4,795 |
137
+ | FINANCIAL_REPORT_AGREEMENT | 4,686 |
138
+ | HARMFUL_ADVICE | 361 |
139
+ | INTERNAL_PRODUCT_ROADMAP_AGREEMENT | 3,740 |
140
+ | LOAN_AND_SECURITY_AGREEMENT | 5,833 |
141
+ | MEDICAL_ADVICE | 643 |
142
+ | MERGER_AGREEMENT | 6,557 |
143
+ | NDA_AGREEMENT | 1,352 |
144
+ | NORMAL_TEXT | 5,811 |
145
+ | PATENT_APPLICATION_FILLINGS_AGREEMENT | 5,608 |
146
+ | PRICE_LIST_AGREEMENT | 5,044 |
147
+ | SETTLEMENT_AGREEMENT | 5,377 |
148
+ | SEXUAL_CONTENT | 4,356 |
149
+ | SEXUAL_INCIDENT_REPORT | 4,750 |
150
 
151
  #### Metrics
152
 
153
+ | Agreement Type | precision | recall | f1-score | support |
154
+ | ------------------------------------- | --------- | ------ | -------- | ------- |
155
+ | BOARD_MEETING_AGREEMENT | 0.92 | 0.95 | 0.93 | 3,975 |
156
+ | CONSULTING_AGREEMENT | 0.81 | 0.85 | 0.83 | 1,430 |
157
+ | CUSTOMER_LIST_AGREEMENT | 0.90 | 0.88 | 0.89 | 4,488 |
158
+ | DISTRIBUTION_PARTNER_AGREEMENT | 0.73 | 0.63 | 0.68 | 6,696 |
159
+ | EMPLOYEE_AGREEMENT | 0.85 | 0.84 | 0.85 | 1,310 |
160
+ | ENTERPRISE_AGREEMENT | 0.18 | 0.70 | 0.29 | 1,501 |
161
+ | ENTERPRISE_LICENSE_AGREEMENT | 0.92 | 0.78 | 0.84 | 7,967 |
162
+ | EXECUTIVE_SEVERANCE_AGREEMENT | 0.97 | 0.88 | 0.92 | 4,795 |
163
+ | FINANCIAL_REPORT_AGREEMENT | 0.93 | 0.99 | 0.96 | 4,686 |
164
+ | HARMFUL_ADVICE | 0.92 | 0.94 | 0.93 | 361 |
165
+ | INTERNAL_PRODUCT_ROADMAP_AGREEMENT | 0.94 | 0.98 | 0.96 | 3,740 |
166
+ | LOAN_AND_SECURITY_AGREEMENT | 0.93 | 0.97 | 0.95 | 5,833 |
167
+ | MEDICAL_ADVICE | 0.93 | 1.00 | 0.96 | 643 |
168
+ | MERGER_AGREEMENT | 0.93 | 0.45 | 0.61 | 6,557 |
169
+ | NDA_AGREEMENT | 0.68 | 0.91 | 0.78 | 1,352 |
170
+ | NORMAL_TEXT | 0.95 | 0.94 | 0.95 | 5,811 |
171
+ | PATENT_APPLICATION_FILLINGS_AGREEMENT | 0.96 | 0.99 | 0.98 | 5,608 |
172
+ | PRICE_LIST_AGREEMENT | 0.76 | 0.79 | 0.77 | 5,044 |
173
+ | SETTLEMENT_AGREEMENT | 0.76 | 0.78 | 0.77 | 5,377 |
174
+ | SEXUAL_CONTENT | 0.92 | 0.97 | 0.94 | 4,356 |
175
+ | SEXUAL_INCIDENT_REPORT | 0.99 | 0.94 | 0.96 | 4,750 |
176
+ | accuracy | | | 0.84 | 86,280 |
177
+ | macro avg | 0.85 | 0.86 | 0.84 | 86,280 |
178
+ | weighted avg | 0.88 | 0.84 | 0.85 | 86,280 |
 
179
 
180
  #### Results
181
 
182
+ The model’s performance is summarized by precision, recall, and f1-score metrics, which are detailed across all 21 labels in the dataset. Based on the test data evaluation results, the model achieved an accuracy of 0.8424, a precision of 0.8794, and a recall of 0.8424. The F1-score, which is the harmonic mean of precision and recall, stands at 0.8505.
 
 
183
 
184
+ The evaluation loss, which measures the discrepancy between the model’s predictions and the actual values, is 0.6815. Lower loss values indicate better model performance.
185
 
186
+ The model was able to process approximately 97.684 samples per second during the evaluation, which took a total runtime of 883.2545 seconds. The model performed approximately 0.764 evaluation steps per second.
config.json CHANGED
@@ -9,7 +9,6 @@
9
  "dropout": 0.1,
10
  "hidden_dim": 3072,
11
  "id2label": {
12
-
13
  "0": "BOARD_MEETING_AGREEMENT",
14
  "1": "CONSULTING_AGREEMENT",
15
  "2": "CUSTOMER_LIST_AGREEMENT",
@@ -29,7 +28,8 @@
29
  "16": "PATENT_APPLICATION_FILLINGS_AGREEMENT",
30
  "17": "PRICE_LIST_AGREEMENT",
31
  "18": "SETTLEMENT_AGREEMENT",
32
- "19": "SEXUAL_HARRASSMENT"
 
33
  },
34
  "initializer_range": 0.02,
35
  "label2id": {
@@ -44,8 +44,9 @@
44
  "PATENT_APPLICATION_FILLINGS_AGREEMENT": 16,
45
  "PRICE_LIST_AGREEMENT": 17,
46
  "SETTLEMENT_AGREEMENT": 18,
47
- "SEXUAL_HARRASSMENT": 19,
48
  "CUSTOMER_LIST_AGREEMENT": 2,
 
49
  "DISTRIBUTION_PARTNER_AGREEMENT": 3,
50
  "EMPLOYEE_AGREEMENT": 4,
51
  "ENTERPRISE_AGREEMENT": 5,
@@ -59,11 +60,12 @@
59
  "n_heads": 12,
60
  "n_layers": 6,
61
  "pad_token_id": 0,
 
62
  "qa_dropout": 0.1,
63
  "seq_classif_dropout": 0.2,
64
  "sinusoidal_pos_embds": false,
65
  "tie_weights_": true,
66
  "torch_dtype": "float32",
67
- "transformers_version": "4.36.2",
68
  "vocab_size": 30522
69
  }
 
9
  "dropout": 0.1,
10
  "hidden_dim": 3072,
11
  "id2label": {
 
12
  "0": "BOARD_MEETING_AGREEMENT",
13
  "1": "CONSULTING_AGREEMENT",
14
  "2": "CUSTOMER_LIST_AGREEMENT",
 
28
  "16": "PATENT_APPLICATION_FILLINGS_AGREEMENT",
29
  "17": "PRICE_LIST_AGREEMENT",
30
  "18": "SETTLEMENT_AGREEMENT",
31
+ "19": "SEXUAL_CONTENT",
32
+ "20": "SEXUAL_INCIDENT_REPORT"
33
  },
34
  "initializer_range": 0.02,
35
  "label2id": {
 
44
  "PATENT_APPLICATION_FILLINGS_AGREEMENT": 16,
45
  "PRICE_LIST_AGREEMENT": 17,
46
  "SETTLEMENT_AGREEMENT": 18,
47
+ "SEXUAL_CONTENT": 19,
48
  "CUSTOMER_LIST_AGREEMENT": 2,
49
+ "SEXUAL_INCIDENT_REPORT": 20,
50
  "DISTRIBUTION_PARTNER_AGREEMENT": 3,
51
  "EMPLOYEE_AGREEMENT": 4,
52
  "ENTERPRISE_AGREEMENT": 5,
 
60
  "n_heads": 12,
61
  "n_layers": 6,
62
  "pad_token_id": 0,
63
+ "problem_type": "single_label_classification",
64
  "qa_dropout": 0.1,
65
  "seq_classif_dropout": 0.2,
66
  "sinusoidal_pos_embds": false,
67
  "tie_weights_": true,
68
  "torch_dtype": "float32",
69
+ "transformers_version": "4.40.2",
70
  "vocab_size": 30522
71
  }
label_encoder.joblib CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:679d6eddc6f1fc6f4f4f58df6d284bf455024e8273567b22557de19dfc8753bb
3
- size 1099
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:f21f9707a92949f7085d2370e49e3be8e7ba71ed1508f0e0f6f21f48f6fbb8e9
3
+ size 1118
pytorch_model.bin CHANGED
@@ -1,3 +1,3 @@
1
  version https://git-lfs.github.com/spec/v1
2
- oid sha256:c110ff2add5c7adf6aadaa01e0e14ce8e140ede610307633fb7172e066fa42fc
3
- size 268209725
 
1
  version https://git-lfs.github.com/spec/v1
2
+ oid sha256:231c166001b51adbeecfddd55563569a52981f7c88e3802364368f76e86279d2
3
+ size 268212925