distily
/

distily_multi_attn_experiment_ortho

Generated from Trainer

Model card Files Files and versions Metrics Training metrics Community

distily_multi_attn_experiment_ortho / logs

1 contributor

History: 8 commits

lapp0's picture

Training in progress, step 61875

5961958 verified 6 months ago

attn_layer_mapper=all, attn_loss_fn=cos, attn_projector=orthogonal, attn_weight=5
Training in progress, step 61875 6 months ago
attn_layer_mapper=all, attn_loss_fn=raw_mse, attn_projector=orthogonal, attn_weight=25.0
Training in progress, step 61875 6 months ago
attn_layer_mapper=all, attn_loss_fn=raw_mse, attn_projector=orthogonal, attn_weight=5
Training in progress, step 61875 6 months ago
attn_layer_mapper=all, attn_loss_fn=raw_mse, attn_weight=5, projector=orthogonal
Training in progress, step 61875 6 months ago
attn_layer_mapper=last, attn_loss_fn=raw_mse, attn_projector=orthogonal, attn_weight=25.0
Training in progress, step 61875 6 months ago
attn_layer_mapper=last, attn_loss_fn=raw_mse, attn_projector=orthogonal, attn_weight=5
Training in progress, step 61875 6 months ago
attn_layer_mapper=last_k_2, attn_loss_fn=raw_mse, attn_projector=orthogonal, attn_weight=25.0
Training in progress, step 61875 6 months ago
attn_layer_mapper=last_k_2, attn_loss_fn=raw_mse, attn_projector=orthogonal, attn_weight=5
Training in progress, step 61875 6 months ago
attn_layer_mapper=layer-2, attn_loss_fn=raw_mse, attn_projector=orthogonal, attn_weight=25.0
Training in progress, step 61875 6 months ago
attn_layer_mapper=layer-2, attn_loss_fn=raw_mse, attn_projector=orthogonal, attn_weight=5
Training in progress, step 61875 6 months ago