Upload 3 files

README.md

@@ -1,3 +1,78 @@
----
-license: apache-2.0
----
+---
+library_name: transformers
+tags:
+- masked-image-modeling
+- generated_from_trainer
+---
+<!-- This model card has been generated automatically according to the information the Trainer had access to. You
+should probably proofread and complete it, then remove this comment. -->
+
+# smb-vision-base-1029
+
+This model is trained from scratch using [VideoMAE](https://huggingface.co/docs/transformers/en/model_doc/videomae) on over 4.7k CT volumes.
+
+## Model description
+
+More information needed
+
+## Intended uses & limitations
+
+More information needed
+
+## Training and evaluation data
+
+More information needed
+
+## Training procedure
+
+### Training hyperparameters
+
+The following hyperparameters were used during training:
+- learning_rate: 3e-04
+- train_batch_size: 32
+- eval_batch_size: 1
+- seed: 42
+- optimizer: Use OptimizerNames.ADAMW_TORCH with betas=(0.9,0.999) and epsilon=1e-08 and optimizer_args=No additional optimizer arguments
+- lr_scheduler_type: cosine
+- num_epochs: 30.0
+
+### Training results
+
+{
+  "_runtime": 54805.860011105,
+  "_step": 4351,
+  "eval/runtime": 17.8428,
+  "eval/samples_per_second": 2.578,
+  "eval/steps_per_second": 2.578,
+  "total_flos": 3.8084565648770335e+21,
+  "train/epoch": 30,
+  "train/global_step": 4350,
+  "train/grad_norm": 0.0735374316573143,
+  "train/learning_rate": 0,
+  "train/loss": 0.5736,
+  "train_loss": 0.5022664608695041,
+  "train_runtime": 54785.1298,
+  "train_samples_per_second": 2.527,
+  "train_steps_per_second": 0.079
+}
+
+
+### Framework versions
+
+- Transformers 4.46.0
+- Pytorch 2.5.0
+- Datasets 3.0.2
+- Tokenizers 0.20.1
+
+### How to use
+```python
+# load data using `dataload.py`
+
+model = VideoMAEForPreTraining.from_pretrained(
+    standardmodelbio/smb-vision-base,
+    trust_remote_code=True,
+)
+
+embedding = model.videomae(batch["image"])
+
+```

config.json

@@ -0,0 +1,30 @@
+{
+  "architectures": [
+    "VideoMAEForPreTraining"
+  ],
+  "attention_probs_dropout_prob": 0.0,
+  "attn_implementation": "flash_attention_2",
+  "decoder_hidden_size": 512,
+  "decoder_intermediate_size": 2048,
+  "decoder_num_attention_heads": 16,
+  "decoder_num_hidden_layers": 8,
+  "hidden_act": "gelu",
+  "hidden_dropout_prob": 0.0,
+  "hidden_size": 1024,
+  "image_size": 384,
+  "initializer_range": 0.02,
+  "intermediate_size": 4096,
+  "layer_norm_eps": 1e-12,
+  "model_type": "videomae",
+  "norm_pix_loss": true,
+  "num_attention_heads": 16,
+  "num_channels": 1,
+  "num_frames": 320,
+  "num_hidden_layers": 24,
+  "patch_size": 16,
+  "qkv_bias": true,
+  "torch_dtype": "float32",
+  "transformers_version": "4.46.3",
+  "tubelet_size": 16,
+  "use_mean_pooling": true
+}

model.safetensors

@@ -0,0 +1,3 @@
+version https://git-lfs.github.com/spec/v1
+oid sha256:d94debd4dd3e0946a93f500657d00485055676e6a5055d168d67f5213abd7c66
+size 1337348480