Model parameters: d_model 768 ffw_size 3072 kv_size 64 n_heads 12 n_layers 15 Megatron-DeepSpeed/pretrain_gpt.py --tensor-model-parallel-size 1 --pipeline-model-parallel-size 1 --num-layers 15 --hidden-size 768 --num-attention-heads 12 --kv-channels 64 --ffn-hidden-size 3072 --seq-length 2048 --max-position-embeddings 2048 --micro-batch-size 4 --global-batch-size 256 --train-samples 1 --vocab-file gpt2/vocab.json --merge-file gpt2/merges.txt --clip-grad 1.0 --kill-switch-path kill-switch-146m1b5100mdedupval --bf16 --optimizer adam --adam-beta1 0.9 --adam-beta2 0.999 --adam-eps 1e-8 --lr 2e-4 --min-lr 2e-5 --lr-decay-style cosine --lr-decay-samples 1 --lr-warmup-samples 0 --clip-grad 1.0 --weight-decay 1e-1 --no-load-optim --reset-progress --override-lr-scheduler --log-interval 10 --save-interval 1000 --eval-interval 1 --eval-iters 100 --eval-only true --tensorboard-dir tensorboard_146m1b5100mdedupval --tensorboard-queue-size 5 --log-timers-to-tensorboard --log-batch-size-to-tensorboard --log-validation-ppl-to-tensorboard --save checkpoints_146m1b5100mdedup --load checkpoints_146m1b5100mdedup --train-weighted-split-paths-path train14b.txt --valid-weighted-split-paths-path val.txt --data-impl mmap --deepspeed --deepspeed_config ds_configs/3406547.json --zero-stage 0 START 3406547: Mon 24 Apr 2023 12:05:08 PM EEST 0: 0: 0: ======================= ROCm System Management Interface ======================= 0: ================================= Concise Info ================================= 0: GPU Temp AvgPwr SCLK MCLK Fan Perf PwrCap VRAM% GPU% 0: 0 48.0c 89.0W 800Mhz 1600Mhz 0% auto 560.0W 0% 0% 0: 1 47.0c N/A 800Mhz 1600Mhz 0% auto 0.0W 0% 0% 0: 2 36.0c 87.0W 800Mhz 1600Mhz 0% auto 560.0W 0% 0% 0: 3 46.0c N/A 800Mhz 1600Mhz 0% auto 0.0W 0% 0% 0: 4 48.0c 85.0W 800Mhz 1600Mhz 0% auto 560.0W 0% 0% 0: 5 48.0c N/A 800Mhz 1600Mhz 0% auto 0.0W 0% 0% 0: 6 41.0c 89.0W 800Mhz 1600Mhz 0% auto 560.0W 0% 0% 0: 7 44.0c N/A 800Mhz 1600Mhz 0% auto 0.0W 0% 0% 0: ================================================================================ 0: ============================= End of ROCm SMI Log ============================== 7: 7: 7: ======================= ROCm System Management Interface ======================= 7: ================================= Concise Info ================================= 7: GPU Temp AvgPwr SCLK MCLK Fan Perf PwrCap VRAM% GPU% 7: 0 49.0c 89.0W 800Mhz 1600Mhz 0% auto 560.0W 0% 0% 7: 1 49.0c N/A 800Mhz 1600Mhz 0% auto 0.0W 0% 0% 7: 2 43.0c 86.0W 800Mhz 1600Mhz 0% auto 560.0W 0% 0% 7: 3 45.0c N/A 800Mhz 1600Mhz 0% auto 0.0W 0% 0% 7: 4 45.0c 92.0W 800Mhz 1600Mhz 0% auto 560.0W 0% 0% 7: 5 47.0c N/A 800Mhz 1600Mhz 0% auto 0.0W 0% 0% 7: 6 41.0c 87.0W 800Mhz 1600Mhz 0% auto 560.0W 0% 0% 7: 7 43.0c N/A 800Mhz 1600Mhz 0% auto 0.0W 0% 0% 7: ================================================================================ 7: ============================= End of ROCm SMI Log ============================== 3: 3: 3: ======================= ROCm System Management Interface ======================= 3: ================================= Concise Info ================================= 3: GPU Temp AvgPwr SCLK MCLK Fan Perf PwrCap VRAM% GPU% 3: 0 47.0c 86.0W 800Mhz 1600Mhz 0% auto 560.0W 0% 0% 3: 1 47.0c N/A 800Mhz 1600Mhz 0% auto 0.0W 0% 0% 3: 2 42.0c 91.0W 800Mhz 1600Mhz 0% auto 560.0W 0% 0% 3: 3 45.0c N/A 800Mhz 1600Mhz 0% auto 0.0W 0% 0% 3: 4 42.0c 89.0W 800Mhz 1600Mhz 0% auto 560.0W 0% 0% 3: 5 49.0c N/A 800Mhz 1600Mhz 0% auto 0.0W 0% 0% 3: 6 43.0c 88.0W 800Mhz 1600Mhz 0% auto 560.0W 0% 0% 3: 7 41.0c N/A 800Mhz 1600Mhz 0% auto 0.0W 0% 0% 3: ================================================================================ 3: ============================= End of ROCm SMI Log ============================== 4: 4: 4: ======================= ROCm System Management Interface ======================= 4: ================================= Concise Info ================================= 4: GPU Temp AvgPwr SCLK MCLK Fan Perf PwrCap VRAM% GPU% 4: 0 47.0c 92.0W 800Mhz 1600Mhz 0% auto 560.0W 0% 0% 4: 1 48.0c N/A 800Mhz 1600Mhz 0% auto 0.0W 0% 0% 4: 2 40.0c 87.0W 800Mhz 1600Mhz 0% auto 560.0W 0% 0% 4: 3 44.0c N/A 800Mhz 1600Mhz 0% auto 0.0W 0% 0% 4: 4 42.0c 88.0W 800Mhz 1600Mhz 0% auto 560.0W 0% 0% 4: 5 42.0c N/A 800Mhz 1600Mhz 0% auto 0.0W 0% 0% 4: 6 41.0c 89.0W 800Mhz 1600Mhz 0% auto 560.0W 0% 0% 4: 7 43.0c N/A 800Mhz 1600Mhz 0% auto 0.0W 0% 0% 4: ================================================================================ 4: ============================= End of ROCm SMI Log ============================== 6: 6: 6: ======================= ROCm System Management Interface ======================= 6: ================================= Concise Info ================================= 6: GPU Temp AvgPwr SCLK MCLK Fan Perf PwrCap VRAM% GPU% 6: 0 45.0c 91.0W 800Mhz 1600Mhz 0% auto 560.0W 0% 0% 6: 1 46.0c N/A 800Mhz 1600Mhz 0% auto 0.0W 0% 0% 6: 2 40.0c 88.0W 800Mhz 1600Mhz 0% auto 560.0W 0% 0% 6: 3 49.0c N/A 800Mhz 1600Mhz 0% auto 0.0W 0% 0% 6: 4 42.0c 93.0W 800Mhz 1600Mhz 0% auto 560.0W 0% 0% 6: 5 44.0c N/A 800Mhz 1600Mhz 0% auto 0.0W 0% 0% 6: 6 37.0c 95.0W 800Mhz 1600Mhz 0% auto 560.0W 0% 0% 6: 7 41.0c N/A 800Mhz 1600Mhz 0% auto 0.0W 0% 0% 6: ================================================================================ 6: ============================= End of ROCm SMI Log ============================== 1: 1: 1: ======================= ROCm System Management Interface ======================= 1: ================================= Concise Info ================================= 1: GPU Temp AvgPwr SCLK MCLK Fan Perf PwrCap VRAM% GPU% 1: 0 40.0c 93.0W 800Mhz 1600Mhz 0% auto 560.0W 0% 0% 1: 1 48.0c N/A 800Mhz 1600Mhz 0% auto 0.0W 0% 0% 1: 2 38.0c 89.0W 800Mhz 1600Mhz 0% auto 560.0W 0% 0% 1: 3 47.0c N/A 800Mhz 1600Mhz 0% auto 0.0W 0% 0% 1: 4 45.0c 86.0W 800Mhz 1600Mhz 0% auto 560.0W 0% 0% 1: 5 47.0c N/A 800Mhz 1600Mhz 0% auto 0.0W 0% 0% 1: 6 43.0c 82.0W 800Mhz 1600Mhz 0% auto 560.0W 0% 0% 1: 7 47.0c N/A 800Mhz 1600Mhz 0% auto 0.0W 0% 0% 1: ================================================================================ 1: ============================= End of ROCm SMI Log ============================== 5: 5: 5: ======================= ROCm System Management Interface ======================= 5: ================================= Concise Info ================================= 5: GPU Temp AvgPwr SCLK MCLK Fan Perf PwrCap VRAM% GPU% 5: 0 47.0c 90.0W 800Mhz 1600Mhz 0% auto 560.0W 0% 0% 5: 1 48.0c N/A 800Mhz 1600Mhz 0% auto 0.0W 0% 0% 5: 2 38.0c 88.0W 800Mhz 1600Mhz 0% auto 560.0W 0% 0% 5: 3 50.0c N/A 800Mhz 1600Mhz 0% auto 0.0W 0% 0% 5: 4 44.0c 85.0W 800Mhz 1600Mhz 0% auto 560.0W 0% 0% 5: 5 45.0c N/A 800Mhz 1600Mhz 0% auto 0.0W 0% 0% 5: 6 41.0c 84.0W 800Mhz 1600Mhz 0% auto 560.0W 0% 0% 5: 7 46.0c N/A 800Mhz 1600Mhz 0% auto 0.0W 0% 0% 5: ================================================================================ 5: ============================= End of ROCm SMI Log ============================== 2: 2: 2: ======================= ROCm System Management Interface ======================= 2: ================================= Concise Info ================================= 2: GPU Temp AvgPwr SCLK MCLK Fan Perf PwrCap VRAM% GPU% 2: 0 44.0c 86.0W 800Mhz 1600Mhz 0% auto 560.0W 0% 0% 2: 1 48.0c N/A 800Mhz 1600Mhz 0% auto 0.0W 0% 0% 2: 2 43.0c 91.0W 800Mhz 1600Mhz 0% auto 560.0W 0% 0% 2: 3 49.0c N/A 800Mhz 1600Mhz 0% auto 0.0W 0% 0% 2: 4 45.0c 86.0W 800Mhz 1600Mhz 0% auto 560.0W 0% 0% 2: 5 42.0c N/A 800Mhz 1600Mhz 0% auto 0.0W 0% 0% 2: 6 43.0c 90.0W 800Mhz 1600Mhz 0% auto 560.0W 0% 0% 2: 7 45.0c N/A 800Mhz 1600Mhz 0% auto 0.0W 0% 0% 2: ================================================================================ 2: ============================= End of ROCm SMI Log ============================== 0: Launching on nid006908 (0/8), master nid006908 port 9999, GPUs 8, CUDA: True 1: Launching on nid006909 (1/8), master nid006908 port 9999, GPUs 8, CUDA: True 2: Launching on nid006910 (2/8), master nid006908 port 9999, GPUs 8, CUDA: True 7: Launching on nid006915 (7/8), master nid006908 port 9999, GPUs 8, CUDA: True 6: Launching on nid006914 (6/8), master nid006908 port 9999, GPUs 8, CUDA: True 3: Launching on nid006911 (3/8), master nid006908 port 9999, GPUs 8, CUDA: True 4: Launching on nid006912 (4/8), master nid006908 port 9999, GPUs 8, CUDA: True 5: Launching on nid006913 (5/8), master nid006908 port 9999, GPUs 8, CUDA: True