Model parameters: d_model 512 ffw_size 2048 kv_size 64 n_heads 8 n_layers 8 Megatron-DeepSpeed/pretrain_gpt.py --tensor-model-parallel-size 1 --pipeline-model-parallel-size 1 --num-layers 8 --hidden-size 512 --num-attention-heads 8 --kv-channels 64 --ffn-hidden-size 2048 --seq-length 2048 --max-position-embeddings 2048 --micro-batch-size 4 --global-batch-size 256 --train-samples 84_762_549 --vocab-file gpt2/vocab.json --merge-file gpt2/merges.txt --loss-scale 12 --clip-grad 1.0 --kill-switch-path kill-switch-44m174b100m --bf16 --checkpoint-activations --optimizer adam --adam-beta1 0.9 --adam-beta2 0.999 --adam-eps 1e-8 --lr 2e-4 --min-lr 2e-5 --lr-decay-style cosine --lr-decay-samples 84_762_549 --lr-warmup-samples 847_625 --clip-grad 1.0 --weight-decay 1e-1 --log-interval 100 --save-interval 10000 --eval-interval 10000 --eval-iters 1 --tensorboard-dir tensorboard_44m174b100m --tensorboard-queue-size 5 --log-timers-to-tensorboard --log-batch-size-to-tensorboard --log-validation-ppl-to-tensorboard --save checkpoints_44m174b100m --load checkpoints_44m174b100m --train-weighted-split-paths-path train100m.txt --valid-weighted-split-paths-path val.txt --data-impl mmap --deepspeed --deepspeed_config ds_configs/3328591.json --zero-stage 0 START 3328591: Fri 17 Mar 2023 09:57:24 AM EET 3: 3: 3: ======================= ROCm System Management Interface ======================= 3: ================================= Concise Info ================================= 3: GPU Temp AvgPwr SCLK MCLK Fan Perf PwrCap VRAM% GPU% 3: 0 45.0c 87.0W 800Mhz 1600Mhz 0% auto 560.0W 0% 0% 3: 1 46.0c N/A 800Mhz 1600Mhz 0% auto 0.0W 0% 0% 3: 2 39.0c 80.0W 800Mhz 1600Mhz 0% auto 560.0W 0% 0% 3: 3 44.0c N/A 800Mhz 1600Mhz 0% auto 0.0W 0% 0% 3: 4 41.0c 87.0W 800Mhz 1600Mhz 0% auto 560.0W 0% 0% 3: 5 46.0c N/A 800Mhz 1600Mhz 0% auto 0.0W 0% 0% 3: 6 37.0c 86.0W 800Mhz 1600Mhz 0% auto 560.0W 0% 0% 3: 7 42.0c N/A 800Mhz 1600Mhz 0% auto 0.0W 0% 0% 3: ================================================================================ 3: ============================= End of ROCm SMI Log ============================== 0: 0: 0: ======================= ROCm System Management Interface ======================= 0: ================================= Concise Info ================================= 0: GPU Temp AvgPwr SCLK MCLK Fan Perf PwrCap VRAM% GPU% 0: 0 47.0c 92.0W 800Mhz 1600Mhz 0% auto 560.0W 0% 0% 0: 1 46.0c N/A 800Mhz 1600Mhz 0% auto 0.0W 0% 0% 0: 2 40.0c 87.0W 800Mhz 1600Mhz 0% auto 560.0W 0% 0% 0: 3 43.0c N/A 800Mhz 1600Mhz 0% auto 0.0W 0% 0% 0: 4 46.0c 87.0W 800Mhz 1600Mhz 0% auto 560.0W 0% 0% 0: 5 43.0c N/A 800Mhz 1600Mhz 0% auto 0.0W 0% 0% 0: 6 43.0c 89.0W 800Mhz 1600Mhz 0% auto 560.0W 0% 0% 0: 7 43.0c N/A 800Mhz 1600Mhz 0% auto 0.0W 0% 0% 0: ================================================================================ 0: ============================= End of ROCm SMI Log ============================== 2: 2: 2: ======================= ROCm System Management Interface ======================= 2: ================================= Concise Info ================================= 2: GPU Temp AvgPwr SCLK MCLK Fan Perf PwrCap VRAM% GPU% 2: 0 46.0c 99.0W 800Mhz 1600Mhz 0% auto 560.0W 0% 0% 2: 1 51.0c N/A 800Mhz 1600Mhz 0% auto 0.0W 0% 0% 2: 2 50.0c 90.0W 800Mhz 1600Mhz 0% auto 560.0W 0% 0% 2: 3 46.0c N/A 800Mhz 1600Mhz 0% auto 0.0W 0% 0% 2: 4 44.0c 92.0W 800Mhz 1600Mhz 0% auto 560.0W 0% 0% 2: 5 45.0c N/A 800Mhz 1600Mhz 0% auto 0.0W 0% 0% 2: 6 41.0c 92.0W 800Mhz 1600Mhz 0% auto 560.0W 0% 0% 2: 7 41.0c N/A 800Mhz 1600Mhz 0% auto 0.0W 0% 0% 2: ================================================================================ 2: ============================= End of ROCm SMI Log ============================== 4: 4: 4: ======================= ROCm System Management Interface ======================= 4: ================================= Concise Info ================================= 4: GPU Temp AvgPwr SCLK MCLK Fan Perf PwrCap VRAM% GPU% 4: 0 45.0c 87.0W 800Mhz 1600Mhz 0% auto 560.0W 0% 0% 4: 1 47.0c N/A 800Mhz 1600Mhz 0% auto 0.0W 0% 0% 4: 2 48.0c 86.0W 800Mhz 1600Mhz 0% auto 560.0W 0% 0% 4: 3 43.0c N/A 800Mhz 1600Mhz 0% auto 0.0W 0% 0% 4: 4 41.0c 87.0W 800Mhz 1600Mhz 0% auto 560.0W 0% 0% 4: 5 45.0c N/A 800Mhz 1600Mhz 0% auto 0.0W 0% 0% 4: 6 42.0c 89.0W 800Mhz 1600Mhz 0% auto 560.0W 0% 0% 4: 7 49.0c N/A 800Mhz 1600Mhz 0% auto 0.0W 0% 0% 4: ================================================================================ 4: ============================= End of ROCm SMI Log ============================== 6: 6: 6: ======================= ROCm System Management Interface ======================= 6: ================================= Concise Info ================================= 6: GPU Temp AvgPwr SCLK MCLK Fan Perf PwrCap VRAM% GPU% 6: 0 44.0c 96.0W 800Mhz 1600Mhz 0% auto 560.0W 0% 0% 6: 1 48.0c N/A 800Mhz 1600Mhz 0% auto 0.0W 0% 0% 6: 2 44.0c 95.0W 800Mhz 1600Mhz 0% auto 560.0W 0% 0% 6: 3 45.0c N/A 800Mhz 1600Mhz 0% auto 0.0W 0% 0% 6: 4 45.0c 88.0W 800Mhz 1600Mhz 0% auto 560.0W 0% 0% 6: 5 48.0c N/A 800Mhz 1600Mhz 0% auto 0.0W 0% 0% 6: 6 40.0c 86.0W 800Mhz 1600Mhz 0% auto 560.0W 0% 0% 6: 7 45.0c N/A 800Mhz 1600Mhz 0% auto 0.0W 0% 0% 6: ================================================================================ 6: ============================= End of ROCm SMI Log ============================== 5: 5: 5: ======================= ROCm System Management Interface ======================= 5: ================================= Concise Info ================================= 5: GPU Temp AvgPwr SCLK MCLK Fan Perf PwrCap VRAM% GPU% 5: 0 40.0c 95.0W 800Mhz 1600Mhz 0% auto 560.0W 0% 0% 5: 1 42.0c N/A 800Mhz 1600Mhz 0% auto 0.0W 0% 0% 5: 2 44.0c 85.0W 800Mhz 1600Mhz 0% auto 560.0W 0% 0% 5: 3 42.0c N/A 800Mhz 1600Mhz 0% auto 0.0W 0% 0% 5: 4 44.0c 83.0W 800Mhz 1600Mhz 0% auto 560.0W 0% 0% 5: 5 47.0c N/A 800Mhz 1600Mhz 0% auto 0.0W 0% 0% 5: 6 34.0c 89.0W 800Mhz 1600Mhz 0% auto 560.0W 0% 0% 5: 7 42.0c N/A 800Mhz 1600Mhz 0% auto 0.0W 0% 0% 5: ================================================================================ 5: ============================= End of ROCm SMI Log ============================== 7: 7: 7: ======================= ROCm System Management Interface ======================= 7: ================================= Concise Info ================================= 7: GPU Temp AvgPwr SCLK MCLK Fan Perf PwrCap VRAM% GPU% 7: 0 43.0c 91.0W 800Mhz 1600Mhz 0% auto 560.0W 0% 0% 7: 1 47.0c N/A 800Mhz 1600Mhz 0% auto 0.0W 0% 0% 7: 2 37.0c 92.0W 800Mhz 1600Mhz 0% auto 560.0W 0% 0% 7: 3 46.0c N/A 800Mhz 1600Mhz 0% auto 0.0W 0% 0% 7: 4 50.0c 89.0W 800Mhz 1600Mhz 0% auto 560.0W 0% 0% 7: 5 44.0c N/A 800Mhz 1600Mhz 0% auto 0.0W 0% 0% 7: 6 44.0c 96.0W 800Mhz 1600Mhz 0% auto 560.0W 0% 0% 7: 7 46.0c N/A 800Mhz 1600Mhz 0% auto 0.0W 0% 0% 7: ================================================================================ 7: ============================= End of ROCm SMI Log ============================== 1: 1: 1: ======================= ROCm System Management Interface ======================= 1: ================================= Concise Info ================================= 1: GPU Temp AvgPwr SCLK MCLK Fan Perf PwrCap VRAM% GPU% 1: 0 45.0c 95.0W 800Mhz 1600Mhz 0% auto 560.0W 0% 0% 1: 1 48.0c N/A 800Mhz 1600Mhz 0% auto 0.0W 0% 0% 1: 2 43.0c 91.0W 800Mhz 1600Mhz 0% auto 560.0W 0% 0% 1: 3 46.0c N/A 800Mhz 1600Mhz 0% auto 0.0W 0% 0% 1: 4 43.0c 89.0W 800Mhz 1600Mhz 0% auto 560.0W 0% 0% 1: 5 44.0c N/A 800Mhz 1600Mhz 0% auto 0.0W 0% 0% 1: 6 50.0c 90.0W 800Mhz 1600Mhz 0% auto 560.0W 0% 0% 1: 7 42.0c N/A 800Mhz 1600Mhz 0% auto 0.0W 0% 0% 1: ================================================================================ 1: ============================= End of ROCm SMI Log ============================== 3: Launching on nid006128 (3/8), master nid006125 port 9999, GPUs 8, CUDA: True 6: Launching on nid006131 (6/8), master nid006125 port 9999, GPUs 8, CUDA: True 2: Launching on nid006127 (2/8), master nid006125 port 9999, GPUs 8, CUDA: True 5: Launching on nid006130 (5/8), master nid006125 port 9999, GPUs 8, CUDA: True 1: Launching on nid006126 (1/8), master nid006125 port 9999, GPUs 8, CUDA: True 4: Launching on nid006129 (4/8), master nid006125 port 9999, GPUs 8, CUDA: True 0: Launching on nid006125 (0/8), master nid006125 port 9999, GPUs 8, CUDA: True 7: Launching on nid006132 (7/8), master nid006125 port 9999, GPUs 8, CUDA: True 0: using world size: 64, data-parallel-size: 64, tensor-model-parallel size: 1, pipeline-model-parallel size: 1 0: accumulate and all-reduce gradients in fp32 for bfloat16 data type. 0: using torch.bfloat16 for parameters ... 0: ------------------------ arguments ------------------------ 0: abort_on_unmet_fused_kernel_constraints ......... False 0: accumulate_allreduce_grads_in_fp32 .............. True 0: adam_beta1 ...................................... 0.9 0: adam_beta2 ...................................... 0.999 0: adam_eps ........................................ 1e-08 0: adlr_autoresume ................................. False 0: adlr_autoresume_interval ........................ 1000 0: apply_query_key_layer_scaling ................... True 0: apply_residual_connection_post_layernorm ........ False 0: attention_dropout ............................... 0.1 0: attention_softmax_in_fp32 ....................... False 0: bert_binary_head ................................ True 0: bert_load ....................................... None 0: bf16 ............................................ True 0: bias_dropout_fusion ............................. True 0: bias_gelu_fusion ................................ True 0: biencoder_projection_dim ........................ 0 0: biencoder_shared_query_context_model ............ False 0: block_data_path ................................. None 0: checkpoint_activations .......................... True 0: checkpoint_in_cpu ............................... False 0: checkpoint_num_layers ........................... 1 0: clip_grad ....................................... 1.0 0: codecarbon_dir .................................. None 0: consumed_train_samples .......................... 0 0: consumed_train_tokens ........................... 0 0: consumed_valid_samples .......................... 0 0: contigious_checkpointing ........................ False 0: cpu_optimizer ................................... False 0: cpu_torch_adam .................................. False 0: curriculum_learning ............................. False 0: data_impl ....................................... mmap 0: data_parallel_size .............................. 64 0: data_path ....................................... None 0: dataloader_type ................................. single 0: DDP_impl ........................................ local 0: decoder_seq_length .............................. None 0: deepscale ....................................... False 0: deepscale_config ................................ None 0: deepspeed ....................................... True 0: deepspeed_activation_checkpointing .............. False 0: deepspeed_config ................................ ds_configs/3328591.json 0: deepspeed_mpi ................................... False 0: distribute_checkpointed_activations ............. False 0: distributed_backend ............................. nccl 0: embed_layernorm ................................. False 0: embedding_path .................................. None 0: encoder_seq_length .............................. 2048 0: eod_mask_loss ................................... False 0: eval_interval ................................... 10000 0: eval_iters ...................................... 1 0: eval_only ....................................... None 0: evidence_data_path .............................. None 0: exit_duration_in_mins ........................... None 0: exit_interval ................................... None 0: ffn_hidden_size ................................. 2048 0: finetune ........................................ False 0: fp16 ............................................ False 0: fp16_lm_cross_entropy ........................... False 0: fp32_residual_connection ........................ False 0: gigaflos_no_embeds .............................. 0 0: global_batch_size ............................... 256 0: glu_activation .................................. None 0: hidden_dropout .................................. 0.1 0: hidden_size ..................................... 512 0: hysteresis ...................................... 2 0: ict_head_size ................................... None 0: ict_load ........................................ None 0: img_dim ......................................... 224 0: indexer_batch_size .............................. 128 0: indexer_log_interval ............................ 1000 0: inference ....................................... False 0: init_method_std ................................. 0.02 0: init_method_xavier_uniform ...................... False 0: initial_loss_scale .............................. 4294967296 0: kill_switch_path ................................ kill-switch-44m174b100m 0: kv_channels ..................................... 64 0: layer_norm_fusion ............................... True 0: layernorm_epsilon ............................... 1e-05 0: lazy_mpu_init ................................... None 0: load ............................................ checkpoints_44m174b100m 0: local_rank ...................................... None 0: log_batch_size_to_tensorboard ................... True 0: log_interval .................................... 100 0: log_learning_rate_to_tensorboard ................ True 0: log_level ....................................... None 0: log_level_replica ............................... None 0: log_loss_scale_to_tensorboard ................... True 0: log_num_zeros_in_grad ........................... False 0: log_params_norm ................................. False 0: log_path ........................................ None 0: log_timers_to_tensorboard ....................... True 0: log_validation_ppl_to_tensorboard ............... True 0: loss_on_targets_only ............................ False 0: loss_scale ...................................... 12.0 0: loss_scale_window ............................... 1000 0: lr .............................................. 0.0002 0: lr_decay_iters .................................. None 0: lr_decay_samples ................................ 84762549 0: lr_decay_style .................................. cosine 0: lr_decay_tokens ................................. None 0: lr_warmup_fraction .............................. None 0: lr_warmup_iters ................................. 0 0: lr_warmup_samples ............................... 847625 0: make_vocab_size_divisible_by .................... 128 0: mask_prob ....................................... 0.15 0: masked_softmax_fusion ........................... True 0: max_position_embeddings ......................... 2048 0: mean_noise_span_length .......................... None 0: memory_centric_tiled_linear ..................... False 0: merge_file ...................................... gpt2/merges.txt 0: micro_batch_size ................................ 4 0: min_loss_scale .................................. 1.0 0: min_lr .......................................... 2e-05 0: mmap_warmup ..................................... False 0: no_load_optim ................................... None 0: no_load_rng ..................................... None 0: no_save_optim ................................... None 0: no_save_rng ..................................... None 0: noise_density ................................... None 0: num_attention_heads ............................. 8 0: num_channels .................................... 3 0: num_classes ..................................... 1000 0: num_layers ...................................... 8 0: num_layers_per_virtual_pipeline_stage ........... None 0: num_workers ..................................... 2 0: onnx_safe ....................................... None 0: openai_gelu ..................................... False 0: optimizer ....................................... adam 0: optimizer_fusion ................................ True 0: override_lr_scheduler ........................... False 0: pad_vocab_size_to ............................... None 0: params_dtype .................................... torch.bfloat16 0: partition_activations ........................... False 0: patch_dim ....................................... 16 0: pipeline_model_parallel_size .................... 1 0: position_embedding_type ......................... PositionEmbeddingType.absolute 0: pp_partition_method ............................. None 0: profile_backward ................................ False 0: query_in_block_prob ............................. 0.1 0: rampup_batch_size ............................... None 0: rank ............................................ 0 0: remote_device ................................... none 0: reset_attention_mask ............................ False 0: reset_position_ids .............................. False 0: reset_progress .................................. None 0: retriever_report_topk_accuracies ................ [] 0: retriever_score_scaling ......................... False 0: retriever_seq_length ............................ 256 0: reweight_loss_based_on_position_frequency ....... False 0: sample_rate ..................................... 1.0 0: save ............................................ checkpoints_44m174b100m 0: save_interval ................................... 10000 0: scatter_gather_tensors_in_pipeline .............. True 0: scattered_embeddings ............................ False 0: seed ............................................ 1234 0: seq_length ...................................... 2048 0: sgd_momentum .................................... 0.9 0: short_seq_prob .................................. 0.1 0: skip_train_iteration_range ...................... None 0: split ........................................... None 0: split_transformers .............................. False 0: sync_tp_duplicated_parameters ................... False 0: synchronize_each_layer .......................... False 0: tensor_model_parallel_size ...................... 1 0: tensorboard_dir ................................. tensorboard_44m174b100m 0: tensorboard_log_interval ........................ 1 0: tensorboard_queue_size .......................... 5 0: test_weighted_split_paths ....................... None 0: test_weighted_split_paths_path .................. None 0: tile_factor ..................................... 1 0: titles_data_path ................................ None 0: tokenizer_name_or_path .......................... None 0: tokenizer_type .................................. GPT2BPETokenizer 0: train_iters ..................................... None 0: train_samples ................................... 84762549 0: train_tokens .................................... None 0: train_weighted_split_names ...................... ['train'] 0: train_weighted_split_paths ...................... [['/scratch/project_462000119/data/c4_subsampled/gpt2tok_c4_en_100M_text_document']] 0: train_weighted_split_paths_path ................. None 0: train_weighted_split_splits ..................... [['0:1']] 0: train_weighted_split_weights .................... [['1.0']] 0: universal_checkpoint ............................ False 0: use_bnb_optimizer ............................... False 0: use_checkpoint_lr_scheduler ..................... False 0: use_contiguous_buffers_in_ddp ................... True 0: use_cpu_initialization .......................... None 0: use_one_sent_docs ............................... False 0: use_pin_memory .................................. False 0: valid_num_workers ............................... 2 0: valid_weighted_split_names ...................... ['validation'] 0: valid_weighted_split_paths ...................... [['/scratch/project_462000119/data/c4_validation/gpt2tok_c4validation_rerun_text_document']] 0: valid_weighted_split_paths_path ................. None 0: valid_weighted_split_splits ..................... [['0:1']] 0: valid_weighted_split_weights .................... [['1.0']] 0: virtual_pipeline_model_parallel_size ............ None 0: vocab_extra_ids ................................. 0 0: vocab_file ...................................... gpt2/vocab.json 0: weight_decay .................................... 0.1 0: world_size ...................................... 64 0: zero_allgather_bucket_size ...................... 0.0 0: zero_contigious_gradients ....................... False 0: zero_reduce_bucket_size ......................... 0.0 0: zero_reduce_scatter ............................. False 0: zero_stage ...................................... 0 0: -------------------- end of arguments --------------------- 0: setting number of micro-batches to constant 1 0: > building GPT2BPETokenizer tokenizer ... 0: > padded vocab (size: 50257) with 47 dummy tokens (new size: 50304) 0: DeepSpeed general environment info: 0: torch install path ............... ['/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch'] 0: torch version .................... 1.13.0+rocm5.2 0: torch cuda version ............... None 0: torch hip version ................ 5.2.21151-afdc89f8 0: nvcc version ..................... None 0: deepspeed install path ........... ['/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/deepspeed'] 0: deepspeed info ................... 0.7.5, unknown, unknown 0: deepspeed wheel compiled w. ...... torch 1.13, hip 5.1 7: > setting tensorboard ... 0: **** Git info for Megatron: git_hash=unknown git_branch=unknown **** 0: > initializing torch distributed ... 0: [2023-03-17 09:59:07,977] [INFO] [comm.py:633:init_distributed] Initializing TorchBackend in DeepSpeed with backend nccl 0: > initializing tensor model parallel with size 1 0: > initializing pipeline model parallel with size 1 0: > setting random seeds to 1234 ... 0: > initializing model parallel cuda seeds on global rank 0, model parallel rank 0, and data parallel rank 0 with model parallel seed: 3952 and data parallel seed: 1234 0: > compiling dataset index builder ... 0: make: Entering directory '/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/data' 0: make: Nothing to be done for 'default'. 0: make: Leaving directory '/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/data' 0: >>> done with dataset index builder. Compilation time: 0.105 seconds 0: > compiling and loading fused kernels ... 0: /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/fused_kernels/scaled_upper_triang_masked_softmax.cpp -> /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/fused_kernels/scaled_upper_triang_masked_softmax_hip.cpp [skipped, already hipified] 0: /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/fused_kernels/scaled_upper_triang_masked_softmax.h -> /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/fused_kernels/scaled_upper_triang_masked_softmax_hip.h [skipped, already hipified] 0: /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/fused_kernels/compat.h -> /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/fused_kernels/compat.h [skipped, no changes] 0: /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/fused_kernels/type_shim.h -> /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/fused_kernels/type_shim.h [skipped, no changes] 0: /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/fused_kernels/scaled_upper_triang_masked_softmax_cuda.cu -> /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/fused_kernels/scaled_upper_triang_masked_softmax_hip.hip [skipped, already hipified] 0: /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/fused_kernels/type_shim.h -> /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/fused_kernels/type_shim.h [skipped, no changes] 0: /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/fused_kernels/compat.h -> /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/fused_kernels/compat.h [skipped, no changes] 0: /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/fused_kernels/scaled_upper_triang_masked_softmax.h -> /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/fused_kernels/scaled_upper_triang_masked_softmax_hip.h [skipped, already hipified] 0: /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/fused_kernels/scaled_masked_softmax.h -> /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/fused_kernels/scaled_masked_softmax_hip.h [skipped, already hipified] 0: Total number of unsupported CUDA function calls: 0 0: 0: 0: Total number of replaced kernel launches: 87 0: [1/1] c++ scaled_upper_triang_masked_softmax_hip.o scaled_upper_triang_masked_softmax_hip.cuda.o -shared -L/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/lib -lc10 -lc10_hip -ltorch_cpu -ltorch_hip -ltorch -ltorch_python -L/pfs/lustrep2/projappl/project_462000125/samantao-public/rocm/rocm-5.2.3/lib -lamdhip64 -o scaled_upper_triang_masked_softmax_cuda.so 0: /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/fused_kernels/scaled_masked_softmax.cpp -> /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/fused_kernels/scaled_masked_softmax_hip.cpp [skipped, already hipified] 0: /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/fused_kernels/scaled_masked_softmax_cuda.cu -> /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/fused_kernels/scaled_masked_softmax_hip.hip [skipped, already hipified] 0: /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/fused_kernels/type_shim.h -> /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/fused_kernels/type_shim.h [skipped, no changes] 0: /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/fused_kernels/compat.h -> /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/fused_kernels/compat.h [skipped, no changes] 0: /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/fused_kernels/scaled_upper_triang_masked_softmax.h -> /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/fused_kernels/scaled_upper_triang_masked_softmax_hip.h [skipped, already hipified] 0: /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/fused_kernels/scaled_masked_softmax.h -> /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/fused_kernels/scaled_masked_softmax_hip.h [skipped, already hipified] 0: Total number of unsupported CUDA function calls: 0 0: 0: 0: Total number of replaced kernel launches: 63 0: ninja: no work to do. 0: /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/fused_kernels/layer_norm_cuda.cpp -> /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/fused_kernels/layer_norm_cuda.cpp [skipped, no changes] 0: /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/fused_kernels/layer_norm_cuda_kernel.cu -> /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/fused_kernels/layer_norm_hip_kernel.hip [skipped, already hipified] 0: /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/fused_kernels/type_shim.h -> /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/fused_kernels/type_shim.h [skipped, no changes] 0: /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/fused_kernels/compat.h -> /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/fused_kernels/compat.h [skipped, no changes] 0: /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/fused_kernels/scaled_upper_triang_masked_softmax.h -> /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/fused_kernels/scaled_upper_triang_masked_softmax_hip.h [skipped, already hipified] 0: /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/fused_kernels/scaled_masked_softmax.h -> /pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/Megatron-DeepSpeed/megatron/fused_kernels/scaled_masked_softmax_hip.h [skipped, already hipified] 0: Total number of unsupported CUDA function calls: 0 0: 0: 0: Total number of replaced kernel launches: 67 0: [1/1] c++ layer_norm_hip_kernel.cuda.o layer_norm_cuda.o -shared -L/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/venv/lib/python3.9/site-packages/torch/lib -lc10 -lc10_hip -ltorch_cpu -ltorch_hip -ltorch -ltorch_python -L/pfs/lustrep2/projappl/project_462000125/samantao-public/rocm/rocm-5.2.3/lib -lamdhip64 -o fused_mix_prec_layer_norm_cuda.so 0: >>> done with compiling and loading fused kernels. Compilation time: 26.963 seconds 0: time to initialize megatron (seconds): 105.701 0: [after megatron is initialized] datetime: 2023-03-17 09:59:37 0: building GPT model ... 0: [2023-03-17 09:59:37,839] [INFO] [utils.py:827:see_memory_usage] Before Building Model 0: [2023-03-17 09:59:37,839] [INFO] [utils.py:828:see_memory_usage] MA 0.0 GB Max_MA 0.0 GB CA 0.0 GB Max_CA 0 GB 0: [2023-03-17 09:59:37,839] [INFO] [utils.py:836:see_memory_usage] CPU Virtual Memory: used = 30.63 GB, percent = 6.1% 0: SEED_LAYERS=False BASE_SEED=1234 SEED_FN=None 0: Using topology: {ProcessCoord(pipe=0, data=0, model=0): 0, ProcessCoord(pipe=0, data=1, model=0): 1, ProcessCoord(pipe=0, data=2, model=0): 2, ProcessCoord(pipe=0, data=3, model=0): 3, ProcessCoord(pipe=0, data=4, model=0): 4, ProcessCoord(pipe=0, data=5, model=0): 5, ProcessCoord(pipe=0, data=6, model=0): 6, ProcessCoord(pipe=0, data=7, model=0): 7, ProcessCoord(pipe=0, data=8, model=0): 8, ProcessCoord(pipe=0, data=9, model=0): 9, ProcessCoord(pipe=0, data=10, model=0): 10, ProcessCoord(pipe=0, data=11, model=0): 11, ProcessCoord(pipe=0, data=12, model=0): 12, ProcessCoord(pipe=0, data=13, model=0): 13, ProcessCoord(pipe=0, data=14, model=0): 14, ProcessCoord(pipe=0, data=15, model=0): 15, ProcessCoord(pipe=0, data=16, model=0): 16, ProcessCoord(pipe=0, data=17, model=0): 17, ProcessCoord(pipe=0, data=18, model=0): 18, ProcessCoord(pipe=0, data=19, model=0): 19, ProcessCoord(pipe=0, data=20, model=0): 20, ProcessCoord(pipe=0, data=21, model=0): 21, ProcessCoord(pipe=0, data=22, model=0): 22, ProcessCoord(pi 0: pe=0, data=23, model=0): 23, ProcessCoord(pipe=0, data=24, model=0): 24, ProcessCoord(pipe=0, data=25, model=0): 25, ProcessCoord(pipe=0, data=26, model=0): 26, ProcessCoord(pipe=0, data=27, model=0): 27, ProcessCoord(pipe=0, data=28, model=0): 28, ProcessCoord(pipe=0, data=29, model=0): 29, ProcessCoord(pipe=0, data=30, model=0): 30, ProcessCoord(pipe=0, data=31, model=0): 31, ProcessCoord(pipe=0, data=32, model=0): 32, ProcessCoord(pipe=0, data=33, model=0): 33, ProcessCoord(pipe=0, data=34, model=0): 34, ProcessCoord(pipe=0, data=35, model=0): 35, ProcessCoord(pipe=0, data=36, model=0): 36, ProcessCoord(pipe=0, data=37, model=0): 37, ProcessCoord(pipe=0, data=38, model=0): 38, ProcessCoord(pipe=0, data=39, model=0): 39, ProcessCoord(pipe=0, data=40, model=0): 40, ProcessCoord(pipe=0, data=41, model=0): 41, ProcessCoord(pipe=0, data=42, model=0): 42, ProcessCoord(pipe=0, data=43, model=0): 43, ProcessCoord(pipe=0, data=44, model=0): 44, ProcessCoord(pipe=0, data=45, model=0): 45, ProcessCoord(pipe=0, data=4 0: 6, model=0): 46, ProcessCoord(pipe=0, data=47, model=0): 47, ProcessCoord(pipe=0, data=48, model=0): 48, ProcessCoord(pipe=0, data=49, model=0): 49, ProcessCoord(pipe=0, data=50, model=0): 50, ProcessCoord(pipe=0, data=51, model=0): 51, ProcessCoord(pipe=0, data=52, model=0): 52, ProcessCoord(pipe=0, data=53, model=0): 53, ProcessCoord(pipe=0, data=54, model=0): 54, ProcessCoord(pipe=0, data=55, model=0): 55, ProcessCoord(pipe=0, data=56, model=0): 56, ProcessCoord(pipe=0, data=57, model=0): 57, ProcessCoord(pipe=0, data=58, model=0): 58, ProcessCoord(pipe=0, data=59, model=0): 59, ProcessCoord(pipe=0, data=60, model=0): 60, ProcessCoord(pipe=0, data=61, model=0): 61, ProcessCoord(pipe=0, data=62, model=0): 62, ProcessCoord(pipe=0, data=63, model=0): 63} 0: [2023-03-17 09:59:39,834] [INFO] [module.py:366:_partition_layers] Partitioning pipeline stages with method type:transformer 0: stage=0 layers=15 0: 0: _to_float16 0: 1: EmbeddingPipe 0: 2: 0: 3: ParallelTransformerLayerPipe 0: 4: ParallelTransformerLayerPipe 0: 5: ParallelTransformerLayerPipe 0: 6: ParallelTransformerLayerPipe 0: 7: ParallelTransformerLayerPipe 0: 8: ParallelTransformerLayerPipe 0: 9: ParallelTransformerLayerPipe 0: 10: ParallelTransformerLayerPipe 0: 11: undo 0: 12: MixedFusedLayerNorm 0: 13: EmbeddingPipe 0: 14: float16_to_fp32 0: loss: CrossEntropy 0: [2023-03-17 09:59:40,173] [INFO] [utils.py:827:see_memory_usage] After Building Model 0: [2023-03-17 09:59:40,174] [INFO] [utils.py:828:see_memory_usage] MA 0.1 GB Max_MA 0.1 GB CA 0.11 GB Max_CA 0 GB 0: [2023-03-17 09:59:40,174] [INFO] [utils.py:836:see_memory_usage] CPU Virtual Memory: used = 30.64 GB, percent = 6.1% 0: setting training iterations to 331103 0: > learning rate decay style: cosine 0: DeepSpeed is enabled. 0: [2023-03-17 09:59:40,175] [INFO] [logging.py:68:log_dist] [Rank 0] DeepSpeed info: version=0.7.5, git-hash=unknown, git-branch=unknown 0: [2023-03-17 09:59:52,853] [INFO] [logging.py:68:log_dist] [Rank 0] DeepSpeed Flops Profiler Enabled: False 0: [2023-03-17 09:59:52,854] [INFO] [logging.py:68:log_dist] [Rank 0] Removing param_group that has no 'params' in the client Optimizer 0: [2023-03-17 09:59:52,854] [INFO] [logging.py:68:log_dist] [Rank 0] Using client Optimizer as basic optimizer 0: [2023-03-17 09:59:52,855] [INFO] [logging.py:68:log_dist] [Rank 0] DeepSpeed Basic Optimizer = FusedAdam 0: [2023-03-17 09:59:52,855] [INFO] [logging.py:68:log_dist] [Rank 0] Creating BF16 optimizer 0: [2023-03-17 09:59:52,970] [INFO] [utils.py:827:see_memory_usage] begin bf16_optimizer 0: [2023-03-17 09:59:52,971] [INFO] [utils.py:828:see_memory_usage] MA 0.1 GB Max_MA 0.1 GB CA 0.11 GB Max_CA 0 GB 0: [2023-03-17 09:59:52,971] [INFO] [utils.py:836:see_memory_usage] CPU Virtual Memory: used = 31.33 GB, percent = 6.2% 0: ninja: no work to do. 0: Time to load utils op: 0.1955721378326416 seconds 0: Time to load utils op: 0.10249209403991699 seconds 4: Time to load utils op: 0.21030664443969727 seconds 5: Time to load utils op: 0.21029162406921387 seconds 0: Time to load utils op: 0.10229825973510742 seconds 0: Time to load utils op: 0.10245156288146973 seconds 0: Time to load utils op: 0.10198116302490234 seconds 0: Time to load utils op: 0.10224366188049316 seconds 0: Time to load utils op: 0.10239362716674805 seconds 0: Time to load utils op: 0.10208654403686523 seconds 3: Time to load utils op: 0.10388755798339844 secondsTime to load utils op: 0.10280632972717285 seconds 3: 3: Time to load utils op: 0.10390424728393555 seconds 3: Time to load utils op: 0.10340523719787598 seconds 3: Time to load utils op: 0.1032717227935791 seconds 3: Time to load utils op: 0.10451650619506836 seconds 4: Time to load utils op: 0.10210394859313965 seconds 3: Time to load utils op: 0.10386276245117188 seconds 4: Time to load utils op: 0.10213232040405273 seconds 3: Time to load utils op: 0.10352063179016113 seconds 4: Time to load utils op: 0.10255742073059082 seconds 4: Time to load utils op: 0.10241818428039551 seconds 4: Time to load utils op: 0.10264778137207031 seconds 4: Time to load utils op: 0.10277628898620605 seconds 4: Time to load utils op: 0.10308980941772461 seconds 5: Time to load utils op: 0.10200905799865723 seconds 5: Time to load utils op: 0.10170149803161621 seconds 5: Time to load utils op: 0.10253500938415527 seconds 5: Time to load utils op: 0.10231161117553711 secondsTime to load utils op: 0.1023101806640625 seconds 5: 5: Time to load utils op: 0.10254311561584473 seconds 5: Time to load utils op: 0.10254931449890137 seconds 1: Time to load utils op: 0.11138200759887695 seconds 1: Time to load utils op: 0.11107039451599121 seconds 1: Time to load utils op: 0.11090207099914551 seconds 1: Time to load utils op: 0.1112215518951416 seconds 1: Time to load utils op: 0.11061763763427734 seconds 1: Time to load utils op: 0.11187314987182617 seconds 1: Time to load utils op: 0.11099362373352051 seconds 1: Time to load utils op: 0.11108040809631348 seconds 2: Time to load utils op: 0.11049580574035645 seconds 2: Time to load utils op: 0.1105048656463623 seconds 2: Time to load utils op: 0.11048150062561035 seconds 2: Time to load utils op: 0.11056923866271973 secondsTime to load utils op: 0.11060476303100586 secondsTime to load utils op: 0.11031746864318848 seconds 2: 2: 2: Time to load utils op: 0.1103508472442627 secondsTime to load utils op: 0.11034965515136719 seconds 2: 3: Time to load utils op: 0.0004425048828125 seconds 3: Time to load utils op: 0.0004105567932128906 seconds 3: Time to load utils op: 0.0003440380096435547 seconds 3: Time to load utils op: 0.00035858154296875 seconds 3: Time to load utils op: 0.0003535747528076172 seconds 3: Time to load utils op: 0.00035572052001953125 seconds 3: Time to load utils op: 0.00034546852111816406 seconds 3: Time to load utils op: 0.00034618377685546875 seconds 7: Time to load utils op: 0.11034870147705078 seconds 7: Time to load utils op: 0.11039590835571289 seconds 7: Time to load utils op: 0.11041545867919922 seconds 7: Time to load utils op: 0.11042118072509766 seconds 6: Time to load utils op: 0.11005854606628418 seconds 7: Time to load utils op: 0.11042952537536621 seconds 7: Time to load utils op: 0.11042380332946777 seconds 7: Time to load utils op: 0.11031150817871094 seconds 6: Time to load utils op: 0.1100924015045166 seconds 6: Time to load utils op: 0.11010313034057617 seconds 6: Time to load utils op: 0.11011958122253418 seconds 6: Time to load utils op: 0.11013221740722656 secondsTime to load utils op: 0.11014413833618164 seconds 7: Time to load utils op: 0.11043334007263184 seconds 6: 6: Time to load utils op: 0.11014556884765625 seconds 6: Time to load utils op: 0.11015462875366211 seconds 0: [2023-03-17 09:59:53,178] [INFO] [utils.py:827:see_memory_usage] before initializing group 0 0: [2023-03-17 09:59:53,178] [INFO] [utils.py:828:see_memory_usage] MA 0.1 GB Max_MA 0.1 GB CA 0.11 GB Max_CA 0 GB 0: [2023-03-17 09:59:53,179] [INFO] [utils.py:836:see_memory_usage] CPU Virtual Memory: used = 31.33 GB, percent = 6.2% 0: Time to load utils op: 0.0005834102630615234 seconds 0: Time to load utils op: 0.0006172657012939453 seconds 0: Time to load utils op: 0.0005242824554443359 seconds 0: Time to load utils op: 0.0005121231079101562 seconds 0: Time to load utils op: 0.0005142688751220703 seconds 0: Time to load utils op: 0.00054931640625 seconds 0: Time to load utils op: 0.0005810260772705078 seconds 1: Time to load utils op: 0.0008418560028076172 seconds 1: Time to load utils op: 0.0007905960083007812 seconds 1: Time to load utils op: 0.0010747909545898438 seconds 1: Time to load utils op: 0.0010378360748291016 seconds 1: Time to load utils op: 0.00107574462890625 seconds 1: Time to load utils op: 0.0010769367218017578 seconds 1: Time to load utils op: 0.0010867118835449219 seconds 1: Time to load utils op: 0.0010967254638671875 seconds 6: Time to load utils op: 0.0007932186126708984 seconds 6: Time to load utils op: 0.0008833408355712891 seconds 6: Time to load utils op: 0.0010385513305664062 seconds 6: Time to load utils op: 0.001123666763305664 seconds 2: Time to load utils op: 0.0007064342498779297 seconds 6: Time to load utils op: 0.001149892807006836 seconds 6: Time to load utils op: 0.0011844635009765625 seconds 6: Time to load utils op: 0.0011649131774902344 seconds 2: Time to load utils op: 0.0009353160858154297 seconds 6: Time to load utils op: 0.0011687278747558594 seconds 2: Time to load utils op: 0.0010211467742919922 seconds 2: Time to load utils op: 0.0011820793151855469 seconds 2: Time to load utils op: 0.00118255615234375 seconds 2: Time to load utils op: 0.0012667179107666016 seconds 2: Time to load utils op: 0.001214742660522461 seconds 2: Time to load utils op: 0.00116729736328125 seconds 4: Time to load utils op: 0.0007421970367431641 seconds 5: Time to load utils op: 0.0008690357208251953 seconds 5: Time to load utils op: 0.0009515285491943359 seconds 4: Time to load utils op: 0.0009248256683349609 seconds 5: Time to load utils op: 0.0011799335479736328 secondsTime to load utils op: 0.0010647773742675781 seconds 5: Time to load utils op: 0.0011153221130371094 seconds 5: 4: Time to load utils op: 0.0009214878082275391 seconds 4: Time to load utils op: 0.0009388923645019531 seconds 5: Time to load utils op: 0.0011851787567138672 seconds 5: Time to load utils op: 0.0011632442474365234 seconds 4: Time to load utils op: 0.0009732246398925781 seconds 5: Time to load utils op: 0.0011973381042480469 seconds 4: Time to load utils op: 0.0011293888092041016 seconds 4: Time to load utils op: 0.0011219978332519531 seconds 4: Time to load utils op: 0.0011107921600341797 seconds 7: Time to load utils op: 0.0008544921875 seconds 7: Time to load utils op: 0.0007431507110595703 secondsTime to load utils op: 0.0011196136474609375 seconds 7: 7: Time to load utils op: 0.001283407211303711 seconds 7: Time to load utils op: 0.0012907981872558594 seconds 7: Time to load utils op: 0.0013089179992675781 secondsTime to load utils op: 0.00130462646484375 seconds 7: 7: Time to load utils op: 0.0011436939239501953 seconds 0: [2023-03-17 09:59:53,315] [INFO] [utils.py:827:see_memory_usage] after initializing group 0 0: [2023-03-17 09:59:53,315] [INFO] [utils.py:828:see_memory_usage] MA 0.24 GB Max_MA 0.24 GB CA 0.33 GB Max_CA 0 GB 0: [2023-03-17 09:59:53,315] [INFO] [utils.py:836:see_memory_usage] CPU Virtual Memory: used = 31.47 GB, percent = 6.3% 0: [2023-03-17 09:59:53,420] [INFO] [utils.py:827:see_memory_usage] before initializing group 1 0: [2023-03-17 09:59:53,420] [INFO] [utils.py:828:see_memory_usage] MA 0.24 GB Max_MA 0.24 GB CA 0.33 GB Max_CA 0 GB 0: [2023-03-17 09:59:53,420] [INFO] [utils.py:836:see_memory_usage] CPU Virtual Memory: used = 31.47 GB, percent = 6.3% 0: [2023-03-17 09:59:53,525] [INFO] [utils.py:827:see_memory_usage] after initializing group 1 0: [2023-03-17 09:59:53,525] [INFO] [utils.py:828:see_memory_usage] MA 0.29 GB Max_MA 0.29 GB CA 0.38 GB Max_CA 0 GB 0: [2023-03-17 09:59:53,525] [INFO] [utils.py:836:see_memory_usage] CPU Virtual Memory: used = 31.47 GB, percent = 6.3% 0: [2023-03-17 09:59:53,627] [INFO] [utils.py:827:see_memory_usage] before initializing group 2 0: [2023-03-17 09:59:53,628] [INFO] [utils.py:828:see_memory_usage] MA 0.29 GB Max_MA 0.29 GB CA 0.38 GB Max_CA 0 GB 0: [2023-03-17 09:59:53,628] [INFO] [utils.py:836:see_memory_usage] CPU Virtual Memory: used = 31.47 GB, percent = 6.3% 0: [2023-03-17 09:59:53,731] [INFO] [utils.py:827:see_memory_usage] after initializing group 2 0: [2023-03-17 09:59:53,732] [INFO] [utils.py:828:see_memory_usage] MA 0.29 GB Max_MA 0.29 GB CA 0.38 GB Max_CA 0 GB 0: [2023-03-17 09:59:53,732] [INFO] [utils.py:836:see_memory_usage] CPU Virtual Memory: used = 31.47 GB, percent = 6.3% 0: [2023-03-17 09:59:53,834] [INFO] [utils.py:827:see_memory_usage] before initialize_optimizer 0: [2023-03-17 09:59:53,834] [INFO] [utils.py:828:see_memory_usage] MA 0.29 GB Max_MA 0.29 GB CA 0.38 GB Max_CA 0 GB 0: [2023-03-17 09:59:53,834] [INFO] [utils.py:836:see_memory_usage] CPU Virtual Memory: used = 31.47 GB, percent = 6.3% 0: [2023-03-17 09:59:53,942] [INFO] [utils.py:827:see_memory_usage] end initialize_optimizer 0: [2023-03-17 09:59:53,943] [INFO] [utils.py:828:see_memory_usage] MA 0.3 GB Max_MA 0.3 GB CA 0.38 GB Max_CA 0 GB 0: [2023-03-17 09:59:53,943] [INFO] [utils.py:836:see_memory_usage] CPU Virtual Memory: used = 31.47 GB, percent = 6.3% 0: [2023-03-17 09:59:54,045] [INFO] [utils.py:827:see_memory_usage] end bf16_optimizer 0: [2023-03-17 09:59:54,045] [INFO] [utils.py:828:see_memory_usage] MA 0.3 GB Max_MA 0.3 GB CA 0.38 GB Max_CA 0 GB 0: [2023-03-17 09:59:54,045] [INFO] [utils.py:836:see_memory_usage] CPU Virtual Memory: used = 31.47 GB, percent = 6.3% 0: [2023-03-17 09:59:54,045] [INFO] [logging.py:68:log_dist] [Rank 0] DeepSpeed Final Optimizer = FusedAdam 0: [2023-03-17 09:59:54,046] [INFO] [logging.py:68:log_dist] [Rank 0] DeepSpeed using client LR scheduler 0: [2023-03-17 09:59:54,046] [INFO] [logging.py:68:log_dist] [Rank 0] DeepSpeed LR Scheduler = 0: [2023-03-17 09:59:54,046] [INFO] [logging.py:68:log_dist] [Rank 0] step=0, skipped=0, lr=[0.0, 0.0, 0.0], mom=[(0.9, 0.999), (0.9, 0.999), (0.9, 0.999)] 0: [2023-03-17 09:59:54,046] [INFO] [config.py:1007:print] DeepSpeedEngine configuration: 0: [2023-03-17 09:59:54,046] [INFO] [config.py:1011:print] activation_checkpointing_config { 0: "partition_activations": false, 0: "contiguous_memory_optimization": false, 0: "cpu_checkpointing": false, 0: "number_checkpoints": null, 0: "synchronize_checkpoint_boundary": false, 0: "profile": false 0: } 0: [2023-03-17 09:59:54,046] [INFO] [config.py:1011:print] aio_config ................... {'block_size': 1048576, 'queue_depth': 8, 'thread_count': 1, 'single_submit': False, 'overlap_events': True} 0: [2023-03-17 09:59:54,046] [INFO] [config.py:1011:print] amp_enabled .................. False 0: [2023-03-17 09:59:54,046] [INFO] [config.py:1011:print] amp_params ................... False 0: [2023-03-17 09:59:54,047] [INFO] [config.py:1011:print] autotuning_config ............ { 0: "enabled": false, 0: "start_step": null, 0: "end_step": null, 0: "metric_path": null, 0: "arg_mappings": null, 0: "metric": "throughput", 0: "model_info": null, 0: "results_dir": "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/autotuning_results", 0: "exps_dir": "/pfs/lustrep4/scratch/project_462000119/muennighoff/nov-2022-bettercom/autotuning_exps", 0: "overwrite": true, 0: "fast": true, 0: "start_profile_step": 3, 0: "end_profile_step": 5, 0: "tuner_type": "gridsearch", 0: "tuner_early_stopping": 5, 0: "tuner_num_trials": 50, 0: "model_info_path": null, 0: "mp_size": 1, 0: "max_train_batch_size": null, 0: "min_train_batch_size": 1, 0: "max_train_micro_batch_size_per_gpu": 1.024000e+03, 0: "min_train_micro_batch_size_per_gpu": 1, 0: "num_tuning_micro_batch_sizes": 3 0: } 0: [2023-03-17 09:59:54,047] [INFO] [config.py:1011:print] bfloat16_enabled ............. True 0: [2023-03-17 09:59:54,047] [INFO] [config.py:1011:print] checkpoint_parallel_write_pipeline False 0: [2023-03-17 09:59:54,047] [INFO] [config.py:1011:print] checkpoint_tag_validation_enabled True 0: [2023-03-17 09:59:54,047] [INFO] [config.py:1011:print] checkpoint_tag_validation_fail False 0: [2023-03-17 09:59:54,047] [INFO] [config.py:1011:print] comms_config ................. 0: [2023-03-17 09:59:54,047] [INFO] [config.py:1011:print] communication_data_type ...... None 0: [2023-03-17 09:59:54,047] [INFO] [config.py:1011:print] compression_config ........... {'weight_quantization': {'shared_parameters': {'enabled': False, 'quantizer_kernel': False, 'schedule_offset': 0, 'quantize_groups': 1, 'quantize_verbose': False, 'quantization_type': 'symmetric', 'quantize_weight_in_forward': False, 'rounding': 'nearest', 'fp16_mixed_quantize': False, 'quantize_change_ratio': 0.001}, 'different_groups': {}}, 'activation_quantization': {'shared_parameters': {'enabled': False, 'quantization_type': 'symmetric', 'range_calibration': 'dynamic', 'schedule_offset': 1000}, 'different_groups': {}}, 'sparse_pruning': {'shared_parameters': {'enabled': False, 'method': 'l1', 'schedule_offset': 1000}, 'different_groups': {}}, 'row_pruning': {'shared_parameters': {'enabled': False, 'method': 'l1', 'schedule_offset': 1000}, 'different_groups': {}}, 'head_pruning': {'shared_parameters': {'enabled': False, 'method': 'topk', 'schedule_offset': 1000}, 'different_groups': {}}, 'channel_pruning': {'shared_pa 0: rameters': {'enabled': False, 'method': 'l1', 'schedule_offset': 1000}, 'different_groups': {}}, 'layer_reduction': {'enabled': False}} 0: [2023-03-17 09:59:54,047] [INFO] [config.py:1011:print] curriculum_enabled ........... False 0: [2023-03-17 09:59:54,047] [INFO] [config.py:1011:print] curriculum_params ............ False 0: [2023-03-17 09:59:54,047] [INFO] [config.py:1011:print] dataloader_drop_last ......... False 0: [2023-03-17 09:59:54,047] [INFO] [config.py:1011:print] disable_allgather ............ False 0: [2023-03-17 09:59:54,047] [INFO] [config.py:1011:print] dump_state ................... False 0: [2023-03-17 09:59:54,047] [INFO] [config.py:1011:print] dynamic_loss_scale_args ...... None 0: [2023-03-17 09:59:54,047] [INFO] [config.py:1011:print] eigenvalue_enabled ........... False 0: [2023-03-17 09:59:54,047] [INFO] [config.py:1011:print] eigenvalue_gas_boundary_resolution 1 0: [2023-03-17 09:59:54,047] [INFO] [config.py:1011:print] eigenvalue_layer_name ........ bert.encoder.layer 0: [2023-03-17 09:59:54,047] [INFO] [config.py:1011:print] eigenvalue_layer_num ......... 0 0: [2023-03-17 09:59:54,047] [INFO] [config.py:1011:print] eigenvalue_max_iter .......... 100 0: [2023-03-17 09:59:54,047] [INFO] [config.py:1011:print] eigenvalue_stability ......... 1e-06 0: [2023-03-17 09:59:54,047] [INFO] [config.py:1011:print] eigenvalue_tol ............... 0.01 0: [2023-03-17 09:59:54,047] [INFO] [config.py:1011:print] eigenvalue_verbose ........... False 0: [2023-03-17 09:59:54,047] [INFO] [config.py:1011:print] elasticity_enabled ........... False 0: [2023-03-17 09:59:54,047] [INFO] [config.py:1011:print] flops_profiler_config ........ { 0: "enabled": false, 0: "profile_step": 1, 0: "module_depth": -1, 0: "top_modules": 1, 0: "detailed": true, 0: "output_file": null 0: } 0: [2023-03-17 09:59:54,047] [INFO] [config.py:1011:print] fp16_auto_cast ............... None 0: [2023-03-17 09:59:54,047] [INFO] [config.py:1011:print] fp16_enabled ................. False 0: [2023-03-17 09:59:54,047] [INFO] [config.py:1011:print] fp16_master_weights_and_gradients False 0: [2023-03-17 09:59:54,047] [INFO] [config.py:1011:print] global_rank .................. 0 0: [2023-03-17 09:59:54,047] [INFO] [config.py:1011:print] gradient_accumulation_steps .. 1 0: [2023-03-17 09:59:54,047] [INFO] [config.py:1011:print] gradient_clipping ............ 1.0 0: [2023-03-17 09:59:54,047] [INFO] [config.py:1011:print] gradient_predivide_factor .... 1.0 0: [2023-03-17 09:59:54,047] [INFO] [config.py:1011:print] initial_dynamic_scale ........ 1 0: [2023-03-17 09:59:54,047] [INFO] [config.py:1011:print] load_universal_checkpoint .... False 0: [2023-03-17 09:59:54,047] [INFO] [config.py:1011:print] loss_scale ................... 1.0 0: [2023-03-17 09:59:54,047] [INFO] [config.py:1011:print] memory_breakdown ............. False 0: [2023-03-17 09:59:54,047] [INFO] [config.py:1011:print] monitor_config ............... 0: [2023-03-17 09:59:54,048] [INFO] [config.py:1011:print] nebula_config ................ { 0: "enabled": false, 0: "persistent_storage_path": null, 0: "persistent_time_interval": 100, 0: "num_of_version_in_retention": 2, 0: "enable_nebula_load": true, 0: "load_path": null 0: } 0: [2023-03-17 09:59:54,048] [INFO] [config.py:1011:print] optimizer_legacy_fusion ...... False 0: [2023-03-17 09:59:54,048] [INFO] [config.py:1011:print] optimizer_name ............... None 0: [2023-03-17 09:59:54,048] [INFO] [config.py:1011:print] optimizer_params ............. None 0: [2023-03-17 09:59:54,048] [INFO] [config.py:1011:print] pipeline ..................... {'stages': 'auto', 'partition': 'best', 'seed_layers': False, 'activation_checkpoint_interval': 0} 0: [2023-03-17 09:59:54,048] [INFO] [config.py:1011:print] pld_enabled .................. False 0: [2023-03-17 09:59:54,048] [INFO] [config.py:1011:print] pld_params ................... False 0: [2023-03-17 09:59:54,048] [INFO] [config.py:1011:print] prescale_gradients ........... False 0: [2023-03-17 09:59:54,048] [INFO] [config.py:1011:print] scheduler_name ............... None 0: [2023-03-17 09:59:54,048] [INFO] [config.py:1011:print] scheduler_params ............. None 0: [2023-03-17 09:59:54,048] [INFO] [config.py:1011:print] sparse_attention ............. None 0: [2023-03-17 09:59:54,048] [INFO] [config.py:1011:print] sparse_gradients_enabled ..... False 0: [2023-03-17 09:59:54,048] [INFO] [config.py:1011:print] steps_per_print .............. 2000 0: [2023-03-17 09:59:54,048] [INFO] [config.py:1011:print] train_batch_size ............. 256 0: [2023-03-17 09:59:54,048] [INFO] [config.py:1011:print] train_micro_batch_size_per_gpu 4 0: [2023-03-17 09:59:54,048] [INFO] [config.py:1011:print] use_node_local_storage ....... False 0: [2023-03-17 09:59:54,048] [INFO] [config.py:1011:print] wall_clock_breakdown ......... False 0: [2023-03-17 09:59:54,048] [INFO] [config.py:1011:print] world_size ................... 64 0: [2023-03-17 09:59:54,048] [INFO] [config.py:1011:print] zero_allow_untested_optimizer False 0: [2023-03-17 09:59:54,048] [INFO] [config.py:1011:print] zero_config .................. stage=0 contiguous_gradients=True reduce_scatter=True reduce_bucket_size=500000000 allgather_partitions=True allgather_bucket_size=500000000 overlap_comm=False load_from_fp32_weights=True elastic_checkpoint=False offload_param=None offload_optimizer=None sub_group_size=1000000000 cpu_offload_param=None cpu_offload_use_pin_memory=None cpu_offload=None prefetch_bucket_size=50000000 param_persistence_threshold=100000 model_persistence_threshold=9223372036854775807 max_live_parameters=1000000000 max_reuse_distance=1000000000 gather_16bit_weights_on_model_save=False stage3_gather_fp16_weights_on_model_save=False ignore_unused_parameters=True legacy_stage1=False round_robin_gradients=False 0: [2023-03-17 09:59:54,048] [INFO] [config.py:1011:print] zero_enabled ................. False 0: [2023-03-17 09:59:54,048] [INFO] [config.py:1011:print] zero_optimization_stage ...... 0 0: [2023-03-17 09:59:54,048] [INFO] [config.py:996:print_user_config] json = { 0: "train_micro_batch_size_per_gpu": 4, 0: "train_batch_size": 256, 0: "gradient_clipping": 1.0, 0: "zero_optimization": { 0: "stage": 0 0: }, 0: "bf16": { 0: "enabled": true 0: }, 0: "steps_per_print": 2.000000e+03, 0: "wall_clock_breakdown": false 0: } 0: Time to load utils op: 0.00043082237243652344 seconds 0: [2023-03-17 09:59:54,049] [INFO] [engine.py:87:__init__] CONFIG: micro_batches=1 micro_batch_size=4 0: [2023-03-17 09:59:54,096] [INFO] [engine.py:145:__init__] RANK=0 STAGE=0 LAYERS=15 [0, 15) STAGE_PARAMS=52024320 (52.024M) TOTAL_PARAMS=52024320 (52.024M) UNIQUE_PARAMS=52024320 (52.024M) 0: [2023-03-17 09:59:54,101] [WARNING] [engine.py:2581:load_checkpoint] Unable to find latest file at checkpoints_44m174b100m/latest, if trying to load latest checkpoint please ensure this file exists or pass an explicit checkpoint tag when loading a checkpoint. 0: WARNING: could not find the metadata file checkpoints_44m174b100m 0: will not load any checkpoints and will start from random 0: [2023-03-17 09:59:54,101] [WARNING] [engine.py:2581:load_checkpoint] Unable to find latest file at checkpoints_44m174b100m/latest, if trying to load latest checkpoint please ensure this file exists or pass an explicit checkpoint tag when loading a checkpoint. 0: [2023-03-17 09:59:54,101] [WARNING] [engine.py:2581:load_checkpoint] Unable to find latest file at checkpoints_44m174b100m/latest, if trying to load latest checkpoint please ensure this file exists or pass an explicit checkpoint tag when loading a checkpoint. 0: [2023-03-17 09:59:54,101] [WARNING] [engine.py:2581:load_checkpoint] Unable to find latest file at checkpoints_44m174b100m/latest, if trying to load latest checkpoint please ensure this file exists or pass an explicit checkpoint tag when loading a checkpoint. 4: [2023-03-17 09:59:54,101] [WARNING] [engine.py:2581:load_checkpoint] Unable to find latest file at checkpoints_44m174b100m/latest, if trying to load latest checkpoint please ensure this file exists or pass an explicit checkpoint tag when loading a checkpoint. 4: [2023-03-17 09:59:54,101] [WARNING] [engine.py:2581:load_checkpoint] Unable to find latest file at checkpoints_44m174b100m/latest, if trying to load latest checkpoint please ensure this file exists or pass an explicit checkpoint tag when loading a checkpoint. 4: [2023-03-17 09:59:54,101] [WARNING] [engine.py:2581:load_checkpoint] Unable to find latest file at checkpoints_44m174b100m/latest, if trying to load latest checkpoint please ensure this file exists or pass an explicit checkpoint tag when loading a checkpoint. 4: [2023-03-17 09:59:54,101] [WARNING] [engine.py:2581:load_checkpoint] Unable to find latest file at checkpoints_44m174b100m/latest, if trying to load latest checkpoint please ensure this file exists or pass an explicit checkpoint tag when loading a checkpoint. 6: [2023-03-17 09:59:54,101] [WARNING] [engine.py:2581:load_checkpoint] Unable to find latest file at checkpoints_44m174b100m/latest, if trying to load latest checkpoint please ensure this file exists or pass an explicit checkpoint tag when loading a checkpoint. 6: [2023-03-17 09:59:54,101] [WARNING] [engine.py:2581:load_checkpoint] Unable to find latest file at checkpoints_44m174b100m/latest, if trying to load latest checkpoint please ensure this file exists or pass an explicit checkpoint tag when loading a checkpoint. 6: [2023-03-17 09:59:54,101] [WARNING] [engine.py:2581:load_checkpoint] Unable to find latest file at checkpoints_44m174b100m/latest, if trying to load latest checkpoint please ensure this file exists or pass an explicit checkpoint tag when loading a checkpoint. 6: [2023-03-17 09:59:54,101] [WARNING] [engine.py:2581:load_checkpoint] Unable to find latest file at checkpoints_44m174b100m/latest, if trying to load latest checkpoint please ensure this file exists or pass an explicit checkpoint tag when loading a checkpoint. 7: [2023-03-17 09:59:54,101] [WARNING] [engine.py:2581:load_checkpoint] Unable to find latest file at checkpoints_44m174b100m/latest, if trying to load latest checkpoint please ensure this file exists or pass an explicit checkpoint tag when loading a checkpoint. 7: [2023-03-17 09:59:54,101] [WARNING] [engine.py:2581:load_checkpoint] Unable to find latest file at checkpoints_44m174b100m/latest, if trying to load latest checkpoint please ensure this file exists or pass an explicit checkpoint tag when loading a checkpoint. 0: [2023-03-17 09:59:54,101] [WARNING] [engine.py:2581:load_checkpoint] Unable to find latest file at checkpoints_44m174b100m/latest, if trying to load latest checkpoint please ensure this file exists or pass an explicit checkpoint tag when loading a checkpoint. 3: [2023-03-17 09:59:54,101] [WARNING] [engine.py:2581:load_checkpoint] Unable to find latest file at checkpoints_44m174b100m/latest, if trying to load latest checkpoint please ensure this file exists or pass an explicit checkpoint tag when loading a checkpoint. 3: [2023-03-17 09:59:54,101] [WARNING] [engine.py:2581:load_checkpoint] Unable to find latest file at checkpoints_44m174b100m/latest, if trying to load latest checkpoint please ensure this file exists or pass an explicit checkpoint tag when loading a checkpoint. 2: [2023-03-17 09:59:54,101] [WARNING] [engine.py:2581:load_checkpoint] Unable to find latest file at checkpoints_44m174b100m/latest, if trying to load latest checkpoint please ensure this file exists or pass an explicit checkpoint tag when loading a checkpoint. 2: [2023-03-17 09:59:54,101] [WARNING] [engine.py:2581:load_checkpoint] Unable to find latest file at checkpoints_44m174b100m/latest, if trying to load latest checkpoint please ensure this file exists or pass an explicit checkpoint tag when loading a checkpoint. 2: [2023-03-17 09:59:54,101] [WARNING] [engine.py:2581:load_checkpoint] Unable to find latest file at checkpoints_44m174b100m/latest, if trying to load latest checkpoint please ensure this file exists or pass an explicit checkpoint tag when loading a checkpoint. 6: [2023-03-17 09:59:54,101] [WARNING] [engine.py:2581:load_checkpoint] Unable to find latest file at checkpoints_44m174b100m/latest, if trying to load latest checkpoint please ensure this file exists or pass an explicit checkpoint tag when loading a checkpoint. 7: [2023-03-17 09:59:54,101] [WARNING] [engine.py:2581:load_checkpoint] Unable to find latest file at checkpoints_44m174b100m/latest, if trying to load latest checkpoint please ensure this file exists or pass an explicit checkpoint tag when loading a checkpoint. 3: [2023-03-17 09:59:54,101] [WARNING] [engine.py:2581:load_checkpoint] Unable to find latest file at checkpoints_44m174b100m/latest, if trying to load latest checkpoint please ensure this file exists or pass an explicit checkpoint tag when loading a checkpoint. 3: [2023-03-17 09:59:54,101] [WARNING] [engine.py:2581:load_checkpoint] Unable to find latest file at checkpoints_44m174b100m/latest, if trying to load latest checkpoint please ensure this file exists or pass an explicit checkpoint tag when loading a checkpoint. 2: [2023-03-17 09:59:54,101] [WARNING] [engine.py:2581:load_checkpoint] Unable to find latest file at checkpoints_44m174b100m/latest, if trying to load latest checkpoint please ensure this file exists or pass an explicit checkpoint tag when loading a checkpoint. 4: [2023-03-17 09:59:54,101] [WARNING] [engine.py:2581:load_checkpoint] Unable to find latest file at checkpoints_44m174b100m/latest, if trying to load latest checkpoint please ensure this file exists or pass an explicit checkpoint tag when loading a checkpoint. 6: [2023-03-17 09:59:54,101] [WARNING] [engine.py:2581:load_checkpoint] Unable to find latest file at checkpoints_44m174b100m/latest, if trying to load latest checkpoint please ensure this file exists or pass an explicit checkpoint tag when loading a checkpoint. 7: [2023-03-17 09:59:54,101] [WARNING] [engine.py:2581:load_checkpoint] Unable to find latest file at checkpoints_44m174b100m/latest, if trying to load latest checkpoint please ensure this file exists or pass an explicit checkpoint tag when loading a checkpoint. 5: [2023-03-17 09:59:54,101] [WARNING] [engine.py:2581:load_checkpoint] Unable to find latest file at checkpoints_44m174b100m/latest, if trying to load latest checkpoint please ensure this file exists or pass an explicit checkpoint tag when loading a checkpoint. 5: [2023-03-17 09:59:54,101] [WARNING] [engine.py:2581:load_checkpoint] Unable to find latest file at checkpoints_44m174b100m/latest, if trying to load latest checkpoint please ensure this file exists or pass an explicit checkpoint tag when loading a checkpoint. 5: [2023-03-17 09:59:54,101] [WARNING] [engine.py:2581:load_checkpoint] Unable to find latest file at checkpoints_44m174b100m/latest, if trying to load latest checkpoint please ensure this file exists or pass an explicit checkpoint tag when loading a checkpoint. 0: [2023-03-17 09:59:54,101] [WARNING] [engine.py:2581:load_checkpoint] Unable to find latest file at checkpoints_44m174b100m/latest, if trying to load latest checkpoint please ensure this file exists or pass an explicit checkpoint tag when loading a checkpoint. 5: [2023-03-17 09:59:54,101] [WARNING] [engine.py:2581:load_checkpoint] Unable to find latest file at checkpoints_44m174b100m/latest, if trying to load latest checkpoint please ensure this file exists or pass an explicit checkpoint tag when loading a checkpoint. 4: [2023-03-17 09:59:54,101] [WARNING] [engine.py:2581:load_checkpoint] Unable to find latest file at checkpoints_44m174b100m/latest, if trying to load latest checkpoint please ensure this file exists or pass an explicit checkpoint tag when loading a checkpoint. 7: [2023-03-17 09:59:54,101] [WARNING] [engine.py:2581:load_checkpoint] Unable to find latest file at checkpoints_44m174b100m/latest, if trying to load latest checkpoint please ensure this file exists or pass an explicit checkpoint tag when loading a checkpoint. 7: [2023-03-17 09:59:54,101] [WARNING] [engine.py:2581:load_checkpoint] Unable to find latest file at checkpoints_44m174b100m/latest, if trying to load latest checkpoint please ensure this file exists or pass an explicit checkpoint tag when loading a checkpoint. 2: [2023-03-17 09:59:54,101] [WARNING] [engine.py:2581:load_checkpoint] Unable to find latest file at checkpoints_44m174b100m/latest, if trying to load latest checkpoint please ensure this file exists or pass an explicit checkpoint tag when loading a checkpoint. 1: [2023-03-17 09:59:54,101] [WARNING] [engine.py:2581:load_checkpoint] Unable to find latest file at checkpoints_44m174b100m/latest, if trying to load latest checkpoint please ensure this file exists or pass an explicit checkpoint tag when loading a checkpoint. 1: [2023-03-17 09:59:54,101] [WARNING] [engine.py:2581:load_checkpoint] Unable to find latest file at checkpoints_44m174b100m/latest, if trying to load latest checkpoint please ensure this file exists or pass an explicit checkpoint tag when loading a checkpoint. 2: [2023-03-17 09:59:54,101] [WARNING] [engine.py:2581:load_checkpoint] Unable to find latest file at checkpoints_44m174b100m/latest, if trying to load latest checkpoint please ensure this file exists or pass an explicit checkpoint tag when loading a checkpoint. 3: [2023-03-17 09:59:54,101] [WARNING] [engine.py:2581:load_checkpoint] Unable to find latest file at checkpoints_44m174b100m/latest, if trying to load latest checkpoint please ensure this file exists or pass an explicit checkpoint tag when loading a checkpoint. 3: [2023-03-17 09:59:54,101] [WARNING] [engine.py:2581:load_checkpoint] Unable to find latest file at checkpoints_44m174b100m/latest, if trying to load latest checkpoint please ensure this file exists or pass an explicit checkpoint tag when loading a checkpoint. 6: [2023-03-17 09:59:54,101] [WARNING] [engine.py:2581:load_checkpoint] Unable to find latest file at checkpoints_44m174b100m/latest, if trying to load latest checkpoint please ensure this file exists or pass an explicit checkpoint tag when loading a checkpoint. 5: [2023-03-17 09:59:54,101] [WARNING] [engine.py:2581:load_checkpoint] Unable to find latest file at checkpoints_44m174b100m/latest, if trying to load latest checkpoint please ensure this file exists or pass an explicit checkpoint tag when loading a checkpoint. 1: [2023-03-17 09:59:54,101] [WARNING] [engine.py:2581:load_checkpoint] Unable to find latest file at checkpoints_44m174b100m/latest, if trying to load latest checkpoint please ensure this file exists or pass an explicit checkpoint tag when loading a checkpoint. 6: [2023-03-17 09:59:54,101] [WARNING] [engine.py:2581:load_checkpoint] Unable to find latest file at checkpoints_44m174b100m/latest, if trying to load latest checkpoint please ensure this file exists or pass an explicit checkpoint tag when loading a checkpoint. 0: [2023-03-17 09:59:54,101] [WARNING] [engine.py:2581:load_checkpoint] Unable to find latest file at checkpoints_44m174b100m/latest, if trying to load latest checkpoint please ensure this file exists or pass an explicit checkpoint tag when loading a checkpoint. 4: [2023-03-17 09:59:54,101] [WARNING] [engine.py:2581:load_checkpoint] Unable to find latest file at checkpoints_44m174b100m/latest, if trying to load latest checkpoint please ensure this file exists or pass an explicit checkpoint tag when loading a checkpoint. 7: [2023-03-17 09:59:54,101] [WARNING] [engine.py:2581:load_checkpoint] Unable to find latest file at checkpoints_44m174b100m/latest, if trying to load latest checkpoint please ensure this file exists or pass an explicit checkpoint tag when loading a checkpoint. 1: [2023-03-17 09:59:54,101] [WARNING] [engine.py:2581:load_checkpoint] Unable to find latest file at checkpoints_44m174b100m/latest, if trying to load latest checkpoint please ensure this file exists or pass an explicit checkpoint tag when loading a checkpoint. 0: [2023-03-17 09:59:54,102] [WARNING] [engine.py:2581:load_checkpoint] Unable to find latest file at checkpoints_44m174b100m/latest, if trying to load latest checkpoint please ensure this file exists or pass an explicit checkpoint tag when loading a checkpoint. 7: [2023-03-17 09:59:54,101] [WARNING] [engine.py:2581:load_checkpoint] Unable to find latest file at checkpoints_44m174b100m/latest, if trying to load latest checkpoint please ensure this file exists or pass an explicit checkpoint tag when loading a checkpoint. 1: [2023-03-17 09:59:54,102] [WARNING] [engine.py:2581:load_checkpoint] Unable to find latest file at checkpoints_44m174b100m/latest, if trying to load latest checkpoint please ensure this file exists or pass an explicit checkpoint tag when loading a checkpoint. 1: [2023-03-17 09:59:54,102] [WARNING] [engine.py:2581:load_checkpoint] Unable to find latest file at checkpoints_44m174b100m/latest, if trying to load latest checkpoint please ensure this file exists or pass an explicit checkpoint tag when loading a checkpoint. 5: [2023-03-17 09:59:54,101] [WARNING] [engine.py:2581:load_checkpoint] Unable to find latest file at checkpoints_44m174b100m/latest, if trying to load latest checkpoint please ensure this file exists or pass an explicit checkpoint tag when loading a checkpoint. 2: [2023-03-17 09:59:54,102] [WARNING] [engine.py:2581:load_checkpoint] Unable to find latest file at checkpoints_44m174b100m/latest, if trying to load latest checkpoint please ensure this file exists or pass an explicit checkpoint tag when loading a checkpoint. 5: [2023-03-17 09:59:54,102] [WARNING] [engine.py:2581:load_checkpoint] Unable to find latest file at checkpoints_44m174b100m/latest, if trying to load latest checkpoint please ensure this file exists or pass an explicit checkpoint tag when loading a checkpoint. 3: [2023-03-17 09:59:54,102] [WARNING] [engine.py:2581:load_checkpoint] Unable to find latest file at checkpoints_44m174b100m/latest, if trying to load latest checkpoint please ensure this file exists or pass an explicit checkpoint tag when loading a checkpoint. 4: [2023-03-17 09:59:54,102] [WARNING] [engine.py:2581:load_checkpoint] Unable to find latest file at checkpoints_44m174b100m/latest, if trying to load latest checkpoint please ensure this file exists or pass an explicit checkpoint tag when loading a checkpoint. 5: [2023-03-17 09:59:54,102] [WARNING] [engine.py:2581:load_checkpoint] Unable to find latest file at checkpoints_44m174b100m/latest, if trying to load latest checkpoint please ensure this file exists or pass an explicit checkpoint tag when loading a checkpoint. 2: [2023-03-17 09:59:54,102] [WARNING] [engine.py:2581:load_checkpoint] Unable to find latest file at checkpoints_44m174b100m/latest, if trying to load latest checkpoint please ensure this file exists or pass an explicit checkpoint tag when loading a checkpoint. 3: [2023-03-17 09:59:54,102] [WARNING] [engine.py:2581:load_checkpoint] Unable to find latest file at checkpoints_44m174b100m/latest, if trying to load latest checkpoint please ensure this file exists or pass an explicit checkpoint tag when loading a checkpoint. 1: [2023-03-17 09:59:54,102] [WARNING] [engine.py:2581:load_checkpoint] Unable to find latest file at checkpoints_44m174b100m/latest, if trying to load latest checkpoint please ensure this file exists or pass an explicit checkpoint tag when loading a checkpoint. 1: [2023-03-17 09:59:54,102] [WARNING] [engine.py:2581:load_checkpoint] Unable to find latest file at checkpoints_44m174b100m/latest, if trying to load latest checkpoint please ensure this file exists or pass an explicit checkpoint tag when loading a checkpoint. 7: time (ms) | load-checkpoint: 8.47 0: estimated model parameters: 0.05202432 0: estimated model parameters without embeddings: 0.025220096 0: [after model, optimizer, and learning rate scheduler are built] datetime: 2023-03-17 09:59:54 0: > building train, validation, and test datasets ... 0: > datasets target sizes (minimum size): 0: train: 84762549 0: validation: 8704 0: test: 256 0: > building train, validation, and test datasets for GPT ... 0: > building dataset index ... 0: reading sizes... 0: reading pointers... 0: reading document index... 0: creating numpy buffer of mmap... 0: creating memory view of numpy buffer... 0: > finished creating indexed dataset in 0.008055 seconds 0: number of documents: 208931 0: > dataset split: 0: train: 0: document indices in [0, 208931) total of 208931 documents 0: > loading doc-idx mapping from /scratch/project_462000119/data/c4_subsampled/gpt2tok_c4_en_100M_text_document_train_indexmap_84762549ns_2048sl_1234s_doc_idx.npy 0: > loading sample-idx mapping from /scratch/project_462000119/data/c4_subsampled/gpt2tok_c4_en_100M_text_document_train_indexmap_84762549ns_2048sl_1234s_sample_idx.npy 0: > loading shuffle-idx mapping from /scratch/project_462000119/data/c4_subsampled/gpt2tok_c4_en_100M_text_document_train_indexmap_84762549ns_2048sl_1234s_shuffle_idx.npy 0: loaded indexed file in 0.088 seconds 0: total number of samples: 84774081 0: total number of epochs: 1737 0: > building dataset index ... 0: reading sizes... 0: reading pointers... 0: reading document index... 0: creating numpy buffer of mmap... 0: creating memory view of numpy buffer... 0: > finished creating indexed dataset in 0.059390 seconds 0: number of documents: 364608 0: > dataset split: 0: validation: 0: document indices in [0, 364608) total of 364608 documents 0: > loading doc-idx mapping from /scratch/project_462000119/data/c4_validation/gpt2tok_c4validation_rerun_text_document_validation_indexmap_8704ns_2048sl_1234s_doc_idx.npy 0: > loading sample-idx mapping from /scratch/project_462000119/data/c4_validation/gpt2tok_c4validation_rerun_text_document_validation_indexmap_8704ns_2048sl_1234s_sample_idx.npy 0: > loading shuffle-idx mapping from /scratch/project_462000119/data/c4_validation/gpt2tok_c4validation_rerun_text_document_validation_indexmap_8704ns_2048sl_1234s_shuffle_idx.npy 0: loaded indexed file in 0.085 seconds 0: total number of samples: 84978 0: total number of epochs: 1 0: > finished creating GPT datasets ... 0: [after dataloaders are built] datetime: 2023-03-17 10:00:08 0: done with setup ... 0: training ... 0: Number of parameters: [tensor rank - pipeline rank] w/ and w/o embeddings: 7: time (ms) | model-and-optimizer-setup: 17097.77 | train/valid/test-data-iterators-setup: 13079.74 0: [000-000] 0.0520B / 0.0252B 0: [before the start of training step] datetime: 2023-03-17 10:00:08 0: [2023-03-17 10:00:09,512] [INFO] [checkpointing.py:553:forward] Activation Checkpointing Information 0: [2023-03-17 10:00:09,513] [INFO] [checkpointing.py:554:forward] ----Partition Activations False, CPU CHECKPOINTING False 0: [2023-03-17 10:00:09,513] [INFO] [checkpointing.py:557:forward] ----contiguous Memory Checkpointing False with None total layers 0: [2023-03-17 10:00:09,513] [INFO] [checkpointing.py:560:forward] ----Synchronization False 0: [2023-03-17 10:00:09,513] [INFO] [checkpointing.py:561:forward] ----Profiling time in checkpointing False 0: [Rank 0] (after 100 iterations) memory (MB) | allocated: 1987.70947265625 | max allocated: 4436.146484375 | reserved: 5476.0 | max reserved: 5476.0 7: iteration 100/ 331103 | consumed samples: 25600 | consumed tokens: 52428800 | elapsed time per iteration (s): 0.28 | learning rate: 6.040E-06 | global batch size: 256 | lm loss: 1.032969E+01 | grad norm: 1.809 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 913.912 | TFLOPs: 14.33 | 7: iteration 200/ 331103 | consumed samples: 51200 | consumed tokens: 104857600 | elapsed time per iteration (s): 0.18 | learning rate: 1.208E-05 | global batch size: 256 | lm loss: 9.362046E+00 | grad norm: 1.684 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1454.357 | TFLOPs: 22.81 | 7: iteration 300/ 331103 | consumed samples: 76800 | consumed tokens: 157286400 | elapsed time per iteration (s): 0.17 | learning rate: 1.812E-05 | global batch size: 256 | lm loss: 8.518806E+00 | grad norm: 1.428 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1513.800 | TFLOPs: 23.74 | 7: iteration 400/ 331103 | consumed samples: 102400 | consumed tokens: 209715200 | elapsed time per iteration (s): 0.17 | learning rate: 2.416E-05 | global batch size: 256 | lm loss: 7.685542E+00 | grad norm: 0.714 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1526.344 | TFLOPs: 23.94 | 7: iteration 500/ 331103 | consumed samples: 128000 | consumed tokens: 262144000 | elapsed time per iteration (s): 0.17 | learning rate: 3.020E-05 | global batch size: 256 | lm loss: 7.127084E+00 | grad norm: 0.506 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1491.156 | TFLOPs: 23.39 | 7: iteration 600/ 331103 | consumed samples: 153600 | consumed tokens: 314572800 | elapsed time per iteration (s): 0.17 | learning rate: 3.624E-05 | global batch size: 256 | lm loss: 6.824040E+00 | grad norm: 0.720 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1511.650 | TFLOPs: 23.71 | 7: iteration 700/ 331103 | consumed samples: 179200 | consumed tokens: 367001600 | elapsed time per iteration (s): 0.17 | learning rate: 4.228E-05 | global batch size: 256 | lm loss: 6.617360E+00 | grad norm: 0.383 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1536.464 | TFLOPs: 24.10 | 7: iteration 800/ 331103 | consumed samples: 204800 | consumed tokens: 419430400 | elapsed time per iteration (s): 0.17 | learning rate: 4.832E-05 | global batch size: 256 | lm loss: 6.471720E+00 | grad norm: 0.473 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1532.859 | TFLOPs: 24.04 | 7: iteration 900/ 331103 | consumed samples: 230400 | consumed tokens: 471859200 | elapsed time per iteration (s): 0.17 | learning rate: 5.436E-05 | global batch size: 256 | lm loss: 6.356984E+00 | grad norm: 0.824 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1525.990 | TFLOPs: 23.93 | 7: iteration 1000/ 331103 | consumed samples: 256000 | consumed tokens: 524288000 | elapsed time per iteration (s): 0.17 | learning rate: 6.040E-05 | global batch size: 256 | lm loss: 6.248834E+00 | grad norm: 1.039 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1510.236 | TFLOPs: 23.68 | 7: iteration 1100/ 331103 | consumed samples: 281600 | consumed tokens: 576716800 | elapsed time per iteration (s): 0.16 | learning rate: 6.644E-05 | global batch size: 256 | lm loss: 6.130931E+00 | grad norm: 1.258 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1553.197 | TFLOPs: 24.36 | 7: iteration 1200/ 331103 | consumed samples: 307200 | consumed tokens: 629145600 | elapsed time per iteration (s): 0.16 | learning rate: 7.248E-05 | global batch size: 256 | lm loss: 6.027696E+00 | grad norm: 1.247 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1577.849 | TFLOPs: 24.74 | 7: iteration 1300/ 331103 | consumed samples: 332800 | consumed tokens: 681574400 | elapsed time per iteration (s): 0.17 | learning rate: 7.853E-05 | global batch size: 256 | lm loss: 5.935196E+00 | grad norm: 1.499 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1547.381 | TFLOPs: 24.27 | 7: iteration 1400/ 331103 | consumed samples: 358400 | consumed tokens: 734003200 | elapsed time per iteration (s): 0.17 | learning rate: 8.457E-05 | global batch size: 256 | lm loss: 5.846610E+00 | grad norm: 1.410 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1521.945 | TFLOPs: 23.87 | 7: iteration 1500/ 331103 | consumed samples: 384000 | consumed tokens: 786432000 | elapsed time per iteration (s): 0.17 | learning rate: 9.061E-05 | global batch size: 256 | lm loss: 5.762782E+00 | grad norm: 0.956 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1517.834 | TFLOPs: 23.80 | 7: iteration 1600/ 331103 | consumed samples: 409600 | consumed tokens: 838860800 | elapsed time per iteration (s): 0.17 | learning rate: 9.665E-05 | global batch size: 256 | lm loss: 5.680977E+00 | grad norm: 1.348 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1541.061 | TFLOPs: 24.17 | 7: iteration 1700/ 331103 | consumed samples: 435200 | consumed tokens: 891289600 | elapsed time per iteration (s): 0.17 | learning rate: 1.027E-04 | global batch size: 256 | lm loss: 5.602249E+00 | grad norm: 0.960 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1526.796 | TFLOPs: 23.94 | 7: iteration 1800/ 331103 | consumed samples: 460800 | consumed tokens: 943718400 | elapsed time per iteration (s): 0.17 | learning rate: 1.087E-04 | global batch size: 256 | lm loss: 5.525040E+00 | grad norm: 1.066 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1539.249 | TFLOPs: 24.14 | 7: iteration 1900/ 331103 | consumed samples: 486400 | consumed tokens: 996147200 | elapsed time per iteration (s): 0.17 | learning rate: 1.148E-04 | global batch size: 256 | lm loss: 5.448262E+00 | grad norm: 1.002 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1525.196 | TFLOPs: 23.92 | 0: [2023-03-17 10:05:55,513] [INFO] [logging.py:68:log_dist] [Rank 0] step=2000, skipped=0, lr=[0.00012080814039227253, 0.00012080814039227253, 0.00012080814039227253], mom=[(0.9, 0.999), (0.9, 0.999), (0.9, 0.999)] 7: iteration 2000/ 331103 | consumed samples: 512000 | consumed tokens: 1048576000 | elapsed time per iteration (s): 0.17 | learning rate: 1.208E-04 | global batch size: 256 | lm loss: 5.381441E+00 | grad norm: 0.915 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1537.807 | TFLOPs: 24.12 | 0: steps: 2000 loss: 5.3339 iter time (s): 0.173 samples/sec: 1483.594 7: iteration 2100/ 331103 | consumed samples: 537600 | consumed tokens: 1101004800 | elapsed time per iteration (s): 0.17 | learning rate: 1.268E-04 | global batch size: 256 | lm loss: 5.310210E+00 | grad norm: 1.018 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1516.302 | TFLOPs: 23.78 | 7: iteration 2200/ 331103 | consumed samples: 563200 | consumed tokens: 1153433600 | elapsed time per iteration (s): 0.17 | learning rate: 1.329E-04 | global batch size: 256 | lm loss: 5.239399E+00 | grad norm: 1.176 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1531.305 | TFLOPs: 24.01 | 7: iteration 2300/ 331103 | consumed samples: 588800 | consumed tokens: 1205862400 | elapsed time per iteration (s): 0.17 | learning rate: 1.389E-04 | global batch size: 256 | lm loss: 5.172229E+00 | grad norm: 1.091 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1548.588 | TFLOPs: 24.29 | 7: iteration 2400/ 331103 | consumed samples: 614400 | consumed tokens: 1258291200 | elapsed time per iteration (s): 0.17 | learning rate: 1.450E-04 | global batch size: 256 | lm loss: 5.100956E+00 | grad norm: 1.034 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1515.605 | TFLOPs: 23.77 | 7: iteration 2500/ 331103 | consumed samples: 640000 | consumed tokens: 1310720000 | elapsed time per iteration (s): 0.17 | learning rate: 1.510E-04 | global batch size: 256 | lm loss: 5.036000E+00 | grad norm: 0.992 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1519.036 | TFLOPs: 23.82 | 7: iteration 2600/ 331103 | consumed samples: 665600 | consumed tokens: 1363148800 | elapsed time per iteration (s): 0.16 | learning rate: 1.571E-04 | global batch size: 256 | lm loss: 4.971732E+00 | grad norm: 0.911 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1554.909 | TFLOPs: 24.38 | 7: iteration 2700/ 331103 | consumed samples: 691200 | consumed tokens: 1415577600 | elapsed time per iteration (s): 0.17 | learning rate: 1.631E-04 | global batch size: 256 | lm loss: 4.916053E+00 | grad norm: 0.989 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1536.022 | TFLOPs: 24.09 | 7: iteration 2800/ 331103 | consumed samples: 716800 | consumed tokens: 1468006400 | elapsed time per iteration (s): 0.17 | learning rate: 1.691E-04 | global batch size: 256 | lm loss: 4.867538E+00 | grad norm: 0.936 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1539.001 | TFLOPs: 24.14 | 7: iteration 2900/ 331103 | consumed samples: 742400 | consumed tokens: 1520435200 | elapsed time per iteration (s): 0.16 | learning rate: 1.752E-04 | global batch size: 256 | lm loss: 4.817440E+00 | grad norm: 0.944 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1573.086 | TFLOPs: 24.67 | 7: iteration 3000/ 331103 | consumed samples: 768000 | consumed tokens: 1572864000 | elapsed time per iteration (s): 0.16 | learning rate: 1.812E-04 | global batch size: 256 | lm loss: 4.782984E+00 | grad norm: 0.802 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1556.793 | TFLOPs: 24.41 | 7: iteration 3100/ 331103 | consumed samples: 793600 | consumed tokens: 1625292800 | elapsed time per iteration (s): 0.16 | learning rate: 1.873E-04 | global batch size: 256 | lm loss: 4.747279E+00 | grad norm: 0.879 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1556.516 | TFLOPs: 24.41 | 7: iteration 3200/ 331103 | consumed samples: 819200 | consumed tokens: 1677721600 | elapsed time per iteration (s): 0.17 | learning rate: 1.933E-04 | global batch size: 256 | lm loss: 4.716347E+00 | grad norm: 0.709 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1532.314 | TFLOPs: 24.03 | 7: iteration 3300/ 331103 | consumed samples: 844800 | consumed tokens: 1730150400 | elapsed time per iteration (s): 0.16 | learning rate: 1.993E-04 | global batch size: 256 | lm loss: 4.684030E+00 | grad norm: 0.890 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1559.301 | TFLOPs: 24.45 | 7: iteration 3400/ 331103 | consumed samples: 870400 | consumed tokens: 1782579200 | elapsed time per iteration (s): 0.16 | learning rate: 2.000E-04 | global batch size: 256 | lm loss: 4.657620E+00 | grad norm: 0.904 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1567.669 | TFLOPs: 24.58 | 7: iteration 3500/ 331103 | consumed samples: 896000 | consumed tokens: 1835008000 | elapsed time per iteration (s): 0.17 | learning rate: 2.000E-04 | global batch size: 256 | lm loss: 4.634519E+00 | grad norm: 0.807 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1531.931 | TFLOPs: 24.02 | 7: iteration 3600/ 331103 | consumed samples: 921600 | consumed tokens: 1887436800 | elapsed time per iteration (s): 0.16 | learning rate: 2.000E-04 | global batch size: 256 | lm loss: 4.625668E+00 | grad norm: 1.110 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1553.826 | TFLOPs: 24.37 | 7: iteration 3700/ 331103 | consumed samples: 947200 | consumed tokens: 1939865600 | elapsed time per iteration (s): 0.16 | learning rate: 2.000E-04 | global batch size: 256 | lm loss: 4.603105E+00 | grad norm: 0.913 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1560.895 | TFLOPs: 24.48 | 7: iteration 3800/ 331103 | consumed samples: 972800 | consumed tokens: 1992294400 | elapsed time per iteration (s): 0.16 | learning rate: 2.000E-04 | global batch size: 256 | lm loss: 4.575419E+00 | grad norm: 0.681 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1580.670 | TFLOPs: 24.79 | 7: iteration 3900/ 331103 | consumed samples: 998400 | consumed tokens: 2044723200 | elapsed time per iteration (s): 0.16 | learning rate: 2.000E-04 | global batch size: 256 | lm loss: 4.541612E+00 | grad norm: 0.829 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1573.183 | TFLOPs: 24.67 | 0: [2023-03-17 10:11:26,295] [INFO] [logging.py:68:log_dist] [Rank 0] step=4000, skipped=0, lr=[0.00019999803796692803, 0.00019999803796692803, 0.00019999803796692803], mom=[(0.9, 0.999), (0.9, 0.999), (0.9, 0.999)] 7: iteration 4000/ 331103 | consumed samples: 1024000 | consumed tokens: 2097152000 | elapsed time per iteration (s): 0.16 | learning rate: 2.000E-04 | global batch size: 256 | lm loss: 4.521265E+00 | grad norm: 0.823 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1554.723 | TFLOPs: 24.38 | 0: steps: 4000 loss: 4.5104 iter time (s): 0.164 samples/sec: 1560.561 7: iteration 4100/ 331103 | consumed samples: 1049600 | consumed tokens: 2149580800 | elapsed time per iteration (s): 0.16 | learning rate: 2.000E-04 | global batch size: 256 | lm loss: 4.502263E+00 | grad norm: 0.737 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1562.968 | TFLOPs: 24.51 | 7: iteration 4200/ 331103 | consumed samples: 1075200 | consumed tokens: 2202009600 | elapsed time per iteration (s): 0.16 | learning rate: 2.000E-04 | global batch size: 256 | lm loss: 4.484799E+00 | grad norm: 0.770 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1559.422 | TFLOPs: 24.46 | 7: iteration 4300/ 331103 | consumed samples: 1100800 | consumed tokens: 2254438400 | elapsed time per iteration (s): 0.17 | learning rate: 2.000E-04 | global batch size: 256 | lm loss: 4.466600E+00 | grad norm: 0.857 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1541.102 | TFLOPs: 24.17 | 7: iteration 4400/ 331103 | consumed samples: 1126400 | consumed tokens: 2306867200 | elapsed time per iteration (s): 0.17 | learning rate: 2.000E-04 | global batch size: 256 | lm loss: 4.451163E+00 | grad norm: 0.820 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1547.903 | TFLOPs: 24.27 | 7: iteration 4500/ 331103 | consumed samples: 1152000 | consumed tokens: 2359296000 | elapsed time per iteration (s): 0.17 | learning rate: 2.000E-04 | global batch size: 256 | lm loss: 4.439550E+00 | grad norm: 0.588 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1529.702 | TFLOPs: 23.99 | 7: iteration 4600/ 331103 | consumed samples: 1177600 | consumed tokens: 2411724800 | elapsed time per iteration (s): 0.17 | learning rate: 2.000E-04 | global batch size: 256 | lm loss: 4.421628E+00 | grad norm: 0.665 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1544.930 | TFLOPs: 24.23 | 7: iteration 4700/ 331103 | consumed samples: 1203200 | consumed tokens: 2464153600 | elapsed time per iteration (s): 0.17 | learning rate: 2.000E-04 | global batch size: 256 | lm loss: 4.407205E+00 | grad norm: 0.492 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1534.246 | TFLOPs: 24.06 | 7: iteration 4800/ 331103 | consumed samples: 1228800 | consumed tokens: 2516582400 | elapsed time per iteration (s): 0.16 | learning rate: 2.000E-04 | global batch size: 256 | lm loss: 4.392427E+00 | grad norm: 0.603 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1569.712 | TFLOPs: 24.62 | 7: iteration 4900/ 331103 | consumed samples: 1254400 | consumed tokens: 2569011200 | elapsed time per iteration (s): 0.16 | learning rate: 2.000E-04 | global batch size: 256 | lm loss: 4.378620E+00 | grad norm: 0.512 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1563.058 | TFLOPs: 24.51 | 7: iteration 5000/ 331103 | consumed samples: 1280000 | consumed tokens: 2621440000 | elapsed time per iteration (s): 0.17 | learning rate: 2.000E-04 | global batch size: 256 | lm loss: 4.360988E+00 | grad norm: 0.557 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1538.503 | TFLOPs: 24.13 | 7: iteration 5100/ 331103 | consumed samples: 1305600 | consumed tokens: 2673868800 | elapsed time per iteration (s): 0.17 | learning rate: 2.000E-04 | global batch size: 256 | lm loss: 4.355483E+00 | grad norm: 0.588 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1551.316 | TFLOPs: 24.33 | 7: iteration 5200/ 331103 | consumed samples: 1331200 | consumed tokens: 2726297600 | elapsed time per iteration (s): 0.17 | learning rate: 2.000E-04 | global batch size: 256 | lm loss: 4.342585E+00 | grad norm: 0.630 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1544.585 | TFLOPs: 24.22 | 7: iteration 5300/ 331103 | consumed samples: 1356800 | consumed tokens: 2778726400 | elapsed time per iteration (s): 0.16 | learning rate: 2.000E-04 | global batch size: 256 | lm loss: 4.330673E+00 | grad norm: 0.569 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1558.353 | TFLOPs: 24.44 | 7: iteration 5400/ 331103 | consumed samples: 1382400 | consumed tokens: 2831155200 | elapsed time per iteration (s): 0.16 | learning rate: 2.000E-04 | global batch size: 256 | lm loss: 4.318574E+00 | grad norm: 0.569 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1555.314 | TFLOPs: 24.39 | 7: iteration 5500/ 331103 | consumed samples: 1408000 | consumed tokens: 2883584000 | elapsed time per iteration (s): 0.16 | learning rate: 2.000E-04 | global batch size: 256 | lm loss: 4.314501E+00 | grad norm: 0.547 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1582.330 | TFLOPs: 24.81 | 7: iteration 5600/ 331103 | consumed samples: 1433600 | consumed tokens: 2936012800 | elapsed time per iteration (s): 0.16 | learning rate: 2.000E-04 | global batch size: 256 | lm loss: 4.302701E+00 | grad norm: 0.585 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1572.826 | TFLOPs: 24.67 | 7: iteration 5700/ 331103 | consumed samples: 1459200 | consumed tokens: 2988441600 | elapsed time per iteration (s): 0.16 | learning rate: 2.000E-04 | global batch size: 256 | lm loss: 4.294689E+00 | grad norm: 0.513 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1572.574 | TFLOPs: 24.66 | 7: iteration 5800/ 331103 | consumed samples: 1484800 | consumed tokens: 3040870400 | elapsed time per iteration (s): 0.16 | learning rate: 2.000E-04 | global batch size: 256 | lm loss: 4.287299E+00 | grad norm: 0.581 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1563.883 | TFLOPs: 24.53 | 7: iteration 5900/ 331103 | consumed samples: 1510400 | consumed tokens: 3093299200 | elapsed time per iteration (s): 0.16 | learning rate: 2.000E-04 | global batch size: 256 | lm loss: 4.277832E+00 | grad norm: 0.583 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1579.434 | TFLOPs: 24.77 | 0: [2023-03-17 10:16:55,191] [INFO] [logging.py:68:log_dist] [Rank 0] step=6000, skipped=0, lr=[0.0001999701145368867, 0.0001999701145368867, 0.0001999701145368867], mom=[(0.9, 0.999), (0.9, 0.999), (0.9, 0.999)] 7: iteration 6000/ 331103 | consumed samples: 1536000 | consumed tokens: 3145728000 | elapsed time per iteration (s): 0.16 | learning rate: 2.000E-04 | global batch size: 256 | lm loss: 4.266526E+00 | grad norm: 0.553 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1561.997 | TFLOPs: 24.50 | 0: steps: 6000 loss: 4.2436 iter time (s): 0.163 samples/sec: 1571.183 7: iteration 6100/ 331103 | consumed samples: 1561600 | consumed tokens: 3198156800 | elapsed time per iteration (s): 0.16 | learning rate: 2.000E-04 | global batch size: 256 | lm loss: 4.258362E+00 | grad norm: 0.549 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1572.860 | TFLOPs: 24.67 | 7: iteration 6200/ 331103 | consumed samples: 1587200 | consumed tokens: 3250585600 | elapsed time per iteration (s): 0.17 | learning rate: 2.000E-04 | global batch size: 256 | lm loss: 4.248731E+00 | grad norm: 0.500 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1550.740 | TFLOPs: 24.32 | 7: iteration 6300/ 331103 | consumed samples: 1612800 | consumed tokens: 3303014400 | elapsed time per iteration (s): 0.16 | learning rate: 2.000E-04 | global batch size: 256 | lm loss: 4.241027E+00 | grad norm: 0.670 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1559.625 | TFLOPs: 24.46 | 7: iteration 6400/ 331103 | consumed samples: 1638400 | consumed tokens: 3355443200 | elapsed time per iteration (s): 0.16 | learning rate: 2.000E-04 | global batch size: 256 | lm loss: 4.233077E+00 | grad norm: 0.564 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1582.016 | TFLOPs: 24.81 | 7: iteration 6500/ 331103 | consumed samples: 1664000 | consumed tokens: 3407872000 | elapsed time per iteration (s): 0.16 | learning rate: 2.000E-04 | global batch size: 256 | lm loss: 4.225160E+00 | grad norm: 0.563 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1589.083 | TFLOPs: 24.92 | 7: iteration 6600/ 331103 | consumed samples: 1689600 | consumed tokens: 3460300800 | elapsed time per iteration (s): 0.16 | learning rate: 2.000E-04 | global batch size: 256 | lm loss: 4.223409E+00 | grad norm: 0.536 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1588.484 | TFLOPs: 24.91 | 7: iteration 6700/ 331103 | consumed samples: 1715200 | consumed tokens: 3512729600 | elapsed time per iteration (s): 0.16 | learning rate: 2.000E-04 | global batch size: 256 | lm loss: 4.216784E+00 | grad norm: 0.474 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1584.974 | TFLOPs: 24.86 | 7: iteration 6800/ 331103 | consumed samples: 1740800 | consumed tokens: 3565158400 | elapsed time per iteration (s): 0.16 | learning rate: 1.999E-04 | global batch size: 256 | lm loss: 4.210370E+00 | grad norm: 0.518 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1582.850 | TFLOPs: 24.82 | 7: iteration 6900/ 331103 | consumed samples: 1766400 | consumed tokens: 3617587200 | elapsed time per iteration (s): 0.16 | learning rate: 1.999E-04 | global batch size: 256 | lm loss: 4.199586E+00 | grad norm: 0.474 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1562.876 | TFLOPs: 24.51 | 7: iteration 7000/ 331103 | consumed samples: 1792000 | consumed tokens: 3670016000 | elapsed time per iteration (s): 0.16 | learning rate: 1.999E-04 | global batch size: 256 | lm loss: 4.195209E+00 | grad norm: 0.619 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1577.414 | TFLOPs: 24.74 | 7: iteration 7100/ 331103 | consumed samples: 1817600 | consumed tokens: 3722444800 | elapsed time per iteration (s): 0.16 | learning rate: 1.999E-04 | global batch size: 256 | lm loss: 4.188061E+00 | grad norm: 0.576 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1572.423 | TFLOPs: 24.66 | 7: iteration 7200/ 331103 | consumed samples: 1843200 | consumed tokens: 3774873600 | elapsed time per iteration (s): 0.16 | learning rate: 1.999E-04 | global batch size: 256 | lm loss: 4.180782E+00 | grad norm: 0.493 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1553.461 | TFLOPs: 24.36 | 7: iteration 7300/ 331103 | consumed samples: 1868800 | consumed tokens: 3827302400 | elapsed time per iteration (s): 0.16 | learning rate: 1.999E-04 | global batch size: 256 | lm loss: 4.172279E+00 | grad norm: 0.571 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1581.566 | TFLOPs: 24.80 | 7: iteration 7400/ 331103 | consumed samples: 1894400 | consumed tokens: 3879731200 | elapsed time per iteration (s): 0.16 | learning rate: 1.999E-04 | global batch size: 256 | lm loss: 4.170287E+00 | grad norm: 0.472 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1595.145 | TFLOPs: 25.02 | 7: iteration 7500/ 331103 | consumed samples: 1920000 | consumed tokens: 3932160000 | elapsed time per iteration (s): 0.17 | learning rate: 1.999E-04 | global batch size: 256 | lm loss: 4.165220E+00 | grad norm: 0.558 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1548.899 | TFLOPs: 24.29 | 7: iteration 7600/ 331103 | consumed samples: 1945600 | consumed tokens: 3984588800 | elapsed time per iteration (s): 0.16 | learning rate: 1.999E-04 | global batch size: 256 | lm loss: 4.160586E+00 | grad norm: 0.487 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1570.944 | TFLOPs: 24.64 | 7: iteration 7700/ 331103 | consumed samples: 1971200 | consumed tokens: 4037017600 | elapsed time per iteration (s): 0.16 | learning rate: 1.999E-04 | global batch size: 256 | lm loss: 4.152704E+00 | grad norm: 0.670 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1573.741 | TFLOPs: 24.68 | 7: iteration 7800/ 331103 | consumed samples: 1996800 | consumed tokens: 4089446400 | elapsed time per iteration (s): 0.16 | learning rate: 1.999E-04 | global batch size: 256 | lm loss: 4.146418E+00 | grad norm: 0.429 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1584.875 | TFLOPs: 24.85 | 7: iteration 7900/ 331103 | consumed samples: 2022400 | consumed tokens: 4141875200 | elapsed time per iteration (s): 0.16 | learning rate: 1.999E-04 | global batch size: 256 | lm loss: 4.146109E+00 | grad norm: 0.446 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1586.648 | TFLOPs: 24.88 | 0: [2023-03-17 10:22:20,402] [INFO] [logging.py:68:log_dist] [Rank 0] step=8000, skipped=0, lr=[0.000199909135416451, 0.000199909135416451, 0.000199909135416451], mom=[(0.9, 0.999), (0.9, 0.999), (0.9, 0.999)] 7: iteration 8000/ 331103 | consumed samples: 2048000 | consumed tokens: 4194304000 | elapsed time per iteration (s): 0.16 | learning rate: 1.999E-04 | global batch size: 256 | lm loss: 4.141363E+00 | grad norm: 0.617 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1574.016 | TFLOPs: 24.68 | 0: steps: 8000 loss: 4.1641 iter time (s): 0.161 samples/sec: 1589.490 7: iteration 8100/ 331103 | consumed samples: 2073600 | consumed tokens: 4246732800 | elapsed time per iteration (s): 0.16 | learning rate: 1.999E-04 | global batch size: 256 | lm loss: 4.135509E+00 | grad norm: 0.432 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1562.484 | TFLOPs: 24.50 | 7: iteration 8200/ 331103 | consumed samples: 2099200 | consumed tokens: 4299161600 | elapsed time per iteration (s): 0.16 | learning rate: 1.999E-04 | global batch size: 256 | lm loss: 4.130533E+00 | grad norm: 0.441 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1560.611 | TFLOPs: 24.47 | 7: iteration 8300/ 331103 | consumed samples: 2124800 | consumed tokens: 4351590400 | elapsed time per iteration (s): 0.16 | learning rate: 1.999E-04 | global batch size: 256 | lm loss: 4.125920E+00 | grad norm: 0.508 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1586.059 | TFLOPs: 24.87 | 7: iteration 8400/ 331103 | consumed samples: 2150400 | consumed tokens: 4404019200 | elapsed time per iteration (s): 0.16 | learning rate: 1.999E-04 | global batch size: 256 | lm loss: 4.120887E+00 | grad norm: 0.636 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1579.102 | TFLOPs: 24.76 | 7: iteration 8500/ 331103 | consumed samples: 2176000 | consumed tokens: 4456448000 | elapsed time per iteration (s): 0.16 | learning rate: 1.999E-04 | global batch size: 256 | lm loss: 4.115660E+00 | grad norm: 0.532 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1561.316 | TFLOPs: 24.49 | 7: iteration 8600/ 331103 | consumed samples: 2201600 | consumed tokens: 4508876800 | elapsed time per iteration (s): 0.16 | learning rate: 1.999E-04 | global batch size: 256 | lm loss: 4.112626E+00 | grad norm: 0.458 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1586.291 | TFLOPs: 24.88 | 7: iteration 8700/ 331103 | consumed samples: 2227200 | consumed tokens: 4561305600 | elapsed time per iteration (s): 0.16 | learning rate: 1.999E-04 | global batch size: 256 | lm loss: 4.103914E+00 | grad norm: 0.452 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1582.498 | TFLOPs: 24.82 | 7: iteration 8800/ 331103 | consumed samples: 2252800 | consumed tokens: 4613734400 | elapsed time per iteration (s): 0.16 | learning rate: 1.999E-04 | global batch size: 256 | lm loss: 4.103002E+00 | grad norm: 0.435 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1577.108 | TFLOPs: 24.73 | 7: iteration 8900/ 331103 | consumed samples: 2278400 | consumed tokens: 4666163200 | elapsed time per iteration (s): 0.16 | learning rate: 1.999E-04 | global batch size: 256 | lm loss: 4.095188E+00 | grad norm: 0.412 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1586.138 | TFLOPs: 24.87 | 7: iteration 9000/ 331103 | consumed samples: 2304000 | consumed tokens: 4718592000 | elapsed time per iteration (s): 0.16 | learning rate: 1.999E-04 | global batch size: 256 | lm loss: 4.095626E+00 | grad norm: 0.470 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1570.036 | TFLOPs: 24.62 | 7: iteration 9100/ 331103 | consumed samples: 2329600 | consumed tokens: 4771020800 | elapsed time per iteration (s): 0.16 | learning rate: 1.999E-04 | global batch size: 256 | lm loss: 4.084980E+00 | grad norm: 0.355 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1582.346 | TFLOPs: 24.82 | 7: iteration 9200/ 331103 | consumed samples: 2355200 | consumed tokens: 4823449600 | elapsed time per iteration (s): 0.16 | learning rate: 1.999E-04 | global batch size: 256 | lm loss: 4.084003E+00 | grad norm: 0.467 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1587.445 | TFLOPs: 24.90 | 7: iteration 9300/ 331103 | consumed samples: 2380800 | consumed tokens: 4875878400 | elapsed time per iteration (s): 0.16 | learning rate: 1.999E-04 | global batch size: 256 | lm loss: 4.085509E+00 | grad norm: 0.453 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1585.183 | TFLOPs: 24.86 | 7: iteration 9400/ 331103 | consumed samples: 2406400 | consumed tokens: 4928307200 | elapsed time per iteration (s): 0.16 | learning rate: 1.998E-04 | global batch size: 256 | lm loss: 4.075407E+00 | grad norm: 0.457 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1601.725 | TFLOPs: 25.12 | 7: iteration 9500/ 331103 | consumed samples: 2432000 | consumed tokens: 4980736000 | elapsed time per iteration (s): 0.16 | learning rate: 1.998E-04 | global batch size: 256 | lm loss: 4.077479E+00 | grad norm: 0.502 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1601.945 | TFLOPs: 25.12 | 7: iteration 9600/ 331103 | consumed samples: 2457600 | consumed tokens: 5033164800 | elapsed time per iteration (s): 0.16 | learning rate: 1.998E-04 | global batch size: 256 | lm loss: 4.074067E+00 | grad norm: 0.389 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1606.417 | TFLOPs: 25.19 | 7: iteration 9700/ 331103 | consumed samples: 2483200 | consumed tokens: 5085593600 | elapsed time per iteration (s): 0.16 | learning rate: 1.998E-04 | global batch size: 256 | lm loss: 4.065799E+00 | grad norm: 0.511 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1607.358 | TFLOPs: 25.21 | 7: iteration 9800/ 331103 | consumed samples: 2508800 | consumed tokens: 5138022400 | elapsed time per iteration (s): 0.16 | learning rate: 1.998E-04 | global batch size: 256 | lm loss: 4.065964E+00 | grad norm: 0.404 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1607.235 | TFLOPs: 25.21 | 7: iteration 9900/ 331103 | consumed samples: 2534400 | consumed tokens: 5190451200 | elapsed time per iteration (s): 0.16 | learning rate: 1.998E-04 | global batch size: 256 | lm loss: 4.064531E+00 | grad norm: 0.386 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1577.445 | TFLOPs: 24.74 | 0: [2023-03-17 10:27:43,322] [INFO] [logging.py:68:log_dist] [Rank 0] step=10000, skipped=0, lr=[0.00019981512300980405, 0.00019981512300980405, 0.00019981512300980405], mom=[(0.9, 0.999), (0.9, 0.999), (0.9, 0.999)] 7: iteration 10000/ 331103 | consumed samples: 2560000 | consumed tokens: 5242880000 | elapsed time per iteration (s): 0.16 | learning rate: 1.998E-04 | global batch size: 256 | lm loss: 4.055241E+00 | grad norm: 0.457 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1604.729 | TFLOPs: 25.17 | 0: steps: 10000 loss: 4.0274 iter time (s): 0.160 samples/sec: 1600.510 7: ------------------------------------------------------------------------------------------------ 7: validation loss at iteration 10000 | lm loss value: 4.119843E+00 | lm loss PPL: 6.154958E+01 | 7: ------------------------------------------------------------------------------------------------ 0: saving checkpoint at iteration 10000 to checkpoints_44m174b100m 0: [2023-03-17 10:27:43,394] [INFO] [logging.py:68:log_dist] [Rank 0] [Torch] Checkpoint global_step10000 is begin to save! 0: [2023-03-17 10:27:43,595] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step10000/layer_01-model_00-model_states.pt... 0: [2023-03-17 10:27:43,658] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step10000/layer_01-model_00-model_states.pt. 0: [2023-03-17 10:27:43,659] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step10000/layer_03-model_00-model_states.pt... 0: [2023-03-17 10:27:43,669] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step10000/layer_03-model_00-model_states.pt. 0: [2023-03-17 10:27:43,669] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step10000/layer_04-model_00-model_states.pt... 0: [2023-03-17 10:27:43,677] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step10000/layer_04-model_00-model_states.pt. 0: [2023-03-17 10:27:43,677] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step10000/layer_05-model_00-model_states.pt... 0: [2023-03-17 10:27:43,685] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step10000/layer_05-model_00-model_states.pt. 0: [2023-03-17 10:27:43,686] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step10000/layer_06-model_00-model_states.pt... 0: [2023-03-17 10:27:43,694] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step10000/layer_06-model_00-model_states.pt. 0: [2023-03-17 10:27:43,694] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step10000/layer_07-model_00-model_states.pt... 0: [2023-03-17 10:27:43,702] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step10000/layer_07-model_00-model_states.pt. 0: [2023-03-17 10:27:43,702] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step10000/layer_08-model_00-model_states.pt... 0: [2023-03-17 10:27:43,710] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step10000/layer_08-model_00-model_states.pt. 0: [2023-03-17 10:27:43,710] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step10000/layer_09-model_00-model_states.pt... 0: [2023-03-17 10:27:43,718] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step10000/layer_09-model_00-model_states.pt. 0: [2023-03-17 10:27:43,718] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step10000/layer_10-model_00-model_states.pt... 0: [2023-03-17 10:27:43,726] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step10000/layer_10-model_00-model_states.pt. 0: [2023-03-17 10:27:43,726] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step10000/layer_12-model_00-model_states.pt... 0: [2023-03-17 10:27:43,727] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step10000/layer_12-model_00-model_states.pt. 0: [2023-03-17 10:27:43,728] [INFO] [logging.py:68:log_dist] [Rank 0] Saving model checkpoint: checkpoints_44m174b100m/global_step10000/mp_rank_00_model_states.pt 0: [2023-03-17 10:27:43,728] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step10000/mp_rank_00_model_states.pt... 0: [2023-03-17 10:27:43,730] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step10000/mp_rank_00_model_states.pt. 0: [2023-03-17 10:27:43,748] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step10000/bf16_zero_pp_rank_0_mp_rank_00_optim_states.pt... 0: [2023-03-17 10:27:43,749] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step10000/bf16_zero_pp_rank_2_mp_rank_00_optim_states.pt... 0: [2023-03-17 10:27:43,749] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step10000/bf16_zero_pp_rank_1_mp_rank_00_optim_states.pt... 0: [2023-03-17 10:27:43,749] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step10000/bf16_zero_pp_rank_7_mp_rank_00_optim_states.pt... 0: [2023-03-17 10:27:43,749] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step10000/bf16_zero_pp_rank_5_mp_rank_00_optim_states.pt... 0: [2023-03-17 10:27:43,749] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step10000/bf16_zero_pp_rank_6_mp_rank_00_optim_states.pt... 0: [2023-03-17 10:27:43,749] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step10000/bf16_zero_pp_rank_3_mp_rank_00_optim_states.pt... 3: [2023-03-17 10:27:43,749] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step10000/bf16_zero_pp_rank_29_mp_rank_00_optim_states.pt... 3: [2023-03-17 10:27:43,749] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step10000/bf16_zero_pp_rank_24_mp_rank_00_optim_states.pt... 3: [2023-03-17 10:27:43,749] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step10000/bf16_zero_pp_rank_30_mp_rank_00_optim_states.pt... 2: [2023-03-17 10:27:43,749] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step10000/bf16_zero_pp_rank_18_mp_rank_00_optim_states.pt... 2: [2023-03-17 10:27:43,749] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step10000/bf16_zero_pp_rank_17_mp_rank_00_optim_states.pt... 2: [2023-03-17 10:27:43,749] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step10000/bf16_zero_pp_rank_23_mp_rank_00_optim_states.pt... 2: [2023-03-17 10:27:43,749] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step10000/bf16_zero_pp_rank_21_mp_rank_00_optim_states.pt... 2: [2023-03-17 10:27:43,749] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step10000/bf16_zero_pp_rank_16_mp_rank_00_optim_states.pt... 4: [2023-03-17 10:27:43,749] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step10000/bf16_zero_pp_rank_36_mp_rank_00_optim_states.pt... 4: [2023-03-17 10:27:43,749] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step10000/bf16_zero_pp_rank_38_mp_rank_00_optim_states.pt... 4: [2023-03-17 10:27:43,749] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step10000/bf16_zero_pp_rank_37_mp_rank_00_optim_states.pt... 6: [2023-03-17 10:27:43,749] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step10000/bf16_zero_pp_rank_51_mp_rank_00_optim_states.pt... 6: [2023-03-17 10:27:43,749] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step10000/bf16_zero_pp_rank_48_mp_rank_00_optim_states.pt... 7: [2023-03-17 10:27:43,749] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step10000/bf16_zero_pp_rank_58_mp_rank_00_optim_states.pt... 7: [2023-03-17 10:27:43,749] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step10000/bf16_zero_pp_rank_57_mp_rank_00_optim_states.pt... 7: [2023-03-17 10:27:43,749] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step10000/bf16_zero_pp_rank_56_mp_rank_00_optim_states.pt... 5: [2023-03-17 10:27:43,749] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step10000/bf16_zero_pp_rank_45_mp_rank_00_optim_states.pt... 5: [2023-03-17 10:27:43,749] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step10000/bf16_zero_pp_rank_44_mp_rank_00_optim_states.pt... 5: [2023-03-17 10:27:43,749] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step10000/bf16_zero_pp_rank_41_mp_rank_00_optim_states.pt... 5: [2023-03-17 10:27:43,749] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step10000/bf16_zero_pp_rank_40_mp_rank_00_optim_states.pt... 5: [2023-03-17 10:27:43,749] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step10000/bf16_zero_pp_rank_43_mp_rank_00_optim_states.pt... 0: [2023-03-17 10:27:43,749] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step10000/bf16_zero_pp_rank_4_mp_rank_00_optim_states.pt... 3: [2023-03-17 10:27:43,749] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step10000/bf16_zero_pp_rank_26_mp_rank_00_optim_states.pt... 3: [2023-03-17 10:27:43,749] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step10000/bf16_zero_pp_rank_25_mp_rank_00_optim_states.pt... 3: [2023-03-17 10:27:43,749] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step10000/bf16_zero_pp_rank_27_mp_rank_00_optim_states.pt... 3: [2023-03-17 10:27:43,749] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step10000/bf16_zero_pp_rank_28_mp_rank_00_optim_states.pt... 3: [2023-03-17 10:27:43,749] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step10000/bf16_zero_pp_rank_31_mp_rank_00_optim_states.pt... 2: [2023-03-17 10:27:43,749] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step10000/bf16_zero_pp_rank_20_mp_rank_00_optim_states.pt... 2: [2023-03-17 10:27:43,749] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step10000/bf16_zero_pp_rank_22_mp_rank_00_optim_states.pt... 4: [2023-03-17 10:27:43,749] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step10000/bf16_zero_pp_rank_33_mp_rank_00_optim_states.pt... 4: [2023-03-17 10:27:43,749] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step10000/bf16_zero_pp_rank_39_mp_rank_00_optim_states.pt... 4: [2023-03-17 10:27:43,749] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step10000/bf16_zero_pp_rank_34_mp_rank_00_optim_states.pt... 4: [2023-03-17 10:27:43,749] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step10000/bf16_zero_pp_rank_32_mp_rank_00_optim_states.pt... 4: [2023-03-17 10:27:43,749] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step10000/bf16_zero_pp_rank_35_mp_rank_00_optim_states.pt... 6: [2023-03-17 10:27:43,749] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step10000/bf16_zero_pp_rank_52_mp_rank_00_optim_states.pt... 6: [2023-03-17 10:27:43,749] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step10000/bf16_zero_pp_rank_50_mp_rank_00_optim_states.pt... 6: [2023-03-17 10:27:43,749] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step10000/bf16_zero_pp_rank_55_mp_rank_00_optim_states.pt... 6: [2023-03-17 10:27:43,749] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step10000/bf16_zero_pp_rank_54_mp_rank_00_optim_states.pt... 6: [2023-03-17 10:27:43,749] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step10000/bf16_zero_pp_rank_49_mp_rank_00_optim_states.pt... 7: [2023-03-17 10:27:43,749] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step10000/bf16_zero_pp_rank_62_mp_rank_00_optim_states.pt... 7: [2023-03-17 10:27:43,749] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step10000/bf16_zero_pp_rank_60_mp_rank_00_optim_states.pt... 7: [2023-03-17 10:27:43,749] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step10000/bf16_zero_pp_rank_59_mp_rank_00_optim_states.pt... 5: [2023-03-17 10:27:43,749] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step10000/bf16_zero_pp_rank_47_mp_rank_00_optim_states.pt... 5: [2023-03-17 10:27:43,749] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step10000/bf16_zero_pp_rank_42_mp_rank_00_optim_states.pt... 1: [2023-03-17 10:27:43,749] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step10000/bf16_zero_pp_rank_12_mp_rank_00_optim_states.pt... 1: [2023-03-17 10:27:43,749] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step10000/bf16_zero_pp_rank_11_mp_rank_00_optim_states.pt... 1: [2023-03-17 10:27:43,749] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step10000/bf16_zero_pp_rank_13_mp_rank_00_optim_states.pt... 1: [2023-03-17 10:27:43,749] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step10000/bf16_zero_pp_rank_10_mp_rank_00_optim_states.pt... 1: [2023-03-17 10:27:43,749] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step10000/bf16_zero_pp_rank_9_mp_rank_00_optim_states.pt... 2: [2023-03-17 10:27:43,749] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step10000/bf16_zero_pp_rank_19_mp_rank_00_optim_states.pt... 6: [2023-03-17 10:27:43,749] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step10000/bf16_zero_pp_rank_53_mp_rank_00_optim_states.pt... 7: [2023-03-17 10:27:43,749] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step10000/bf16_zero_pp_rank_63_mp_rank_00_optim_states.pt... 5: [2023-03-17 10:27:43,749] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step10000/bf16_zero_pp_rank_46_mp_rank_00_optim_states.pt... 1: [2023-03-17 10:27:43,749] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step10000/bf16_zero_pp_rank_15_mp_rank_00_optim_states.pt... 1: [2023-03-17 10:27:43,749] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step10000/bf16_zero_pp_rank_8_mp_rank_00_optim_states.pt... 1: [2023-03-17 10:27:43,749] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step10000/bf16_zero_pp_rank_14_mp_rank_00_optim_states.pt... 7: [2023-03-17 10:27:43,749] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step10000/bf16_zero_pp_rank_61_mp_rank_00_optim_states.pt... 0: [2023-03-17 10:27:43,761] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step10000/bf16_zero_pp_rank_0_mp_rank_00_optim_states.pt. 0: [2023-03-17 10:27:43,762] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step10000/bf16_zero_pp_rank_1_mp_rank_00_optim_states.pt. 0: [2023-03-17 10:27:43,762] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step10000/bf16_zero_pp_rank_1_mp_rank_00_optim_states.pt 0: [2023-03-17 10:27:43,762] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step10000 is ready now! 0: [2023-03-17 10:27:43,763] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step10000/bf16_zero_pp_rank_3_mp_rank_00_optim_states.pt. 0: [2023-03-17 10:27:43,763] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step10000/bf16_zero_pp_rank_4_mp_rank_00_optim_states.pt. 0: [2023-03-17 10:27:43,763] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step10000/bf16_zero_pp_rank_3_mp_rank_00_optim_states.pt 0: [2023-03-17 10:27:43,763] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step10000/bf16_zero_pp_rank_4_mp_rank_00_optim_states.pt 0: [2023-03-17 10:27:43,764] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step10000 is ready now! 0: [2023-03-17 10:27:43,764] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step10000 is ready now! 0: [2023-03-17 10:27:43,764] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step10000/bf16_zero_pp_rank_6_mp_rank_00_optim_states.pt. 0: [2023-03-17 10:27:43,764] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step10000/bf16_zero_pp_rank_6_mp_rank_00_optim_states.pt 0: [2023-03-17 10:27:43,764] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step10000/bf16_zero_pp_rank_0_mp_rank_00_optim_states.pt 0: [2023-03-17 10:27:43,764] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step10000 is ready now! 0: [2023-03-17 10:27:43,764] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step10000 is ready now! 3: [2023-03-17 10:27:43,764] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step10000/bf16_zero_pp_rank_31_mp_rank_00_optim_states.pt. 3: [2023-03-17 10:27:43,764] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step10000/bf16_zero_pp_rank_31_mp_rank_00_optim_states.pt 3: [2023-03-17 10:27:43,764] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step10000 is ready now! 3: [2023-03-17 10:27:43,764] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step10000/bf16_zero_pp_rank_28_mp_rank_00_optim_states.pt. 3: [2023-03-17 10:27:43,765] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step10000/bf16_zero_pp_rank_28_mp_rank_00_optim_states.pt 3: [2023-03-17 10:27:43,765] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step10000 is ready now! 3: [2023-03-17 10:27:43,765] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step10000/bf16_zero_pp_rank_27_mp_rank_00_optim_states.pt. 3: [2023-03-17 10:27:43,765] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step10000/bf16_zero_pp_rank_29_mp_rank_00_optim_states.pt. 3: [2023-03-17 10:27:43,765] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step10000/bf16_zero_pp_rank_27_mp_rank_00_optim_states.pt 3: [2023-03-17 10:27:43,765] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step10000/bf16_zero_pp_rank_26_mp_rank_00_optim_states.pt. 3: [2023-03-17 10:27:43,765] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step10000/bf16_zero_pp_rank_29_mp_rank_00_optim_states.pt 3: [2023-03-17 10:27:43,765] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step10000 is ready now! 3: [2023-03-17 10:27:43,765] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step10000/bf16_zero_pp_rank_26_mp_rank_00_optim_states.pt 3: [2023-03-17 10:27:43,765] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step10000 is ready now! 3: [2023-03-17 10:27:43,765] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step10000 is ready now! 0: [2023-03-17 10:27:43,765] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step10000/bf16_zero_pp_rank_5_mp_rank_00_optim_states.pt. 0: [2023-03-17 10:27:43,765] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step10000/bf16_zero_pp_rank_5_mp_rank_00_optim_states.pt 0: [2023-03-17 10:27:43,765] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step10000 is ready now! 3: [2023-03-17 10:27:43,765] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step10000/bf16_zero_pp_rank_30_mp_rank_00_optim_states.pt. 3: [2023-03-17 10:27:43,766] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step10000/bf16_zero_pp_rank_30_mp_rank_00_optim_states.pt 3: [2023-03-17 10:27:43,766] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step10000 is ready now! 0: [2023-03-17 10:27:43,766] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step10000/bf16_zero_pp_rank_7_mp_rank_00_optim_states.pt. 1: [2023-03-17 10:27:43,766] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step10000/bf16_zero_pp_rank_11_mp_rank_00_optim_states.pt. 0: [2023-03-17 10:27:43,767] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step10000/bf16_zero_pp_rank_7_mp_rank_00_optim_states.pt 1: [2023-03-17 10:27:43,767] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step10000/bf16_zero_pp_rank_11_mp_rank_00_optim_states.pt 0: [2023-03-17 10:27:43,767] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step10000 is ready now! 1: [2023-03-17 10:27:43,767] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step10000/bf16_zero_pp_rank_12_mp_rank_00_optim_states.pt. 1: [2023-03-17 10:27:43,767] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step10000 is ready now! 1: [2023-03-17 10:27:43,767] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step10000/bf16_zero_pp_rank_12_mp_rank_00_optim_states.pt 1: [2023-03-17 10:27:43,767] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step10000 is ready now! 3: [2023-03-17 10:27:43,767] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step10000/bf16_zero_pp_rank_25_mp_rank_00_optim_states.pt. 1: [2023-03-17 10:27:43,767] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step10000/bf16_zero_pp_rank_15_mp_rank_00_optim_states.pt. 3: [2023-03-17 10:27:43,767] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step10000/bf16_zero_pp_rank_25_mp_rank_00_optim_states.pt 1: [2023-03-17 10:27:43,767] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step10000/bf16_zero_pp_rank_15_mp_rank_00_optim_states.pt 3: [2023-03-17 10:27:43,767] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step10000 is ready now! 0: [2023-03-17 10:27:43,767] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step10000/bf16_zero_pp_rank_2_mp_rank_00_optim_states.pt. 2: [2023-03-17 10:27:43,767] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step10000/bf16_zero_pp_rank_16_mp_rank_00_optim_states.pt. 1: [2023-03-17 10:27:43,767] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step10000 is ready now! 0: [2023-03-17 10:27:43,767] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step10000/bf16_zero_pp_rank_2_mp_rank_00_optim_states.pt 2: [2023-03-17 10:27:43,767] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step10000/bf16_zero_pp_rank_16_mp_rank_00_optim_states.pt 4: [2023-03-17 10:27:43,767] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step10000/bf16_zero_pp_rank_36_mp_rank_00_optim_states.pt. 2: [2023-03-17 10:27:43,767] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step10000 is ready now! 1: [2023-03-17 10:27:43,767] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step10000/bf16_zero_pp_rank_14_mp_rank_00_optim_states.pt. 1: [2023-03-17 10:27:43,767] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step10000/bf16_zero_pp_rank_8_mp_rank_00_optim_states.pt. 0: [2023-03-17 10:27:43,767] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step10000 is ready now! 4: [2023-03-17 10:27:43,767] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step10000/bf16_zero_pp_rank_36_mp_rank_00_optim_states.pt 1: [2023-03-17 10:27:43,767] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step10000/bf16_zero_pp_rank_14_mp_rank_00_optim_states.pt 4: [2023-03-17 10:27:43,767] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step10000 is ready now! 1: [2023-03-17 10:27:43,767] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step10000/bf16_zero_pp_rank_8_mp_rank_00_optim_states.pt 1: [2023-03-17 10:27:43,767] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step10000 is ready now! 1: [2023-03-17 10:27:43,767] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step10000 is ready now! 2: [2023-03-17 10:27:43,767] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step10000/bf16_zero_pp_rank_20_mp_rank_00_optim_states.pt. 2: [2023-03-17 10:27:43,767] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step10000/bf16_zero_pp_rank_18_mp_rank_00_optim_states.pt. 7: [2023-03-17 10:27:43,767] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step10000/bf16_zero_pp_rank_63_mp_rank_00_optim_states.pt. 7: [2023-03-17 10:27:43,767] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step10000/bf16_zero_pp_rank_58_mp_rank_00_optim_states.pt. 4: [2023-03-17 10:27:43,767] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step10000/bf16_zero_pp_rank_33_mp_rank_00_optim_states.pt. 2: [2023-03-17 10:27:43,767] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step10000/bf16_zero_pp_rank_18_mp_rank_00_optim_states.pt 2: [2023-03-17 10:27:43,767] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step10000/bf16_zero_pp_rank_20_mp_rank_00_optim_states.pt 7: [2023-03-17 10:27:43,767] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step10000/bf16_zero_pp_rank_60_mp_rank_00_optim_states.pt. 4: [2023-03-17 10:27:43,767] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step10000/bf16_zero_pp_rank_33_mp_rank_00_optim_states.pt 7: [2023-03-17 10:27:43,767] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step10000/bf16_zero_pp_rank_63_mp_rank_00_optim_states.pt 7: [2023-03-17 10:27:43,767] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step10000/bf16_zero_pp_rank_58_mp_rank_00_optim_states.pt 7: [2023-03-17 10:27:43,767] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step10000/bf16_zero_pp_rank_61_mp_rank_00_optim_states.pt. 7: [2023-03-17 10:27:43,767] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step10000/bf16_zero_pp_rank_60_mp_rank_00_optim_states.pt 4: [2023-03-17 10:27:43,767] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step10000 is ready now! 7: [2023-03-17 10:27:43,767] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step10000 is ready now! 7: [2023-03-17 10:27:43,767] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step10000 is ready now! 7: [2023-03-17 10:27:43,767] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step10000 is ready now! 4: [2023-03-17 10:27:43,767] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step10000/bf16_zero_pp_rank_32_mp_rank_00_optim_states.pt. 7: [2023-03-17 10:27:43,767] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step10000/bf16_zero_pp_rank_61_mp_rank_00_optim_states.pt 4: [2023-03-17 10:27:43,767] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step10000/bf16_zero_pp_rank_32_mp_rank_00_optim_states.pt 4: [2023-03-17 10:27:43,767] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step10000 is ready now! 7: [2023-03-17 10:27:43,767] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step10000/bf16_zero_pp_rank_59_mp_rank_00_optim_states.pt. 7: [2023-03-17 10:27:43,767] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step10000 is ready now! 7: [2023-03-17 10:27:43,767] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step10000/bf16_zero_pp_rank_59_mp_rank_00_optim_states.pt 2: [2023-03-17 10:27:43,767] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step10000 is ready now! 2: [2023-03-17 10:27:43,767] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step10000 is ready now! 2: [2023-03-17 10:27:43,767] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step10000/bf16_zero_pp_rank_21_mp_rank_00_optim_states.pt. 2: [2023-03-17 10:27:43,767] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step10000/bf16_zero_pp_rank_21_mp_rank_00_optim_states.pt 2: [2023-03-17 10:27:43,767] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step10000 is ready now! 6: [2023-03-17 10:27:43,767] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step10000/bf16_zero_pp_rank_50_mp_rank_00_optim_states.pt. 7: [2023-03-17 10:27:43,767] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step10000 is ready now! 6: [2023-03-17 10:27:43,767] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step10000/bf16_zero_pp_rank_54_mp_rank_00_optim_states.pt. 6: [2023-03-17 10:27:43,767] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step10000/bf16_zero_pp_rank_55_mp_rank_00_optim_states.pt. 6: [2023-03-17 10:27:43,767] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step10000/bf16_zero_pp_rank_50_mp_rank_00_optim_states.pt 6: [2023-03-17 10:27:43,767] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step10000/bf16_zero_pp_rank_54_mp_rank_00_optim_states.pt 6: [2023-03-17 10:27:43,767] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step10000/bf16_zero_pp_rank_55_mp_rank_00_optim_states.pt 6: [2023-03-17 10:27:43,767] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step10000 is ready now! 6: [2023-03-17 10:27:43,767] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step10000/bf16_zero_pp_rank_51_mp_rank_00_optim_states.pt. 6: [2023-03-17 10:27:43,768] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step10000 is ready now! 6: [2023-03-17 10:27:43,768] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step10000 is ready now! 6: [2023-03-17 10:27:43,767] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step10000/bf16_zero_pp_rank_53_mp_rank_00_optim_states.pt. 6: [2023-03-17 10:27:43,768] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step10000/bf16_zero_pp_rank_51_mp_rank_00_optim_states.pt 6: [2023-03-17 10:27:43,768] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step10000/bf16_zero_pp_rank_53_mp_rank_00_optim_states.pt 6: [2023-03-17 10:27:43,768] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step10000 is ready now! 2: [2023-03-17 10:27:43,768] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step10000/bf16_zero_pp_rank_17_mp_rank_00_optim_states.pt. 6: [2023-03-17 10:27:43,768] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step10000 is ready now! 3: [2023-03-17 10:27:43,768] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step10000/bf16_zero_pp_rank_24_mp_rank_00_optim_states.pt. 2: [2023-03-17 10:27:43,768] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step10000/bf16_zero_pp_rank_17_mp_rank_00_optim_states.pt 2: [2023-03-17 10:27:43,768] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step10000/bf16_zero_pp_rank_22_mp_rank_00_optim_states.pt. 1: [2023-03-17 10:27:43,768] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step10000/bf16_zero_pp_rank_9_mp_rank_00_optim_states.pt. 3: [2023-03-17 10:27:43,768] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step10000/bf16_zero_pp_rank_24_mp_rank_00_optim_states.pt 2: [2023-03-17 10:27:43,768] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step10000 is ready now! 2: [2023-03-17 10:27:43,768] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step10000/bf16_zero_pp_rank_22_mp_rank_00_optim_states.pt 3: [2023-03-17 10:27:43,768] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step10000 is ready now! 2: [2023-03-17 10:27:43,768] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step10000 is ready now! 4: [2023-03-17 10:27:43,768] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step10000/bf16_zero_pp_rank_39_mp_rank_00_optim_states.pt. 6: [2023-03-17 10:27:43,768] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step10000/bf16_zero_pp_rank_48_mp_rank_00_optim_states.pt. 1: [2023-03-17 10:27:43,768] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step10000/bf16_zero_pp_rank_9_mp_rank_00_optim_states.pt 1: [2023-03-17 10:27:43,768] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step10000 is ready now! 2: [2023-03-17 10:27:43,768] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step10000/bf16_zero_pp_rank_19_mp_rank_00_optim_states.pt. 4: [2023-03-17 10:27:43,768] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step10000/bf16_zero_pp_rank_39_mp_rank_00_optim_states.pt 6: [2023-03-17 10:27:43,768] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step10000/bf16_zero_pp_rank_48_mp_rank_00_optim_states.pt 4: [2023-03-17 10:27:43,768] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step10000 is ready now! 2: [2023-03-17 10:27:43,768] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step10000/bf16_zero_pp_rank_19_mp_rank_00_optim_states.pt 6: [2023-03-17 10:27:43,768] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step10000 is ready now! 4: [2023-03-17 10:27:43,768] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step10000/bf16_zero_pp_rank_35_mp_rank_00_optim_states.pt. 2: [2023-03-17 10:27:43,768] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step10000 is ready now! 4: [2023-03-17 10:27:43,768] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step10000/bf16_zero_pp_rank_35_mp_rank_00_optim_states.pt 4: [2023-03-17 10:27:43,768] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step10000/bf16_zero_pp_rank_34_mp_rank_00_optim_states.pt. 4: [2023-03-17 10:27:43,768] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step10000 is ready now! 4: [2023-03-17 10:27:43,768] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step10000/bf16_zero_pp_rank_34_mp_rank_00_optim_states.pt 4: [2023-03-17 10:27:43,768] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step10000 is ready now! 7: [2023-03-17 10:27:43,768] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step10000/bf16_zero_pp_rank_62_mp_rank_00_optim_states.pt. 1: [2023-03-17 10:27:43,768] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step10000/bf16_zero_pp_rank_10_mp_rank_00_optim_states.pt. 7: [2023-03-17 10:27:43,768] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step10000/bf16_zero_pp_rank_62_mp_rank_00_optim_states.pt 7: [2023-03-17 10:27:43,768] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step10000 is ready now! 1: [2023-03-17 10:27:43,768] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step10000/bf16_zero_pp_rank_10_mp_rank_00_optim_states.pt 1: [2023-03-17 10:27:43,768] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step10000 is ready now! 4: [2023-03-17 10:27:43,768] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step10000/bf16_zero_pp_rank_37_mp_rank_00_optim_states.pt. 4: [2023-03-17 10:27:43,768] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step10000/bf16_zero_pp_rank_37_mp_rank_00_optim_states.pt 7: [2023-03-17 10:27:43,768] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step10000/bf16_zero_pp_rank_57_mp_rank_00_optim_states.pt. 4: [2023-03-17 10:27:43,769] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step10000 is ready now! 7: [2023-03-17 10:27:43,769] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step10000/bf16_zero_pp_rank_57_mp_rank_00_optim_states.pt 7: [2023-03-17 10:27:43,769] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step10000 is ready now! 2: [2023-03-17 10:27:43,769] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step10000/bf16_zero_pp_rank_23_mp_rank_00_optim_states.pt. 2: [2023-03-17 10:27:43,769] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step10000/bf16_zero_pp_rank_23_mp_rank_00_optim_states.pt 2: [2023-03-17 10:27:43,769] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step10000 is ready now! 1: [2023-03-17 10:27:43,769] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step10000/bf16_zero_pp_rank_13_mp_rank_00_optim_states.pt. 6: [2023-03-17 10:27:43,769] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step10000/bf16_zero_pp_rank_52_mp_rank_00_optim_states.pt. 1: [2023-03-17 10:27:43,769] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step10000/bf16_zero_pp_rank_13_mp_rank_00_optim_states.pt 1: [2023-03-17 10:27:43,769] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step10000 is ready now! 4: [2023-03-17 10:27:43,769] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step10000/bf16_zero_pp_rank_38_mp_rank_00_optim_states.pt. 6: [2023-03-17 10:27:43,769] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step10000/bf16_zero_pp_rank_52_mp_rank_00_optim_states.pt 4: [2023-03-17 10:27:43,769] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step10000/bf16_zero_pp_rank_38_mp_rank_00_optim_states.pt 6: [2023-03-17 10:27:43,769] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step10000 is ready now! 4: [2023-03-17 10:27:43,769] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step10000 is ready now! 6: [2023-03-17 10:27:43,769] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step10000/bf16_zero_pp_rank_49_mp_rank_00_optim_states.pt. 6: [2023-03-17 10:27:43,769] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step10000/bf16_zero_pp_rank_49_mp_rank_00_optim_states.pt 7: [2023-03-17 10:27:43,769] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step10000/bf16_zero_pp_rank_56_mp_rank_00_optim_states.pt. 6: [2023-03-17 10:27:43,769] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step10000 is ready now! 7: [2023-03-17 10:27:43,769] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step10000/bf16_zero_pp_rank_56_mp_rank_00_optim_states.pt 7: [2023-03-17 10:27:43,769] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step10000 is ready now! 5: [2023-03-17 10:27:43,791] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step10000/bf16_zero_pp_rank_44_mp_rank_00_optim_states.pt. 5: [2023-03-17 10:27:43,791] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step10000/bf16_zero_pp_rank_46_mp_rank_00_optim_states.pt. 5: [2023-03-17 10:27:43,791] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step10000/bf16_zero_pp_rank_41_mp_rank_00_optim_states.pt. 5: [2023-03-17 10:27:43,792] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step10000/bf16_zero_pp_rank_44_mp_rank_00_optim_states.pt 5: [2023-03-17 10:27:43,792] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step10000/bf16_zero_pp_rank_46_mp_rank_00_optim_states.pt 5: [2023-03-17 10:27:43,791] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step10000/bf16_zero_pp_rank_47_mp_rank_00_optim_states.pt. 5: [2023-03-17 10:27:43,792] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step10000/bf16_zero_pp_rank_41_mp_rank_00_optim_states.pt 5: [2023-03-17 10:27:43,792] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step10000 is ready now! 5: [2023-03-17 10:27:43,792] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step10000 is ready now! 5: [2023-03-17 10:27:43,792] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step10000 is ready now! 5: [2023-03-17 10:27:43,792] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step10000/bf16_zero_pp_rank_47_mp_rank_00_optim_states.pt 5: [2023-03-17 10:27:43,792] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step10000 is ready now! 5: [2023-03-17 10:27:43,792] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step10000/bf16_zero_pp_rank_42_mp_rank_00_optim_states.pt. 5: [2023-03-17 10:27:43,792] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step10000/bf16_zero_pp_rank_45_mp_rank_00_optim_states.pt. 5: [2023-03-17 10:27:43,792] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step10000/bf16_zero_pp_rank_42_mp_rank_00_optim_states.pt 5: [2023-03-17 10:27:43,792] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step10000 is ready now! 5: [2023-03-17 10:27:43,792] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step10000/bf16_zero_pp_rank_45_mp_rank_00_optim_states.pt 5: [2023-03-17 10:27:43,792] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step10000/bf16_zero_pp_rank_43_mp_rank_00_optim_states.pt. 5: [2023-03-17 10:27:43,792] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step10000 is ready now! 5: [2023-03-17 10:27:43,792] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step10000/bf16_zero_pp_rank_40_mp_rank_00_optim_states.pt. 5: [2023-03-17 10:27:43,792] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step10000/bf16_zero_pp_rank_43_mp_rank_00_optim_states.pt 5: [2023-03-17 10:27:43,792] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step10000/bf16_zero_pp_rank_40_mp_rank_00_optim_states.pt 5: [2023-03-17 10:27:43,792] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step10000 is ready now! 5: [2023-03-17 10:27:43,792] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step10000 is ready now! 0: successfully saved checkpoint at iteration 10000 to checkpoints_44m174b100m 7: time (ms) | save-checkpoint: 403.66 7: iteration 10100/ 331103 | consumed samples: 2585600 | consumed tokens: 5295308800 | elapsed time per iteration (s): 0.16 | learning rate: 1.998E-04 | global batch size: 256 | lm loss: 4.053796E+00 | grad norm: 0.476 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1566.337 | TFLOPs: 24.56 | 7: iteration 10200/ 331103 | consumed samples: 2611200 | consumed tokens: 5347737600 | elapsed time per iteration (s): 0.16 | learning rate: 1.998E-04 | global batch size: 256 | lm loss: 4.051366E+00 | grad norm: 0.395 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1577.172 | TFLOPs: 24.73 | 7: iteration 10300/ 331103 | consumed samples: 2636800 | consumed tokens: 5400166400 | elapsed time per iteration (s): 0.16 | learning rate: 1.998E-04 | global batch size: 256 | lm loss: 4.047337E+00 | grad norm: 0.516 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1611.950 | TFLOPs: 25.28 | 7: iteration 10400/ 331103 | consumed samples: 2662400 | consumed tokens: 5452595200 | elapsed time per iteration (s): 0.16 | learning rate: 1.998E-04 | global batch size: 256 | lm loss: 4.047060E+00 | grad norm: 0.429 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1596.480 | TFLOPs: 25.04 | 7: iteration 10500/ 331103 | consumed samples: 2688000 | consumed tokens: 5505024000 | elapsed time per iteration (s): 0.16 | learning rate: 1.998E-04 | global batch size: 256 | lm loss: 4.044305E+00 | grad norm: 0.356 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1605.513 | TFLOPs: 25.18 | 7: iteration 10600/ 331103 | consumed samples: 2713600 | consumed tokens: 5557452800 | elapsed time per iteration (s): 0.16 | learning rate: 1.998E-04 | global batch size: 256 | lm loss: 4.036382E+00 | grad norm: 0.385 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1597.143 | TFLOPs: 25.05 | 7: iteration 10700/ 331103 | consumed samples: 2739200 | consumed tokens: 5609881600 | elapsed time per iteration (s): 0.16 | learning rate: 1.998E-04 | global batch size: 256 | lm loss: 4.032678E+00 | grad norm: 0.370 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1592.806 | TFLOPs: 24.98 | 7: iteration 10800/ 331103 | consumed samples: 2764800 | consumed tokens: 5662310400 | elapsed time per iteration (s): 0.16 | learning rate: 1.998E-04 | global batch size: 256 | lm loss: 4.035749E+00 | grad norm: 0.462 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1607.486 | TFLOPs: 25.21 | 7: iteration 10900/ 331103 | consumed samples: 2790400 | consumed tokens: 5714739200 | elapsed time per iteration (s): 0.16 | learning rate: 1.998E-04 | global batch size: 256 | lm loss: 4.034238E+00 | grad norm: 0.436 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1615.024 | TFLOPs: 25.33 | 7: iteration 11000/ 331103 | consumed samples: 2816000 | consumed tokens: 5767168000 | elapsed time per iteration (s): 0.16 | learning rate: 1.998E-04 | global batch size: 256 | lm loss: 4.023975E+00 | grad norm: 0.302 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1616.194 | TFLOPs: 25.35 | 7: iteration 11100/ 331103 | consumed samples: 2841600 | consumed tokens: 5819596800 | elapsed time per iteration (s): 0.16 | learning rate: 1.997E-04 | global batch size: 256 | lm loss: 4.027915E+00 | grad norm: 0.396 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1594.329 | TFLOPs: 25.00 | 7: iteration 11200/ 331103 | consumed samples: 2867200 | consumed tokens: 5872025600 | elapsed time per iteration (s): 0.16 | learning rate: 1.997E-04 | global batch size: 256 | lm loss: 4.019272E+00 | grad norm: 0.295 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1620.132 | TFLOPs: 25.41 | 7: iteration 11300/ 331103 | consumed samples: 2892800 | consumed tokens: 5924454400 | elapsed time per iteration (s): 0.16 | learning rate: 1.997E-04 | global batch size: 256 | lm loss: 4.021964E+00 | grad norm: 0.375 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1620.746 | TFLOPs: 25.42 | 7: iteration 11400/ 331103 | consumed samples: 2918400 | consumed tokens: 5976883200 | elapsed time per iteration (s): 0.16 | learning rate: 1.997E-04 | global batch size: 256 | lm loss: 4.016543E+00 | grad norm: 0.443 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1604.848 | TFLOPs: 25.17 | 7: iteration 11500/ 331103 | consumed samples: 2944000 | consumed tokens: 6029312000 | elapsed time per iteration (s): 0.16 | learning rate: 1.997E-04 | global batch size: 256 | lm loss: 4.016207E+00 | grad norm: 0.383 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1598.086 | TFLOPs: 25.06 | 7: iteration 11600/ 331103 | consumed samples: 2969600 | consumed tokens: 6081740800 | elapsed time per iteration (s): 0.16 | learning rate: 1.997E-04 | global batch size: 256 | lm loss: 4.010818E+00 | grad norm: 0.506 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1602.456 | TFLOPs: 25.13 | 7: iteration 11700/ 331103 | consumed samples: 2995200 | consumed tokens: 6134169600 | elapsed time per iteration (s): 0.16 | learning rate: 1.997E-04 | global batch size: 256 | lm loss: 4.013252E+00 | grad norm: 0.383 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1595.003 | TFLOPs: 25.01 | 7: iteration 11800/ 331103 | consumed samples: 3020800 | consumed tokens: 6186598400 | elapsed time per iteration (s): 0.16 | learning rate: 1.997E-04 | global batch size: 256 | lm loss: 4.007014E+00 | grad norm: 0.435 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1621.032 | TFLOPs: 25.42 | 7: iteration 11900/ 331103 | consumed samples: 3046400 | consumed tokens: 6239027200 | elapsed time per iteration (s): 0.16 | learning rate: 1.997E-04 | global batch size: 256 | lm loss: 4.007841E+00 | grad norm: 0.376 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1628.504 | TFLOPs: 25.54 | 0: [2023-03-17 10:33:02,354] [INFO] [logging.py:68:log_dist] [Rank 0] step=12000, skipped=0, lr=[0.00019968811185780457, 0.00019968811185780457, 0.00019968811185780457], mom=[(0.9, 0.999), (0.9, 0.999), (0.9, 0.999)] 7: iteration 12000/ 331103 | consumed samples: 3072000 | consumed tokens: 6291456000 | elapsed time per iteration (s): 0.16 | learning rate: 1.997E-04 | global batch size: 256 | lm loss: 4.001406E+00 | grad norm: 0.409 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1628.993 | TFLOPs: 25.55 | 0: steps: 12000 loss: 4.0054 iter time (s): 0.158 samples/sec: 1621.747 7: iteration 12100/ 331103 | consumed samples: 3097600 | consumed tokens: 6343884800 | elapsed time per iteration (s): 0.16 | learning rate: 1.997E-04 | global batch size: 256 | lm loss: 4.003288E+00 | grad norm: 0.315 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1617.302 | TFLOPs: 25.36 | 7: iteration 12200/ 331103 | consumed samples: 3123200 | consumed tokens: 6396313600 | elapsed time per iteration (s): 0.16 | learning rate: 1.997E-04 | global batch size: 256 | lm loss: 3.991168E+00 | grad norm: 0.403 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1623.476 | TFLOPs: 25.46 | 7: iteration 12300/ 331103 | consumed samples: 3148800 | consumed tokens: 6448742400 | elapsed time per iteration (s): 0.16 | learning rate: 1.997E-04 | global batch size: 256 | lm loss: 3.997861E+00 | grad norm: 0.447 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1620.119 | TFLOPs: 25.41 | 7: iteration 12400/ 331103 | consumed samples: 3174400 | consumed tokens: 6501171200 | elapsed time per iteration (s): 0.16 | learning rate: 1.997E-04 | global batch size: 256 | lm loss: 3.993774E+00 | grad norm: 0.518 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1613.023 | TFLOPs: 25.30 | 7: iteration 12500/ 331103 | consumed samples: 3200000 | consumed tokens: 6553600000 | elapsed time per iteration (s): 0.16 | learning rate: 1.997E-04 | global batch size: 256 | lm loss: 3.991158E+00 | grad norm: 0.437 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1619.650 | TFLOPs: 25.40 | 7: iteration 12600/ 331103 | consumed samples: 3225600 | consumed tokens: 6606028800 | elapsed time per iteration (s): 0.16 | learning rate: 1.996E-04 | global batch size: 256 | lm loss: 3.989981E+00 | grad norm: 0.321 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1614.782 | TFLOPs: 25.32 | 7: iteration 12700/ 331103 | consumed samples: 3251200 | consumed tokens: 6658457600 | elapsed time per iteration (s): 0.16 | learning rate: 1.996E-04 | global batch size: 256 | lm loss: 3.984923E+00 | grad norm: 0.420 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1619.928 | TFLOPs: 25.40 | 7: iteration 12800/ 331103 | consumed samples: 3276800 | consumed tokens: 6710886400 | elapsed time per iteration (s): 0.16 | learning rate: 1.996E-04 | global batch size: 256 | lm loss: 3.985205E+00 | grad norm: 0.366 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1619.161 | TFLOPs: 25.39 | 7: iteration 12900/ 331103 | consumed samples: 3302400 | consumed tokens: 6763315200 | elapsed time per iteration (s): 0.16 | learning rate: 1.996E-04 | global batch size: 256 | lm loss: 3.985256E+00 | grad norm: 0.446 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1613.278 | TFLOPs: 25.30 | 7: iteration 13000/ 331103 | consumed samples: 3328000 | consumed tokens: 6815744000 | elapsed time per iteration (s): 0.16 | learning rate: 1.996E-04 | global batch size: 256 | lm loss: 3.978413E+00 | grad norm: 0.342 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1608.782 | TFLOPs: 25.23 | 7: iteration 13100/ 331103 | consumed samples: 3353600 | consumed tokens: 6868172800 | elapsed time per iteration (s): 0.16 | learning rate: 1.996E-04 | global batch size: 256 | lm loss: 3.978840E+00 | grad norm: 0.349 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1613.739 | TFLOPs: 25.31 | 7: iteration 13200/ 331103 | consumed samples: 3379200 | consumed tokens: 6920601600 | elapsed time per iteration (s): 0.16 | learning rate: 1.996E-04 | global batch size: 256 | lm loss: 3.978529E+00 | grad norm: 0.356 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1607.518 | TFLOPs: 25.21 | 7: iteration 13300/ 331103 | consumed samples: 3404800 | consumed tokens: 6973030400 | elapsed time per iteration (s): 0.16 | learning rate: 1.996E-04 | global batch size: 256 | lm loss: 3.972823E+00 | grad norm: 0.450 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1629.062 | TFLOPs: 25.55 | 7: iteration 13400/ 331103 | consumed samples: 3430400 | consumed tokens: 7025459200 | elapsed time per iteration (s): 0.16 | learning rate: 1.996E-04 | global batch size: 256 | lm loss: 3.974167E+00 | grad norm: 0.361 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1647.699 | TFLOPs: 25.84 | 7: iteration 13500/ 331103 | consumed samples: 3456000 | consumed tokens: 7077888000 | elapsed time per iteration (s): 0.16 | learning rate: 1.996E-04 | global batch size: 256 | lm loss: 3.963086E+00 | grad norm: 0.371 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1627.600 | TFLOPs: 25.52 | 7: iteration 13600/ 331103 | consumed samples: 3481600 | consumed tokens: 7130316800 | elapsed time per iteration (s): 0.16 | learning rate: 1.996E-04 | global batch size: 256 | lm loss: 3.969101E+00 | grad norm: 0.364 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1622.529 | TFLOPs: 25.45 | 7: iteration 13700/ 331103 | consumed samples: 3507200 | consumed tokens: 7182745600 | elapsed time per iteration (s): 0.16 | learning rate: 1.996E-04 | global batch size: 256 | lm loss: 3.965058E+00 | grad norm: 0.374 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1615.644 | TFLOPs: 25.34 | 7: iteration 13800/ 331103 | consumed samples: 3532800 | consumed tokens: 7235174400 | elapsed time per iteration (s): 0.16 | learning rate: 1.995E-04 | global batch size: 256 | lm loss: 3.965342E+00 | grad norm: 0.363 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1622.436 | TFLOPs: 25.44 | 7: iteration 13900/ 331103 | consumed samples: 3558400 | consumed tokens: 7287603200 | elapsed time per iteration (s): 0.16 | learning rate: 1.995E-04 | global batch size: 256 | lm loss: 3.959620E+00 | grad norm: 0.386 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1612.431 | TFLOPs: 25.29 | 0: [2023-03-17 10:38:18,484] [INFO] [logging.py:68:log_dist] [Rank 0] step=14000, skipped=0, lr=[0.00019952814862529602, 0.00019952814862529602, 0.00019952814862529602], mom=[(0.9, 0.999), (0.9, 0.999), (0.9, 0.999)] 7: iteration 14000/ 331103 | consumed samples: 3584000 | consumed tokens: 7340032000 | elapsed time per iteration (s): 0.16 | learning rate: 1.995E-04 | global batch size: 256 | lm loss: 3.957062E+00 | grad norm: 0.330 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1624.561 | TFLOPs: 25.48 | 0: steps: 14000 loss: 3.9410 iter time (s): 0.157 samples/sec: 1635.342 7: iteration 14100/ 331103 | consumed samples: 3609600 | consumed tokens: 7392460800 | elapsed time per iteration (s): 0.16 | learning rate: 1.995E-04 | global batch size: 256 | lm loss: 3.962184E+00 | grad norm: 0.346 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1622.750 | TFLOPs: 25.45 | 7: iteration 14200/ 331103 | consumed samples: 3635200 | consumed tokens: 7444889600 | elapsed time per iteration (s): 0.24 | learning rate: 1.995E-04 | global batch size: 256 | lm loss: 3.957191E+00 | grad norm: 0.381 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1082.576 | TFLOPs: 16.98 | 7: iteration 14300/ 331103 | consumed samples: 3660800 | consumed tokens: 7497318400 | elapsed time per iteration (s): 0.22 | learning rate: 1.995E-04 | global batch size: 256 | lm loss: 3.952648E+00 | grad norm: 0.388 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1166.841 | TFLOPs: 18.30 | 7: iteration 14400/ 331103 | consumed samples: 3686400 | consumed tokens: 7549747200 | elapsed time per iteration (s): 0.16 | learning rate: 1.995E-04 | global batch size: 256 | lm loss: 3.955305E+00 | grad norm: 0.365 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1634.578 | TFLOPs: 25.63 | 7: iteration 14500/ 331103 | consumed samples: 3712000 | consumed tokens: 7602176000 | elapsed time per iteration (s): 0.16 | learning rate: 1.995E-04 | global batch size: 256 | lm loss: 3.955842E+00 | grad norm: 0.313 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1635.412 | TFLOPs: 25.65 | 7: iteration 14600/ 331103 | consumed samples: 3737600 | consumed tokens: 7654604800 | elapsed time per iteration (s): 0.16 | learning rate: 1.995E-04 | global batch size: 256 | lm loss: 3.949105E+00 | grad norm: 0.422 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1636.166 | TFLOPs: 25.66 | 7: iteration 14700/ 331103 | consumed samples: 3763200 | consumed tokens: 7707033600 | elapsed time per iteration (s): 0.16 | learning rate: 1.995E-04 | global batch size: 256 | lm loss: 3.951932E+00 | grad norm: 0.336 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1617.857 | TFLOPs: 25.37 | 7: iteration 14800/ 331103 | consumed samples: 3788800 | consumed tokens: 7759462400 | elapsed time per iteration (s): 0.16 | learning rate: 1.995E-04 | global batch size: 256 | lm loss: 3.952508E+00 | grad norm: 0.327 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1618.030 | TFLOPs: 25.37 | 7: iteration 14900/ 331103 | consumed samples: 3814400 | consumed tokens: 7811891200 | elapsed time per iteration (s): 0.16 | learning rate: 1.994E-04 | global batch size: 256 | lm loss: 3.946388E+00 | grad norm: 0.419 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1631.995 | TFLOPs: 25.59 | 7: iteration 15000/ 331103 | consumed samples: 3840000 | consumed tokens: 7864320000 | elapsed time per iteration (s): 0.16 | learning rate: 1.994E-04 | global batch size: 256 | lm loss: 3.943889E+00 | grad norm: 0.376 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1631.617 | TFLOPs: 25.59 | 7: iteration 15100/ 331103 | consumed samples: 3865600 | consumed tokens: 7916748800 | elapsed time per iteration (s): 0.16 | learning rate: 1.994E-04 | global batch size: 256 | lm loss: 3.940260E+00 | grad norm: 0.340 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1637.682 | TFLOPs: 25.68 | 7: iteration 15200/ 331103 | consumed samples: 3891200 | consumed tokens: 7969177600 | elapsed time per iteration (s): 0.16 | learning rate: 1.994E-04 | global batch size: 256 | lm loss: 3.941477E+00 | grad norm: 0.351 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1632.552 | TFLOPs: 25.60 | 7: iteration 15300/ 331103 | consumed samples: 3916800 | consumed tokens: 8021606400 | elapsed time per iteration (s): 0.16 | learning rate: 1.994E-04 | global batch size: 256 | lm loss: 3.939484E+00 | grad norm: 0.334 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1620.351 | TFLOPs: 25.41 | 7: iteration 15400/ 331103 | consumed samples: 3942400 | consumed tokens: 8074035200 | elapsed time per iteration (s): 0.16 | learning rate: 1.994E-04 | global batch size: 256 | lm loss: 3.940862E+00 | grad norm: 0.376 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1630.999 | TFLOPs: 25.58 | 7: iteration 15500/ 331103 | consumed samples: 3968000 | consumed tokens: 8126464000 | elapsed time per iteration (s): 0.16 | learning rate: 1.994E-04 | global batch size: 256 | lm loss: 3.934151E+00 | grad norm: 0.328 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1629.082 | TFLOPs: 25.55 | 7: iteration 15600/ 331103 | consumed samples: 3993600 | consumed tokens: 8178892800 | elapsed time per iteration (s): 0.16 | learning rate: 1.994E-04 | global batch size: 256 | lm loss: 3.935645E+00 | grad norm: 0.329 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1642.342 | TFLOPs: 25.76 | 7: iteration 15700/ 331103 | consumed samples: 4019200 | consumed tokens: 8231321600 | elapsed time per iteration (s): 0.16 | learning rate: 1.994E-04 | global batch size: 256 | lm loss: 3.934573E+00 | grad norm: 0.388 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1612.933 | TFLOPs: 25.29 | 7: iteration 15800/ 331103 | consumed samples: 4044800 | consumed tokens: 8283750400 | elapsed time per iteration (s): 0.16 | learning rate: 1.994E-04 | global batch size: 256 | lm loss: 3.932506E+00 | grad norm: 0.375 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1648.500 | TFLOPs: 25.85 | 7: iteration 15900/ 331103 | consumed samples: 4070400 | consumed tokens: 8336179200 | elapsed time per iteration (s): 0.16 | learning rate: 1.993E-04 | global batch size: 256 | lm loss: 3.934685E+00 | grad norm: 0.296 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1640.939 | TFLOPs: 25.73 | 0: [2023-03-17 10:43:46,573] [INFO] [logging.py:68:log_dist] [Rank 0] step=16000, skipped=0, lr=[0.00019933529208396184, 0.00019933529208396184, 0.00019933529208396184], mom=[(0.9, 0.999), (0.9, 0.999), (0.9, 0.999)] 7: iteration 16000/ 331103 | consumed samples: 4096000 | consumed tokens: 8388608000 | elapsed time per iteration (s): 0.16 | learning rate: 1.993E-04 | global batch size: 256 | lm loss: 3.932782E+00 | grad norm: 0.344 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1637.556 | TFLOPs: 25.68 | 0: steps: 16000 loss: 3.9343 iter time (s): 0.163 samples/sec: 1574.624 7: iteration 16100/ 331103 | consumed samples: 4121600 | consumed tokens: 8441036800 | elapsed time per iteration (s): 0.16 | learning rate: 1.993E-04 | global batch size: 256 | lm loss: 3.932262E+00 | grad norm: 0.380 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1623.835 | TFLOPs: 25.47 | 7: iteration 16200/ 331103 | consumed samples: 4147200 | consumed tokens: 8493465600 | elapsed time per iteration (s): 0.16 | learning rate: 1.993E-04 | global batch size: 256 | lm loss: 3.928919E+00 | grad norm: 0.350 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1623.209 | TFLOPs: 25.46 | 7: iteration 16300/ 331103 | consumed samples: 4172800 | consumed tokens: 8545894400 | elapsed time per iteration (s): 0.16 | learning rate: 1.993E-04 | global batch size: 256 | lm loss: 3.926278E+00 | grad norm: 0.377 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1646.466 | TFLOPs: 25.82 | 7: iteration 16400/ 331103 | consumed samples: 4198400 | consumed tokens: 8598323200 | elapsed time per iteration (s): 0.16 | learning rate: 1.993E-04 | global batch size: 256 | lm loss: 3.925863E+00 | grad norm: 0.334 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1645.470 | TFLOPs: 25.81 | 7: iteration 16500/ 331103 | consumed samples: 4224000 | consumed tokens: 8650752000 | elapsed time per iteration (s): 0.16 | learning rate: 1.993E-04 | global batch size: 256 | lm loss: 3.925933E+00 | grad norm: 0.314 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1645.551 | TFLOPs: 25.81 | 7: iteration 16600/ 331103 | consumed samples: 4249600 | consumed tokens: 8703180800 | elapsed time per iteration (s): 0.16 | learning rate: 1.993E-04 | global batch size: 256 | lm loss: 3.924902E+00 | grad norm: 0.341 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1623.415 | TFLOPs: 25.46 | 7: iteration 16700/ 331103 | consumed samples: 4275200 | consumed tokens: 8755609600 | elapsed time per iteration (s): 0.16 | learning rate: 1.993E-04 | global batch size: 256 | lm loss: 3.919805E+00 | grad norm: 0.325 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1622.174 | TFLOPs: 25.44 | 7: iteration 16800/ 331103 | consumed samples: 4300800 | consumed tokens: 8808038400 | elapsed time per iteration (s): 0.16 | learning rate: 1.992E-04 | global batch size: 256 | lm loss: 3.917948E+00 | grad norm: 0.340 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1625.263 | TFLOPs: 25.49 | 7: iteration 16900/ 331103 | consumed samples: 4326400 | consumed tokens: 8860467200 | elapsed time per iteration (s): 0.16 | learning rate: 1.992E-04 | global batch size: 256 | lm loss: 3.917692E+00 | grad norm: 0.361 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1644.698 | TFLOPs: 25.79 | 7: iteration 17000/ 331103 | consumed samples: 4352000 | consumed tokens: 8912896000 | elapsed time per iteration (s): 0.16 | learning rate: 1.992E-04 | global batch size: 256 | lm loss: 3.917712E+00 | grad norm: 0.369 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1620.023 | TFLOPs: 25.41 | 7: iteration 17100/ 331103 | consumed samples: 4377600 | consumed tokens: 8965324800 | elapsed time per iteration (s): 0.16 | learning rate: 1.992E-04 | global batch size: 256 | lm loss: 3.917498E+00 | grad norm: 0.310 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1625.940 | TFLOPs: 25.50 | 7: iteration 17200/ 331103 | consumed samples: 4403200 | consumed tokens: 9017753600 | elapsed time per iteration (s): 0.16 | learning rate: 1.992E-04 | global batch size: 256 | lm loss: 3.916953E+00 | grad norm: 0.344 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1613.062 | TFLOPs: 25.30 | 7: iteration 17300/ 331103 | consumed samples: 4428800 | consumed tokens: 9070182400 | elapsed time per iteration (s): 0.16 | learning rate: 1.992E-04 | global batch size: 256 | lm loss: 3.915765E+00 | grad norm: 0.314 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1636.993 | TFLOPs: 25.67 | 7: iteration 17400/ 331103 | consumed samples: 4454400 | consumed tokens: 9122611200 | elapsed time per iteration (s): 0.16 | learning rate: 1.992E-04 | global batch size: 256 | lm loss: 3.910309E+00 | grad norm: 0.332 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1635.919 | TFLOPs: 25.66 | 7: iteration 17500/ 331103 | consumed samples: 4480000 | consumed tokens: 9175040000 | elapsed time per iteration (s): 0.16 | learning rate: 1.992E-04 | global batch size: 256 | lm loss: 3.912882E+00 | grad norm: 0.342 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1622.166 | TFLOPs: 25.44 | 7: iteration 17600/ 331103 | consumed samples: 4505600 | consumed tokens: 9227468800 | elapsed time per iteration (s): 0.16 | learning rate: 1.992E-04 | global batch size: 256 | lm loss: 3.912144E+00 | grad norm: 0.367 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1636.427 | TFLOPs: 25.66 | 7: iteration 17700/ 331103 | consumed samples: 4531200 | consumed tokens: 9279897600 | elapsed time per iteration (s): 0.16 | learning rate: 1.991E-04 | global batch size: 256 | lm loss: 3.913989E+00 | grad norm: 0.387 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1641.124 | TFLOPs: 25.74 | 7: iteration 17800/ 331103 | consumed samples: 4556800 | consumed tokens: 9332326400 | elapsed time per iteration (s): 0.16 | learning rate: 1.991E-04 | global batch size: 256 | lm loss: 3.906084E+00 | grad norm: 0.382 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1628.941 | TFLOPs: 25.55 | 7: iteration 17900/ 331103 | consumed samples: 4582400 | consumed tokens: 9384755200 | elapsed time per iteration (s): 0.16 | learning rate: 1.991E-04 | global batch size: 256 | lm loss: 3.902943E+00 | grad norm: 0.351 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1633.420 | TFLOPs: 25.62 | 0: [2023-03-17 10:49:00,510] [INFO] [logging.py:68:log_dist] [Rank 0] step=18000, skipped=0, lr=[0.00019910961309073215, 0.00019910961309073215, 0.00019910961309073215], mom=[(0.9, 0.999), (0.9, 0.999), (0.9, 0.999)] 7: iteration 18000/ 331103 | consumed samples: 4608000 | consumed tokens: 9437184000 | elapsed time per iteration (s): 0.16 | learning rate: 1.991E-04 | global batch size: 256 | lm loss: 3.905841E+00 | grad norm: 0.338 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1625.204 | TFLOPs: 25.49 | 0: steps: 18000 loss: 3.9099 iter time (s): 0.155 samples/sec: 1647.380 7: iteration 18100/ 331103 | consumed samples: 4633600 | consumed tokens: 9489612800 | elapsed time per iteration (s): 0.16 | learning rate: 1.991E-04 | global batch size: 256 | lm loss: 3.906725E+00 | grad norm: 0.303 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1608.263 | TFLOPs: 25.22 | 7: iteration 18200/ 331103 | consumed samples: 4659200 | consumed tokens: 9542041600 | elapsed time per iteration (s): 0.16 | learning rate: 1.991E-04 | global batch size: 256 | lm loss: 3.906817E+00 | grad norm: 0.335 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1643.042 | TFLOPs: 25.77 | 7: iteration 18300/ 331103 | consumed samples: 4684800 | consumed tokens: 9594470400 | elapsed time per iteration (s): 0.16 | learning rate: 1.991E-04 | global batch size: 256 | lm loss: 3.902694E+00 | grad norm: 0.340 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1644.090 | TFLOPs: 25.78 | 7: iteration 18400/ 331103 | consumed samples: 4710400 | consumed tokens: 9646899200 | elapsed time per iteration (s): 0.16 | learning rate: 1.991E-04 | global batch size: 256 | lm loss: 3.902517E+00 | grad norm: 0.366 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1613.584 | TFLOPs: 25.31 | 7: iteration 18500/ 331103 | consumed samples: 4736000 | consumed tokens: 9699328000 | elapsed time per iteration (s): 0.16 | learning rate: 1.990E-04 | global batch size: 256 | lm loss: 3.898546E+00 | grad norm: 0.370 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1624.770 | TFLOPs: 25.48 | 7: iteration 18600/ 331103 | consumed samples: 4761600 | consumed tokens: 9751756800 | elapsed time per iteration (s): 0.16 | learning rate: 1.990E-04 | global batch size: 256 | lm loss: 3.899608E+00 | grad norm: 0.343 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1635.293 | TFLOPs: 25.65 | 7: iteration 18700/ 331103 | consumed samples: 4787200 | consumed tokens: 9804185600 | elapsed time per iteration (s): 0.16 | learning rate: 1.990E-04 | global batch size: 256 | lm loss: 3.898123E+00 | grad norm: 0.342 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1634.431 | TFLOPs: 25.63 | 7: iteration 18800/ 331103 | consumed samples: 4812800 | consumed tokens: 9856614400 | elapsed time per iteration (s): 0.15 | learning rate: 1.990E-04 | global batch size: 256 | lm loss: 3.896628E+00 | grad norm: 0.330 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1661.489 | TFLOPs: 26.06 | 7: iteration 18900/ 331103 | consumed samples: 4838400 | consumed tokens: 9909043200 | elapsed time per iteration (s): 0.16 | learning rate: 1.990E-04 | global batch size: 256 | lm loss: 3.896649E+00 | grad norm: 0.328 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1632.138 | TFLOPs: 25.60 | 7: iteration 19000/ 331103 | consumed samples: 4864000 | consumed tokens: 9961472000 | elapsed time per iteration (s): 0.16 | learning rate: 1.990E-04 | global batch size: 256 | lm loss: 3.896582E+00 | grad norm: 0.385 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1646.378 | TFLOPs: 25.82 | 7: iteration 19100/ 331103 | consumed samples: 4889600 | consumed tokens: 10013900800 | elapsed time per iteration (s): 0.16 | learning rate: 1.990E-04 | global batch size: 256 | lm loss: 3.897778E+00 | grad norm: 0.344 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1628.475 | TFLOPs: 25.54 | 7: iteration 19200/ 331103 | consumed samples: 4915200 | consumed tokens: 10066329600 | elapsed time per iteration (s): 0.16 | learning rate: 1.990E-04 | global batch size: 256 | lm loss: 3.894473E+00 | grad norm: 0.326 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1634.031 | TFLOPs: 25.63 | 7: iteration 19300/ 331103 | consumed samples: 4940800 | consumed tokens: 10118758400 | elapsed time per iteration (s): 0.16 | learning rate: 1.989E-04 | global batch size: 256 | lm loss: 3.891491E+00 | grad norm: 0.331 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1649.529 | TFLOPs: 25.87 | 7: iteration 19400/ 331103 | consumed samples: 4966400 | consumed tokens: 10171187200 | elapsed time per iteration (s): 0.16 | learning rate: 1.989E-04 | global batch size: 256 | lm loss: 3.891590E+00 | grad norm: 0.285 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1646.784 | TFLOPs: 25.83 | 7: iteration 19500/ 331103 | consumed samples: 4992000 | consumed tokens: 10223616000 | elapsed time per iteration (s): 0.16 | learning rate: 1.989E-04 | global batch size: 256 | lm loss: 3.886781E+00 | grad norm: 0.338 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1637.752 | TFLOPs: 25.68 | 7: iteration 19600/ 331103 | consumed samples: 5017600 | consumed tokens: 10276044800 | elapsed time per iteration (s): 0.16 | learning rate: 1.989E-04 | global batch size: 256 | lm loss: 3.889518E+00 | grad norm: 0.314 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1628.283 | TFLOPs: 25.54 | 7: iteration 19700/ 331103 | consumed samples: 5043200 | consumed tokens: 10328473600 | elapsed time per iteration (s): 0.16 | learning rate: 1.989E-04 | global batch size: 256 | lm loss: 3.894202E+00 | grad norm: 0.311 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1639.011 | TFLOPs: 25.70 | 7: iteration 19800/ 331103 | consumed samples: 5068800 | consumed tokens: 10380902400 | elapsed time per iteration (s): 0.16 | learning rate: 1.989E-04 | global batch size: 256 | lm loss: 3.884542E+00 | grad norm: 0.369 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1633.359 | TFLOPs: 25.62 | 7: iteration 19900/ 331103 | consumed samples: 5094400 | consumed tokens: 10433331200 | elapsed time per iteration (s): 0.16 | learning rate: 1.989E-04 | global batch size: 256 | lm loss: 3.891487E+00 | grad norm: 0.385 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1633.284 | TFLOPs: 25.61 | 0: [2023-03-17 10:54:13,550] [INFO] [logging.py:68:log_dist] [Rank 0] step=20000, skipped=0, lr=[0.00019885119456175047, 0.00019885119456175047, 0.00019885119456175047], mom=[(0.9, 0.999), (0.9, 0.999), (0.9, 0.999)] 7: iteration 20000/ 331103 | consumed samples: 5120000 | consumed tokens: 10485760000 | elapsed time per iteration (s): 0.16 | learning rate: 1.989E-04 | global batch size: 256 | lm loss: 3.886676E+00 | grad norm: 0.354 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1639.219 | TFLOPs: 25.71 | 0: steps: 20000 loss: 3.8956 iter time (s): 0.155 samples/sec: 1651.439 7: ------------------------------------------------------------------------------------------------ 7: validation loss at iteration 20000 | lm loss value: 3.956715E+00 | lm loss PPL: 5.228530E+01 | 7: ------------------------------------------------------------------------------------------------ 0: saving checkpoint at iteration 20000 to checkpoints_44m174b100m 0: [2023-03-17 10:54:13,623] [INFO] [logging.py:68:log_dist] [Rank 0] [Torch] Checkpoint global_step20000 is begin to save! 0: [2023-03-17 10:54:13,626] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step20000/layer_01-model_00-model_states.pt... 0: [2023-03-17 10:54:13,685] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step20000/layer_01-model_00-model_states.pt. 0: [2023-03-17 10:54:13,685] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step20000/layer_03-model_00-model_states.pt... 0: [2023-03-17 10:54:13,696] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step20000/layer_03-model_00-model_states.pt. 0: [2023-03-17 10:54:13,697] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step20000/layer_04-model_00-model_states.pt... 0: [2023-03-17 10:54:13,705] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step20000/layer_04-model_00-model_states.pt. 0: [2023-03-17 10:54:13,706] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step20000/layer_05-model_00-model_states.pt... 0: [2023-03-17 10:54:13,714] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step20000/layer_05-model_00-model_states.pt. 0: [2023-03-17 10:54:13,714] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step20000/layer_06-model_00-model_states.pt... 0: [2023-03-17 10:54:13,722] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step20000/layer_06-model_00-model_states.pt. 0: [2023-03-17 10:54:13,722] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step20000/layer_07-model_00-model_states.pt... 0: [2023-03-17 10:54:13,730] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step20000/layer_07-model_00-model_states.pt. 0: [2023-03-17 10:54:13,731] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step20000/layer_08-model_00-model_states.pt... 0: [2023-03-17 10:54:13,738] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step20000/layer_08-model_00-model_states.pt. 0: [2023-03-17 10:54:13,739] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step20000/layer_09-model_00-model_states.pt... 0: [2023-03-17 10:54:13,747] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step20000/layer_09-model_00-model_states.pt. 0: [2023-03-17 10:54:13,747] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step20000/layer_10-model_00-model_states.pt... 0: [2023-03-17 10:54:13,755] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step20000/layer_10-model_00-model_states.pt. 0: [2023-03-17 10:54:13,755] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step20000/layer_12-model_00-model_states.pt... 0: [2023-03-17 10:54:13,756] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step20000/layer_12-model_00-model_states.pt. 0: [2023-03-17 10:54:13,756] [INFO] [logging.py:68:log_dist] [Rank 0] Saving model checkpoint: checkpoints_44m174b100m/global_step20000/mp_rank_00_model_states.pt 0: [2023-03-17 10:54:13,756] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step20000/mp_rank_00_model_states.pt... 0: [2023-03-17 10:54:13,758] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step20000/mp_rank_00_model_states.pt. 0: [2023-03-17 10:54:13,775] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step20000/bf16_zero_pp_rank_0_mp_rank_00_optim_states.pt... 0: [2023-03-17 10:54:13,775] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step20000/bf16_zero_pp_rank_1_mp_rank_00_optim_states.pt... 0: [2023-03-17 10:54:13,775] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step20000/bf16_zero_pp_rank_4_mp_rank_00_optim_states.pt... 0: [2023-03-17 10:54:13,775] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step20000/bf16_zero_pp_rank_5_mp_rank_00_optim_states.pt... 0: [2023-03-17 10:54:13,775] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step20000/bf16_zero_pp_rank_2_mp_rank_00_optim_states.pt... 0: [2023-03-17 10:54:13,775] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step20000/bf16_zero_pp_rank_6_mp_rank_00_optim_states.pt... 0: [2023-03-17 10:54:13,775] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step20000/bf16_zero_pp_rank_3_mp_rank_00_optim_states.pt... 3: [2023-03-17 10:54:13,775] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step20000/bf16_zero_pp_rank_30_mp_rank_00_optim_states.pt... 3: [2023-03-17 10:54:13,775] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step20000/bf16_zero_pp_rank_29_mp_rank_00_optim_states.pt... 3: [2023-03-17 10:54:13,775] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step20000/bf16_zero_pp_rank_28_mp_rank_00_optim_states.pt... 3: [2023-03-17 10:54:13,775] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step20000/bf16_zero_pp_rank_24_mp_rank_00_optim_states.pt... 3: [2023-03-17 10:54:13,775] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step20000/bf16_zero_pp_rank_31_mp_rank_00_optim_states.pt... 4: [2023-03-17 10:54:13,775] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step20000/bf16_zero_pp_rank_37_mp_rank_00_optim_states.pt... 4: [2023-03-17 10:54:13,775] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step20000/bf16_zero_pp_rank_38_mp_rank_00_optim_states.pt... 4: [2023-03-17 10:54:13,775] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step20000/bf16_zero_pp_rank_32_mp_rank_00_optim_states.pt... 4: [2023-03-17 10:54:13,775] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step20000/bf16_zero_pp_rank_33_mp_rank_00_optim_states.pt... 4: [2023-03-17 10:54:13,775] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step20000/bf16_zero_pp_rank_39_mp_rank_00_optim_states.pt... 7: [2023-03-17 10:54:13,775] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step20000/bf16_zero_pp_rank_62_mp_rank_00_optim_states.pt... 7: [2023-03-17 10:54:13,775] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step20000/bf16_zero_pp_rank_56_mp_rank_00_optim_states.pt... 7: [2023-03-17 10:54:13,775] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step20000/bf16_zero_pp_rank_60_mp_rank_00_optim_states.pt... 7: [2023-03-17 10:54:13,775] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step20000/bf16_zero_pp_rank_61_mp_rank_00_optim_states.pt... 7: [2023-03-17 10:54:13,775] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step20000/bf16_zero_pp_rank_63_mp_rank_00_optim_states.pt... 5: [2023-03-17 10:54:13,775] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step20000/bf16_zero_pp_rank_46_mp_rank_00_optim_states.pt... 5: [2023-03-17 10:54:13,775] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step20000/bf16_zero_pp_rank_45_mp_rank_00_optim_states.pt... 5: [2023-03-17 10:54:13,775] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step20000/bf16_zero_pp_rank_41_mp_rank_00_optim_states.pt... 5: [2023-03-17 10:54:13,775] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step20000/bf16_zero_pp_rank_43_mp_rank_00_optim_states.pt... 5: [2023-03-17 10:54:13,775] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step20000/bf16_zero_pp_rank_44_mp_rank_00_optim_states.pt... 1: [2023-03-17 10:54:13,775] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step20000/bf16_zero_pp_rank_8_mp_rank_00_optim_states.pt... 1: [2023-03-17 10:54:13,775] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step20000/bf16_zero_pp_rank_13_mp_rank_00_optim_states.pt... 1: [2023-03-17 10:54:13,775] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step20000/bf16_zero_pp_rank_9_mp_rank_00_optim_states.pt... 1: [2023-03-17 10:54:13,775] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step20000/bf16_zero_pp_rank_12_mp_rank_00_optim_states.pt... 1: [2023-03-17 10:54:13,775] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step20000/bf16_zero_pp_rank_14_mp_rank_00_optim_states.pt... 0: [2023-03-17 10:54:13,775] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step20000/bf16_zero_pp_rank_7_mp_rank_00_optim_states.pt... 3: [2023-03-17 10:54:13,775] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step20000/bf16_zero_pp_rank_26_mp_rank_00_optim_states.pt... 3: [2023-03-17 10:54:13,775] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step20000/bf16_zero_pp_rank_25_mp_rank_00_optim_states.pt... 3: [2023-03-17 10:54:13,775] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step20000/bf16_zero_pp_rank_27_mp_rank_00_optim_states.pt... 2: [2023-03-17 10:54:13,775] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step20000/bf16_zero_pp_rank_16_mp_rank_00_optim_states.pt... 2: [2023-03-17 10:54:13,775] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step20000/bf16_zero_pp_rank_23_mp_rank_00_optim_states.pt... 2: [2023-03-17 10:54:13,775] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step20000/bf16_zero_pp_rank_19_mp_rank_00_optim_states.pt... 2: [2023-03-17 10:54:13,775] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step20000/bf16_zero_pp_rank_22_mp_rank_00_optim_states.pt... 2: [2023-03-17 10:54:13,775] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step20000/bf16_zero_pp_rank_17_mp_rank_00_optim_states.pt... 4: [2023-03-17 10:54:13,775] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step20000/bf16_zero_pp_rank_35_mp_rank_00_optim_states.pt... 4: [2023-03-17 10:54:13,775] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step20000/bf16_zero_pp_rank_36_mp_rank_00_optim_states.pt... 4: [2023-03-17 10:54:13,775] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step20000/bf16_zero_pp_rank_34_mp_rank_00_optim_states.pt... 6: [2023-03-17 10:54:13,775] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step20000/bf16_zero_pp_rank_49_mp_rank_00_optim_states.pt... 6: [2023-03-17 10:54:13,775] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step20000/bf16_zero_pp_rank_51_mp_rank_00_optim_states.pt... 6: [2023-03-17 10:54:13,775] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step20000/bf16_zero_pp_rank_48_mp_rank_00_optim_states.pt... 6: [2023-03-17 10:54:13,775] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step20000/bf16_zero_pp_rank_53_mp_rank_00_optim_states.pt... 6: [2023-03-17 10:54:13,775] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step20000/bf16_zero_pp_rank_54_mp_rank_00_optim_states.pt... 5: [2023-03-17 10:54:13,775] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step20000/bf16_zero_pp_rank_40_mp_rank_00_optim_states.pt... 5: [2023-03-17 10:54:13,775] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step20000/bf16_zero_pp_rank_47_mp_rank_00_optim_states.pt... 5: [2023-03-17 10:54:13,775] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step20000/bf16_zero_pp_rank_42_mp_rank_00_optim_states.pt... 1: [2023-03-17 10:54:13,775] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step20000/bf16_zero_pp_rank_15_mp_rank_00_optim_states.pt... 2: [2023-03-17 10:54:13,775] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step20000/bf16_zero_pp_rank_21_mp_rank_00_optim_states.pt... 2: [2023-03-17 10:54:13,775] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step20000/bf16_zero_pp_rank_20_mp_rank_00_optim_states.pt... 6: [2023-03-17 10:54:13,775] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step20000/bf16_zero_pp_rank_50_mp_rank_00_optim_states.pt... 6: [2023-03-17 10:54:13,775] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step20000/bf16_zero_pp_rank_55_mp_rank_00_optim_states.pt... 6: [2023-03-17 10:54:13,775] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step20000/bf16_zero_pp_rank_52_mp_rank_00_optim_states.pt... 7: [2023-03-17 10:54:13,775] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step20000/bf16_zero_pp_rank_57_mp_rank_00_optim_states.pt... 7: [2023-03-17 10:54:13,775] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step20000/bf16_zero_pp_rank_59_mp_rank_00_optim_states.pt... 7: [2023-03-17 10:54:13,775] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step20000/bf16_zero_pp_rank_58_mp_rank_00_optim_states.pt... 1: [2023-03-17 10:54:13,775] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step20000/bf16_zero_pp_rank_10_mp_rank_00_optim_states.pt... 1: [2023-03-17 10:54:13,775] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step20000/bf16_zero_pp_rank_11_mp_rank_00_optim_states.pt... 2: [2023-03-17 10:54:13,775] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step20000/bf16_zero_pp_rank_18_mp_rank_00_optim_states.pt... 0: [2023-03-17 10:54:13,788] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step20000/bf16_zero_pp_rank_0_mp_rank_00_optim_states.pt. 2: [2023-03-17 10:54:13,789] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step20000/bf16_zero_pp_rank_16_mp_rank_00_optim_states.pt. 2: [2023-03-17 10:54:13,789] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step20000/bf16_zero_pp_rank_16_mp_rank_00_optim_states.pt 6: [2023-03-17 10:54:13,789] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step20000/bf16_zero_pp_rank_49_mp_rank_00_optim_states.pt. 2: [2023-03-17 10:54:13,789] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step20000 is ready now! 6: [2023-03-17 10:54:13,789] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step20000/bf16_zero_pp_rank_49_mp_rank_00_optim_states.pt 6: [2023-03-17 10:54:13,789] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step20000 is ready now! 1: [2023-03-17 10:54:13,789] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step20000/bf16_zero_pp_rank_8_mp_rank_00_optim_states.pt. 1: [2023-03-17 10:54:13,789] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step20000/bf16_zero_pp_rank_8_mp_rank_00_optim_states.pt 1: [2023-03-17 10:54:13,789] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step20000 is ready now! 7: [2023-03-17 10:54:13,789] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step20000/bf16_zero_pp_rank_62_mp_rank_00_optim_states.pt. 7: [2023-03-17 10:54:13,789] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step20000/bf16_zero_pp_rank_56_mp_rank_00_optim_states.pt. 7: [2023-03-17 10:54:13,789] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step20000/bf16_zero_pp_rank_62_mp_rank_00_optim_states.pt 4: [2023-03-17 10:54:13,789] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step20000/bf16_zero_pp_rank_38_mp_rank_00_optim_states.pt. 7: [2023-03-17 10:54:13,789] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step20000/bf16_zero_pp_rank_56_mp_rank_00_optim_states.pt 7: [2023-03-17 10:54:13,789] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step20000 is ready now! 7: [2023-03-17 10:54:13,789] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step20000 is ready now! 4: [2023-03-17 10:54:13,789] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step20000/bf16_zero_pp_rank_38_mp_rank_00_optim_states.pt 4: [2023-03-17 10:54:13,789] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step20000 is ready now! 0: [2023-03-17 10:54:13,790] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step20000/bf16_zero_pp_rank_0_mp_rank_00_optim_states.pt 0: [2023-03-17 10:54:13,790] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step20000 is ready now! 3: [2023-03-17 10:54:13,790] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step20000/bf16_zero_pp_rank_30_mp_rank_00_optim_states.pt. 3: [2023-03-17 10:54:13,790] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step20000/bf16_zero_pp_rank_30_mp_rank_00_optim_states.pt 3: [2023-03-17 10:54:13,790] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step20000 is ready now! 5: [2023-03-17 10:54:13,790] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step20000/bf16_zero_pp_rank_46_mp_rank_00_optim_states.pt. 5: [2023-03-17 10:54:13,790] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step20000/bf16_zero_pp_rank_46_mp_rank_00_optim_states.pt 5: [2023-03-17 10:54:13,790] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step20000 is ready now! 2: [2023-03-17 10:54:13,790] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step20000/bf16_zero_pp_rank_23_mp_rank_00_optim_states.pt. 1: [2023-03-17 10:54:13,790] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step20000/bf16_zero_pp_rank_9_mp_rank_00_optim_states.pt. 2: [2023-03-17 10:54:13,790] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step20000/bf16_zero_pp_rank_23_mp_rank_00_optim_states.pt 1: [2023-03-17 10:54:13,790] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step20000/bf16_zero_pp_rank_9_mp_rank_00_optim_states.pt 1: [2023-03-17 10:54:13,790] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step20000 is ready now! 2: [2023-03-17 10:54:13,790] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step20000 is ready now! 4: [2023-03-17 10:54:13,790] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step20000/bf16_zero_pp_rank_32_mp_rank_00_optim_states.pt. 4: [2023-03-17 10:54:13,790] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step20000/bf16_zero_pp_rank_32_mp_rank_00_optim_states.pt 0: [2023-03-17 10:54:13,790] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step20000/bf16_zero_pp_rank_1_mp_rank_00_optim_states.pt. 4: [2023-03-17 10:54:13,791] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step20000 is ready now! 0: [2023-03-17 10:54:13,791] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step20000/bf16_zero_pp_rank_1_mp_rank_00_optim_states.pt 5: [2023-03-17 10:54:13,791] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step20000/bf16_zero_pp_rank_45_mp_rank_00_optim_states.pt. 0: [2023-03-17 10:54:13,791] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step20000 is ready now! 5: [2023-03-17 10:54:13,791] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step20000/bf16_zero_pp_rank_45_mp_rank_00_optim_states.pt 5: [2023-03-17 10:54:13,791] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step20000 is ready now! 6: [2023-03-17 10:54:13,791] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step20000/bf16_zero_pp_rank_53_mp_rank_00_optim_states.pt. 1: [2023-03-17 10:54:13,791] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step20000/bf16_zero_pp_rank_13_mp_rank_00_optim_states.pt. 6: [2023-03-17 10:54:13,791] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step20000/bf16_zero_pp_rank_53_mp_rank_00_optim_states.pt 6: [2023-03-17 10:54:13,791] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step20000 is ready now! 1: [2023-03-17 10:54:13,791] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step20000/bf16_zero_pp_rank_13_mp_rank_00_optim_states.pt 6: [2023-03-17 10:54:13,791] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step20000/bf16_zero_pp_rank_51_mp_rank_00_optim_states.pt. 1: [2023-03-17 10:54:13,791] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step20000 is ready now! 6: [2023-03-17 10:54:13,791] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step20000/bf16_zero_pp_rank_51_mp_rank_00_optim_states.pt 6: [2023-03-17 10:54:13,791] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step20000 is ready now! 3: [2023-03-17 10:54:13,791] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step20000/bf16_zero_pp_rank_28_mp_rank_00_optim_states.pt. 2: [2023-03-17 10:54:13,791] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step20000/bf16_zero_pp_rank_19_mp_rank_00_optim_states.pt. 3: [2023-03-17 10:54:13,791] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step20000/bf16_zero_pp_rank_28_mp_rank_00_optim_states.pt 2: [2023-03-17 10:54:13,791] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step20000/bf16_zero_pp_rank_19_mp_rank_00_optim_states.pt 3: [2023-03-17 10:54:13,791] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step20000 is ready now! 2: [2023-03-17 10:54:13,791] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step20000 is ready now! 0: [2023-03-17 10:54:13,792] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step20000/bf16_zero_pp_rank_3_mp_rank_00_optim_states.pt. 0: [2023-03-17 10:54:13,792] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step20000/bf16_zero_pp_rank_3_mp_rank_00_optim_states.pt 0: [2023-03-17 10:54:13,792] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step20000 is ready now! 5: [2023-03-17 10:54:13,792] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step20000/bf16_zero_pp_rank_41_mp_rank_00_optim_states.pt. 5: [2023-03-17 10:54:13,792] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step20000/bf16_zero_pp_rank_41_mp_rank_00_optim_states.pt 5: [2023-03-17 10:54:13,792] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step20000 is ready now! 4: [2023-03-17 10:54:13,792] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step20000/bf16_zero_pp_rank_37_mp_rank_00_optim_states.pt. 1: [2023-03-17 10:54:13,792] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step20000/bf16_zero_pp_rank_12_mp_rank_00_optim_states.pt. 4: [2023-03-17 10:54:13,792] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step20000/bf16_zero_pp_rank_37_mp_rank_00_optim_states.pt 4: [2023-03-17 10:54:13,792] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step20000 is ready now! 1: [2023-03-17 10:54:13,792] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step20000/bf16_zero_pp_rank_12_mp_rank_00_optim_states.pt 1: [2023-03-17 10:54:13,792] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step20000 is ready now! 3: [2023-03-17 10:54:13,793] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step20000/bf16_zero_pp_rank_24_mp_rank_00_optim_states.pt. 3: [2023-03-17 10:54:13,793] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step20000/bf16_zero_pp_rank_24_mp_rank_00_optim_states.pt 3: [2023-03-17 10:54:13,793] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step20000 is ready now! 2: [2023-03-17 10:54:13,793] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step20000/bf16_zero_pp_rank_22_mp_rank_00_optim_states.pt. 2: [2023-03-17 10:54:13,793] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step20000/bf16_zero_pp_rank_22_mp_rank_00_optim_states.pt 2: [2023-03-17 10:54:13,793] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step20000 is ready now! 7: [2023-03-17 10:54:13,793] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step20000/bf16_zero_pp_rank_57_mp_rank_00_optim_states.pt. 7: [2023-03-17 10:54:13,793] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step20000/bf16_zero_pp_rank_57_mp_rank_00_optim_states.pt 7: [2023-03-17 10:54:13,793] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step20000 is ready now! 7: [2023-03-17 10:54:13,793] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step20000/bf16_zero_pp_rank_63_mp_rank_00_optim_states.pt. 7: [2023-03-17 10:54:13,793] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step20000/bf16_zero_pp_rank_63_mp_rank_00_optim_states.pt 7: [2023-03-17 10:54:13,793] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step20000 is ready now! 5: [2023-03-17 10:54:13,793] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step20000/bf16_zero_pp_rank_43_mp_rank_00_optim_states.pt. 5: [2023-03-17 10:54:13,794] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step20000/bf16_zero_pp_rank_43_mp_rank_00_optim_states.pt 5: [2023-03-17 10:54:13,794] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step20000 is ready now! 0: [2023-03-17 10:54:13,794] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step20000/bf16_zero_pp_rank_7_mp_rank_00_optim_states.pt. 0: [2023-03-17 10:54:13,794] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step20000/bf16_zero_pp_rank_7_mp_rank_00_optim_states.pt 0: [2023-03-17 10:54:13,794] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step20000 is ready now! 4: [2023-03-17 10:54:13,794] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step20000/bf16_zero_pp_rank_39_mp_rank_00_optim_states.pt. 1: [2023-03-17 10:54:13,794] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step20000/bf16_zero_pp_rank_11_mp_rank_00_optim_states.pt. 4: [2023-03-17 10:54:13,794] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step20000/bf16_zero_pp_rank_39_mp_rank_00_optim_states.pt 4: [2023-03-17 10:54:13,794] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step20000 is ready now! 1: [2023-03-17 10:54:13,794] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step20000/bf16_zero_pp_rank_11_mp_rank_00_optim_states.pt 6: [2023-03-17 10:54:13,794] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step20000/bf16_zero_pp_rank_54_mp_rank_00_optim_states.pt. 1: [2023-03-17 10:54:13,794] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step20000 is ready now! 6: [2023-03-17 10:54:13,794] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step20000/bf16_zero_pp_rank_54_mp_rank_00_optim_states.pt 6: [2023-03-17 10:54:13,794] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step20000 is ready now! 2: [2023-03-17 10:54:13,794] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step20000/bf16_zero_pp_rank_21_mp_rank_00_optim_states.pt. 7: [2023-03-17 10:54:13,794] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step20000/bf16_zero_pp_rank_60_mp_rank_00_optim_states.pt. 2: [2023-03-17 10:54:13,794] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step20000/bf16_zero_pp_rank_21_mp_rank_00_optim_states.pt 7: [2023-03-17 10:54:13,794] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step20000/bf16_zero_pp_rank_60_mp_rank_00_optim_states.pt 2: [2023-03-17 10:54:13,794] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step20000 is ready now! 7: [2023-03-17 10:54:13,794] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step20000 is ready now! 0: [2023-03-17 10:54:13,794] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step20000/bf16_zero_pp_rank_6_mp_rank_00_optim_states.pt. 3: [2023-03-17 10:54:13,794] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step20000/bf16_zero_pp_rank_31_mp_rank_00_optim_states.pt. 0: [2023-03-17 10:54:13,794] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step20000/bf16_zero_pp_rank_6_mp_rank_00_optim_states.pt 3: [2023-03-17 10:54:13,794] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step20000/bf16_zero_pp_rank_31_mp_rank_00_optim_states.pt 0: [2023-03-17 10:54:13,794] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step20000 is ready now! 3: [2023-03-17 10:54:13,794] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step20000 is ready now! 1: [2023-03-17 10:54:13,795] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step20000/bf16_zero_pp_rank_14_mp_rank_00_optim_states.pt. 1: [2023-03-17 10:54:13,795] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step20000/bf16_zero_pp_rank_14_mp_rank_00_optim_states.pt 1: [2023-03-17 10:54:13,795] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step20000 is ready now! 4: [2023-03-17 10:54:13,795] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step20000/bf16_zero_pp_rank_33_mp_rank_00_optim_states.pt. 4: [2023-03-17 10:54:13,795] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step20000/bf16_zero_pp_rank_33_mp_rank_00_optim_states.pt 4: [2023-03-17 10:54:13,795] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step20000 is ready now! 5: [2023-03-17 10:54:13,795] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step20000/bf16_zero_pp_rank_44_mp_rank_00_optim_states.pt. 6: [2023-03-17 10:54:13,795] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step20000/bf16_zero_pp_rank_48_mp_rank_00_optim_states.pt. 5: [2023-03-17 10:54:13,795] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step20000/bf16_zero_pp_rank_44_mp_rank_00_optim_states.pt 6: [2023-03-17 10:54:13,795] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step20000/bf16_zero_pp_rank_48_mp_rank_00_optim_states.pt 5: [2023-03-17 10:54:13,795] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step20000 is ready now! 6: [2023-03-17 10:54:13,795] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step20000 is ready now! 3: [2023-03-17 10:54:13,795] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step20000/bf16_zero_pp_rank_25_mp_rank_00_optim_states.pt. 3: [2023-03-17 10:54:13,795] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step20000/bf16_zero_pp_rank_25_mp_rank_00_optim_states.pt 3: [2023-03-17 10:54:13,795] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step20000 is ready now! 2: [2023-03-17 10:54:13,795] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step20000/bf16_zero_pp_rank_20_mp_rank_00_optim_states.pt. 2: [2023-03-17 10:54:13,795] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step20000/bf16_zero_pp_rank_20_mp_rank_00_optim_states.pt 2: [2023-03-17 10:54:13,795] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step20000 is ready now! 0: [2023-03-17 10:54:13,796] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step20000/bf16_zero_pp_rank_2_mp_rank_00_optim_states.pt. 0: [2023-03-17 10:54:13,796] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step20000/bf16_zero_pp_rank_2_mp_rank_00_optim_states.pt 1: [2023-03-17 10:54:13,796] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step20000/bf16_zero_pp_rank_15_mp_rank_00_optim_states.pt. 5: [2023-03-17 10:54:13,796] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step20000/bf16_zero_pp_rank_40_mp_rank_00_optim_states.pt. 1: [2023-03-17 10:54:13,796] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step20000/bf16_zero_pp_rank_15_mp_rank_00_optim_states.pt 5: [2023-03-17 10:54:13,796] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step20000/bf16_zero_pp_rank_40_mp_rank_00_optim_states.pt 1: [2023-03-17 10:54:13,796] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step20000 is ready now! 0: [2023-03-17 10:54:13,796] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step20000 is ready now! 5: [2023-03-17 10:54:13,796] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step20000 is ready now! 4: [2023-03-17 10:54:13,796] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step20000/bf16_zero_pp_rank_36_mp_rank_00_optim_states.pt. 4: [2023-03-17 10:54:13,796] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step20000/bf16_zero_pp_rank_36_mp_rank_00_optim_states.pt 4: [2023-03-17 10:54:13,797] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step20000 is ready now! 6: [2023-03-17 10:54:13,797] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step20000/bf16_zero_pp_rank_55_mp_rank_00_optim_states.pt. 6: [2023-03-17 10:54:13,797] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step20000/bf16_zero_pp_rank_55_mp_rank_00_optim_states.pt 6: [2023-03-17 10:54:13,797] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step20000 is ready now! 0: [2023-03-17 10:54:13,797] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step20000/bf16_zero_pp_rank_5_mp_rank_00_optim_states.pt. 0: [2023-03-17 10:54:13,797] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step20000/bf16_zero_pp_rank_5_mp_rank_00_optim_states.pt 0: [2023-03-17 10:54:13,797] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step20000 is ready now! 2: [2023-03-17 10:54:13,797] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step20000/bf16_zero_pp_rank_17_mp_rank_00_optim_states.pt. 4: [2023-03-17 10:54:13,797] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step20000/bf16_zero_pp_rank_35_mp_rank_00_optim_states.pt. 2: [2023-03-17 10:54:13,797] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step20000/bf16_zero_pp_rank_17_mp_rank_00_optim_states.pt 4: [2023-03-17 10:54:13,797] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step20000/bf16_zero_pp_rank_35_mp_rank_00_optim_states.pt 2: [2023-03-17 10:54:13,797] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step20000 is ready now! 4: [2023-03-17 10:54:13,797] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step20000 is ready now! 3: [2023-03-17 10:54:13,797] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step20000/bf16_zero_pp_rank_26_mp_rank_00_optim_states.pt. 3: [2023-03-17 10:54:13,797] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step20000/bf16_zero_pp_rank_26_mp_rank_00_optim_states.pt 7: [2023-03-17 10:54:13,797] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step20000/bf16_zero_pp_rank_61_mp_rank_00_optim_states.pt. 3: [2023-03-17 10:54:13,797] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step20000 is ready now! 7: [2023-03-17 10:54:13,797] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step20000/bf16_zero_pp_rank_58_mp_rank_00_optim_states.pt. 7: [2023-03-17 10:54:13,797] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step20000/bf16_zero_pp_rank_61_mp_rank_00_optim_states.pt 7: [2023-03-17 10:54:13,797] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step20000/bf16_zero_pp_rank_58_mp_rank_00_optim_states.pt 7: [2023-03-17 10:54:13,797] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step20000 is ready now! 7: [2023-03-17 10:54:13,797] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step20000 is ready now! 0: [2023-03-17 10:54:13,797] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step20000/bf16_zero_pp_rank_4_mp_rank_00_optim_states.pt. 1: [2023-03-17 10:54:13,798] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step20000/bf16_zero_pp_rank_10_mp_rank_00_optim_states.pt. 0: [2023-03-17 10:54:13,798] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step20000/bf16_zero_pp_rank_4_mp_rank_00_optim_states.pt 0: [2023-03-17 10:54:13,798] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step20000 is ready now! 1: [2023-03-17 10:54:13,798] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step20000/bf16_zero_pp_rank_10_mp_rank_00_optim_states.pt 1: [2023-03-17 10:54:13,798] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step20000 is ready now! 7: [2023-03-17 10:54:13,798] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step20000/bf16_zero_pp_rank_59_mp_rank_00_optim_states.pt. 4: [2023-03-17 10:54:13,798] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step20000/bf16_zero_pp_rank_34_mp_rank_00_optim_states.pt. 7: [2023-03-17 10:54:13,798] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step20000/bf16_zero_pp_rank_59_mp_rank_00_optim_states.pt 5: [2023-03-17 10:54:13,798] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step20000/bf16_zero_pp_rank_47_mp_rank_00_optim_states.pt. 7: [2023-03-17 10:54:13,798] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step20000 is ready now! 4: [2023-03-17 10:54:13,798] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step20000/bf16_zero_pp_rank_34_mp_rank_00_optim_states.pt 5: [2023-03-17 10:54:13,798] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step20000/bf16_zero_pp_rank_47_mp_rank_00_optim_states.pt 4: [2023-03-17 10:54:13,798] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step20000 is ready now! 5: [2023-03-17 10:54:13,798] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step20000 is ready now! 5: [2023-03-17 10:54:13,798] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step20000/bf16_zero_pp_rank_42_mp_rank_00_optim_states.pt. 2: [2023-03-17 10:54:13,798] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step20000/bf16_zero_pp_rank_18_mp_rank_00_optim_states.pt. 5: [2023-03-17 10:54:13,798] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step20000/bf16_zero_pp_rank_42_mp_rank_00_optim_states.pt 2: [2023-03-17 10:54:13,798] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step20000/bf16_zero_pp_rank_18_mp_rank_00_optim_states.pt 5: [2023-03-17 10:54:13,798] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step20000 is ready now! 2: [2023-03-17 10:54:13,798] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step20000 is ready now! 3: [2023-03-17 10:54:13,798] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step20000/bf16_zero_pp_rank_29_mp_rank_00_optim_states.pt. 3: [2023-03-17 10:54:13,798] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step20000/bf16_zero_pp_rank_27_mp_rank_00_optim_states.pt. 3: [2023-03-17 10:54:13,798] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step20000/bf16_zero_pp_rank_29_mp_rank_00_optim_states.pt 3: [2023-03-17 10:54:13,798] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step20000 is ready now! 3: [2023-03-17 10:54:13,798] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step20000/bf16_zero_pp_rank_27_mp_rank_00_optim_states.pt 3: [2023-03-17 10:54:13,798] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step20000 is ready now! 6: [2023-03-17 10:54:13,799] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step20000/bf16_zero_pp_rank_50_mp_rank_00_optim_states.pt. 6: [2023-03-17 10:54:13,799] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step20000/bf16_zero_pp_rank_52_mp_rank_00_optim_states.pt. 6: [2023-03-17 10:54:13,799] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step20000/bf16_zero_pp_rank_50_mp_rank_00_optim_states.pt 6: [2023-03-17 10:54:13,799] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step20000/bf16_zero_pp_rank_52_mp_rank_00_optim_states.pt 6: [2023-03-17 10:54:13,799] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step20000 is ready now! 6: [2023-03-17 10:54:13,799] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step20000 is ready now! 0: successfully saved checkpoint at iteration 20000 to checkpoints_44m174b100m 7: time (ms) | save-checkpoint: 179.71 7: iteration 20100/ 331103 | consumed samples: 5145600 | consumed tokens: 10538188800 | elapsed time per iteration (s): 0.16 | learning rate: 1.988E-04 | global batch size: 256 | lm loss: 3.886956E+00 | grad norm: 0.319 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1614.233 | TFLOPs: 25.32 | 7: iteration 20200/ 331103 | consumed samples: 5171200 | consumed tokens: 10590617600 | elapsed time per iteration (s): 0.15 | learning rate: 1.988E-04 | global batch size: 256 | lm loss: 3.889131E+00 | grad norm: 0.368 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1655.285 | TFLOPs: 25.96 | 7: iteration 20300/ 331103 | consumed samples: 5196800 | consumed tokens: 10643046400 | elapsed time per iteration (s): 0.16 | learning rate: 1.988E-04 | global batch size: 256 | lm loss: 3.884304E+00 | grad norm: 0.267 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1645.802 | TFLOPs: 25.81 | 7: iteration 20400/ 331103 | consumed samples: 5222400 | consumed tokens: 10695475200 | elapsed time per iteration (s): 0.16 | learning rate: 1.988E-04 | global batch size: 256 | lm loss: 3.884441E+00 | grad norm: 0.328 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1651.173 | TFLOPs: 25.89 | 7: iteration 20500/ 331103 | consumed samples: 5248000 | consumed tokens: 10747904000 | elapsed time per iteration (s): 0.16 | learning rate: 1.988E-04 | global batch size: 256 | lm loss: 3.882257E+00 | grad norm: 0.326 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1636.852 | TFLOPs: 25.67 | 7: iteration 20600/ 331103 | consumed samples: 5273600 | consumed tokens: 10800332800 | elapsed time per iteration (s): 0.15 | learning rate: 1.988E-04 | global batch size: 256 | lm loss: 3.878691E+00 | grad norm: 0.373 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1651.640 | TFLOPs: 25.90 | 7: iteration 20700/ 331103 | consumed samples: 5299200 | consumed tokens: 10852761600 | elapsed time per iteration (s): 0.16 | learning rate: 1.988E-04 | global batch size: 256 | lm loss: 3.882394E+00 | grad norm: 0.324 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1624.649 | TFLOPs: 25.48 | 7: iteration 20800/ 331103 | consumed samples: 5324800 | consumed tokens: 10905190400 | elapsed time per iteration (s): 0.16 | learning rate: 1.987E-04 | global batch size: 256 | lm loss: 3.877575E+00 | grad norm: 0.307 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1650.615 | TFLOPs: 25.89 | 7: iteration 20900/ 331103 | consumed samples: 5350400 | consumed tokens: 10957619200 | elapsed time per iteration (s): 0.15 | learning rate: 1.987E-04 | global batch size: 256 | lm loss: 3.876461E+00 | grad norm: 0.324 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1654.151 | TFLOPs: 25.94 | 7: iteration 21000/ 331103 | consumed samples: 5376000 | consumed tokens: 11010048000 | elapsed time per iteration (s): 0.16 | learning rate: 1.987E-04 | global batch size: 256 | lm loss: 3.877610E+00 | grad norm: 0.410 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1639.305 | TFLOPs: 25.71 | 7: iteration 21100/ 331103 | consumed samples: 5401600 | consumed tokens: 11062476800 | elapsed time per iteration (s): 0.16 | learning rate: 1.987E-04 | global batch size: 256 | lm loss: 3.883143E+00 | grad norm: 0.349 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1640.637 | TFLOPs: 25.73 | 7: iteration 21200/ 331103 | consumed samples: 5427200 | consumed tokens: 11114905600 | elapsed time per iteration (s): 0.15 | learning rate: 1.987E-04 | global batch size: 256 | lm loss: 3.878052E+00 | grad norm: 0.364 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1661.189 | TFLOPs: 26.05 | 7: iteration 21300/ 331103 | consumed samples: 5452800 | consumed tokens: 11167334400 | elapsed time per iteration (s): 0.15 | learning rate: 1.987E-04 | global batch size: 256 | lm loss: 3.876416E+00 | grad norm: 0.308 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1663.135 | TFLOPs: 26.08 | 7: iteration 21400/ 331103 | consumed samples: 5478400 | consumed tokens: 11219763200 | elapsed time per iteration (s): 0.16 | learning rate: 1.987E-04 | global batch size: 256 | lm loss: 3.877230E+00 | grad norm: 0.406 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1642.431 | TFLOPs: 25.76 | 7: iteration 21500/ 331103 | consumed samples: 5504000 | consumed tokens: 11272192000 | elapsed time per iteration (s): 0.16 | learning rate: 1.986E-04 | global batch size: 256 | lm loss: 3.875130E+00 | grad norm: 0.305 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1650.763 | TFLOPs: 25.89 | 7: iteration 21600/ 331103 | consumed samples: 5529600 | consumed tokens: 11324620800 | elapsed time per iteration (s): 0.16 | learning rate: 1.986E-04 | global batch size: 256 | lm loss: 3.873380E+00 | grad norm: 0.325 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1632.756 | TFLOPs: 25.61 | 7: iteration 21700/ 331103 | consumed samples: 5555200 | consumed tokens: 11377049600 | elapsed time per iteration (s): 0.15 | learning rate: 1.986E-04 | global batch size: 256 | lm loss: 3.870882E+00 | grad norm: 0.432 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1656.850 | TFLOPs: 25.98 | 7: iteration 21800/ 331103 | consumed samples: 5580800 | consumed tokens: 11429478400 | elapsed time per iteration (s): 0.16 | learning rate: 1.986E-04 | global batch size: 256 | lm loss: 3.872178E+00 | grad norm: 0.333 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1635.608 | TFLOPs: 25.65 | 7: iteration 21900/ 331103 | consumed samples: 5606400 | consumed tokens: 11481907200 | elapsed time per iteration (s): 0.16 | learning rate: 1.986E-04 | global batch size: 256 | lm loss: 3.869696E+00 | grad norm: 0.341 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1647.972 | TFLOPs: 25.84 | 0: [2023-03-17 10:59:24,664] [INFO] [logging.py:68:log_dist] [Rank 0] step=22000, skipped=0, lr=[0.0001985601314419097, 0.0001985601314419097, 0.0001985601314419097], mom=[(0.9, 0.999), (0.9, 0.999), (0.9, 0.999)] 7: iteration 22000/ 331103 | consumed samples: 5632000 | consumed tokens: 11534336000 | elapsed time per iteration (s): 0.15 | learning rate: 1.986E-04 | global batch size: 256 | lm loss: 3.871158E+00 | grad norm: 0.361 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1660.817 | TFLOPs: 26.05 | 0: steps: 22000 loss: 3.8518 iter time (s): 0.154 samples/sec: 1663.370 7: iteration 22100/ 331103 | consumed samples: 5657600 | consumed tokens: 11586764800 | elapsed time per iteration (s): 0.16 | learning rate: 1.985E-04 | global batch size: 256 | lm loss: 3.869656E+00 | grad norm: 0.299 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1637.878 | TFLOPs: 25.69 | 7: iteration 22200/ 331103 | consumed samples: 5683200 | consumed tokens: 11639193600 | elapsed time per iteration (s): 0.15 | learning rate: 1.985E-04 | global batch size: 256 | lm loss: 3.866912E+00 | grad norm: 0.314 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1663.390 | TFLOPs: 26.09 | 7: iteration 22300/ 331103 | consumed samples: 5708800 | consumed tokens: 11691622400 | elapsed time per iteration (s): 0.15 | learning rate: 1.985E-04 | global batch size: 256 | lm loss: 3.864391E+00 | grad norm: 0.324 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1663.787 | TFLOPs: 26.09 | 7: iteration 22400/ 331103 | consumed samples: 5734400 | consumed tokens: 11744051200 | elapsed time per iteration (s): 0.15 | learning rate: 1.985E-04 | global batch size: 256 | lm loss: 3.866714E+00 | grad norm: 0.334 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1657.595 | TFLOPs: 26.00 | 7: iteration 22500/ 331103 | consumed samples: 5760000 | consumed tokens: 11796480000 | elapsed time per iteration (s): 0.16 | learning rate: 1.985E-04 | global batch size: 256 | lm loss: 3.870866E+00 | grad norm: 0.380 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1643.259 | TFLOPs: 25.77 | 7: iteration 22600/ 331103 | consumed samples: 5785600 | consumed tokens: 11848908800 | elapsed time per iteration (s): 0.16 | learning rate: 1.985E-04 | global batch size: 256 | lm loss: 3.867113E+00 | grad norm: 0.356 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1648.073 | TFLOPs: 25.85 | 7: iteration 22700/ 331103 | consumed samples: 5811200 | consumed tokens: 11901337600 | elapsed time per iteration (s): 0.15 | learning rate: 1.985E-04 | global batch size: 256 | lm loss: 3.866344E+00 | grad norm: 0.378 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1654.304 | TFLOPs: 25.94 | 7: iteration 22800/ 331103 | consumed samples: 5836800 | consumed tokens: 11953766400 | elapsed time per iteration (s): 0.16 | learning rate: 1.984E-04 | global batch size: 256 | lm loss: 3.863145E+00 | grad norm: 0.297 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1645.004 | TFLOPs: 25.80 | 7: iteration 22900/ 331103 | consumed samples: 5862400 | consumed tokens: 12006195200 | elapsed time per iteration (s): 0.15 | learning rate: 1.984E-04 | global batch size: 256 | lm loss: 3.862398E+00 | grad norm: 0.397 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1653.625 | TFLOPs: 25.93 | 7: iteration 23000/ 331103 | consumed samples: 5888000 | consumed tokens: 12058624000 | elapsed time per iteration (s): 0.16 | learning rate: 1.984E-04 | global batch size: 256 | lm loss: 3.865450E+00 | grad norm: 0.345 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1646.924 | TFLOPs: 25.83 | 7: iteration 23100/ 331103 | consumed samples: 5913600 | consumed tokens: 12111052800 | elapsed time per iteration (s): 0.15 | learning rate: 1.984E-04 | global batch size: 256 | lm loss: 3.862246E+00 | grad norm: 0.376 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1670.372 | TFLOPs: 26.20 | 7: iteration 23200/ 331103 | consumed samples: 5939200 | consumed tokens: 12163481600 | elapsed time per iteration (s): 0.16 | learning rate: 1.984E-04 | global batch size: 256 | lm loss: 3.862797E+00 | grad norm: 0.359 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1628.813 | TFLOPs: 25.54 | 7: iteration 23300/ 331103 | consumed samples: 5964800 | consumed tokens: 12215910400 | elapsed time per iteration (s): 0.16 | learning rate: 1.984E-04 | global batch size: 256 | lm loss: 3.864138E+00 | grad norm: 0.341 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1597.886 | TFLOPs: 25.06 | 7: iteration 23400/ 331103 | consumed samples: 5990400 | consumed tokens: 12268339200 | elapsed time per iteration (s): 0.15 | learning rate: 1.983E-04 | global batch size: 256 | lm loss: 3.859944E+00 | grad norm: 0.324 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1659.894 | TFLOPs: 26.03 | 7: iteration 23500/ 331103 | consumed samples: 6016000 | consumed tokens: 12320768000 | elapsed time per iteration (s): 0.16 | learning rate: 1.983E-04 | global batch size: 256 | lm loss: 3.858650E+00 | grad norm: 0.300 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1628.601 | TFLOPs: 25.54 | 7: iteration 23600/ 331103 | consumed samples: 6041600 | consumed tokens: 12373196800 | elapsed time per iteration (s): 0.16 | learning rate: 1.983E-04 | global batch size: 256 | lm loss: 3.857465E+00 | grad norm: 0.355 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1616.480 | TFLOPs: 25.35 | 7: iteration 23700/ 331103 | consumed samples: 6067200 | consumed tokens: 12425625600 | elapsed time per iteration (s): 0.15 | learning rate: 1.983E-04 | global batch size: 256 | lm loss: 3.853207E+00 | grad norm: 0.360 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1654.213 | TFLOPs: 25.94 | 7: iteration 23800/ 331103 | consumed samples: 6092800 | consumed tokens: 12478054400 | elapsed time per iteration (s): 0.15 | learning rate: 1.983E-04 | global batch size: 256 | lm loss: 3.858434E+00 | grad norm: 0.296 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1657.578 | TFLOPs: 25.99 | 7: iteration 23900/ 331103 | consumed samples: 6118400 | consumed tokens: 12530483200 | elapsed time per iteration (s): 0.17 | learning rate: 1.983E-04 | global batch size: 256 | lm loss: 3.859976E+00 | grad norm: 0.286 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1544.583 | TFLOPs: 24.22 | 0: [2023-03-17 11:04:37,130] [INFO] [logging.py:68:log_dist] [Rank 0] step=24000, skipped=0, lr=[0.0001982365306699686, 0.0001982365306699686, 0.0001982365306699686], mom=[(0.9, 0.999), (0.9, 0.999), (0.9, 0.999)] 7: iteration 24000/ 331103 | consumed samples: 6144000 | consumed tokens: 12582912000 | elapsed time per iteration (s): 0.16 | learning rate: 1.982E-04 | global batch size: 256 | lm loss: 3.857140E+00 | grad norm: 0.331 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1609.645 | TFLOPs: 25.24 | 0: steps: 24000 loss: 3.8769 iter time (s): 0.155 samples/sec: 1655.696 7: iteration 24100/ 331103 | consumed samples: 6169600 | consumed tokens: 12635340800 | elapsed time per iteration (s): 0.18 | learning rate: 1.982E-04 | global batch size: 256 | lm loss: 3.852972E+00 | grad norm: 0.352 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1452.614 | TFLOPs: 22.78 | 7: iteration 24200/ 331103 | consumed samples: 6195200 | consumed tokens: 12687769600 | elapsed time per iteration (s): 0.19 | learning rate: 1.982E-04 | global batch size: 256 | lm loss: 3.854726E+00 | grad norm: 0.357 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1335.287 | TFLOPs: 20.94 | 7: iteration 24300/ 331103 | consumed samples: 6220800 | consumed tokens: 12740198400 | elapsed time per iteration (s): 0.17 | learning rate: 1.982E-04 | global batch size: 256 | lm loss: 3.858229E+00 | grad norm: 0.325 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1519.872 | TFLOPs: 23.84 | 7: iteration 24400/ 331103 | consumed samples: 6246400 | consumed tokens: 12792627200 | elapsed time per iteration (s): 0.17 | learning rate: 1.982E-04 | global batch size: 256 | lm loss: 3.845603E+00 | grad norm: 0.353 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1547.832 | TFLOPs: 24.27 | 7: iteration 24500/ 331103 | consumed samples: 6272000 | consumed tokens: 12845056000 | elapsed time per iteration (s): 0.16 | learning rate: 1.982E-04 | global batch size: 256 | lm loss: 3.858075E+00 | grad norm: 0.381 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1592.770 | TFLOPs: 24.98 | 7: iteration 24600/ 331103 | consumed samples: 6297600 | consumed tokens: 12897484800 | elapsed time per iteration (s): 0.17 | learning rate: 1.981E-04 | global batch size: 256 | lm loss: 3.850780E+00 | grad norm: 0.349 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1539.142 | TFLOPs: 24.14 | 7: iteration 24700/ 331103 | consumed samples: 6323200 | consumed tokens: 12949913600 | elapsed time per iteration (s): 0.16 | learning rate: 1.981E-04 | global batch size: 256 | lm loss: 3.852273E+00 | grad norm: 0.334 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1602.227 | TFLOPs: 25.13 | 7: iteration 24800/ 331103 | consumed samples: 6348800 | consumed tokens: 13002342400 | elapsed time per iteration (s): 0.16 | learning rate: 1.981E-04 | global batch size: 256 | lm loss: 3.852013E+00 | grad norm: 0.346 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1609.316 | TFLOPs: 25.24 | 7: iteration 24900/ 331103 | consumed samples: 6374400 | consumed tokens: 13054771200 | elapsed time per iteration (s): 0.16 | learning rate: 1.981E-04 | global batch size: 256 | lm loss: 3.852939E+00 | grad norm: 0.334 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1619.596 | TFLOPs: 25.40 | 7: iteration 25000/ 331103 | consumed samples: 6400000 | consumed tokens: 13107200000 | elapsed time per iteration (s): 0.16 | learning rate: 1.981E-04 | global batch size: 256 | lm loss: 3.849902E+00 | grad norm: 0.290 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1651.038 | TFLOPs: 25.89 | 7: iteration 25100/ 331103 | consumed samples: 6425600 | consumed tokens: 13159628800 | elapsed time per iteration (s): 0.16 | learning rate: 1.980E-04 | global batch size: 256 | lm loss: 3.848777E+00 | grad norm: 0.323 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1638.600 | TFLOPs: 25.70 | 7: iteration 25200/ 331103 | consumed samples: 6451200 | consumed tokens: 13212057600 | elapsed time per iteration (s): 0.16 | learning rate: 1.980E-04 | global batch size: 256 | lm loss: 3.846746E+00 | grad norm: 0.304 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1632.732 | TFLOPs: 25.61 | 7: iteration 25300/ 331103 | consumed samples: 6476800 | consumed tokens: 13264486400 | elapsed time per iteration (s): 0.16 | learning rate: 1.980E-04 | global batch size: 256 | lm loss: 3.846705E+00 | grad norm: 0.371 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1633.734 | TFLOPs: 25.62 | 7: iteration 25400/ 331103 | consumed samples: 6502400 | consumed tokens: 13316915200 | elapsed time per iteration (s): 0.16 | learning rate: 1.980E-04 | global batch size: 256 | lm loss: 3.844183E+00 | grad norm: 0.291 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1608.272 | TFLOPs: 25.22 | 7: iteration 25500/ 331103 | consumed samples: 6528000 | consumed tokens: 13369344000 | elapsed time per iteration (s): 0.16 | learning rate: 1.980E-04 | global batch size: 256 | lm loss: 3.846151E+00 | grad norm: 0.289 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1630.810 | TFLOPs: 25.58 | 7: iteration 25600/ 331103 | consumed samples: 6553600 | consumed tokens: 13421772800 | elapsed time per iteration (s): 0.16 | learning rate: 1.980E-04 | global batch size: 256 | lm loss: 3.847375E+00 | grad norm: 0.331 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1610.067 | TFLOPs: 25.25 | 7: iteration 25700/ 331103 | consumed samples: 6579200 | consumed tokens: 13474201600 | elapsed time per iteration (s): 0.16 | learning rate: 1.979E-04 | global batch size: 256 | lm loss: 3.848939E+00 | grad norm: 0.343 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1621.777 | TFLOPs: 25.43 | 7: iteration 25800/ 331103 | consumed samples: 6604800 | consumed tokens: 13526630400 | elapsed time per iteration (s): 0.16 | learning rate: 1.979E-04 | global batch size: 256 | lm loss: 3.850204E+00 | grad norm: 0.405 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1641.878 | TFLOPs: 25.75 | 7: iteration 25900/ 331103 | consumed samples: 6630400 | consumed tokens: 13579059200 | elapsed time per iteration (s): 0.16 | learning rate: 1.979E-04 | global batch size: 256 | lm loss: 3.844893E+00 | grad norm: 0.337 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1649.271 | TFLOPs: 25.86 | 0: [2023-03-17 11:10:00,025] [INFO] [logging.py:68:log_dist] [Rank 0] step=26000, skipped=0, lr=[0.00019788051113926172, 0.00019788051113926172, 0.00019788051113926172], mom=[(0.9, 0.999), (0.9, 0.999), (0.9, 0.999)] 7: iteration 26000/ 331103 | consumed samples: 6656000 | consumed tokens: 13631488000 | elapsed time per iteration (s): 0.15 | learning rate: 1.979E-04 | global batch size: 256 | lm loss: 3.841921E+00 | grad norm: 0.314 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1659.325 | TFLOPs: 26.02 | 0: steps: 26000 loss: 3.8193 iter time (s): 0.160 samples/sec: 1601.323 7: iteration 26100/ 331103 | consumed samples: 6681600 | consumed tokens: 13683916800 | elapsed time per iteration (s): 0.15 | learning rate: 1.979E-04 | global batch size: 256 | lm loss: 3.845394E+00 | grad norm: 0.323 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1662.062 | TFLOPs: 26.07 | 7: iteration 26200/ 331103 | consumed samples: 6707200 | consumed tokens: 13736345600 | elapsed time per iteration (s): 0.16 | learning rate: 1.978E-04 | global batch size: 256 | lm loss: 3.843037E+00 | grad norm: 0.318 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1645.699 | TFLOPs: 25.81 | 7: iteration 26300/ 331103 | consumed samples: 6732800 | consumed tokens: 13788774400 | elapsed time per iteration (s): 0.15 | learning rate: 1.978E-04 | global batch size: 256 | lm loss: 3.843332E+00 | grad norm: 0.330 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1666.237 | TFLOPs: 26.13 | 7: iteration 26400/ 331103 | consumed samples: 6758400 | consumed tokens: 13841203200 | elapsed time per iteration (s): 0.16 | learning rate: 1.978E-04 | global batch size: 256 | lm loss: 3.844610E+00 | grad norm: 0.293 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1644.392 | TFLOPs: 25.79 | 7: iteration 26500/ 331103 | consumed samples: 6784000 | consumed tokens: 13893632000 | elapsed time per iteration (s): 0.15 | learning rate: 1.978E-04 | global batch size: 256 | lm loss: 3.843902E+00 | grad norm: 0.314 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1653.713 | TFLOPs: 25.93 | 7: iteration 26600/ 331103 | consumed samples: 6809600 | consumed tokens: 13946060800 | elapsed time per iteration (s): 0.16 | learning rate: 1.978E-04 | global batch size: 256 | lm loss: 3.844882E+00 | grad norm: 0.317 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1649.948 | TFLOPs: 25.88 | 7: iteration 26700/ 331103 | consumed samples: 6835200 | consumed tokens: 13998489600 | elapsed time per iteration (s): 0.15 | learning rate: 1.977E-04 | global batch size: 256 | lm loss: 3.841587E+00 | grad norm: 0.337 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1662.044 | TFLOPs: 26.07 | 7: iteration 26800/ 331103 | consumed samples: 6860800 | consumed tokens: 14050918400 | elapsed time per iteration (s): 0.15 | learning rate: 1.977E-04 | global batch size: 256 | lm loss: 3.837327E+00 | grad norm: 0.337 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1665.395 | TFLOPs: 26.12 | 7: iteration 26900/ 331103 | consumed samples: 6886400 | consumed tokens: 14103347200 | elapsed time per iteration (s): 0.15 | learning rate: 1.977E-04 | global batch size: 256 | lm loss: 3.842203E+00 | grad norm: 0.304 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1656.907 | TFLOPs: 25.98 | 7: iteration 27000/ 331103 | consumed samples: 6912000 | consumed tokens: 14155776000 | elapsed time per iteration (s): 0.16 | learning rate: 1.977E-04 | global batch size: 256 | lm loss: 3.839190E+00 | grad norm: 0.358 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1633.707 | TFLOPs: 25.62 | 7: iteration 27100/ 331103 | consumed samples: 6937600 | consumed tokens: 14208204800 | elapsed time per iteration (s): 0.15 | learning rate: 1.977E-04 | global batch size: 256 | lm loss: 3.839077E+00 | grad norm: 0.313 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1662.915 | TFLOPs: 26.08 | 7: iteration 27200/ 331103 | consumed samples: 6963200 | consumed tokens: 14260633600 | elapsed time per iteration (s): 0.15 | learning rate: 1.977E-04 | global batch size: 256 | lm loss: 3.839130E+00 | grad norm: 0.331 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1662.005 | TFLOPs: 26.06 | 7: iteration 27300/ 331103 | consumed samples: 6988800 | consumed tokens: 14313062400 | elapsed time per iteration (s): 0.16 | learning rate: 1.976E-04 | global batch size: 256 | lm loss: 3.842966E+00 | grad norm: 0.380 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1641.229 | TFLOPs: 25.74 | 7: iteration 27400/ 331103 | consumed samples: 7014400 | consumed tokens: 14365491200 | elapsed time per iteration (s): 0.15 | learning rate: 1.976E-04 | global batch size: 256 | lm loss: 3.837889E+00 | grad norm: 0.295 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1660.435 | TFLOPs: 26.04 | 7: iteration 27500/ 331103 | consumed samples: 7040000 | consumed tokens: 14417920000 | elapsed time per iteration (s): 0.15 | learning rate: 1.976E-04 | global batch size: 256 | lm loss: 3.837119E+00 | grad norm: 0.314 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1667.135 | TFLOPs: 26.14 | 7: iteration 27600/ 331103 | consumed samples: 7065600 | consumed tokens: 14470348800 | elapsed time per iteration (s): 0.15 | learning rate: 1.976E-04 | global batch size: 256 | lm loss: 3.833174E+00 | grad norm: 0.352 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1663.780 | TFLOPs: 26.09 | 7: iteration 27700/ 331103 | consumed samples: 7091200 | consumed tokens: 14522777600 | elapsed time per iteration (s): 0.15 | learning rate: 1.976E-04 | global batch size: 256 | lm loss: 3.828977E+00 | grad norm: 0.366 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1665.295 | TFLOPs: 26.12 | 7: iteration 27800/ 331103 | consumed samples: 7116800 | consumed tokens: 14575206400 | elapsed time per iteration (s): 0.15 | learning rate: 1.975E-04 | global batch size: 256 | lm loss: 3.832897E+00 | grad norm: 0.273 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1661.838 | TFLOPs: 26.06 | 7: iteration 27900/ 331103 | consumed samples: 7142400 | consumed tokens: 14627635200 | elapsed time per iteration (s): 0.15 | learning rate: 1.975E-04 | global batch size: 256 | lm loss: 3.832972E+00 | grad norm: 0.352 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1652.331 | TFLOPs: 25.91 | 0: [2023-03-17 11:15:09,043] [INFO] [logging.py:68:log_dist] [Rank 0] step=28000, skipped=0, lr=[0.00019749220365401708, 0.00019749220365401708, 0.00019749220365401708], mom=[(0.9, 0.999), (0.9, 0.999), (0.9, 0.999)] 7: iteration 28000/ 331103 | consumed samples: 7168000 | consumed tokens: 14680064000 | elapsed time per iteration (s): 0.15 | learning rate: 1.975E-04 | global batch size: 256 | lm loss: 3.831279E+00 | grad norm: 0.342 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1661.228 | TFLOPs: 26.05 | 0: steps: 28000 loss: 3.7820 iter time (s): 0.153 samples/sec: 1672.229 7: iteration 28100/ 331103 | consumed samples: 7193600 | consumed tokens: 14732492800 | elapsed time per iteration (s): 0.15 | learning rate: 1.975E-04 | global batch size: 256 | lm loss: 3.833808E+00 | grad norm: 0.410 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1654.390 | TFLOPs: 25.94 | 7: iteration 28200/ 331103 | consumed samples: 7219200 | consumed tokens: 14784921600 | elapsed time per iteration (s): 0.15 | learning rate: 1.975E-04 | global batch size: 256 | lm loss: 3.834608E+00 | grad norm: 0.319 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1654.118 | TFLOPs: 25.94 | 7: iteration 28300/ 331103 | consumed samples: 7244800 | consumed tokens: 14837350400 | elapsed time per iteration (s): 0.15 | learning rate: 1.974E-04 | global batch size: 256 | lm loss: 3.831270E+00 | grad norm: 0.388 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1667.415 | TFLOPs: 26.15 | 7: iteration 28400/ 331103 | consumed samples: 7270400 | consumed tokens: 14889779200 | elapsed time per iteration (s): 0.15 | learning rate: 1.974E-04 | global batch size: 256 | lm loss: 3.829955E+00 | grad norm: 0.304 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1668.351 | TFLOPs: 26.16 | 7: iteration 28500/ 331103 | consumed samples: 7296000 | consumed tokens: 14942208000 | elapsed time per iteration (s): 0.15 | learning rate: 1.974E-04 | global batch size: 256 | lm loss: 3.830292E+00 | grad norm: 0.336 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1662.621 | TFLOPs: 26.07 | 7: iteration 28600/ 331103 | consumed samples: 7321600 | consumed tokens: 14994636800 | elapsed time per iteration (s): 0.16 | learning rate: 1.974E-04 | global batch size: 256 | lm loss: 3.832829E+00 | grad norm: 0.319 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1635.011 | TFLOPs: 25.64 | 7: iteration 28700/ 331103 | consumed samples: 7347200 | consumed tokens: 15047065600 | elapsed time per iteration (s): 0.15 | learning rate: 1.973E-04 | global batch size: 256 | lm loss: 3.829835E+00 | grad norm: 0.338 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1668.491 | TFLOPs: 26.17 | 7: iteration 28800/ 331103 | consumed samples: 7372800 | consumed tokens: 15099494400 | elapsed time per iteration (s): 0.15 | learning rate: 1.973E-04 | global batch size: 256 | lm loss: 3.826750E+00 | grad norm: 0.348 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1662.448 | TFLOPs: 26.07 | 7: iteration 28900/ 331103 | consumed samples: 7398400 | consumed tokens: 15151923200 | elapsed time per iteration (s): 0.15 | learning rate: 1.973E-04 | global batch size: 256 | lm loss: 3.825896E+00 | grad norm: 0.333 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1667.979 | TFLOPs: 26.16 | 7: iteration 29000/ 331103 | consumed samples: 7424000 | consumed tokens: 15204352000 | elapsed time per iteration (s): 0.15 | learning rate: 1.973E-04 | global batch size: 256 | lm loss: 3.826927E+00 | grad norm: 0.393 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1665.466 | TFLOPs: 26.12 | 7: iteration 29100/ 331103 | consumed samples: 7449600 | consumed tokens: 15256780800 | elapsed time per iteration (s): 0.15 | learning rate: 1.973E-04 | global batch size: 256 | lm loss: 3.831330E+00 | grad norm: 0.302 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1658.435 | TFLOPs: 26.01 | 7: iteration 29200/ 331103 | consumed samples: 7475200 | consumed tokens: 15309209600 | elapsed time per iteration (s): 0.15 | learning rate: 1.972E-04 | global batch size: 256 | lm loss: 3.825835E+00 | grad norm: 0.329 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1661.528 | TFLOPs: 26.06 | 7: iteration 29300/ 331103 | consumed samples: 7500800 | consumed tokens: 15361638400 | elapsed time per iteration (s): 0.15 | learning rate: 1.972E-04 | global batch size: 256 | lm loss: 3.829061E+00 | grad norm: 0.288 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1666.191 | TFLOPs: 26.13 | 7: iteration 29400/ 331103 | consumed samples: 7526400 | consumed tokens: 15414067200 | elapsed time per iteration (s): 0.15 | learning rate: 1.972E-04 | global batch size: 256 | lm loss: 3.825789E+00 | grad norm: 0.312 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1663.424 | TFLOPs: 26.09 | 7: iteration 29500/ 331103 | consumed samples: 7552000 | consumed tokens: 15466496000 | elapsed time per iteration (s): 0.15 | learning rate: 1.972E-04 | global batch size: 256 | lm loss: 3.825765E+00 | grad norm: 0.335 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1663.290 | TFLOPs: 26.08 | 7: iteration 29600/ 331103 | consumed samples: 7577600 | consumed tokens: 15518924800 | elapsed time per iteration (s): 0.15 | learning rate: 1.972E-04 | global batch size: 256 | lm loss: 3.824774E+00 | grad norm: 0.353 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1660.661 | TFLOPs: 26.04 | 7: iteration 29700/ 331103 | consumed samples: 7603200 | consumed tokens: 15571353600 | elapsed time per iteration (s): 0.15 | learning rate: 1.971E-04 | global batch size: 256 | lm loss: 3.823217E+00 | grad norm: 0.339 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1658.733 | TFLOPs: 26.01 | 7: iteration 29800/ 331103 | consumed samples: 7628800 | consumed tokens: 15623782400 | elapsed time per iteration (s): 0.15 | learning rate: 1.971E-04 | global batch size: 256 | lm loss: 3.827996E+00 | grad norm: 0.279 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1662.442 | TFLOPs: 26.07 | 7: iteration 29900/ 331103 | consumed samples: 7654400 | consumed tokens: 15676211200 | elapsed time per iteration (s): 0.15 | learning rate: 1.971E-04 | global batch size: 256 | lm loss: 3.829593E+00 | grad norm: 0.316 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1663.479 | TFLOPs: 26.09 | 0: [2023-03-17 11:20:17,217] [INFO] [logging.py:68:log_dist] [Rank 0] step=30000, skipped=0, lr=[0.0001970717508812979, 0.0001970717508812979, 0.0001970717508812979], mom=[(0.9, 0.999), (0.9, 0.999), (0.9, 0.999)] 7: iteration 30000/ 331103 | consumed samples: 7680000 | consumed tokens: 15728640000 | elapsed time per iteration (s): 0.15 | learning rate: 1.971E-04 | global batch size: 256 | lm loss: 3.818916E+00 | grad norm: 0.314 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1664.192 | TFLOPs: 26.10 | 0: steps: 30000 loss: 3.7998 iter time (s): 0.153 samples/sec: 1678.471 7: ------------------------------------------------------------------------------------------------ 7: validation loss at iteration 30000 | lm loss value: 3.915038E+00 | lm loss PPL: 5.015099E+01 | 7: ------------------------------------------------------------------------------------------------ 0: saving checkpoint at iteration 30000 to checkpoints_44m174b100m 0: [2023-03-17 11:20:17,311] [INFO] [logging.py:68:log_dist] [Rank 0] [Torch] Checkpoint global_step30000 is begin to save! 0: [2023-03-17 11:20:17,317] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step30000/layer_01-model_00-model_states.pt... 0: [2023-03-17 11:20:17,389] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step30000/layer_01-model_00-model_states.pt. 0: [2023-03-17 11:20:17,390] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step30000/layer_03-model_00-model_states.pt... 0: [2023-03-17 11:20:17,398] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step30000/layer_03-model_00-model_states.pt. 0: [2023-03-17 11:20:17,398] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step30000/layer_04-model_00-model_states.pt... 0: [2023-03-17 11:20:17,407] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step30000/layer_04-model_00-model_states.pt. 0: [2023-03-17 11:20:17,407] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step30000/layer_05-model_00-model_states.pt... 0: [2023-03-17 11:20:17,415] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step30000/layer_05-model_00-model_states.pt. 0: [2023-03-17 11:20:17,415] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step30000/layer_06-model_00-model_states.pt... 0: [2023-03-17 11:20:17,423] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step30000/layer_06-model_00-model_states.pt. 0: [2023-03-17 11:20:17,423] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step30000/layer_07-model_00-model_states.pt... 0: [2023-03-17 11:20:17,431] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step30000/layer_07-model_00-model_states.pt. 0: [2023-03-17 11:20:17,431] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step30000/layer_08-model_00-model_states.pt... 0: [2023-03-17 11:20:17,439] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step30000/layer_08-model_00-model_states.pt. 0: [2023-03-17 11:20:17,439] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step30000/layer_09-model_00-model_states.pt... 0: [2023-03-17 11:20:17,447] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step30000/layer_09-model_00-model_states.pt. 0: [2023-03-17 11:20:17,448] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step30000/layer_10-model_00-model_states.pt... 0: [2023-03-17 11:20:17,456] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step30000/layer_10-model_00-model_states.pt. 0: [2023-03-17 11:20:17,456] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step30000/layer_12-model_00-model_states.pt... 0: [2023-03-17 11:20:17,457] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step30000/layer_12-model_00-model_states.pt. 0: [2023-03-17 11:20:17,457] [INFO] [logging.py:68:log_dist] [Rank 0] Saving model checkpoint: checkpoints_44m174b100m/global_step30000/mp_rank_00_model_states.pt 0: [2023-03-17 11:20:17,457] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step30000/mp_rank_00_model_states.pt... 0: [2023-03-17 11:20:17,459] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step30000/mp_rank_00_model_states.pt. 0: [2023-03-17 11:20:17,476] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step30000/bf16_zero_pp_rank_0_mp_rank_00_optim_states.pt... 0: [2023-03-17 11:20:17,476] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step30000/bf16_zero_pp_rank_5_mp_rank_00_optim_states.pt... 0: [2023-03-17 11:20:17,476] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step30000/bf16_zero_pp_rank_2_mp_rank_00_optim_states.pt... 0: [2023-03-17 11:20:17,476] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step30000/bf16_zero_pp_rank_3_mp_rank_00_optim_states.pt... 0: [2023-03-17 11:20:17,476] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step30000/bf16_zero_pp_rank_6_mp_rank_00_optim_states.pt... 0: [2023-03-17 11:20:17,476] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step30000/bf16_zero_pp_rank_1_mp_rank_00_optim_states.pt... 0: [2023-03-17 11:20:17,476] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step30000/bf16_zero_pp_rank_7_mp_rank_00_optim_states.pt... 2: [2023-03-17 11:20:17,476] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step30000/bf16_zero_pp_rank_20_mp_rank_00_optim_states.pt... 2: [2023-03-17 11:20:17,476] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step30000/bf16_zero_pp_rank_19_mp_rank_00_optim_states.pt... 2: [2023-03-17 11:20:17,476] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step30000/bf16_zero_pp_rank_16_mp_rank_00_optim_states.pt... 2: [2023-03-17 11:20:17,476] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step30000/bf16_zero_pp_rank_17_mp_rank_00_optim_states.pt... 2: [2023-03-17 11:20:17,476] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step30000/bf16_zero_pp_rank_23_mp_rank_00_optim_states.pt... 4: [2023-03-17 11:20:17,476] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step30000/bf16_zero_pp_rank_32_mp_rank_00_optim_states.pt... 4: [2023-03-17 11:20:17,476] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step30000/bf16_zero_pp_rank_37_mp_rank_00_optim_states.pt... 4: [2023-03-17 11:20:17,476] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step30000/bf16_zero_pp_rank_33_mp_rank_00_optim_states.pt... 4: [2023-03-17 11:20:17,476] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step30000/bf16_zero_pp_rank_38_mp_rank_00_optim_states.pt... 4: [2023-03-17 11:20:17,476] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step30000/bf16_zero_pp_rank_36_mp_rank_00_optim_states.pt... 6: [2023-03-17 11:20:17,476] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step30000/bf16_zero_pp_rank_48_mp_rank_00_optim_states.pt... 6: [2023-03-17 11:20:17,476] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step30000/bf16_zero_pp_rank_49_mp_rank_00_optim_states.pt... 6: [2023-03-17 11:20:17,476] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step30000/bf16_zero_pp_rank_51_mp_rank_00_optim_states.pt... 6: [2023-03-17 11:20:17,476] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step30000/bf16_zero_pp_rank_53_mp_rank_00_optim_states.pt... 6: [2023-03-17 11:20:17,476] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step30000/bf16_zero_pp_rank_52_mp_rank_00_optim_states.pt... 7: [2023-03-17 11:20:17,476] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step30000/bf16_zero_pp_rank_62_mp_rank_00_optim_states.pt... 7: [2023-03-17 11:20:17,476] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step30000/bf16_zero_pp_rank_57_mp_rank_00_optim_states.pt... 7: [2023-03-17 11:20:17,476] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step30000/bf16_zero_pp_rank_63_mp_rank_00_optim_states.pt... 7: [2023-03-17 11:20:17,476] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step30000/bf16_zero_pp_rank_60_mp_rank_00_optim_states.pt... 7: [2023-03-17 11:20:17,476] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step30000/bf16_zero_pp_rank_56_mp_rank_00_optim_states.pt... 5: [2023-03-17 11:20:17,476] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step30000/bf16_zero_pp_rank_41_mp_rank_00_optim_states.pt... 5: [2023-03-17 11:20:17,476] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step30000/bf16_zero_pp_rank_40_mp_rank_00_optim_states.pt... 5: [2023-03-17 11:20:17,476] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step30000/bf16_zero_pp_rank_46_mp_rank_00_optim_states.pt... 5: [2023-03-17 11:20:17,476] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step30000/bf16_zero_pp_rank_44_mp_rank_00_optim_states.pt... 5: [2023-03-17 11:20:17,476] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step30000/bf16_zero_pp_rank_42_mp_rank_00_optim_states.pt... 0: [2023-03-17 11:20:17,476] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step30000/bf16_zero_pp_rank_4_mp_rank_00_optim_states.pt... 3: [2023-03-17 11:20:17,476] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step30000/bf16_zero_pp_rank_25_mp_rank_00_optim_states.pt... 3: [2023-03-17 11:20:17,476] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step30000/bf16_zero_pp_rank_31_mp_rank_00_optim_states.pt... 3: [2023-03-17 11:20:17,476] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step30000/bf16_zero_pp_rank_29_mp_rank_00_optim_states.pt... 3: [2023-03-17 11:20:17,476] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step30000/bf16_zero_pp_rank_27_mp_rank_00_optim_states.pt... 3: [2023-03-17 11:20:17,476] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step30000/bf16_zero_pp_rank_28_mp_rank_00_optim_states.pt... 2: [2023-03-17 11:20:17,476] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step30000/bf16_zero_pp_rank_21_mp_rank_00_optim_states.pt... 2: [2023-03-17 11:20:17,476] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step30000/bf16_zero_pp_rank_18_mp_rank_00_optim_states.pt... 2: [2023-03-17 11:20:17,476] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step30000/bf16_zero_pp_rank_22_mp_rank_00_optim_states.pt... 4: [2023-03-17 11:20:17,476] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step30000/bf16_zero_pp_rank_39_mp_rank_00_optim_states.pt... 4: [2023-03-17 11:20:17,476] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step30000/bf16_zero_pp_rank_35_mp_rank_00_optim_states.pt... 4: [2023-03-17 11:20:17,476] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step30000/bf16_zero_pp_rank_34_mp_rank_00_optim_states.pt... 6: [2023-03-17 11:20:17,476] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step30000/bf16_zero_pp_rank_54_mp_rank_00_optim_states.pt... 6: [2023-03-17 11:20:17,476] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step30000/bf16_zero_pp_rank_55_mp_rank_00_optim_states.pt... 6: [2023-03-17 11:20:17,476] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step30000/bf16_zero_pp_rank_50_mp_rank_00_optim_states.pt... 5: [2023-03-17 11:20:17,476] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step30000/bf16_zero_pp_rank_43_mp_rank_00_optim_states.pt... 5: [2023-03-17 11:20:17,476] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step30000/bf16_zero_pp_rank_45_mp_rank_00_optim_states.pt... 5: [2023-03-17 11:20:17,476] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step30000/bf16_zero_pp_rank_47_mp_rank_00_optim_states.pt... 1: [2023-03-17 11:20:17,476] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step30000/bf16_zero_pp_rank_13_mp_rank_00_optim_states.pt... 1: [2023-03-17 11:20:17,476] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step30000/bf16_zero_pp_rank_8_mp_rank_00_optim_states.pt... 1: [2023-03-17 11:20:17,476] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step30000/bf16_zero_pp_rank_14_mp_rank_00_optim_states.pt... 1: [2023-03-17 11:20:17,476] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step30000/bf16_zero_pp_rank_12_mp_rank_00_optim_states.pt... 1: [2023-03-17 11:20:17,476] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step30000/bf16_zero_pp_rank_11_mp_rank_00_optim_states.pt... 3: [2023-03-17 11:20:17,476] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step30000/bf16_zero_pp_rank_24_mp_rank_00_optim_states.pt... 3: [2023-03-17 11:20:17,476] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step30000/bf16_zero_pp_rank_30_mp_rank_00_optim_states.pt... 3: [2023-03-17 11:20:17,476] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step30000/bf16_zero_pp_rank_26_mp_rank_00_optim_states.pt... 7: [2023-03-17 11:20:17,476] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step30000/bf16_zero_pp_rank_61_mp_rank_00_optim_states.pt... 7: [2023-03-17 11:20:17,476] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step30000/bf16_zero_pp_rank_59_mp_rank_00_optim_states.pt... 7: [2023-03-17 11:20:17,476] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step30000/bf16_zero_pp_rank_58_mp_rank_00_optim_states.pt... 1: [2023-03-17 11:20:17,476] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step30000/bf16_zero_pp_rank_9_mp_rank_00_optim_states.pt... 1: [2023-03-17 11:20:17,476] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step30000/bf16_zero_pp_rank_10_mp_rank_00_optim_states.pt... 1: [2023-03-17 11:20:17,476] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step30000/bf16_zero_pp_rank_15_mp_rank_00_optim_states.pt... 0: [2023-03-17 11:20:17,488] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step30000/bf16_zero_pp_rank_0_mp_rank_00_optim_states.pt. 0: [2023-03-17 11:20:17,489] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step30000/bf16_zero_pp_rank_5_mp_rank_00_optim_states.pt. 0: [2023-03-17 11:20:17,489] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step30000/bf16_zero_pp_rank_5_mp_rank_00_optim_states.pt 0: [2023-03-17 11:20:17,490] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step30000 is ready now! 0: [2023-03-17 11:20:17,490] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step30000/bf16_zero_pp_rank_6_mp_rank_00_optim_states.pt. 0: [2023-03-17 11:20:17,490] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step30000/bf16_zero_pp_rank_6_mp_rank_00_optim_states.pt 0: [2023-03-17 11:20:17,490] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step30000 is ready now! 0: [2023-03-17 11:20:17,490] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step30000/bf16_zero_pp_rank_4_mp_rank_00_optim_states.pt. 0: [2023-03-17 11:20:17,490] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step30000/bf16_zero_pp_rank_4_mp_rank_00_optim_states.pt 0: [2023-03-17 11:20:17,490] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step30000 is ready now! 0: [2023-03-17 11:20:17,491] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step30000/bf16_zero_pp_rank_7_mp_rank_00_optim_states.pt. 0: [2023-03-17 11:20:17,491] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step30000/bf16_zero_pp_rank_7_mp_rank_00_optim_states.pt 0: [2023-03-17 11:20:17,491] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step30000 is ready now! 0: [2023-03-17 11:20:17,491] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step30000/bf16_zero_pp_rank_1_mp_rank_00_optim_states.pt. 0: [2023-03-17 11:20:17,491] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step30000/bf16_zero_pp_rank_1_mp_rank_00_optim_states.pt 0: [2023-03-17 11:20:17,491] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step30000 is ready now! 0: [2023-03-17 11:20:17,491] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step30000/bf16_zero_pp_rank_3_mp_rank_00_optim_states.pt. 0: [2023-03-17 11:20:17,491] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step30000/bf16_zero_pp_rank_3_mp_rank_00_optim_states.pt 0: [2023-03-17 11:20:17,491] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step30000 is ready now! 0: [2023-03-17 11:20:17,492] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step30000/bf16_zero_pp_rank_0_mp_rank_00_optim_states.pt 0: [2023-03-17 11:20:17,492] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step30000 is ready now! 0: [2023-03-17 11:20:17,492] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step30000/bf16_zero_pp_rank_2_mp_rank_00_optim_states.pt. 0: [2023-03-17 11:20:17,492] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step30000/bf16_zero_pp_rank_2_mp_rank_00_optim_states.pt 0: [2023-03-17 11:20:17,492] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step30000 is ready now! 6: [2023-03-17 11:20:17,499] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step30000/bf16_zero_pp_rank_52_mp_rank_00_optim_states.pt. 6: [2023-03-17 11:20:17,499] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step30000/bf16_zero_pp_rank_48_mp_rank_00_optim_states.pt. 6: [2023-03-17 11:20:17,499] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step30000/bf16_zero_pp_rank_52_mp_rank_00_optim_states.pt 6: [2023-03-17 11:20:17,499] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step30000/bf16_zero_pp_rank_48_mp_rank_00_optim_states.pt 6: [2023-03-17 11:20:17,499] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step30000 is ready now! 6: [2023-03-17 11:20:17,499] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step30000 is ready now! 6: [2023-03-17 11:20:17,499] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step30000/bf16_zero_pp_rank_55_mp_rank_00_optim_states.pt. 6: [2023-03-17 11:20:17,499] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step30000/bf16_zero_pp_rank_55_mp_rank_00_optim_states.pt 6: [2023-03-17 11:20:17,499] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step30000 is ready now! 6: [2023-03-17 11:20:17,500] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step30000/bf16_zero_pp_rank_51_mp_rank_00_optim_states.pt. 6: [2023-03-17 11:20:17,500] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step30000/bf16_zero_pp_rank_51_mp_rank_00_optim_states.pt 6: [2023-03-17 11:20:17,500] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step30000 is ready now! 6: [2023-03-17 11:20:17,500] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step30000/bf16_zero_pp_rank_54_mp_rank_00_optim_states.pt. 6: [2023-03-17 11:20:17,500] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step30000/bf16_zero_pp_rank_49_mp_rank_00_optim_states.pt. 6: [2023-03-17 11:20:17,500] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step30000/bf16_zero_pp_rank_54_mp_rank_00_optim_states.pt 2: [2023-03-17 11:20:17,500] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step30000/bf16_zero_pp_rank_16_mp_rank_00_optim_states.pt. 2: [2023-03-17 11:20:17,500] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step30000/bf16_zero_pp_rank_23_mp_rank_00_optim_states.pt. 2: [2023-03-17 11:20:17,500] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step30000/bf16_zero_pp_rank_21_mp_rank_00_optim_states.pt. 2: [2023-03-17 11:20:17,500] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step30000/bf16_zero_pp_rank_22_mp_rank_00_optim_states.pt. 3: [2023-03-17 11:20:17,500] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step30000/bf16_zero_pp_rank_29_mp_rank_00_optim_states.pt. 3: [2023-03-17 11:20:17,500] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step30000/bf16_zero_pp_rank_27_mp_rank_00_optim_states.pt. 3: [2023-03-17 11:20:17,500] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step30000/bf16_zero_pp_rank_28_mp_rank_00_optim_states.pt. 3: [2023-03-17 11:20:17,500] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step30000/bf16_zero_pp_rank_24_mp_rank_00_optim_states.pt. 3: [2023-03-17 11:20:17,500] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step30000/bf16_zero_pp_rank_30_mp_rank_00_optim_states.pt. 2: [2023-03-17 11:20:17,500] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step30000/bf16_zero_pp_rank_17_mp_rank_00_optim_states.pt. 2: [2023-03-17 11:20:17,500] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step30000/bf16_zero_pp_rank_18_mp_rank_00_optim_states.pt. 2: [2023-03-17 11:20:17,500] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step30000/bf16_zero_pp_rank_20_mp_rank_00_optim_states.pt. 2: [2023-03-17 11:20:17,500] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step30000/bf16_zero_pp_rank_19_mp_rank_00_optim_states.pt. 2: [2023-03-17 11:20:17,500] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step30000/bf16_zero_pp_rank_21_mp_rank_00_optim_states.pt 2: [2023-03-17 11:20:17,500] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step30000/bf16_zero_pp_rank_23_mp_rank_00_optim_states.pt 2: [2023-03-17 11:20:17,500] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step30000/bf16_zero_pp_rank_16_mp_rank_00_optim_states.pt 3: [2023-03-17 11:20:17,500] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step30000/bf16_zero_pp_rank_26_mp_rank_00_optim_states.pt. 3: [2023-03-17 11:20:17,500] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step30000/bf16_zero_pp_rank_25_mp_rank_00_optim_states.pt. 3: [2023-03-17 11:20:17,500] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step30000/bf16_zero_pp_rank_31_mp_rank_00_optim_states.pt. 2: [2023-03-17 11:20:17,500] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step30000/bf16_zero_pp_rank_22_mp_rank_00_optim_states.pt 6: [2023-03-17 11:20:17,500] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step30000/bf16_zero_pp_rank_53_mp_rank_00_optim_states.pt. 3: [2023-03-17 11:20:17,500] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step30000/bf16_zero_pp_rank_27_mp_rank_00_optim_states.pt 3: [2023-03-17 11:20:17,500] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step30000/bf16_zero_pp_rank_29_mp_rank_00_optim_states.pt 3: [2023-03-17 11:20:17,500] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step30000/bf16_zero_pp_rank_28_mp_rank_00_optim_states.pt 3: [2023-03-17 11:20:17,500] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step30000/bf16_zero_pp_rank_24_mp_rank_00_optim_states.pt 3: [2023-03-17 11:20:17,500] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step30000/bf16_zero_pp_rank_30_mp_rank_00_optim_states.pt 2: [2023-03-17 11:20:17,500] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step30000/bf16_zero_pp_rank_17_mp_rank_00_optim_states.pt 6: [2023-03-17 11:20:17,500] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step30000 is ready now! 6: [2023-03-17 11:20:17,500] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step30000/bf16_zero_pp_rank_49_mp_rank_00_optim_states.pt 6: [2023-03-17 11:20:17,500] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step30000/bf16_zero_pp_rank_53_mp_rank_00_optim_states.pt 6: [2023-03-17 11:20:17,500] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step30000 is ready now! 6: [2023-03-17 11:20:17,500] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step30000 is ready now! 6: [2023-03-17 11:20:17,500] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step30000/bf16_zero_pp_rank_50_mp_rank_00_optim_states.pt. 3: [2023-03-17 11:20:17,500] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step30000/bf16_zero_pp_rank_26_mp_rank_00_optim_states.pt 2: [2023-03-17 11:20:17,500] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step30000/bf16_zero_pp_rank_18_mp_rank_00_optim_states.pt 6: [2023-03-17 11:20:17,500] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step30000/bf16_zero_pp_rank_50_mp_rank_00_optim_states.pt 3: [2023-03-17 11:20:17,500] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step30000/bf16_zero_pp_rank_31_mp_rank_00_optim_states.pt 3: [2023-03-17 11:20:17,500] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step30000/bf16_zero_pp_rank_25_mp_rank_00_optim_states.pt 3: [2023-03-17 11:20:17,500] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step30000 is ready now! 3: [2023-03-17 11:20:17,500] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step30000 is ready now! 2: [2023-03-17 11:20:17,500] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step30000 is ready now! 2: [2023-03-17 11:20:17,500] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step30000 is ready now! 2: [2023-03-17 11:20:17,500] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step30000/bf16_zero_pp_rank_20_mp_rank_00_optim_states.pt 2: [2023-03-17 11:20:17,500] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step30000/bf16_zero_pp_rank_19_mp_rank_00_optim_states.pt 6: [2023-03-17 11:20:17,500] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step30000 is ready now! 3: [2023-03-17 11:20:17,500] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step30000 is ready now! 3: [2023-03-17 11:20:17,500] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step30000 is ready now! 3: [2023-03-17 11:20:17,500] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step30000 is ready now! 2: [2023-03-17 11:20:17,500] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step30000 is ready now! 3: [2023-03-17 11:20:17,500] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step30000 is ready now! 2: [2023-03-17 11:20:17,500] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step30000 is ready now! 3: [2023-03-17 11:20:17,500] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step30000 is ready now! 3: [2023-03-17 11:20:17,500] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step30000 is ready now! 2: [2023-03-17 11:20:17,500] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step30000 is ready now! 2: [2023-03-17 11:20:17,500] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step30000 is ready now! 2: [2023-03-17 11:20:17,500] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step30000 is ready now! 2: [2023-03-17 11:20:17,500] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step30000 is ready now! 7: [2023-03-17 11:20:17,657] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step30000/bf16_zero_pp_rank_56_mp_rank_00_optim_states.pt. 4: [2023-03-17 11:20:17,657] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step30000/bf16_zero_pp_rank_33_mp_rank_00_optim_states.pt. 4: [2023-03-17 11:20:17,657] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step30000/bf16_zero_pp_rank_38_mp_rank_00_optim_states.pt. 7: [2023-03-17 11:20:17,657] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step30000/bf16_zero_pp_rank_56_mp_rank_00_optim_states.pt 7: [2023-03-17 11:20:17,657] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step30000 is ready now! 4: [2023-03-17 11:20:17,657] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step30000/bf16_zero_pp_rank_38_mp_rank_00_optim_states.pt 4: [2023-03-17 11:20:17,657] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step30000/bf16_zero_pp_rank_33_mp_rank_00_optim_states.pt 4: [2023-03-17 11:20:17,657] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step30000 is ready now! 4: [2023-03-17 11:20:17,657] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step30000 is ready now! 4: [2023-03-17 11:20:17,657] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step30000/bf16_zero_pp_rank_34_mp_rank_00_optim_states.pt. 4: [2023-03-17 11:20:17,657] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step30000/bf16_zero_pp_rank_34_mp_rank_00_optim_states.pt 7: [2023-03-17 11:20:17,657] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step30000/bf16_zero_pp_rank_60_mp_rank_00_optim_states.pt. 4: [2023-03-17 11:20:17,657] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step30000 is ready now! 7: [2023-03-17 11:20:17,657] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step30000/bf16_zero_pp_rank_58_mp_rank_00_optim_states.pt. 7: [2023-03-17 11:20:17,657] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step30000/bf16_zero_pp_rank_60_mp_rank_00_optim_states.pt 7: [2023-03-17 11:20:17,657] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step30000/bf16_zero_pp_rank_58_mp_rank_00_optim_states.pt 7: [2023-03-17 11:20:17,657] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step30000/bf16_zero_pp_rank_57_mp_rank_00_optim_states.pt. 7: [2023-03-17 11:20:17,657] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step30000 is ready now! 4: [2023-03-17 11:20:17,657] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step30000/bf16_zero_pp_rank_32_mp_rank_00_optim_states.pt. 7: [2023-03-17 11:20:17,657] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step30000 is ready now! 7: [2023-03-17 11:20:17,657] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step30000/bf16_zero_pp_rank_57_mp_rank_00_optim_states.pt 7: [2023-03-17 11:20:17,657] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step30000 is ready now! 4: [2023-03-17 11:20:17,657] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step30000/bf16_zero_pp_rank_37_mp_rank_00_optim_states.pt. 4: [2023-03-17 11:20:17,658] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step30000/bf16_zero_pp_rank_32_mp_rank_00_optim_states.pt 4: [2023-03-17 11:20:17,658] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step30000/bf16_zero_pp_rank_37_mp_rank_00_optim_states.pt 7: [2023-03-17 11:20:17,657] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step30000/bf16_zero_pp_rank_63_mp_rank_00_optim_states.pt. 7: [2023-03-17 11:20:17,658] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step30000/bf16_zero_pp_rank_61_mp_rank_00_optim_states.pt. 7: [2023-03-17 11:20:17,658] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step30000/bf16_zero_pp_rank_63_mp_rank_00_optim_states.pt 4: [2023-03-17 11:20:17,658] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step30000 is ready now! 4: [2023-03-17 11:20:17,658] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step30000/bf16_zero_pp_rank_36_mp_rank_00_optim_states.pt. 4: [2023-03-17 11:20:17,658] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step30000 is ready now! 7: [2023-03-17 11:20:17,658] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step30000/bf16_zero_pp_rank_61_mp_rank_00_optim_states.pt 7: [2023-03-17 11:20:17,658] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step30000 is ready now! 4: [2023-03-17 11:20:17,658] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step30000/bf16_zero_pp_rank_36_mp_rank_00_optim_states.pt 7: [2023-03-17 11:20:17,658] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step30000 is ready now! 7: [2023-03-17 11:20:17,658] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step30000/bf16_zero_pp_rank_59_mp_rank_00_optim_states.pt. 4: [2023-03-17 11:20:17,658] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step30000 is ready now! 7: [2023-03-17 11:20:17,658] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step30000/bf16_zero_pp_rank_62_mp_rank_00_optim_states.pt. 7: [2023-03-17 11:20:17,658] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step30000/bf16_zero_pp_rank_59_mp_rank_00_optim_states.pt 7: [2023-03-17 11:20:17,658] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step30000/bf16_zero_pp_rank_62_mp_rank_00_optim_states.pt 7: [2023-03-17 11:20:17,658] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step30000 is ready now! 7: [2023-03-17 11:20:17,658] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step30000 is ready now! 4: [2023-03-17 11:20:17,658] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step30000/bf16_zero_pp_rank_35_mp_rank_00_optim_states.pt. 4: [2023-03-17 11:20:17,658] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step30000/bf16_zero_pp_rank_35_mp_rank_00_optim_states.pt 4: [2023-03-17 11:20:17,658] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step30000 is ready now! 4: [2023-03-17 11:20:17,658] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step30000/bf16_zero_pp_rank_39_mp_rank_00_optim_states.pt. 5: [2023-03-17 11:20:17,658] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step30000/bf16_zero_pp_rank_41_mp_rank_00_optim_states.pt. 5: [2023-03-17 11:20:17,658] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step30000/bf16_zero_pp_rank_46_mp_rank_00_optim_states.pt. 5: [2023-03-17 11:20:17,658] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step30000/bf16_zero_pp_rank_44_mp_rank_00_optim_states.pt. 5: [2023-03-17 11:20:17,658] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step30000/bf16_zero_pp_rank_40_mp_rank_00_optim_states.pt. 5: [2023-03-17 11:20:17,658] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step30000/bf16_zero_pp_rank_43_mp_rank_00_optim_states.pt. 4: [2023-03-17 11:20:17,658] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step30000/bf16_zero_pp_rank_39_mp_rank_00_optim_states.pt 5: [2023-03-17 11:20:17,658] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step30000/bf16_zero_pp_rank_41_mp_rank_00_optim_states.pt 5: [2023-03-17 11:20:17,658] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step30000/bf16_zero_pp_rank_45_mp_rank_00_optim_states.pt. 5: [2023-03-17 11:20:17,658] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step30000/bf16_zero_pp_rank_46_mp_rank_00_optim_states.pt 5: [2023-03-17 11:20:17,658] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step30000/bf16_zero_pp_rank_44_mp_rank_00_optim_states.pt 5: [2023-03-17 11:20:17,658] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step30000/bf16_zero_pp_rank_40_mp_rank_00_optim_states.pt 5: [2023-03-17 11:20:17,658] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step30000/bf16_zero_pp_rank_43_mp_rank_00_optim_states.pt 5: [2023-03-17 11:20:17,658] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step30000 is ready now! 4: [2023-03-17 11:20:17,658] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step30000 is ready now! 5: [2023-03-17 11:20:17,658] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step30000/bf16_zero_pp_rank_45_mp_rank_00_optim_states.pt 5: [2023-03-17 11:20:17,658] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step30000 is ready now! 5: [2023-03-17 11:20:17,658] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step30000 is ready now! 5: [2023-03-17 11:20:17,658] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step30000 is ready now! 5: [2023-03-17 11:20:17,658] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step30000 is ready now! 5: [2023-03-17 11:20:17,658] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step30000 is ready now! 5: [2023-03-17 11:20:17,659] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step30000/bf16_zero_pp_rank_47_mp_rank_00_optim_states.pt. 5: [2023-03-17 11:20:17,659] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step30000/bf16_zero_pp_rank_42_mp_rank_00_optim_states.pt. 5: [2023-03-17 11:20:17,659] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step30000/bf16_zero_pp_rank_47_mp_rank_00_optim_states.pt 5: [2023-03-17 11:20:17,659] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step30000/bf16_zero_pp_rank_42_mp_rank_00_optim_states.pt 5: [2023-03-17 11:20:17,659] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step30000 is ready now! 5: [2023-03-17 11:20:17,659] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step30000 is ready now! 1: [2023-03-17 11:20:17,659] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step30000/bf16_zero_pp_rank_10_mp_rank_00_optim_states.pt. 1: [2023-03-17 11:20:17,659] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step30000/bf16_zero_pp_rank_12_mp_rank_00_optim_states.pt. 1: [2023-03-17 11:20:17,659] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step30000/bf16_zero_pp_rank_11_mp_rank_00_optim_states.pt. 1: [2023-03-17 11:20:17,659] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step30000/bf16_zero_pp_rank_9_mp_rank_00_optim_states.pt. 1: [2023-03-17 11:20:17,659] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step30000/bf16_zero_pp_rank_13_mp_rank_00_optim_states.pt. 1: [2023-03-17 11:20:17,659] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step30000/bf16_zero_pp_rank_14_mp_rank_00_optim_states.pt. 1: [2023-03-17 11:20:17,659] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step30000/bf16_zero_pp_rank_8_mp_rank_00_optim_states.pt. 1: [2023-03-17 11:20:17,659] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step30000/bf16_zero_pp_rank_15_mp_rank_00_optim_states.pt. 1: [2023-03-17 11:20:17,659] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step30000/bf16_zero_pp_rank_10_mp_rank_00_optim_states.pt 1: [2023-03-17 11:20:17,659] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step30000/bf16_zero_pp_rank_12_mp_rank_00_optim_states.pt 1: [2023-03-17 11:20:17,659] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step30000/bf16_zero_pp_rank_11_mp_rank_00_optim_states.pt 1: [2023-03-17 11:20:17,659] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step30000/bf16_zero_pp_rank_9_mp_rank_00_optim_states.pt 1: [2023-03-17 11:20:17,659] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step30000/bf16_zero_pp_rank_13_mp_rank_00_optim_states.pt 1: [2023-03-17 11:20:17,659] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step30000/bf16_zero_pp_rank_14_mp_rank_00_optim_states.pt 1: [2023-03-17 11:20:17,659] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step30000 is ready now! 1: [2023-03-17 11:20:17,659] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step30000 is ready now! 1: [2023-03-17 11:20:17,659] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step30000/bf16_zero_pp_rank_15_mp_rank_00_optim_states.pt 1: [2023-03-17 11:20:17,659] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step30000/bf16_zero_pp_rank_8_mp_rank_00_optim_states.pt 1: [2023-03-17 11:20:17,659] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step30000 is ready now! 1: [2023-03-17 11:20:17,659] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step30000 is ready now! 1: [2023-03-17 11:20:17,659] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step30000 is ready now! 1: [2023-03-17 11:20:17,659] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step30000 is ready now! 1: [2023-03-17 11:20:17,659] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step30000 is ready now! 1: [2023-03-17 11:20:17,659] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step30000 is ready now! 0: successfully saved checkpoint at iteration 30000 to checkpoints_44m174b100m 7: time (ms) | save-checkpoint: 374.40 7: iteration 30100/ 331103 | consumed samples: 7705600 | consumed tokens: 15781068800 | elapsed time per iteration (s): 0.16 | learning rate: 1.970E-04 | global batch size: 256 | lm loss: 3.826924E+00 | grad norm: 0.316 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1613.878 | TFLOPs: 25.31 | 7: iteration 30200/ 331103 | consumed samples: 7731200 | consumed tokens: 15833497600 | elapsed time per iteration (s): 0.15 | learning rate: 1.970E-04 | global batch size: 256 | lm loss: 3.822425E+00 | grad norm: 0.293 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1661.178 | TFLOPs: 26.05 | 7: iteration 30300/ 331103 | consumed samples: 7756800 | consumed tokens: 15885926400 | elapsed time per iteration (s): 0.15 | learning rate: 1.970E-04 | global batch size: 256 | lm loss: 3.822907E+00 | grad norm: 0.322 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1661.466 | TFLOPs: 26.06 | 7: iteration 30400/ 331103 | consumed samples: 7782400 | consumed tokens: 15938355200 | elapsed time per iteration (s): 0.15 | learning rate: 1.970E-04 | global batch size: 256 | lm loss: 3.819632E+00 | grad norm: 0.323 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1658.522 | TFLOPs: 26.01 | 7: iteration 30500/ 331103 | consumed samples: 7808000 | consumed tokens: 15990784000 | elapsed time per iteration (s): 0.15 | learning rate: 1.970E-04 | global batch size: 256 | lm loss: 3.820110E+00 | grad norm: 0.347 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1660.928 | TFLOPs: 26.05 | 7: iteration 30600/ 331103 | consumed samples: 7833600 | consumed tokens: 16043212800 | elapsed time per iteration (s): 0.15 | learning rate: 1.969E-04 | global batch size: 256 | lm loss: 3.825713E+00 | grad norm: 0.350 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1668.937 | TFLOPs: 26.17 | 7: iteration 30700/ 331103 | consumed samples: 7859200 | consumed tokens: 16095641600 | elapsed time per iteration (s): 0.15 | learning rate: 1.969E-04 | global batch size: 256 | lm loss: 3.820035E+00 | grad norm: 0.295 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1663.670 | TFLOPs: 26.09 | 7: iteration 30800/ 331103 | consumed samples: 7884800 | consumed tokens: 16148070400 | elapsed time per iteration (s): 0.15 | learning rate: 1.969E-04 | global batch size: 256 | lm loss: 3.821803E+00 | grad norm: 0.315 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1652.831 | TFLOPs: 25.92 | 7: iteration 30900/ 331103 | consumed samples: 7910400 | consumed tokens: 16200499200 | elapsed time per iteration (s): 0.16 | learning rate: 1.969E-04 | global batch size: 256 | lm loss: 3.818453E+00 | grad norm: 0.338 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1648.016 | TFLOPs: 25.85 | 7: iteration 31000/ 331103 | consumed samples: 7936000 | consumed tokens: 16252928000 | elapsed time per iteration (s): 0.15 | learning rate: 1.968E-04 | global batch size: 256 | lm loss: 3.817333E+00 | grad norm: 0.310 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1662.924 | TFLOPs: 26.08 | 7: iteration 31100/ 331103 | consumed samples: 7961600 | consumed tokens: 16305356800 | elapsed time per iteration (s): 0.15 | learning rate: 1.968E-04 | global batch size: 256 | lm loss: 3.820338E+00 | grad norm: 0.328 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1661.709 | TFLOPs: 26.06 | 7: iteration 31200/ 331103 | consumed samples: 7987200 | consumed tokens: 16357785600 | elapsed time per iteration (s): 0.15 | learning rate: 1.968E-04 | global batch size: 256 | lm loss: 3.818674E+00 | grad norm: 0.306 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1656.283 | TFLOPs: 25.97 | 7: iteration 31300/ 331103 | consumed samples: 8012800 | consumed tokens: 16410214400 | elapsed time per iteration (s): 0.15 | learning rate: 1.968E-04 | global batch size: 256 | lm loss: 3.821581E+00 | grad norm: 0.339 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1657.484 | TFLOPs: 25.99 | 7: iteration 31400/ 331103 | consumed samples: 8038400 | consumed tokens: 16462643200 | elapsed time per iteration (s): 0.15 | learning rate: 1.968E-04 | global batch size: 256 | lm loss: 3.816821E+00 | grad norm: 0.312 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1655.673 | TFLOPs: 25.97 | 7: iteration 31500/ 331103 | consumed samples: 8064000 | consumed tokens: 16515072000 | elapsed time per iteration (s): 0.15 | learning rate: 1.967E-04 | global batch size: 256 | lm loss: 3.816173E+00 | grad norm: 0.332 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1664.905 | TFLOPs: 26.11 | 7: iteration 31600/ 331103 | consumed samples: 8089600 | consumed tokens: 16567500800 | elapsed time per iteration (s): 0.15 | learning rate: 1.967E-04 | global batch size: 256 | lm loss: 3.817582E+00 | grad norm: 0.318 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1664.645 | TFLOPs: 26.11 | 7: iteration 31700/ 331103 | consumed samples: 8115200 | consumed tokens: 16619929600 | elapsed time per iteration (s): 0.15 | learning rate: 1.967E-04 | global batch size: 256 | lm loss: 3.820266E+00 | grad norm: 0.306 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1657.377 | TFLOPs: 25.99 | 7: iteration 31800/ 331103 | consumed samples: 8140800 | consumed tokens: 16672358400 | elapsed time per iteration (s): 0.15 | learning rate: 1.967E-04 | global batch size: 256 | lm loss: 3.811594E+00 | grad norm: 0.299 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1654.248 | TFLOPs: 25.94 | 7: iteration 31900/ 331103 | consumed samples: 8166400 | consumed tokens: 16724787200 | elapsed time per iteration (s): 0.15 | learning rate: 1.966E-04 | global batch size: 256 | lm loss: 3.810526E+00 | grad norm: 0.286 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1655.257 | TFLOPs: 25.96 | 0: [2023-03-17 11:25:26,206] [INFO] [logging.py:68:log_dist] [Rank 0] step=32000, skipped=0, lr=[0.00019661930729858538, 0.00019661930729858538, 0.00019661930729858538], mom=[(0.9, 0.999), (0.9, 0.999), (0.9, 0.999)] 7: iteration 32000/ 331103 | consumed samples: 8192000 | consumed tokens: 16777216000 | elapsed time per iteration (s): 0.15 | learning rate: 1.966E-04 | global batch size: 256 | lm loss: 3.816432E+00 | grad norm: 0.320 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1661.847 | TFLOPs: 26.06 | 0: steps: 32000 loss: 3.8242 iter time (s): 0.153 samples/sec: 1669.096 7: iteration 32100/ 331103 | consumed samples: 8217600 | consumed tokens: 16829644800 | elapsed time per iteration (s): 0.15 | learning rate: 1.966E-04 | global batch size: 256 | lm loss: 3.806801E+00 | grad norm: 0.313 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1657.144 | TFLOPs: 25.99 | 7: iteration 32200/ 331103 | consumed samples: 8243200 | consumed tokens: 16882073600 | elapsed time per iteration (s): 0.15 | learning rate: 1.966E-04 | global batch size: 256 | lm loss: 3.818968E+00 | grad norm: 0.294 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1662.793 | TFLOPs: 26.08 | 7: iteration 32300/ 331103 | consumed samples: 8268800 | consumed tokens: 16934502400 | elapsed time per iteration (s): 0.15 | learning rate: 1.965E-04 | global batch size: 256 | lm loss: 3.810085E+00 | grad norm: 0.313 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1666.769 | TFLOPs: 26.14 | 7: iteration 32400/ 331103 | consumed samples: 8294400 | consumed tokens: 16986931200 | elapsed time per iteration (s): 0.15 | learning rate: 1.965E-04 | global batch size: 256 | lm loss: 3.812942E+00 | grad norm: 0.323 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1665.530 | TFLOPs: 26.12 | 7: iteration 32500/ 331103 | consumed samples: 8320000 | consumed tokens: 17039360000 | elapsed time per iteration (s): 0.15 | learning rate: 1.965E-04 | global batch size: 256 | lm loss: 3.809056E+00 | grad norm: 0.279 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1659.469 | TFLOPs: 26.02 | 7: iteration 32600/ 331103 | consumed samples: 8345600 | consumed tokens: 17091788800 | elapsed time per iteration (s): 0.15 | learning rate: 1.965E-04 | global batch size: 256 | lm loss: 3.813540E+00 | grad norm: 0.303 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1668.424 | TFLOPs: 26.17 | 7: iteration 32700/ 331103 | consumed samples: 8371200 | consumed tokens: 17144217600 | elapsed time per iteration (s): 0.15 | learning rate: 1.965E-04 | global batch size: 256 | lm loss: 3.814139E+00 | grad norm: 0.277 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1661.813 | TFLOPs: 26.06 | 7: iteration 32800/ 331103 | consumed samples: 8396800 | consumed tokens: 17196646400 | elapsed time per iteration (s): 0.16 | learning rate: 1.964E-04 | global batch size: 256 | lm loss: 3.813374E+00 | grad norm: 0.312 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1644.880 | TFLOPs: 25.80 | 7: iteration 32900/ 331103 | consumed samples: 8422400 | consumed tokens: 17249075200 | elapsed time per iteration (s): 0.16 | learning rate: 1.964E-04 | global batch size: 256 | lm loss: 3.807367E+00 | grad norm: 0.297 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1650.707 | TFLOPs: 25.89 | 7: iteration 33000/ 331103 | consumed samples: 8448000 | consumed tokens: 17301504000 | elapsed time per iteration (s): 0.15 | learning rate: 1.964E-04 | global batch size: 256 | lm loss: 3.809254E+00 | grad norm: 0.319 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1657.179 | TFLOPs: 25.99 | 7: iteration 33100/ 331103 | consumed samples: 8473600 | consumed tokens: 17353932800 | elapsed time per iteration (s): 0.16 | learning rate: 1.964E-04 | global batch size: 256 | lm loss: 3.811617E+00 | grad norm: 0.359 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1650.993 | TFLOPs: 25.89 | 7: iteration 33200/ 331103 | consumed samples: 8499200 | consumed tokens: 17406361600 | elapsed time per iteration (s): 0.16 | learning rate: 1.963E-04 | global batch size: 256 | lm loss: 3.807158E+00 | grad norm: 0.290 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1649.106 | TFLOPs: 25.86 | 7: iteration 33300/ 331103 | consumed samples: 8524800 | consumed tokens: 17458790400 | elapsed time per iteration (s): 0.15 | learning rate: 1.963E-04 | global batch size: 256 | lm loss: 3.811475E+00 | grad norm: 0.347 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1666.643 | TFLOPs: 26.14 | 7: iteration 33400/ 331103 | consumed samples: 8550400 | consumed tokens: 17511219200 | elapsed time per iteration (s): 0.16 | learning rate: 1.963E-04 | global batch size: 256 | lm loss: 3.807485E+00 | grad norm: 0.291 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1646.514 | TFLOPs: 25.82 | 7: iteration 33500/ 331103 | consumed samples: 8576000 | consumed tokens: 17563648000 | elapsed time per iteration (s): 0.15 | learning rate: 1.963E-04 | global batch size: 256 | lm loss: 3.811863E+00 | grad norm: 0.329 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1651.706 | TFLOPs: 25.90 | 7: iteration 33600/ 331103 | consumed samples: 8601600 | consumed tokens: 17616076800 | elapsed time per iteration (s): 0.17 | learning rate: 1.962E-04 | global batch size: 256 | lm loss: 3.806690E+00 | grad norm: 0.341 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1472.026 | TFLOPs: 23.09 | 7: iteration 33700/ 331103 | consumed samples: 8627200 | consumed tokens: 17668505600 | elapsed time per iteration (s): 0.16 | learning rate: 1.962E-04 | global batch size: 256 | lm loss: 3.811810E+00 | grad norm: 0.317 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1646.240 | TFLOPs: 25.82 | 7: iteration 33800/ 331103 | consumed samples: 8652800 | consumed tokens: 17720934400 | elapsed time per iteration (s): 0.15 | learning rate: 1.962E-04 | global batch size: 256 | lm loss: 3.808783E+00 | grad norm: 0.314 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1669.460 | TFLOPs: 26.18 | 7: iteration 33900/ 331103 | consumed samples: 8678400 | consumed tokens: 17773363200 | elapsed time per iteration (s): 0.15 | learning rate: 1.962E-04 | global batch size: 256 | lm loss: 3.808904E+00 | grad norm: 0.289 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1667.076 | TFLOPs: 26.14 | 0: [2023-03-17 11:30:36,966] [INFO] [logging.py:68:log_dist] [Rank 0] step=34000, skipped=0, lr=[0.0001961350391370227, 0.0001961350391370227, 0.0001961350391370227], mom=[(0.9, 0.999), (0.9, 0.999), (0.9, 0.999)] 7: iteration 34000/ 331103 | consumed samples: 8704000 | consumed tokens: 17825792000 | elapsed time per iteration (s): 0.15 | learning rate: 1.961E-04 | global batch size: 256 | lm loss: 3.809847E+00 | grad norm: 0.290 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1660.105 | TFLOPs: 26.03 | 0: steps: 34000 loss: 3.7933 iter time (s): 0.154 samples/sec: 1657.193 7: iteration 34100/ 331103 | consumed samples: 8729600 | consumed tokens: 17878220800 | elapsed time per iteration (s): 0.15 | learning rate: 1.961E-04 | global batch size: 256 | lm loss: 3.809294E+00 | grad norm: 0.335 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1654.133 | TFLOPs: 25.94 | 7: iteration 34200/ 331103 | consumed samples: 8755200 | consumed tokens: 17930649600 | elapsed time per iteration (s): 0.15 | learning rate: 1.961E-04 | global batch size: 256 | lm loss: 3.809586E+00 | grad norm: 0.281 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1665.141 | TFLOPs: 26.11 | 7: iteration 34300/ 331103 | consumed samples: 8780800 | consumed tokens: 17983078400 | elapsed time per iteration (s): 0.15 | learning rate: 1.961E-04 | global batch size: 256 | lm loss: 3.812692E+00 | grad norm: 0.266 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1659.245 | TFLOPs: 26.02 | 7: iteration 34400/ 331103 | consumed samples: 8806400 | consumed tokens: 18035507200 | elapsed time per iteration (s): 0.15 | learning rate: 1.960E-04 | global batch size: 256 | lm loss: 3.805591E+00 | grad norm: 0.303 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1663.772 | TFLOPs: 26.09 | 7: iteration 34500/ 331103 | consumed samples: 8832000 | consumed tokens: 18087936000 | elapsed time per iteration (s): 0.15 | learning rate: 1.960E-04 | global batch size: 256 | lm loss: 3.800517E+00 | grad norm: 0.333 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1657.961 | TFLOPs: 26.00 | 7: iteration 34600/ 331103 | consumed samples: 8857600 | consumed tokens: 18140364800 | elapsed time per iteration (s): 0.15 | learning rate: 1.960E-04 | global batch size: 256 | lm loss: 3.806266E+00 | grad norm: 0.310 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1665.362 | TFLOPs: 26.12 | 7: iteration 34700/ 331103 | consumed samples: 8883200 | consumed tokens: 18192793600 | elapsed time per iteration (s): 0.15 | learning rate: 1.960E-04 | global batch size: 256 | lm loss: 3.804657E+00 | grad norm: 0.314 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1656.671 | TFLOPs: 25.98 | 7: iteration 34800/ 331103 | consumed samples: 8908800 | consumed tokens: 18245222400 | elapsed time per iteration (s): 0.15 | learning rate: 1.959E-04 | global batch size: 256 | lm loss: 3.801911E+00 | grad norm: 0.307 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1661.762 | TFLOPs: 26.06 | 7: iteration 34900/ 331103 | consumed samples: 8934400 | consumed tokens: 18297651200 | elapsed time per iteration (s): 0.15 | learning rate: 1.959E-04 | global batch size: 256 | lm loss: 3.800631E+00 | grad norm: 0.348 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1652.077 | TFLOPs: 25.91 | 7: iteration 35000/ 331103 | consumed samples: 8960000 | consumed tokens: 18350080000 | elapsed time per iteration (s): 0.15 | learning rate: 1.959E-04 | global batch size: 256 | lm loss: 3.805911E+00 | grad norm: 0.303 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1662.539 | TFLOPs: 26.07 | 7: iteration 35100/ 331103 | consumed samples: 8985600 | consumed tokens: 18402508800 | elapsed time per iteration (s): 0.15 | learning rate: 1.959E-04 | global batch size: 256 | lm loss: 3.800790E+00 | grad norm: 0.280 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1665.105 | TFLOPs: 26.11 | 7: iteration 35200/ 331103 | consumed samples: 9011200 | consumed tokens: 18454937600 | elapsed time per iteration (s): 0.15 | learning rate: 1.958E-04 | global batch size: 256 | lm loss: 3.807446E+00 | grad norm: 0.340 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1664.774 | TFLOPs: 26.11 | 7: iteration 35300/ 331103 | consumed samples: 9036800 | consumed tokens: 18507366400 | elapsed time per iteration (s): 0.15 | learning rate: 1.958E-04 | global batch size: 256 | lm loss: 3.801234E+00 | grad norm: 0.288 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1664.282 | TFLOPs: 26.10 | 7: iteration 35400/ 331103 | consumed samples: 9062400 | consumed tokens: 18559795200 | elapsed time per iteration (s): 0.16 | learning rate: 1.958E-04 | global batch size: 256 | lm loss: 3.805229E+00 | grad norm: 0.309 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1629.550 | TFLOPs: 25.56 | 7: iteration 35500/ 331103 | consumed samples: 9088000 | consumed tokens: 18612224000 | elapsed time per iteration (s): 0.16 | learning rate: 1.958E-04 | global batch size: 256 | lm loss: 3.803124E+00 | grad norm: 0.309 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1638.981 | TFLOPs: 25.70 | 7: iteration 35600/ 331103 | consumed samples: 9113600 | consumed tokens: 18664652800 | elapsed time per iteration (s): 0.16 | learning rate: 1.957E-04 | global batch size: 256 | lm loss: 3.801339E+00 | grad norm: 0.358 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1644.722 | TFLOPs: 25.79 | 7: iteration 35700/ 331103 | consumed samples: 9139200 | consumed tokens: 18717081600 | elapsed time per iteration (s): 0.15 | learning rate: 1.957E-04 | global batch size: 256 | lm loss: 3.803651E+00 | grad norm: 0.302 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1655.734 | TFLOPs: 25.97 | 7: iteration 35800/ 331103 | consumed samples: 9164800 | consumed tokens: 18769510400 | elapsed time per iteration (s): 0.16 | learning rate: 1.957E-04 | global batch size: 256 | lm loss: 3.800175E+00 | grad norm: 0.330 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1646.692 | TFLOPs: 25.82 | 7: iteration 35900/ 331103 | consumed samples: 9190400 | consumed tokens: 18821939200 | elapsed time per iteration (s): 0.15 | learning rate: 1.956E-04 | global batch size: 256 | lm loss: 3.807513E+00 | grad norm: 0.357 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1659.487 | TFLOPs: 26.02 | 0: [2023-03-17 11:35:46,063] [INFO] [logging.py:68:log_dist] [Rank 0] step=36000, skipped=0, lr=[0.00019561912432034038, 0.00019561912432034038, 0.00019561912432034038], mom=[(0.9, 0.999), (0.9, 0.999), (0.9, 0.999)] 7: iteration 36000/ 331103 | consumed samples: 9216000 | consumed tokens: 18874368000 | elapsed time per iteration (s): 0.15 | learning rate: 1.956E-04 | global batch size: 256 | lm loss: 3.796548E+00 | grad norm: 0.284 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1661.823 | TFLOPs: 26.06 | 0: steps: 36000 loss: 3.7538 iter time (s): 0.154 samples/sec: 1666.058 7: iteration 36100/ 331103 | consumed samples: 9241600 | consumed tokens: 18926796800 | elapsed time per iteration (s): 0.16 | learning rate: 1.956E-04 | global batch size: 256 | lm loss: 3.798423E+00 | grad norm: 0.328 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1650.922 | TFLOPs: 25.89 | 7: iteration 36200/ 331103 | consumed samples: 9267200 | consumed tokens: 18979225600 | elapsed time per iteration (s): 0.15 | learning rate: 1.956E-04 | global batch size: 256 | lm loss: 3.796633E+00 | grad norm: 0.324 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1656.038 | TFLOPs: 25.97 | 7: iteration 36300/ 331103 | consumed samples: 9292800 | consumed tokens: 19031654400 | elapsed time per iteration (s): 0.15 | learning rate: 1.955E-04 | global batch size: 256 | lm loss: 3.798434E+00 | grad norm: 0.289 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1652.544 | TFLOPs: 25.92 | 7: iteration 36400/ 331103 | consumed samples: 9318400 | consumed tokens: 19084083200 | elapsed time per iteration (s): 0.16 | learning rate: 1.955E-04 | global batch size: 256 | lm loss: 3.802041E+00 | grad norm: 0.306 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1651.054 | TFLOPs: 25.89 | 7: iteration 36500/ 331103 | consumed samples: 9344000 | consumed tokens: 19136512000 | elapsed time per iteration (s): 0.16 | learning rate: 1.955E-04 | global batch size: 256 | lm loss: 3.797764E+00 | grad norm: 0.303 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1644.236 | TFLOPs: 25.79 | 7: iteration 36600/ 331103 | consumed samples: 9369600 | consumed tokens: 19188940800 | elapsed time per iteration (s): 0.16 | learning rate: 1.955E-04 | global batch size: 256 | lm loss: 3.795846E+00 | grad norm: 0.316 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1645.003 | TFLOPs: 25.80 | 7: iteration 36700/ 331103 | consumed samples: 9395200 | consumed tokens: 19241369600 | elapsed time per iteration (s): 0.15 | learning rate: 1.954E-04 | global batch size: 256 | lm loss: 3.800725E+00 | grad norm: 0.376 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1661.872 | TFLOPs: 26.06 | 7: iteration 36800/ 331103 | consumed samples: 9420800 | consumed tokens: 19293798400 | elapsed time per iteration (s): 0.15 | learning rate: 1.954E-04 | global batch size: 256 | lm loss: 3.797778E+00 | grad norm: 0.274 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1673.103 | TFLOPs: 26.24 | 7: iteration 36900/ 331103 | consumed samples: 9446400 | consumed tokens: 19346227200 | elapsed time per iteration (s): 0.15 | learning rate: 1.954E-04 | global batch size: 256 | lm loss: 3.798614E+00 | grad norm: 0.309 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1660.564 | TFLOPs: 26.04 | 7: iteration 37000/ 331103 | consumed samples: 9472000 | consumed tokens: 19398656000 | elapsed time per iteration (s): 0.15 | learning rate: 1.953E-04 | global batch size: 256 | lm loss: 3.800693E+00 | grad norm: 0.313 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1653.996 | TFLOPs: 25.94 | 7: iteration 37100/ 331103 | consumed samples: 9497600 | consumed tokens: 19451084800 | elapsed time per iteration (s): 0.16 | learning rate: 1.953E-04 | global batch size: 256 | lm loss: 3.792787E+00 | grad norm: 0.320 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1635.354 | TFLOPs: 25.65 | 7: iteration 37200/ 331103 | consumed samples: 9523200 | consumed tokens: 19503513600 | elapsed time per iteration (s): 0.15 | learning rate: 1.953E-04 | global batch size: 256 | lm loss: 3.799473E+00 | grad norm: 0.336 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1656.199 | TFLOPs: 25.97 | 7: iteration 37300/ 331103 | consumed samples: 9548800 | consumed tokens: 19555942400 | elapsed time per iteration (s): 0.16 | learning rate: 1.953E-04 | global batch size: 256 | lm loss: 3.800047E+00 | grad norm: 0.325 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1635.865 | TFLOPs: 25.65 | 7: iteration 37400/ 331103 | consumed samples: 9574400 | consumed tokens: 19608371200 | elapsed time per iteration (s): 0.15 | learning rate: 1.952E-04 | global batch size: 256 | lm loss: 3.797474E+00 | grad norm: 0.312 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1663.898 | TFLOPs: 26.09 | 7: iteration 37500/ 331103 | consumed samples: 9600000 | consumed tokens: 19660800000 | elapsed time per iteration (s): 0.16 | learning rate: 1.952E-04 | global batch size: 256 | lm loss: 3.798620E+00 | grad norm: 0.287 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1641.069 | TFLOPs: 25.74 | 7: iteration 37600/ 331103 | consumed samples: 9625600 | consumed tokens: 19713228800 | elapsed time per iteration (s): 0.15 | learning rate: 1.952E-04 | global batch size: 256 | lm loss: 3.798758E+00 | grad norm: 0.318 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1654.965 | TFLOPs: 25.95 | 7: iteration 37700/ 331103 | consumed samples: 9651200 | consumed tokens: 19765657600 | elapsed time per iteration (s): 0.15 | learning rate: 1.952E-04 | global batch size: 256 | lm loss: 3.792908E+00 | grad norm: 0.308 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1665.666 | TFLOPs: 26.12 | 7: iteration 37800/ 331103 | consumed samples: 9676800 | consumed tokens: 19818086400 | elapsed time per iteration (s): 0.15 | learning rate: 1.951E-04 | global batch size: 256 | lm loss: 3.792779E+00 | grad norm: 0.288 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1664.185 | TFLOPs: 26.10 | 7: iteration 37900/ 331103 | consumed samples: 9702400 | consumed tokens: 19870515200 | elapsed time per iteration (s): 0.15 | learning rate: 1.951E-04 | global batch size: 256 | lm loss: 3.795125E+00 | grad norm: 0.312 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1660.705 | TFLOPs: 26.04 | 0: [2023-03-17 11:40:55,611] [INFO] [logging.py:68:log_dist] [Rank 0] step=38000, skipped=0, lr=[0.0001950717523994859, 0.0001950717523994859, 0.0001950717523994859], mom=[(0.9, 0.999), (0.9, 0.999), (0.9, 0.999)] 7: iteration 38000/ 331103 | consumed samples: 9728000 | consumed tokens: 19922944000 | elapsed time per iteration (s): 0.15 | learning rate: 1.951E-04 | global batch size: 256 | lm loss: 3.796673E+00 | grad norm: 0.293 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1654.461 | TFLOPs: 25.95 | 0: steps: 38000 loss: 3.7624 iter time (s): 0.154 samples/sec: 1663.953 7: iteration 38100/ 331103 | consumed samples: 9753600 | consumed tokens: 19975372800 | elapsed time per iteration (s): 0.15 | learning rate: 1.950E-04 | global batch size: 256 | lm loss: 3.797327E+00 | grad norm: 0.342 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1658.030 | TFLOPs: 26.00 | 7: iteration 38200/ 331103 | consumed samples: 9779200 | consumed tokens: 20027801600 | elapsed time per iteration (s): 0.15 | learning rate: 1.950E-04 | global batch size: 256 | lm loss: 3.793367E+00 | grad norm: 0.332 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1664.280 | TFLOPs: 26.10 | 7: iteration 38300/ 331103 | consumed samples: 9804800 | consumed tokens: 20080230400 | elapsed time per iteration (s): 0.15 | learning rate: 1.950E-04 | global batch size: 256 | lm loss: 3.792845E+00 | grad norm: 0.333 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1658.674 | TFLOPs: 26.01 | 7: iteration 38400/ 331103 | consumed samples: 9830400 | consumed tokens: 20132659200 | elapsed time per iteration (s): 0.15 | learning rate: 1.950E-04 | global batch size: 256 | lm loss: 3.790394E+00 | grad norm: 0.304 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1662.626 | TFLOPs: 26.07 | 7: iteration 38500/ 331103 | consumed samples: 9856000 | consumed tokens: 20185088000 | elapsed time per iteration (s): 0.15 | learning rate: 1.949E-04 | global batch size: 256 | lm loss: 3.796718E+00 | grad norm: 0.291 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1663.106 | TFLOPs: 26.08 | 7: iteration 38600/ 331103 | consumed samples: 9881600 | consumed tokens: 20237516800 | elapsed time per iteration (s): 0.15 | learning rate: 1.949E-04 | global batch size: 256 | lm loss: 3.793406E+00 | grad norm: 0.279 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1659.496 | TFLOPs: 26.03 | 7: iteration 38700/ 331103 | consumed samples: 9907200 | consumed tokens: 20289945600 | elapsed time per iteration (s): 0.16 | learning rate: 1.949E-04 | global batch size: 256 | lm loss: 3.795109E+00 | grad norm: 0.304 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1614.466 | TFLOPs: 25.32 | 7: iteration 38800/ 331103 | consumed samples: 9932800 | consumed tokens: 20342374400 | elapsed time per iteration (s): 0.15 | learning rate: 1.948E-04 | global batch size: 256 | lm loss: 3.795443E+00 | grad norm: 0.343 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1664.644 | TFLOPs: 26.11 | 7: iteration 38900/ 331103 | consumed samples: 9958400 | consumed tokens: 20394803200 | elapsed time per iteration (s): 0.15 | learning rate: 1.948E-04 | global batch size: 256 | lm loss: 3.796027E+00 | grad norm: 0.322 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1662.946 | TFLOPs: 26.08 | 7: iteration 39000/ 331103 | consumed samples: 9984000 | consumed tokens: 20447232000 | elapsed time per iteration (s): 0.16 | learning rate: 1.948E-04 | global batch size: 256 | lm loss: 3.796516E+00 | grad norm: 0.311 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1646.869 | TFLOPs: 25.83 | 7: iteration 39100/ 331103 | consumed samples: 10009600 | consumed tokens: 20499660800 | elapsed time per iteration (s): 0.15 | learning rate: 1.948E-04 | global batch size: 256 | lm loss: 3.791613E+00 | grad norm: 0.296 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1662.379 | TFLOPs: 26.07 | 7: iteration 39200/ 331103 | consumed samples: 10035200 | consumed tokens: 20552089600 | elapsed time per iteration (s): 0.15 | learning rate: 1.947E-04 | global batch size: 256 | lm loss: 3.792505E+00 | grad norm: 0.278 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1660.475 | TFLOPs: 26.04 | 7: iteration 39300/ 331103 | consumed samples: 10060800 | consumed tokens: 20604518400 | elapsed time per iteration (s): 0.15 | learning rate: 1.947E-04 | global batch size: 256 | lm loss: 3.788555E+00 | grad norm: 0.282 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1659.649 | TFLOPs: 26.03 | 7: iteration 39400/ 331103 | consumed samples: 10086400 | consumed tokens: 20656947200 | elapsed time per iteration (s): 0.17 | learning rate: 1.947E-04 | global batch size: 256 | lm loss: 3.788400E+00 | grad norm: 0.299 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1491.779 | TFLOPs: 23.39 | 7: iteration 39500/ 331103 | consumed samples: 10112000 | consumed tokens: 20709376000 | elapsed time per iteration (s): 0.16 | learning rate: 1.946E-04 | global batch size: 256 | lm loss: 3.789710E+00 | grad norm: 0.304 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1637.298 | TFLOPs: 25.68 | 7: iteration 39600/ 331103 | consumed samples: 10137600 | consumed tokens: 20761804800 | elapsed time per iteration (s): 0.15 | learning rate: 1.946E-04 | global batch size: 256 | lm loss: 3.788962E+00 | grad norm: 0.281 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1665.023 | TFLOPs: 26.11 | 7: iteration 39700/ 331103 | consumed samples: 10163200 | consumed tokens: 20814233600 | elapsed time per iteration (s): 0.15 | learning rate: 1.946E-04 | global batch size: 256 | lm loss: 3.792566E+00 | grad norm: 0.318 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1672.569 | TFLOPs: 26.23 | 7: iteration 39800/ 331103 | consumed samples: 10188800 | consumed tokens: 20866662400 | elapsed time per iteration (s): 0.15 | learning rate: 1.946E-04 | global batch size: 256 | lm loss: 3.789366E+00 | grad norm: 0.307 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1669.181 | TFLOPs: 26.18 | 7: iteration 39900/ 331103 | consumed samples: 10214400 | consumed tokens: 20919091200 | elapsed time per iteration (s): 0.15 | learning rate: 1.945E-04 | global batch size: 256 | lm loss: 3.785238E+00 | grad norm: 0.326 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1668.854 | TFLOPs: 26.17 | 0: [2023-03-17 11:46:06,082] [INFO] [logging.py:68:log_dist] [Rank 0] step=40000, skipped=0, lr=[0.00019449312448298106, 0.00019449312448298106, 0.00019449312448298106], mom=[(0.9, 0.999), (0.9, 0.999), (0.9, 0.999)] 7: iteration 40000/ 331103 | consumed samples: 10240000 | consumed tokens: 20971520000 | elapsed time per iteration (s): 0.15 | learning rate: 1.945E-04 | global batch size: 256 | lm loss: 3.790216E+00 | grad norm: 0.276 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1659.325 | TFLOPs: 26.02 | 0: steps: 40000 loss: 3.8001 iter time (s): 0.154 samples/sec: 1658.422 7: ------------------------------------------------------------------------------------------------ 7: validation loss at iteration 40000 | lm loss value: 3.923141E+00 | lm loss PPL: 5.055903E+01 | 7: ------------------------------------------------------------------------------------------------ 0: saving checkpoint at iteration 40000 to checkpoints_44m174b100m 0: [2023-03-17 11:46:06,156] [INFO] [logging.py:68:log_dist] [Rank 0] [Torch] Checkpoint global_step40000 is begin to save! 0: [2023-03-17 11:46:06,159] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step40000/layer_01-model_00-model_states.pt... 0: [2023-03-17 11:46:06,223] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step40000/layer_01-model_00-model_states.pt. 0: [2023-03-17 11:46:06,224] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step40000/layer_03-model_00-model_states.pt... 0: [2023-03-17 11:46:06,232] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step40000/layer_03-model_00-model_states.pt. 0: [2023-03-17 11:46:06,233] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step40000/layer_04-model_00-model_states.pt... 0: [2023-03-17 11:46:06,241] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step40000/layer_04-model_00-model_states.pt. 0: [2023-03-17 11:46:06,241] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step40000/layer_05-model_00-model_states.pt... 0: [2023-03-17 11:46:06,249] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step40000/layer_05-model_00-model_states.pt. 0: [2023-03-17 11:46:06,249] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step40000/layer_06-model_00-model_states.pt... 0: [2023-03-17 11:46:06,257] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step40000/layer_06-model_00-model_states.pt. 0: [2023-03-17 11:46:06,257] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step40000/layer_07-model_00-model_states.pt... 0: [2023-03-17 11:46:06,265] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step40000/layer_07-model_00-model_states.pt. 0: [2023-03-17 11:46:06,265] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step40000/layer_08-model_00-model_states.pt... 0: [2023-03-17 11:46:06,273] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step40000/layer_08-model_00-model_states.pt. 0: [2023-03-17 11:46:06,273] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step40000/layer_09-model_00-model_states.pt... 0: [2023-03-17 11:46:06,281] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step40000/layer_09-model_00-model_states.pt. 0: [2023-03-17 11:46:06,282] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step40000/layer_10-model_00-model_states.pt... 0: [2023-03-17 11:46:06,289] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step40000/layer_10-model_00-model_states.pt. 0: [2023-03-17 11:46:06,289] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step40000/layer_12-model_00-model_states.pt... 0: [2023-03-17 11:46:06,290] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step40000/layer_12-model_00-model_states.pt. 0: [2023-03-17 11:46:06,291] [INFO] [logging.py:68:log_dist] [Rank 0] Saving model checkpoint: checkpoints_44m174b100m/global_step40000/mp_rank_00_model_states.pt 0: [2023-03-17 11:46:06,291] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step40000/mp_rank_00_model_states.pt... 0: [2023-03-17 11:46:06,294] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step40000/mp_rank_00_model_states.pt. 0: [2023-03-17 11:46:06,311] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step40000/bf16_zero_pp_rank_0_mp_rank_00_optim_states.pt... 0: [2023-03-17 11:46:06,311] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step40000/bf16_zero_pp_rank_5_mp_rank_00_optim_states.pt... 0: [2023-03-17 11:46:06,311] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step40000/bf16_zero_pp_rank_6_mp_rank_00_optim_states.pt... 0: [2023-03-17 11:46:06,311] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step40000/bf16_zero_pp_rank_3_mp_rank_00_optim_states.pt... 0: [2023-03-17 11:46:06,311] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step40000/bf16_zero_pp_rank_2_mp_rank_00_optim_states.pt... 0: [2023-03-17 11:46:06,311] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step40000/bf16_zero_pp_rank_7_mp_rank_00_optim_states.pt... 7: [2023-03-17 11:46:06,311] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step40000/bf16_zero_pp_rank_56_mp_rank_00_optim_states.pt... 7: [2023-03-17 11:46:06,311] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step40000/bf16_zero_pp_rank_60_mp_rank_00_optim_states.pt... 7: [2023-03-17 11:46:06,311] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step40000/bf16_zero_pp_rank_63_mp_rank_00_optim_states.pt... 7: [2023-03-17 11:46:06,311] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step40000/bf16_zero_pp_rank_57_mp_rank_00_optim_states.pt... 7: [2023-03-17 11:46:06,311] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step40000/bf16_zero_pp_rank_61_mp_rank_00_optim_states.pt... 0: [2023-03-17 11:46:06,311] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step40000/bf16_zero_pp_rank_1_mp_rank_00_optim_states.pt... 2: [2023-03-17 11:46:06,311] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step40000/bf16_zero_pp_rank_20_mp_rank_00_optim_states.pt... 2: [2023-03-17 11:46:06,311] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step40000/bf16_zero_pp_rank_19_mp_rank_00_optim_states.pt... 2: [2023-03-17 11:46:06,311] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step40000/bf16_zero_pp_rank_21_mp_rank_00_optim_states.pt... 2: [2023-03-17 11:46:06,311] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step40000/bf16_zero_pp_rank_23_mp_rank_00_optim_states.pt... 2: [2023-03-17 11:46:06,311] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step40000/bf16_zero_pp_rank_16_mp_rank_00_optim_states.pt... 4: [2023-03-17 11:46:06,311] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step40000/bf16_zero_pp_rank_39_mp_rank_00_optim_states.pt... 4: [2023-03-17 11:46:06,311] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step40000/bf16_zero_pp_rank_38_mp_rank_00_optim_states.pt... 4: [2023-03-17 11:46:06,311] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step40000/bf16_zero_pp_rank_35_mp_rank_00_optim_states.pt... 4: [2023-03-17 11:46:06,311] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step40000/bf16_zero_pp_rank_32_mp_rank_00_optim_states.pt... 4: [2023-03-17 11:46:06,311] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step40000/bf16_zero_pp_rank_33_mp_rank_00_optim_states.pt... 6: [2023-03-17 11:46:06,311] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step40000/bf16_zero_pp_rank_52_mp_rank_00_optim_states.pt... 6: [2023-03-17 11:46:06,311] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step40000/bf16_zero_pp_rank_54_mp_rank_00_optim_states.pt... 6: [2023-03-17 11:46:06,311] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step40000/bf16_zero_pp_rank_49_mp_rank_00_optim_states.pt... 6: [2023-03-17 11:46:06,311] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step40000/bf16_zero_pp_rank_48_mp_rank_00_optim_states.pt... 6: [2023-03-17 11:46:06,311] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step40000/bf16_zero_pp_rank_55_mp_rank_00_optim_states.pt... 5: [2023-03-17 11:46:06,311] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step40000/bf16_zero_pp_rank_46_mp_rank_00_optim_states.pt... 5: [2023-03-17 11:46:06,311] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step40000/bf16_zero_pp_rank_44_mp_rank_00_optim_states.pt... 5: [2023-03-17 11:46:06,311] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step40000/bf16_zero_pp_rank_40_mp_rank_00_optim_states.pt... 5: [2023-03-17 11:46:06,311] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step40000/bf16_zero_pp_rank_41_mp_rank_00_optim_states.pt... 5: [2023-03-17 11:46:06,311] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step40000/bf16_zero_pp_rank_45_mp_rank_00_optim_states.pt... 1: [2023-03-17 11:46:06,311] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step40000/bf16_zero_pp_rank_15_mp_rank_00_optim_states.pt... 1: [2023-03-17 11:46:06,311] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step40000/bf16_zero_pp_rank_12_mp_rank_00_optim_states.pt... 1: [2023-03-17 11:46:06,311] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step40000/bf16_zero_pp_rank_13_mp_rank_00_optim_states.pt... 1: [2023-03-17 11:46:06,311] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step40000/bf16_zero_pp_rank_10_mp_rank_00_optim_states.pt... 1: [2023-03-17 11:46:06,311] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step40000/bf16_zero_pp_rank_9_mp_rank_00_optim_states.pt... 0: [2023-03-17 11:46:06,311] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step40000/bf16_zero_pp_rank_4_mp_rank_00_optim_states.pt... 3: [2023-03-17 11:46:06,311] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step40000/bf16_zero_pp_rank_31_mp_rank_00_optim_states.pt... 3: [2023-03-17 11:46:06,311] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step40000/bf16_zero_pp_rank_30_mp_rank_00_optim_states.pt... 3: [2023-03-17 11:46:06,311] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step40000/bf16_zero_pp_rank_29_mp_rank_00_optim_states.pt... 3: [2023-03-17 11:46:06,311] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step40000/bf16_zero_pp_rank_24_mp_rank_00_optim_states.pt... 3: [2023-03-17 11:46:06,311] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step40000/bf16_zero_pp_rank_28_mp_rank_00_optim_states.pt... 2: [2023-03-17 11:46:06,311] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step40000/bf16_zero_pp_rank_17_mp_rank_00_optim_states.pt... 2: [2023-03-17 11:46:06,311] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step40000/bf16_zero_pp_rank_22_mp_rank_00_optim_states.pt... 2: [2023-03-17 11:46:06,311] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step40000/bf16_zero_pp_rank_18_mp_rank_00_optim_states.pt... 4: [2023-03-17 11:46:06,311] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step40000/bf16_zero_pp_rank_36_mp_rank_00_optim_states.pt... 4: [2023-03-17 11:46:06,311] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step40000/bf16_zero_pp_rank_34_mp_rank_00_optim_states.pt... 6: [2023-03-17 11:46:06,311] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step40000/bf16_zero_pp_rank_51_mp_rank_00_optim_states.pt... 6: [2023-03-17 11:46:06,311] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step40000/bf16_zero_pp_rank_50_mp_rank_00_optim_states.pt... 6: [2023-03-17 11:46:06,311] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step40000/bf16_zero_pp_rank_53_mp_rank_00_optim_states.pt... 7: [2023-03-17 11:46:06,311] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step40000/bf16_zero_pp_rank_59_mp_rank_00_optim_states.pt... 5: [2023-03-17 11:46:06,311] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step40000/bf16_zero_pp_rank_42_mp_rank_00_optim_states.pt... 5: [2023-03-17 11:46:06,311] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step40000/bf16_zero_pp_rank_43_mp_rank_00_optim_states.pt... 5: [2023-03-17 11:46:06,311] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step40000/bf16_zero_pp_rank_47_mp_rank_00_optim_states.pt... 1: [2023-03-17 11:46:06,311] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step40000/bf16_zero_pp_rank_11_mp_rank_00_optim_states.pt... 1: [2023-03-17 11:46:06,311] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step40000/bf16_zero_pp_rank_8_mp_rank_00_optim_states.pt... 1: [2023-03-17 11:46:06,311] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step40000/bf16_zero_pp_rank_14_mp_rank_00_optim_states.pt... 3: [2023-03-17 11:46:06,311] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step40000/bf16_zero_pp_rank_25_mp_rank_00_optim_states.pt... 3: [2023-03-17 11:46:06,311] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step40000/bf16_zero_pp_rank_26_mp_rank_00_optim_states.pt... 3: [2023-03-17 11:46:06,311] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step40000/bf16_zero_pp_rank_27_mp_rank_00_optim_states.pt... 4: [2023-03-17 11:46:06,311] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step40000/bf16_zero_pp_rank_37_mp_rank_00_optim_states.pt... 7: [2023-03-17 11:46:06,311] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step40000/bf16_zero_pp_rank_58_mp_rank_00_optim_states.pt... 7: [2023-03-17 11:46:06,311] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step40000/bf16_zero_pp_rank_62_mp_rank_00_optim_states.pt... 0: [2023-03-17 11:46:06,324] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step40000/bf16_zero_pp_rank_0_mp_rank_00_optim_states.pt. 0: [2023-03-17 11:46:06,325] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step40000/bf16_zero_pp_rank_0_mp_rank_00_optim_states.pt 0: [2023-03-17 11:46:06,325] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step40000 is ready now! 2: [2023-03-17 11:46:06,326] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step40000/bf16_zero_pp_rank_19_mp_rank_00_optim_states.pt. 1: [2023-03-17 11:46:06,326] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step40000/bf16_zero_pp_rank_15_mp_rank_00_optim_states.pt. 2: [2023-03-17 11:46:06,326] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step40000/bf16_zero_pp_rank_19_mp_rank_00_optim_states.pt 1: [2023-03-17 11:46:06,326] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step40000/bf16_zero_pp_rank_15_mp_rank_00_optim_states.pt 2: [2023-03-17 11:46:06,326] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step40000 is ready now! 1: [2023-03-17 11:46:06,326] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step40000 is ready now! 2: [2023-03-17 11:46:06,326] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step40000/bf16_zero_pp_rank_20_mp_rank_00_optim_states.pt. 2: [2023-03-17 11:46:06,326] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step40000/bf16_zero_pp_rank_20_mp_rank_00_optim_states.pt 3: [2023-03-17 11:46:06,326] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step40000/bf16_zero_pp_rank_31_mp_rank_00_optim_states.pt. 2: [2023-03-17 11:46:06,326] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step40000 is ready now! 3: [2023-03-17 11:46:06,326] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step40000/bf16_zero_pp_rank_31_mp_rank_00_optim_states.pt 3: [2023-03-17 11:46:06,326] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step40000 is ready now! 0: [2023-03-17 11:46:06,326] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step40000/bf16_zero_pp_rank_5_mp_rank_00_optim_states.pt. 0: [2023-03-17 11:46:06,326] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step40000/bf16_zero_pp_rank_5_mp_rank_00_optim_states.pt 0: [2023-03-17 11:46:06,326] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step40000 is ready now! 1: [2023-03-17 11:46:06,327] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step40000/bf16_zero_pp_rank_12_mp_rank_00_optim_states.pt. 1: [2023-03-17 11:46:06,327] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step40000/bf16_zero_pp_rank_12_mp_rank_00_optim_states.pt 1: [2023-03-17 11:46:06,327] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step40000 is ready now! 7: [2023-03-17 11:46:06,327] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step40000/bf16_zero_pp_rank_56_mp_rank_00_optim_states.pt. 7: [2023-03-17 11:46:06,327] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step40000/bf16_zero_pp_rank_60_mp_rank_00_optim_states.pt. 7: [2023-03-17 11:46:06,327] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step40000/bf16_zero_pp_rank_57_mp_rank_00_optim_states.pt. 7: [2023-03-17 11:46:06,327] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step40000/bf16_zero_pp_rank_56_mp_rank_00_optim_states.pt 7: [2023-03-17 11:46:06,327] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step40000/bf16_zero_pp_rank_60_mp_rank_00_optim_states.pt 7: [2023-03-17 11:46:06,327] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step40000/bf16_zero_pp_rank_57_mp_rank_00_optim_states.pt 7: [2023-03-17 11:46:06,327] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step40000 is ready now! 7: [2023-03-17 11:46:06,328] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step40000 is ready now! 7: [2023-03-17 11:46:06,328] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step40000 is ready now! 2: [2023-03-17 11:46:06,328] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step40000/bf16_zero_pp_rank_23_mp_rank_00_optim_states.pt. 4: [2023-03-17 11:46:06,328] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step40000/bf16_zero_pp_rank_39_mp_rank_00_optim_states.pt. 4: [2023-03-17 11:46:06,328] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step40000/bf16_zero_pp_rank_38_mp_rank_00_optim_states.pt. 4: [2023-03-17 11:46:06,328] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step40000/bf16_zero_pp_rank_39_mp_rank_00_optim_states.pt 2: [2023-03-17 11:46:06,328] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step40000/bf16_zero_pp_rank_23_mp_rank_00_optim_states.pt 4: [2023-03-17 11:46:06,328] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step40000/bf16_zero_pp_rank_38_mp_rank_00_optim_states.pt 4: [2023-03-17 11:46:06,328] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step40000 is ready now! 2: [2023-03-17 11:46:06,328] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step40000 is ready now! 4: [2023-03-17 11:46:06,328] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step40000 is ready now! 3: [2023-03-17 11:46:06,328] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step40000/bf16_zero_pp_rank_30_mp_rank_00_optim_states.pt. 3: [2023-03-17 11:46:06,328] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step40000/bf16_zero_pp_rank_30_mp_rank_00_optim_states.pt 4: [2023-03-17 11:46:06,328] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step40000/bf16_zero_pp_rank_36_mp_rank_00_optim_states.pt. 3: [2023-03-17 11:46:06,328] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step40000 is ready now! 4: [2023-03-17 11:46:06,328] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step40000/bf16_zero_pp_rank_36_mp_rank_00_optim_states.pt 4: [2023-03-17 11:46:06,328] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step40000 is ready now! 0: [2023-03-17 11:46:06,328] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step40000/bf16_zero_pp_rank_4_mp_rank_00_optim_states.pt. 0: [2023-03-17 11:46:06,328] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step40000/bf16_zero_pp_rank_4_mp_rank_00_optim_states.pt 0: [2023-03-17 11:46:06,328] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step40000 is ready now! 7: [2023-03-17 11:46:06,328] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step40000/bf16_zero_pp_rank_63_mp_rank_00_optim_states.pt. 7: [2023-03-17 11:46:06,328] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step40000/bf16_zero_pp_rank_63_mp_rank_00_optim_states.pt 5: [2023-03-17 11:46:06,328] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step40000/bf16_zero_pp_rank_46_mp_rank_00_optim_states.pt. 5: [2023-03-17 11:46:06,328] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step40000/bf16_zero_pp_rank_44_mp_rank_00_optim_states.pt. 7: [2023-03-17 11:46:06,328] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step40000 is ready now! 5: [2023-03-17 11:46:06,328] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step40000/bf16_zero_pp_rank_44_mp_rank_00_optim_states.pt 5: [2023-03-17 11:46:06,328] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step40000/bf16_zero_pp_rank_46_mp_rank_00_optim_states.pt 5: [2023-03-17 11:46:06,328] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step40000 is ready now! 5: [2023-03-17 11:46:06,328] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step40000 is ready now! 1: [2023-03-17 11:46:06,328] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step40000/bf16_zero_pp_rank_13_mp_rank_00_optim_states.pt. 1: [2023-03-17 11:46:06,329] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step40000/bf16_zero_pp_rank_13_mp_rank_00_optim_states.pt 6: [2023-03-17 11:46:06,328] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step40000/bf16_zero_pp_rank_52_mp_rank_00_optim_states.pt. 6: [2023-03-17 11:46:06,328] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step40000/bf16_zero_pp_rank_51_mp_rank_00_optim_states.pt. 1: [2023-03-17 11:46:06,329] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step40000 is ready now! 6: [2023-03-17 11:46:06,329] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step40000/bf16_zero_pp_rank_51_mp_rank_00_optim_states.pt 6: [2023-03-17 11:46:06,329] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step40000/bf16_zero_pp_rank_52_mp_rank_00_optim_states.pt 6: [2023-03-17 11:46:06,329] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step40000 is ready now! 6: [2023-03-17 11:46:06,329] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step40000 is ready now! 5: [2023-03-17 11:46:06,329] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step40000/bf16_zero_pp_rank_40_mp_rank_00_optim_states.pt. 5: [2023-03-17 11:46:06,329] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step40000/bf16_zero_pp_rank_40_mp_rank_00_optim_states.pt 5: [2023-03-17 11:46:06,329] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step40000 is ready now! 6: [2023-03-17 11:46:06,329] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step40000/bf16_zero_pp_rank_54_mp_rank_00_optim_states.pt. 6: [2023-03-17 11:46:06,329] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step40000/bf16_zero_pp_rank_54_mp_rank_00_optim_states.pt 6: [2023-03-17 11:46:06,329] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step40000 is ready now! 4: [2023-03-17 11:46:06,329] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step40000/bf16_zero_pp_rank_35_mp_rank_00_optim_states.pt. 4: [2023-03-17 11:46:06,329] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step40000/bf16_zero_pp_rank_35_mp_rank_00_optim_states.pt 4: [2023-03-17 11:46:06,329] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step40000 is ready now! 7: [2023-03-17 11:46:06,329] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step40000/bf16_zero_pp_rank_61_mp_rank_00_optim_states.pt. 7: [2023-03-17 11:46:06,330] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step40000/bf16_zero_pp_rank_61_mp_rank_00_optim_states.pt 7: [2023-03-17 11:46:06,330] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step40000 is ready now! 2: [2023-03-17 11:46:06,330] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step40000/bf16_zero_pp_rank_16_mp_rank_00_optim_states.pt. 2: [2023-03-17 11:46:06,330] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step40000/bf16_zero_pp_rank_16_mp_rank_00_optim_states.pt 2: [2023-03-17 11:46:06,330] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step40000 is ready now! 0: [2023-03-17 11:46:06,330] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step40000/bf16_zero_pp_rank_1_mp_rank_00_optim_states.pt. 5: [2023-03-17 11:46:06,330] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step40000/bf16_zero_pp_rank_41_mp_rank_00_optim_states.pt. 3: [2023-03-17 11:46:06,330] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step40000/bf16_zero_pp_rank_24_mp_rank_00_optim_states.pt. 0: [2023-03-17 11:46:06,330] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step40000/bf16_zero_pp_rank_1_mp_rank_00_optim_states.pt 3: [2023-03-17 11:46:06,330] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step40000/bf16_zero_pp_rank_24_mp_rank_00_optim_states.pt 5: [2023-03-17 11:46:06,330] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step40000/bf16_zero_pp_rank_41_mp_rank_00_optim_states.pt 0: [2023-03-17 11:46:06,330] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step40000 is ready now! 6: [2023-03-17 11:46:06,330] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step40000/bf16_zero_pp_rank_49_mp_rank_00_optim_states.pt. 5: [2023-03-17 11:46:06,330] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step40000 is ready now! 3: [2023-03-17 11:46:06,330] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step40000 is ready now! 6: [2023-03-17 11:46:06,330] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step40000/bf16_zero_pp_rank_49_mp_rank_00_optim_states.pt 6: [2023-03-17 11:46:06,330] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step40000 is ready now! 1: [2023-03-17 11:46:06,330] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step40000/bf16_zero_pp_rank_10_mp_rank_00_optim_states.pt. 1: [2023-03-17 11:46:06,330] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step40000/bf16_zero_pp_rank_10_mp_rank_00_optim_states.pt 1: [2023-03-17 11:46:06,331] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step40000 is ready now! 4: [2023-03-17 11:46:06,330] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step40000/bf16_zero_pp_rank_32_mp_rank_00_optim_states.pt. 2: [2023-03-17 11:46:06,331] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step40000/bf16_zero_pp_rank_21_mp_rank_00_optim_states.pt. 4: [2023-03-17 11:46:06,331] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step40000/bf16_zero_pp_rank_32_mp_rank_00_optim_states.pt 2: [2023-03-17 11:46:06,331] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step40000/bf16_zero_pp_rank_21_mp_rank_00_optim_states.pt 4: [2023-03-17 11:46:06,331] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step40000 is ready now! 2: [2023-03-17 11:46:06,331] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step40000 is ready now! 0: [2023-03-17 11:46:06,331] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step40000/bf16_zero_pp_rank_7_mp_rank_00_optim_states.pt. 3: [2023-03-17 11:46:06,331] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step40000/bf16_zero_pp_rank_27_mp_rank_00_optim_states.pt. 0: [2023-03-17 11:46:06,331] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step40000/bf16_zero_pp_rank_7_mp_rank_00_optim_states.pt 3: [2023-03-17 11:46:06,331] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step40000/bf16_zero_pp_rank_27_mp_rank_00_optim_states.pt 0: [2023-03-17 11:46:06,331] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step40000 is ready now! 3: [2023-03-17 11:46:06,331] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step40000 is ready now! 7: [2023-03-17 11:46:06,331] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step40000/bf16_zero_pp_rank_59_mp_rank_00_optim_states.pt. 5: [2023-03-17 11:46:06,331] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step40000/bf16_zero_pp_rank_42_mp_rank_00_optim_states.pt. 7: [2023-03-17 11:46:06,331] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step40000/bf16_zero_pp_rank_59_mp_rank_00_optim_states.pt 5: [2023-03-17 11:46:06,331] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step40000/bf16_zero_pp_rank_42_mp_rank_00_optim_states.pt 1: [2023-03-17 11:46:06,331] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step40000/bf16_zero_pp_rank_9_mp_rank_00_optim_states.pt. 7: [2023-03-17 11:46:06,331] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step40000 is ready now! 5: [2023-03-17 11:46:06,332] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step40000 is ready now! 1: [2023-03-17 11:46:06,332] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step40000/bf16_zero_pp_rank_9_mp_rank_00_optim_states.pt 1: [2023-03-17 11:46:06,332] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step40000 is ready now! 0: [2023-03-17 11:46:06,332] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step40000/bf16_zero_pp_rank_3_mp_rank_00_optim_states.pt. 0: [2023-03-17 11:46:06,332] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step40000/bf16_zero_pp_rank_3_mp_rank_00_optim_states.pt 0: [2023-03-17 11:46:06,332] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step40000 is ready now! 2: [2023-03-17 11:46:06,332] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step40000/bf16_zero_pp_rank_17_mp_rank_00_optim_states.pt. 2: [2023-03-17 11:46:06,332] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step40000/bf16_zero_pp_rank_17_mp_rank_00_optim_states.pt 3: [2023-03-17 11:46:06,332] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step40000/bf16_zero_pp_rank_25_mp_rank_00_optim_states.pt. 2: [2023-03-17 11:46:06,332] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step40000 is ready now! 4: [2023-03-17 11:46:06,332] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step40000/bf16_zero_pp_rank_34_mp_rank_00_optim_states.pt. 3: [2023-03-17 11:46:06,332] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step40000/bf16_zero_pp_rank_25_mp_rank_00_optim_states.pt 3: [2023-03-17 11:46:06,332] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step40000 is ready now! 4: [2023-03-17 11:46:06,332] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step40000/bf16_zero_pp_rank_34_mp_rank_00_optim_states.pt 4: [2023-03-17 11:46:06,332] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step40000 is ready now! 6: [2023-03-17 11:46:06,332] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step40000/bf16_zero_pp_rank_48_mp_rank_00_optim_states.pt. 6: [2023-03-17 11:46:06,332] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step40000/bf16_zero_pp_rank_48_mp_rank_00_optim_states.pt 6: [2023-03-17 11:46:06,333] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step40000 is ready now! 6: [2023-03-17 11:46:06,333] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step40000/bf16_zero_pp_rank_55_mp_rank_00_optim_states.pt. 6: [2023-03-17 11:46:06,333] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step40000/bf16_zero_pp_rank_55_mp_rank_00_optim_states.pt 6: [2023-03-17 11:46:06,333] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step40000 is ready now! 7: [2023-03-17 11:46:06,333] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step40000/bf16_zero_pp_rank_62_mp_rank_00_optim_states.pt. 7: [2023-03-17 11:46:06,333] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step40000/bf16_zero_pp_rank_62_mp_rank_00_optim_states.pt 5: [2023-03-17 11:46:06,333] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step40000/bf16_zero_pp_rank_45_mp_rank_00_optim_states.pt. 7: [2023-03-17 11:46:06,333] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step40000 is ready now! 5: [2023-03-17 11:46:06,333] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step40000/bf16_zero_pp_rank_45_mp_rank_00_optim_states.pt 5: [2023-03-17 11:46:06,333] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step40000 is ready now! 1: [2023-03-17 11:46:06,333] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step40000/bf16_zero_pp_rank_11_mp_rank_00_optim_states.pt. 1: [2023-03-17 11:46:06,333] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step40000/bf16_zero_pp_rank_11_mp_rank_00_optim_states.pt 1: [2023-03-17 11:46:06,333] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step40000 is ready now! 0: [2023-03-17 11:46:06,333] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step40000/bf16_zero_pp_rank_6_mp_rank_00_optim_states.pt. 0: [2023-03-17 11:46:06,333] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step40000/bf16_zero_pp_rank_6_mp_rank_00_optim_states.pt 0: [2023-03-17 11:46:06,333] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step40000 is ready now! 4: [2023-03-17 11:46:06,333] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step40000/bf16_zero_pp_rank_33_mp_rank_00_optim_states.pt. 4: [2023-03-17 11:46:06,333] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step40000/bf16_zero_pp_rank_33_mp_rank_00_optim_states.pt 2: [2023-03-17 11:46:06,333] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step40000/bf16_zero_pp_rank_18_mp_rank_00_optim_states.pt. 4: [2023-03-17 11:46:06,333] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step40000 is ready now! 2: [2023-03-17 11:46:06,334] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step40000/bf16_zero_pp_rank_18_mp_rank_00_optim_states.pt 2: [2023-03-17 11:46:06,334] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step40000 is ready now! 1: [2023-03-17 11:46:06,334] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step40000/bf16_zero_pp_rank_14_mp_rank_00_optim_states.pt. 1: [2023-03-17 11:46:06,334] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step40000/bf16_zero_pp_rank_14_mp_rank_00_optim_states.pt 1: [2023-03-17 11:46:06,334] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step40000 is ready now! 3: [2023-03-17 11:46:06,334] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step40000/bf16_zero_pp_rank_26_mp_rank_00_optim_states.pt. 7: [2023-03-17 11:46:06,334] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step40000/bf16_zero_pp_rank_58_mp_rank_00_optim_states.pt. 3: [2023-03-17 11:46:06,334] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step40000/bf16_zero_pp_rank_26_mp_rank_00_optim_states.pt 7: [2023-03-17 11:46:06,334] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step40000/bf16_zero_pp_rank_58_mp_rank_00_optim_states.pt 1: [2023-03-17 11:46:06,334] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step40000/bf16_zero_pp_rank_8_mp_rank_00_optim_states.pt. 3: [2023-03-17 11:46:06,334] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step40000 is ready now! 7: [2023-03-17 11:46:06,334] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step40000 is ready now! 1: [2023-03-17 11:46:06,334] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step40000/bf16_zero_pp_rank_8_mp_rank_00_optim_states.pt 1: [2023-03-17 11:46:06,334] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step40000 is ready now! 3: [2023-03-17 11:46:06,334] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step40000/bf16_zero_pp_rank_29_mp_rank_00_optim_states.pt. 3: [2023-03-17 11:46:06,334] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step40000/bf16_zero_pp_rank_29_mp_rank_00_optim_states.pt 2: [2023-03-17 11:46:06,334] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step40000/bf16_zero_pp_rank_22_mp_rank_00_optim_states.pt. 3: [2023-03-17 11:46:06,334] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step40000 is ready now! 2: [2023-03-17 11:46:06,334] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step40000/bf16_zero_pp_rank_22_mp_rank_00_optim_states.pt 2: [2023-03-17 11:46:06,334] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step40000 is ready now! 6: [2023-03-17 11:46:06,334] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step40000/bf16_zero_pp_rank_53_mp_rank_00_optim_states.pt. 6: [2023-03-17 11:46:06,334] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step40000/bf16_zero_pp_rank_50_mp_rank_00_optim_states.pt. 6: [2023-03-17 11:46:06,334] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step40000/bf16_zero_pp_rank_50_mp_rank_00_optim_states.pt 6: [2023-03-17 11:46:06,334] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step40000/bf16_zero_pp_rank_53_mp_rank_00_optim_states.pt 6: [2023-03-17 11:46:06,334] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step40000 is ready now! 6: [2023-03-17 11:46:06,334] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step40000 is ready now! 5: [2023-03-17 11:46:06,334] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step40000/bf16_zero_pp_rank_47_mp_rank_00_optim_states.pt. 5: [2023-03-17 11:46:06,335] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step40000/bf16_zero_pp_rank_43_mp_rank_00_optim_states.pt. 5: [2023-03-17 11:46:06,335] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step40000/bf16_zero_pp_rank_47_mp_rank_00_optim_states.pt 5: [2023-03-17 11:46:06,335] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step40000 is ready now! 5: [2023-03-17 11:46:06,335] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step40000/bf16_zero_pp_rank_43_mp_rank_00_optim_states.pt 0: [2023-03-17 11:46:06,335] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step40000/bf16_zero_pp_rank_2_mp_rank_00_optim_states.pt. 5: [2023-03-17 11:46:06,335] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step40000 is ready now! 0: [2023-03-17 11:46:06,335] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step40000/bf16_zero_pp_rank_2_mp_rank_00_optim_states.pt 4: [2023-03-17 11:46:06,335] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step40000/bf16_zero_pp_rank_37_mp_rank_00_optim_states.pt. 0: [2023-03-17 11:46:06,335] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step40000 is ready now! 4: [2023-03-17 11:46:06,335] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step40000/bf16_zero_pp_rank_37_mp_rank_00_optim_states.pt 4: [2023-03-17 11:46:06,335] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step40000 is ready now! 3: [2023-03-17 11:46:06,335] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step40000/bf16_zero_pp_rank_28_mp_rank_00_optim_states.pt. 3: [2023-03-17 11:46:06,335] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step40000/bf16_zero_pp_rank_28_mp_rank_00_optim_states.pt 3: [2023-03-17 11:46:06,335] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step40000 is ready now! 0: successfully saved checkpoint at iteration 40000 to checkpoints_44m174b100m 7: time (ms) | save-checkpoint: 182.97 7: iteration 40100/ 331103 | consumed samples: 10265600 | consumed tokens: 21023948800 | elapsed time per iteration (s): 0.16 | learning rate: 1.945E-04 | global batch size: 256 | lm loss: 3.785896E+00 | grad norm: 0.322 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1634.805 | TFLOPs: 25.64 | 7: iteration 40200/ 331103 | consumed samples: 10291200 | consumed tokens: 21076377600 | elapsed time per iteration (s): 0.15 | learning rate: 1.944E-04 | global batch size: 256 | lm loss: 3.789151E+00 | grad norm: 0.334 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1665.118 | TFLOPs: 26.11 | 7: iteration 40300/ 331103 | consumed samples: 10316800 | consumed tokens: 21128806400 | elapsed time per iteration (s): 0.15 | learning rate: 1.944E-04 | global batch size: 256 | lm loss: 3.791356E+00 | grad norm: 0.414 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1654.300 | TFLOPs: 25.94 | 7: iteration 40400/ 331103 | consumed samples: 10342400 | consumed tokens: 21181235200 | elapsed time per iteration (s): 0.18 | learning rate: 1.944E-04 | global batch size: 256 | lm loss: 3.785977E+00 | grad norm: 0.301 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1407.903 | TFLOPs: 22.08 | 7: iteration 40500/ 331103 | consumed samples: 10368000 | consumed tokens: 21233664000 | elapsed time per iteration (s): 0.15 | learning rate: 1.943E-04 | global batch size: 256 | lm loss: 3.788598E+00 | grad norm: 0.288 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1663.608 | TFLOPs: 26.09 | 7: iteration 40600/ 331103 | consumed samples: 10393600 | consumed tokens: 21286092800 | elapsed time per iteration (s): 0.16 | learning rate: 1.943E-04 | global batch size: 256 | lm loss: 3.788507E+00 | grad norm: 0.294 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1637.159 | TFLOPs: 25.67 | 7: iteration 40700/ 331103 | consumed samples: 10419200 | consumed tokens: 21338521600 | elapsed time per iteration (s): 0.15 | learning rate: 1.943E-04 | global batch size: 256 | lm loss: 3.784997E+00 | grad norm: 0.324 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1656.486 | TFLOPs: 25.98 | 7: iteration 40800/ 331103 | consumed samples: 10444800 | consumed tokens: 21390950400 | elapsed time per iteration (s): 0.16 | learning rate: 1.943E-04 | global batch size: 256 | lm loss: 3.784278E+00 | grad norm: 0.314 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1572.599 | TFLOPs: 24.66 | 7: iteration 40900/ 331103 | consumed samples: 10470400 | consumed tokens: 21443379200 | elapsed time per iteration (s): 0.15 | learning rate: 1.942E-04 | global batch size: 256 | lm loss: 3.786794E+00 | grad norm: 0.290 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1654.818 | TFLOPs: 25.95 | 7: iteration 41000/ 331103 | consumed samples: 10496000 | consumed tokens: 21495808000 | elapsed time per iteration (s): 0.15 | learning rate: 1.942E-04 | global batch size: 256 | lm loss: 3.784331E+00 | grad norm: 0.268 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1663.182 | TFLOPs: 26.08 | 7: iteration 41100/ 331103 | consumed samples: 10521600 | consumed tokens: 21548236800 | elapsed time per iteration (s): 0.16 | learning rate: 1.942E-04 | global batch size: 256 | lm loss: 3.789532E+00 | grad norm: 0.308 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1638.228 | TFLOPs: 25.69 | 7: iteration 41200/ 331103 | consumed samples: 10547200 | consumed tokens: 21600665600 | elapsed time per iteration (s): 0.15 | learning rate: 1.941E-04 | global batch size: 256 | lm loss: 3.782340E+00 | grad norm: 0.361 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1664.523 | TFLOPs: 26.10 | 7: iteration 41300/ 331103 | consumed samples: 10572800 | consumed tokens: 21653094400 | elapsed time per iteration (s): 0.15 | learning rate: 1.941E-04 | global batch size: 256 | lm loss: 3.785514E+00 | grad norm: 0.350 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1662.188 | TFLOPs: 26.07 | 7: iteration 41400/ 331103 | consumed samples: 10598400 | consumed tokens: 21705523200 | elapsed time per iteration (s): 0.15 | learning rate: 1.941E-04 | global batch size: 256 | lm loss: 3.782597E+00 | grad norm: 0.311 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1660.532 | TFLOPs: 26.04 | 7: iteration 41500/ 331103 | consumed samples: 10624000 | consumed tokens: 21757952000 | elapsed time per iteration (s): 0.16 | learning rate: 1.940E-04 | global batch size: 256 | lm loss: 3.782377E+00 | grad norm: 0.269 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1648.335 | TFLOPs: 25.85 | 7: iteration 41600/ 331103 | consumed samples: 10649600 | consumed tokens: 21810380800 | elapsed time per iteration (s): 0.15 | learning rate: 1.940E-04 | global batch size: 256 | lm loss: 3.784053E+00 | grad norm: 0.299 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1653.011 | TFLOPs: 25.92 | 7: iteration 41700/ 331103 | consumed samples: 10675200 | consumed tokens: 21862809600 | elapsed time per iteration (s): 0.15 | learning rate: 1.940E-04 | global batch size: 256 | lm loss: 3.787060E+00 | grad norm: 0.327 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1658.413 | TFLOPs: 26.01 | 7: iteration 41800/ 331103 | consumed samples: 10700800 | consumed tokens: 21915238400 | elapsed time per iteration (s): 0.16 | learning rate: 1.939E-04 | global batch size: 256 | lm loss: 3.783591E+00 | grad norm: 0.332 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1597.292 | TFLOPs: 25.05 | 7: iteration 41900/ 331103 | consumed samples: 10726400 | consumed tokens: 21967667200 | elapsed time per iteration (s): 0.15 | learning rate: 1.939E-04 | global batch size: 256 | lm loss: 3.780845E+00 | grad norm: 0.287 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1672.706 | TFLOPs: 26.23 | 0: [2023-03-17 11:51:19,435] [INFO] [logging.py:68:log_dist] [Rank 0] step=42000, skipped=0, lr=[0.00019388345316303344, 0.00019388345316303344, 0.00019388345316303344], mom=[(0.9, 0.999), (0.9, 0.999), (0.9, 0.999)] 7: iteration 42000/ 331103 | consumed samples: 10752000 | consumed tokens: 22020096000 | elapsed time per iteration (s): 0.15 | learning rate: 1.939E-04 | global batch size: 256 | lm loss: 3.784592E+00 | grad norm: 0.320 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1659.099 | TFLOPs: 26.02 | 0: steps: 42000 loss: 3.7757 iter time (s): 0.156 samples/sec: 1644.561 7: iteration 42100/ 331103 | consumed samples: 10777600 | consumed tokens: 22072524800 | elapsed time per iteration (s): 0.15 | learning rate: 1.939E-04 | global batch size: 256 | lm loss: 3.781696E+00 | grad norm: 0.276 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1659.525 | TFLOPs: 26.03 | 7: iteration 42200/ 331103 | consumed samples: 10803200 | consumed tokens: 22124953600 | elapsed time per iteration (s): 0.15 | learning rate: 1.938E-04 | global batch size: 256 | lm loss: 3.785829E+00 | grad norm: 0.312 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1666.248 | TFLOPs: 26.13 | 7: iteration 42300/ 331103 | consumed samples: 10828800 | consumed tokens: 22177382400 | elapsed time per iteration (s): 0.15 | learning rate: 1.938E-04 | global batch size: 256 | lm loss: 3.783217E+00 | grad norm: 0.359 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1672.569 | TFLOPs: 26.23 | 7: iteration 42400/ 331103 | consumed samples: 10854400 | consumed tokens: 22229811200 | elapsed time per iteration (s): 0.15 | learning rate: 1.938E-04 | global batch size: 256 | lm loss: 3.783378E+00 | grad norm: 0.301 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1664.549 | TFLOPs: 26.10 | 7: iteration 42500/ 331103 | consumed samples: 10880000 | consumed tokens: 22282240000 | elapsed time per iteration (s): 0.15 | learning rate: 1.937E-04 | global batch size: 256 | lm loss: 3.782375E+00 | grad norm: 0.280 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1670.859 | TFLOPs: 26.20 | 7: iteration 42600/ 331103 | consumed samples: 10905600 | consumed tokens: 22334668800 | elapsed time per iteration (s): 0.15 | learning rate: 1.937E-04 | global batch size: 256 | lm loss: 3.782087E+00 | grad norm: 0.296 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1670.073 | TFLOPs: 26.19 | 7: iteration 42700/ 331103 | consumed samples: 10931200 | consumed tokens: 22387097600 | elapsed time per iteration (s): 0.15 | learning rate: 1.937E-04 | global batch size: 256 | lm loss: 3.785580E+00 | grad norm: 0.277 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1668.948 | TFLOPs: 26.17 | 7: iteration 42800/ 331103 | consumed samples: 10956800 | consumed tokens: 22439526400 | elapsed time per iteration (s): 0.15 | learning rate: 1.936E-04 | global batch size: 256 | lm loss: 3.783334E+00 | grad norm: 0.322 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1671.128 | TFLOPs: 26.21 | 7: iteration 42900/ 331103 | consumed samples: 10982400 | consumed tokens: 22491955200 | elapsed time per iteration (s): 0.15 | learning rate: 1.936E-04 | global batch size: 256 | lm loss: 3.779323E+00 | grad norm: 0.311 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1665.577 | TFLOPs: 26.12 | 7: iteration 43000/ 331103 | consumed samples: 11008000 | consumed tokens: 22544384000 | elapsed time per iteration (s): 0.15 | learning rate: 1.936E-04 | global batch size: 256 | lm loss: 3.786009E+00 | grad norm: 0.289 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1662.945 | TFLOPs: 26.08 | 7: iteration 43100/ 331103 | consumed samples: 11033600 | consumed tokens: 22596812800 | elapsed time per iteration (s): 0.15 | learning rate: 1.935E-04 | global batch size: 256 | lm loss: 3.775918E+00 | grad norm: 0.296 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1663.258 | TFLOPs: 26.08 | 7: iteration 43200/ 331103 | consumed samples: 11059200 | consumed tokens: 22649241600 | elapsed time per iteration (s): 0.15 | learning rate: 1.935E-04 | global batch size: 256 | lm loss: 3.783565E+00 | grad norm: 0.311 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1664.796 | TFLOPs: 26.11 | 7: iteration 43300/ 331103 | consumed samples: 11084800 | consumed tokens: 22701670400 | elapsed time per iteration (s): 0.15 | learning rate: 1.935E-04 | global batch size: 256 | lm loss: 3.781381E+00 | grad norm: 0.288 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1663.259 | TFLOPs: 26.08 | 7: iteration 43400/ 331103 | consumed samples: 11110400 | consumed tokens: 22754099200 | elapsed time per iteration (s): 0.16 | learning rate: 1.934E-04 | global batch size: 256 | lm loss: 3.781817E+00 | grad norm: 0.298 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1575.983 | TFLOPs: 24.72 | 7: iteration 43500/ 331103 | consumed samples: 11136000 | consumed tokens: 22806528000 | elapsed time per iteration (s): 0.15 | learning rate: 1.934E-04 | global batch size: 256 | lm loss: 3.778536E+00 | grad norm: 0.355 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1661.170 | TFLOPs: 26.05 | 7: iteration 43600/ 331103 | consumed samples: 11161600 | consumed tokens: 22858956800 | elapsed time per iteration (s): 0.15 | learning rate: 1.934E-04 | global batch size: 256 | lm loss: 3.782423E+00 | grad norm: 0.302 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1660.901 | TFLOPs: 26.05 | 7: iteration 43700/ 331103 | consumed samples: 11187200 | consumed tokens: 22911385600 | elapsed time per iteration (s): 0.15 | learning rate: 1.933E-04 | global batch size: 256 | lm loss: 3.776835E+00 | grad norm: 0.291 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1661.189 | TFLOPs: 26.05 | 7: iteration 43800/ 331103 | consumed samples: 11212800 | consumed tokens: 22963814400 | elapsed time per iteration (s): 0.15 | learning rate: 1.933E-04 | global batch size: 256 | lm loss: 3.777882E+00 | grad norm: 0.313 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1661.886 | TFLOPs: 26.06 | 7: iteration 43900/ 331103 | consumed samples: 11238400 | consumed tokens: 23016243200 | elapsed time per iteration (s): 0.15 | learning rate: 1.933E-04 | global batch size: 256 | lm loss: 3.779422E+00 | grad norm: 0.305 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1668.881 | TFLOPs: 26.17 | 0: [2023-03-17 11:56:27,665] [INFO] [logging.py:68:log_dist] [Rank 0] step=44000, skipped=0, lr=[0.00019324296243742825, 0.00019324296243742825, 0.00019324296243742825], mom=[(0.9, 0.999), (0.9, 0.999), (0.9, 0.999)] 7: iteration 44000/ 331103 | consumed samples: 11264000 | consumed tokens: 23068672000 | elapsed time per iteration (s): 0.15 | learning rate: 1.932E-04 | global batch size: 256 | lm loss: 3.776537E+00 | grad norm: 0.317 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1673.308 | TFLOPs: 26.24 | 0: steps: 44000 loss: 3.8097 iter time (s): 0.153 samples/sec: 1670.853 7: iteration 44100/ 331103 | consumed samples: 11289600 | consumed tokens: 23121100800 | elapsed time per iteration (s): 0.15 | learning rate: 1.932E-04 | global batch size: 256 | lm loss: 3.782855E+00 | grad norm: 0.306 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1675.416 | TFLOPs: 26.27 | 7: iteration 44200/ 331103 | consumed samples: 11315200 | consumed tokens: 23173529600 | elapsed time per iteration (s): 0.15 | learning rate: 1.932E-04 | global batch size: 256 | lm loss: 3.776697E+00 | grad norm: 0.299 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1663.954 | TFLOPs: 26.09 | 7: iteration 44300/ 331103 | consumed samples: 11340800 | consumed tokens: 23225958400 | elapsed time per iteration (s): 0.15 | learning rate: 1.931E-04 | global batch size: 256 | lm loss: 3.775213E+00 | grad norm: 0.332 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1656.932 | TFLOPs: 25.98 | 7: iteration 44400/ 331103 | consumed samples: 11366400 | consumed tokens: 23278387200 | elapsed time per iteration (s): 0.15 | learning rate: 1.931E-04 | global batch size: 256 | lm loss: 3.775516E+00 | grad norm: 0.323 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1664.691 | TFLOPs: 26.11 | 7: iteration 44500/ 331103 | consumed samples: 11392000 | consumed tokens: 23330816000 | elapsed time per iteration (s): 0.15 | learning rate: 1.931E-04 | global batch size: 256 | lm loss: 3.778291E+00 | grad norm: 0.297 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1657.086 | TFLOPs: 25.99 | 7: iteration 44600/ 331103 | consumed samples: 11417600 | consumed tokens: 23383244800 | elapsed time per iteration (s): 0.15 | learning rate: 1.930E-04 | global batch size: 256 | lm loss: 3.776949E+00 | grad norm: 0.339 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1667.126 | TFLOPs: 26.14 | 7: iteration 44700/ 331103 | consumed samples: 11443200 | consumed tokens: 23435673600 | elapsed time per iteration (s): 0.15 | learning rate: 1.930E-04 | global batch size: 256 | lm loss: 3.777064E+00 | grad norm: 0.297 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1659.851 | TFLOPs: 26.03 | 7: iteration 44800/ 331103 | consumed samples: 11468800 | consumed tokens: 23488102400 | elapsed time per iteration (s): 0.16 | learning rate: 1.930E-04 | global batch size: 256 | lm loss: 3.776448E+00 | grad norm: 0.289 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1645.514 | TFLOPs: 25.81 | 7: iteration 44900/ 331103 | consumed samples: 11494400 | consumed tokens: 23540531200 | elapsed time per iteration (s): 0.15 | learning rate: 1.929E-04 | global batch size: 256 | lm loss: 3.772399E+00 | grad norm: 0.306 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1668.429 | TFLOPs: 26.17 | 7: iteration 45000/ 331103 | consumed samples: 11520000 | consumed tokens: 23592960000 | elapsed time per iteration (s): 0.15 | learning rate: 1.929E-04 | global batch size: 256 | lm loss: 3.776942E+00 | grad norm: 0.309 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1658.743 | TFLOPs: 26.01 | 7: iteration 45100/ 331103 | consumed samples: 11545600 | consumed tokens: 23645388800 | elapsed time per iteration (s): 0.15 | learning rate: 1.929E-04 | global batch size: 256 | lm loss: 3.774450E+00 | grad norm: 0.324 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1657.824 | TFLOPs: 26.00 | 7: iteration 45200/ 331103 | consumed samples: 11571200 | consumed tokens: 23697817600 | elapsed time per iteration (s): 0.15 | learning rate: 1.928E-04 | global batch size: 256 | lm loss: 3.775164E+00 | grad norm: 0.312 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1664.455 | TFLOPs: 26.10 | 7: iteration 45300/ 331103 | consumed samples: 11596800 | consumed tokens: 23750246400 | elapsed time per iteration (s): 0.15 | learning rate: 1.928E-04 | global batch size: 256 | lm loss: 3.779476E+00 | grad norm: 0.302 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1672.011 | TFLOPs: 26.22 | 7: iteration 45400/ 331103 | consumed samples: 11622400 | consumed tokens: 23802675200 | elapsed time per iteration (s): 0.15 | learning rate: 1.928E-04 | global batch size: 256 | lm loss: 3.774526E+00 | grad norm: 0.316 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1669.585 | TFLOPs: 26.18 | 7: iteration 45500/ 331103 | consumed samples: 11648000 | consumed tokens: 23855104000 | elapsed time per iteration (s): 0.15 | learning rate: 1.927E-04 | global batch size: 256 | lm loss: 3.776742E+00 | grad norm: 0.296 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1671.667 | TFLOPs: 26.22 | 7: iteration 45600/ 331103 | consumed samples: 11673600 | consumed tokens: 23907532800 | elapsed time per iteration (s): 0.15 | learning rate: 1.927E-04 | global batch size: 256 | lm loss: 3.776574E+00 | grad norm: 0.315 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1665.638 | TFLOPs: 26.12 | 7: iteration 45700/ 331103 | consumed samples: 11699200 | consumed tokens: 23959961600 | elapsed time per iteration (s): 0.15 | learning rate: 1.927E-04 | global batch size: 256 | lm loss: 3.772908E+00 | grad norm: 0.296 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1669.074 | TFLOPs: 26.18 | 7: iteration 45800/ 331103 | consumed samples: 11724800 | consumed tokens: 24012390400 | elapsed time per iteration (s): 0.15 | learning rate: 1.926E-04 | global batch size: 256 | lm loss: 3.774787E+00 | grad norm: 0.319 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1666.374 | TFLOPs: 26.13 | 7: iteration 45900/ 331103 | consumed samples: 11750400 | consumed tokens: 24064819200 | elapsed time per iteration (s): 0.15 | learning rate: 1.926E-04 | global batch size: 256 | lm loss: 3.776692E+00 | grad norm: 0.308 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1665.306 | TFLOPs: 26.12 | 0: [2023-03-17 12:01:35,336] [INFO] [logging.py:68:log_dist] [Rank 0] step=46000, skipped=0, lr=[0.00019257188762723004, 0.00019257188762723004, 0.00019257188762723004], mom=[(0.9, 0.999), (0.9, 0.999), (0.9, 0.999)] 7: iteration 46000/ 331103 | consumed samples: 11776000 | consumed tokens: 24117248000 | elapsed time per iteration (s): 0.15 | learning rate: 1.926E-04 | global batch size: 256 | lm loss: 3.772827E+00 | grad norm: 0.274 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1663.153 | TFLOPs: 26.08 | 0: steps: 46000 loss: 3.7945 iter time (s): 0.153 samples/sec: 1677.215 7: iteration 46100/ 331103 | consumed samples: 11801600 | consumed tokens: 24169676800 | elapsed time per iteration (s): 0.15 | learning rate: 1.925E-04 | global batch size: 256 | lm loss: 3.772570E+00 | grad norm: 0.348 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1653.850 | TFLOPs: 25.94 | 7: iteration 46200/ 331103 | consumed samples: 11827200 | consumed tokens: 24222105600 | elapsed time per iteration (s): 0.15 | learning rate: 1.925E-04 | global batch size: 256 | lm loss: 3.773744E+00 | grad norm: 0.303 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1664.489 | TFLOPs: 26.10 | 7: iteration 46300/ 331103 | consumed samples: 11852800 | consumed tokens: 24274534400 | elapsed time per iteration (s): 0.15 | learning rate: 1.925E-04 | global batch size: 256 | lm loss: 3.776871E+00 | grad norm: 0.307 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1657.551 | TFLOPs: 25.99 | 7: iteration 46400/ 331103 | consumed samples: 11878400 | consumed tokens: 24326963200 | elapsed time per iteration (s): 0.15 | learning rate: 1.924E-04 | global batch size: 256 | lm loss: 3.772568E+00 | grad norm: 0.296 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1652.949 | TFLOPs: 25.92 | 7: iteration 46500/ 331103 | consumed samples: 11904000 | consumed tokens: 24379392000 | elapsed time per iteration (s): 0.15 | learning rate: 1.924E-04 | global batch size: 256 | lm loss: 3.773520E+00 | grad norm: 0.351 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1666.378 | TFLOPs: 26.13 | 7: iteration 46600/ 331103 | consumed samples: 11929600 | consumed tokens: 24431820800 | elapsed time per iteration (s): 0.15 | learning rate: 1.924E-04 | global batch size: 256 | lm loss: 3.773485E+00 | grad norm: 0.329 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1659.139 | TFLOPs: 26.02 | 7: iteration 46700/ 331103 | consumed samples: 11955200 | consumed tokens: 24484249600 | elapsed time per iteration (s): 0.16 | learning rate: 1.923E-04 | global batch size: 256 | lm loss: 3.771286E+00 | grad norm: 0.304 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1646.784 | TFLOPs: 25.83 | 7: iteration 46800/ 331103 | consumed samples: 11980800 | consumed tokens: 24536678400 | elapsed time per iteration (s): 0.15 | learning rate: 1.923E-04 | global batch size: 256 | lm loss: 3.780977E+00 | grad norm: 0.313 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1660.402 | TFLOPs: 26.04 | 7: iteration 46900/ 331103 | consumed samples: 12006400 | consumed tokens: 24589107200 | elapsed time per iteration (s): 0.16 | learning rate: 1.923E-04 | global batch size: 256 | lm loss: 3.770778E+00 | grad norm: 0.305 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1642.576 | TFLOPs: 25.76 | 7: iteration 47000/ 331103 | consumed samples: 12032000 | consumed tokens: 24641536000 | elapsed time per iteration (s): 0.15 | learning rate: 1.922E-04 | global batch size: 256 | lm loss: 3.771763E+00 | grad norm: 0.336 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1656.189 | TFLOPs: 25.97 | 7: iteration 47100/ 331103 | consumed samples: 12057600 | consumed tokens: 24693964800 | elapsed time per iteration (s): 0.16 | learning rate: 1.922E-04 | global batch size: 256 | lm loss: 3.768873E+00 | grad norm: 0.300 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1644.167 | TFLOPs: 25.78 | 7: iteration 47200/ 331103 | consumed samples: 12083200 | consumed tokens: 24746393600 | elapsed time per iteration (s): 0.15 | learning rate: 1.922E-04 | global batch size: 256 | lm loss: 3.771955E+00 | grad norm: 0.313 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1660.345 | TFLOPs: 26.04 | 7: iteration 47300/ 331103 | consumed samples: 12108800 | consumed tokens: 24798822400 | elapsed time per iteration (s): 0.15 | learning rate: 1.921E-04 | global batch size: 256 | lm loss: 3.766070E+00 | grad norm: 0.311 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1664.359 | TFLOPs: 26.10 | 7: iteration 47400/ 331103 | consumed samples: 12134400 | consumed tokens: 24851251200 | elapsed time per iteration (s): 0.15 | learning rate: 1.921E-04 | global batch size: 256 | lm loss: 3.774593E+00 | grad norm: 0.296 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1664.983 | TFLOPs: 26.11 | 7: iteration 47500/ 331103 | consumed samples: 12160000 | consumed tokens: 24903680000 | elapsed time per iteration (s): 0.15 | learning rate: 1.920E-04 | global batch size: 256 | lm loss: 3.773785E+00 | grad norm: 0.285 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1670.072 | TFLOPs: 26.19 | 7: iteration 47600/ 331103 | consumed samples: 12185600 | consumed tokens: 24956108800 | elapsed time per iteration (s): 0.15 | learning rate: 1.920E-04 | global batch size: 256 | lm loss: 3.771846E+00 | grad norm: 0.279 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1670.467 | TFLOPs: 26.20 | 7: iteration 47700/ 331103 | consumed samples: 12211200 | consumed tokens: 25008537600 | elapsed time per iteration (s): 0.15 | learning rate: 1.920E-04 | global batch size: 256 | lm loss: 3.771859E+00 | grad norm: 0.313 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1672.333 | TFLOPs: 26.23 | 7: iteration 47800/ 331103 | consumed samples: 12236800 | consumed tokens: 25060966400 | elapsed time per iteration (s): 0.15 | learning rate: 1.919E-04 | global batch size: 256 | lm loss: 3.766718E+00 | grad norm: 0.315 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1674.421 | TFLOPs: 26.26 | 7: iteration 47900/ 331103 | consumed samples: 12262400 | consumed tokens: 25113395200 | elapsed time per iteration (s): 0.15 | learning rate: 1.919E-04 | global batch size: 256 | lm loss: 3.774585E+00 | grad norm: 0.272 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1675.364 | TFLOPs: 26.27 | 0: [2023-03-17 12:06:43,536] [INFO] [logging.py:68:log_dist] [Rank 0] step=48000, skipped=0, lr=[0.0001918704752903238, 0.0001918704752903238, 0.0001918704752903238], mom=[(0.9, 0.999), (0.9, 0.999), (0.9, 0.999)] 7: iteration 48000/ 331103 | consumed samples: 12288000 | consumed tokens: 25165824000 | elapsed time per iteration (s): 0.15 | learning rate: 1.919E-04 | global batch size: 256 | lm loss: 3.770280E+00 | grad norm: 0.286 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1669.540 | TFLOPs: 26.18 | 0: steps: 48000 loss: 3.8005 iter time (s): 0.153 samples/sec: 1675.568 7: iteration 48100/ 331103 | consumed samples: 12313600 | consumed tokens: 25218252800 | elapsed time per iteration (s): 0.15 | learning rate: 1.918E-04 | global batch size: 256 | lm loss: 3.770985E+00 | grad norm: 0.317 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1666.690 | TFLOPs: 26.14 | 7: iteration 48200/ 331103 | consumed samples: 12339200 | consumed tokens: 25270681600 | elapsed time per iteration (s): 0.15 | learning rate: 1.918E-04 | global batch size: 256 | lm loss: 3.769477E+00 | grad norm: 0.314 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1664.432 | TFLOPs: 26.10 | 7: iteration 48300/ 331103 | consumed samples: 12364800 | consumed tokens: 25323110400 | elapsed time per iteration (s): 0.15 | learning rate: 1.918E-04 | global batch size: 256 | lm loss: 3.767178E+00 | grad norm: 0.293 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1669.232 | TFLOPs: 26.18 | 7: iteration 48400/ 331103 | consumed samples: 12390400 | consumed tokens: 25375539200 | elapsed time per iteration (s): 0.15 | learning rate: 1.917E-04 | global batch size: 256 | lm loss: 3.772295E+00 | grad norm: 0.281 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1657.429 | TFLOPs: 25.99 | 7: iteration 48500/ 331103 | consumed samples: 12416000 | consumed tokens: 25427968000 | elapsed time per iteration (s): 0.15 | learning rate: 1.917E-04 | global batch size: 256 | lm loss: 3.767960E+00 | grad norm: 0.294 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1675.305 | TFLOPs: 26.27 | 7: iteration 48600/ 331103 | consumed samples: 12441600 | consumed tokens: 25480396800 | elapsed time per iteration (s): 0.16 | learning rate: 1.917E-04 | global batch size: 256 | lm loss: 3.764729E+00 | grad norm: 0.303 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1639.145 | TFLOPs: 25.71 | 7: iteration 48700/ 331103 | consumed samples: 12467200 | consumed tokens: 25532825600 | elapsed time per iteration (s): 0.15 | learning rate: 1.916E-04 | global batch size: 256 | lm loss: 3.768759E+00 | grad norm: 0.316 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1652.550 | TFLOPs: 25.92 | 7: iteration 48800/ 331103 | consumed samples: 12492800 | consumed tokens: 25585254400 | elapsed time per iteration (s): 0.16 | learning rate: 1.916E-04 | global batch size: 256 | lm loss: 3.767416E+00 | grad norm: 0.319 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1559.168 | TFLOPs: 24.45 | 7: iteration 48900/ 331103 | consumed samples: 12518400 | consumed tokens: 25637683200 | elapsed time per iteration (s): 0.18 | learning rate: 1.915E-04 | global batch size: 256 | lm loss: 3.768195E+00 | grad norm: 0.294 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1457.500 | TFLOPs: 22.86 | 7: iteration 49000/ 331103 | consumed samples: 12544000 | consumed tokens: 25690112000 | elapsed time per iteration (s): 0.16 | learning rate: 1.915E-04 | global batch size: 256 | lm loss: 3.767155E+00 | grad norm: 0.312 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1566.555 | TFLOPs: 24.57 | 7: iteration 49100/ 331103 | consumed samples: 12569600 | consumed tokens: 25742540800 | elapsed time per iteration (s): 0.16 | learning rate: 1.915E-04 | global batch size: 256 | lm loss: 3.764107E+00 | grad norm: 0.314 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1599.185 | TFLOPs: 25.08 | 7: iteration 49200/ 331103 | consumed samples: 12595200 | consumed tokens: 25794969600 | elapsed time per iteration (s): 0.17 | learning rate: 1.914E-04 | global batch size: 256 | lm loss: 3.765661E+00 | grad norm: 0.309 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1515.088 | TFLOPs: 23.76 | 7: iteration 49300/ 331103 | consumed samples: 12620800 | consumed tokens: 25847398400 | elapsed time per iteration (s): 0.17 | learning rate: 1.914E-04 | global batch size: 256 | lm loss: 3.768185E+00 | grad norm: 0.278 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1521.982 | TFLOPs: 23.87 | 7: iteration 49400/ 331103 | consumed samples: 12646400 | consumed tokens: 25899827200 | elapsed time per iteration (s): 0.17 | learning rate: 1.914E-04 | global batch size: 256 | lm loss: 3.763523E+00 | grad norm: 0.301 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1548.439 | TFLOPs: 24.28 | 7: iteration 49500/ 331103 | consumed samples: 12672000 | consumed tokens: 25952256000 | elapsed time per iteration (s): 0.16 | learning rate: 1.913E-04 | global batch size: 256 | lm loss: 3.768863E+00 | grad norm: 0.328 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1560.852 | TFLOPs: 24.48 | 7: iteration 49600/ 331103 | consumed samples: 12697600 | consumed tokens: 26004684800 | elapsed time per iteration (s): 0.16 | learning rate: 1.913E-04 | global batch size: 256 | lm loss: 3.764901E+00 | grad norm: 0.281 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1564.537 | TFLOPs: 24.54 | 7: iteration 49700/ 331103 | consumed samples: 12723200 | consumed tokens: 26057113600 | elapsed time per iteration (s): 0.16 | learning rate: 1.913E-04 | global batch size: 256 | lm loss: 3.771206E+00 | grad norm: 0.312 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1566.824 | TFLOPs: 24.57 | 7: iteration 49800/ 331103 | consumed samples: 12748800 | consumed tokens: 26109542400 | elapsed time per iteration (s): 0.16 | learning rate: 1.912E-04 | global batch size: 256 | lm loss: 3.767267E+00 | grad norm: 0.305 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1569.255 | TFLOPs: 24.61 | 7: iteration 49900/ 331103 | consumed samples: 12774400 | consumed tokens: 26161971200 | elapsed time per iteration (s): 0.16 | learning rate: 1.912E-04 | global batch size: 256 | lm loss: 3.766339E+00 | grad norm: 0.321 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1609.558 | TFLOPs: 25.24 | 0: [2023-03-17 12:12:05,516] [INFO] [logging.py:68:log_dist] [Rank 0] step=50000, skipped=0, lr=[0.00019113898313082793, 0.00019113898313082793, 0.00019113898313082793], mom=[(0.9, 0.999), (0.9, 0.999), (0.9, 0.999)] 7: iteration 50000/ 331103 | consumed samples: 12800000 | consumed tokens: 26214400000 | elapsed time per iteration (s): 0.16 | learning rate: 1.911E-04 | global batch size: 256 | lm loss: 3.767703E+00 | grad norm: 0.328 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1583.772 | TFLOPs: 24.84 | 0: steps: 50000 loss: 3.7790 iter time (s): 0.160 samples/sec: 1604.881 7: ------------------------------------------------------------------------------------------------ 7: validation loss at iteration 50000 | lm loss value: 3.886840E+00 | lm loss PPL: 4.875655E+01 | 7: ------------------------------------------------------------------------------------------------ 0: saving checkpoint at iteration 50000 to checkpoints_44m174b100m 0: [2023-03-17 12:12:05,684] [INFO] [logging.py:68:log_dist] [Rank 0] [Torch] Checkpoint global_step50000 is begin to save! 0: [2023-03-17 12:12:05,689] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step50000/layer_01-model_00-model_states.pt... 0: [2023-03-17 12:12:05,758] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step50000/layer_01-model_00-model_states.pt. 0: [2023-03-17 12:12:05,759] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step50000/layer_03-model_00-model_states.pt... 0: [2023-03-17 12:12:05,768] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step50000/layer_03-model_00-model_states.pt. 0: [2023-03-17 12:12:05,768] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step50000/layer_04-model_00-model_states.pt... 0: [2023-03-17 12:12:05,778] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step50000/layer_04-model_00-model_states.pt. 0: [2023-03-17 12:12:05,778] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step50000/layer_05-model_00-model_states.pt... 0: [2023-03-17 12:12:05,786] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step50000/layer_05-model_00-model_states.pt. 0: [2023-03-17 12:12:05,786] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step50000/layer_06-model_00-model_states.pt... 0: [2023-03-17 12:12:05,794] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step50000/layer_06-model_00-model_states.pt. 0: [2023-03-17 12:12:05,795] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step50000/layer_07-model_00-model_states.pt... 0: [2023-03-17 12:12:05,802] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step50000/layer_07-model_00-model_states.pt. 0: [2023-03-17 12:12:05,803] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step50000/layer_08-model_00-model_states.pt... 0: [2023-03-17 12:12:05,811] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step50000/layer_08-model_00-model_states.pt. 0: [2023-03-17 12:12:05,811] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step50000/layer_09-model_00-model_states.pt... 0: [2023-03-17 12:12:05,819] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step50000/layer_09-model_00-model_states.pt. 0: [2023-03-17 12:12:05,819] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step50000/layer_10-model_00-model_states.pt... 0: [2023-03-17 12:12:05,827] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step50000/layer_10-model_00-model_states.pt. 0: [2023-03-17 12:12:05,827] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step50000/layer_12-model_00-model_states.pt... 0: [2023-03-17 12:12:05,828] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step50000/layer_12-model_00-model_states.pt. 0: [2023-03-17 12:12:05,828] [INFO] [logging.py:68:log_dist] [Rank 0] Saving model checkpoint: checkpoints_44m174b100m/global_step50000/mp_rank_00_model_states.pt 0: [2023-03-17 12:12:05,828] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step50000/mp_rank_00_model_states.pt... 0: [2023-03-17 12:12:05,830] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step50000/mp_rank_00_model_states.pt. 0: [2023-03-17 12:12:05,847] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step50000/bf16_zero_pp_rank_0_mp_rank_00_optim_states.pt... 0: [2023-03-17 12:12:05,847] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step50000/bf16_zero_pp_rank_5_mp_rank_00_optim_states.pt... 0: [2023-03-17 12:12:05,847] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step50000/bf16_zero_pp_rank_2_mp_rank_00_optim_states.pt... 0: [2023-03-17 12:12:05,847] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step50000/bf16_zero_pp_rank_6_mp_rank_00_optim_states.pt... 3: [2023-03-17 12:12:05,847] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step50000/bf16_zero_pp_rank_25_mp_rank_00_optim_states.pt... 3: [2023-03-17 12:12:05,847] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step50000/bf16_zero_pp_rank_24_mp_rank_00_optim_states.pt... 3: [2023-03-17 12:12:05,847] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step50000/bf16_zero_pp_rank_31_mp_rank_00_optim_states.pt... 3: [2023-03-17 12:12:05,847] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step50000/bf16_zero_pp_rank_30_mp_rank_00_optim_states.pt... 3: [2023-03-17 12:12:05,847] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step50000/bf16_zero_pp_rank_29_mp_rank_00_optim_states.pt... 7: [2023-03-17 12:12:05,847] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step50000/bf16_zero_pp_rank_56_mp_rank_00_optim_states.pt... 7: [2023-03-17 12:12:05,847] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step50000/bf16_zero_pp_rank_60_mp_rank_00_optim_states.pt... 7: [2023-03-17 12:12:05,847] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step50000/bf16_zero_pp_rank_58_mp_rank_00_optim_states.pt... 7: [2023-03-17 12:12:05,847] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step50000/bf16_zero_pp_rank_59_mp_rank_00_optim_states.pt... 7: [2023-03-17 12:12:05,847] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step50000/bf16_zero_pp_rank_61_mp_rank_00_optim_states.pt... 0: [2023-03-17 12:12:05,847] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step50000/bf16_zero_pp_rank_3_mp_rank_00_optim_states.pt... 2: [2023-03-17 12:12:05,847] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step50000/bf16_zero_pp_rank_20_mp_rank_00_optim_states.pt... 2: [2023-03-17 12:12:05,847] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step50000/bf16_zero_pp_rank_16_mp_rank_00_optim_states.pt... 2: [2023-03-17 12:12:05,847] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step50000/bf16_zero_pp_rank_21_mp_rank_00_optim_states.pt... 2: [2023-03-17 12:12:05,847] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step50000/bf16_zero_pp_rank_22_mp_rank_00_optim_states.pt... 2: [2023-03-17 12:12:05,847] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step50000/bf16_zero_pp_rank_19_mp_rank_00_optim_states.pt... 4: [2023-03-17 12:12:05,847] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step50000/bf16_zero_pp_rank_37_mp_rank_00_optim_states.pt... 4: [2023-03-17 12:12:05,847] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step50000/bf16_zero_pp_rank_38_mp_rank_00_optim_states.pt... 4: [2023-03-17 12:12:05,847] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step50000/bf16_zero_pp_rank_35_mp_rank_00_optim_states.pt... 4: [2023-03-17 12:12:05,847] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step50000/bf16_zero_pp_rank_33_mp_rank_00_optim_states.pt... 4: [2023-03-17 12:12:05,847] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step50000/bf16_zero_pp_rank_34_mp_rank_00_optim_states.pt... 6: [2023-03-17 12:12:05,847] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step50000/bf16_zero_pp_rank_53_mp_rank_00_optim_states.pt... 6: [2023-03-17 12:12:05,847] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step50000/bf16_zero_pp_rank_49_mp_rank_00_optim_states.pt... 6: [2023-03-17 12:12:05,847] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step50000/bf16_zero_pp_rank_48_mp_rank_00_optim_states.pt... 6: [2023-03-17 12:12:05,847] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step50000/bf16_zero_pp_rank_50_mp_rank_00_optim_states.pt... 6: [2023-03-17 12:12:05,847] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step50000/bf16_zero_pp_rank_55_mp_rank_00_optim_states.pt... 5: [2023-03-17 12:12:05,847] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step50000/bf16_zero_pp_rank_41_mp_rank_00_optim_states.pt... 5: [2023-03-17 12:12:05,847] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step50000/bf16_zero_pp_rank_47_mp_rank_00_optim_states.pt... 5: [2023-03-17 12:12:05,847] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step50000/bf16_zero_pp_rank_46_mp_rank_00_optim_states.pt... 5: [2023-03-17 12:12:05,847] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step50000/bf16_zero_pp_rank_40_mp_rank_00_optim_states.pt... 5: [2023-03-17 12:12:05,847] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step50000/bf16_zero_pp_rank_44_mp_rank_00_optim_states.pt... 1: [2023-03-17 12:12:05,847] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step50000/bf16_zero_pp_rank_8_mp_rank_00_optim_states.pt... 1: [2023-03-17 12:12:05,847] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step50000/bf16_zero_pp_rank_12_mp_rank_00_optim_states.pt... 1: [2023-03-17 12:12:05,847] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step50000/bf16_zero_pp_rank_13_mp_rank_00_optim_states.pt... 1: [2023-03-17 12:12:05,847] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step50000/bf16_zero_pp_rank_15_mp_rank_00_optim_states.pt... 1: [2023-03-17 12:12:05,847] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step50000/bf16_zero_pp_rank_10_mp_rank_00_optim_states.pt... 0: [2023-03-17 12:12:05,847] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step50000/bf16_zero_pp_rank_4_mp_rank_00_optim_states.pt... 3: [2023-03-17 12:12:05,847] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step50000/bf16_zero_pp_rank_27_mp_rank_00_optim_states.pt... 2: [2023-03-17 12:12:05,847] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step50000/bf16_zero_pp_rank_17_mp_rank_00_optim_states.pt... 2: [2023-03-17 12:12:05,847] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step50000/bf16_zero_pp_rank_23_mp_rank_00_optim_states.pt... 4: [2023-03-17 12:12:05,847] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step50000/bf16_zero_pp_rank_32_mp_rank_00_optim_states.pt... 4: [2023-03-17 12:12:05,847] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step50000/bf16_zero_pp_rank_39_mp_rank_00_optim_states.pt... 6: [2023-03-17 12:12:05,847] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step50000/bf16_zero_pp_rank_54_mp_rank_00_optim_states.pt... 6: [2023-03-17 12:12:05,847] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step50000/bf16_zero_pp_rank_51_mp_rank_00_optim_states.pt... 6: [2023-03-17 12:12:05,847] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step50000/bf16_zero_pp_rank_52_mp_rank_00_optim_states.pt... 7: [2023-03-17 12:12:05,847] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step50000/bf16_zero_pp_rank_57_mp_rank_00_optim_states.pt... 7: [2023-03-17 12:12:05,847] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step50000/bf16_zero_pp_rank_63_mp_rank_00_optim_states.pt... 7: [2023-03-17 12:12:05,847] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step50000/bf16_zero_pp_rank_62_mp_rank_00_optim_states.pt... 5: [2023-03-17 12:12:05,847] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step50000/bf16_zero_pp_rank_45_mp_rank_00_optim_states.pt... 5: [2023-03-17 12:12:05,847] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step50000/bf16_zero_pp_rank_43_mp_rank_00_optim_states.pt... 5: [2023-03-17 12:12:05,847] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step50000/bf16_zero_pp_rank_42_mp_rank_00_optim_states.pt... 1: [2023-03-17 12:12:05,847] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step50000/bf16_zero_pp_rank_9_mp_rank_00_optim_states.pt... 1: [2023-03-17 12:12:05,847] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step50000/bf16_zero_pp_rank_14_mp_rank_00_optim_states.pt... 1: [2023-03-17 12:12:05,847] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step50000/bf16_zero_pp_rank_11_mp_rank_00_optim_states.pt... 0: [2023-03-17 12:12:05,847] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step50000/bf16_zero_pp_rank_1_mp_rank_00_optim_states.pt... 3: [2023-03-17 12:12:05,847] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step50000/bf16_zero_pp_rank_26_mp_rank_00_optim_states.pt... 2: [2023-03-17 12:12:05,847] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step50000/bf16_zero_pp_rank_18_mp_rank_00_optim_states.pt... 4: [2023-03-17 12:12:05,847] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step50000/bf16_zero_pp_rank_36_mp_rank_00_optim_states.pt... 0: [2023-03-17 12:12:05,847] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step50000/bf16_zero_pp_rank_7_mp_rank_00_optim_states.pt... 3: [2023-03-17 12:12:05,847] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step50000/bf16_zero_pp_rank_28_mp_rank_00_optim_states.pt... 0: [2023-03-17 12:12:05,860] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step50000/bf16_zero_pp_rank_0_mp_rank_00_optim_states.pt. 0: [2023-03-17 12:12:05,862] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step50000/bf16_zero_pp_rank_5_mp_rank_00_optim_states.pt. 0: [2023-03-17 12:12:05,862] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step50000/bf16_zero_pp_rank_5_mp_rank_00_optim_states.pt 0: [2023-03-17 12:12:05,862] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step50000 is ready now! 0: [2023-03-17 12:12:05,862] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step50000/bf16_zero_pp_rank_7_mp_rank_00_optim_states.pt. 0: [2023-03-17 12:12:05,862] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step50000/bf16_zero_pp_rank_7_mp_rank_00_optim_states.pt 0: [2023-03-17 12:12:05,862] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step50000 is ready now! 0: [2023-03-17 12:12:05,863] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step50000/bf16_zero_pp_rank_0_mp_rank_00_optim_states.pt 0: [2023-03-17 12:12:05,863] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step50000 is ready now! 0: [2023-03-17 12:12:05,863] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step50000/bf16_zero_pp_rank_4_mp_rank_00_optim_states.pt. 0: [2023-03-17 12:12:05,863] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step50000/bf16_zero_pp_rank_4_mp_rank_00_optim_states.pt 0: [2023-03-17 12:12:05,863] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step50000 is ready now! 0: [2023-03-17 12:12:05,863] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step50000/bf16_zero_pp_rank_1_mp_rank_00_optim_states.pt. 0: [2023-03-17 12:12:05,863] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step50000/bf16_zero_pp_rank_1_mp_rank_00_optim_states.pt 0: [2023-03-17 12:12:05,863] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step50000 is ready now! 0: [2023-03-17 12:12:05,864] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step50000/bf16_zero_pp_rank_6_mp_rank_00_optim_states.pt. 0: [2023-03-17 12:12:05,864] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step50000/bf16_zero_pp_rank_6_mp_rank_00_optim_states.pt 0: [2023-03-17 12:12:05,864] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step50000 is ready now! 0: [2023-03-17 12:12:05,864] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step50000/bf16_zero_pp_rank_3_mp_rank_00_optim_states.pt. 0: [2023-03-17 12:12:05,864] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step50000/bf16_zero_pp_rank_3_mp_rank_00_optim_states.pt 0: [2023-03-17 12:12:05,864] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step50000 is ready now! 0: [2023-03-17 12:12:05,864] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step50000/bf16_zero_pp_rank_2_mp_rank_00_optim_states.pt. 0: [2023-03-17 12:12:05,864] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step50000/bf16_zero_pp_rank_2_mp_rank_00_optim_states.pt 0: [2023-03-17 12:12:05,864] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step50000 is ready now! 7: [2023-03-17 12:12:05,914] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step50000/bf16_zero_pp_rank_57_mp_rank_00_optim_states.pt. 7: [2023-03-17 12:12:05,914] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step50000/bf16_zero_pp_rank_61_mp_rank_00_optim_states.pt. 7: [2023-03-17 12:12:05,914] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step50000/bf16_zero_pp_rank_57_mp_rank_00_optim_states.pt 7: [2023-03-17 12:12:05,914] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step50000/bf16_zero_pp_rank_61_mp_rank_00_optim_states.pt 7: [2023-03-17 12:12:05,914] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step50000 is ready now! 7: [2023-03-17 12:12:05,914] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step50000 is ready now! 7: [2023-03-17 12:12:05,914] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step50000/bf16_zero_pp_rank_56_mp_rank_00_optim_states.pt. 7: [2023-03-17 12:12:05,914] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step50000/bf16_zero_pp_rank_58_mp_rank_00_optim_states.pt. 7: [2023-03-17 12:12:05,914] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step50000/bf16_zero_pp_rank_56_mp_rank_00_optim_states.pt 7: [2023-03-17 12:12:05,914] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step50000/bf16_zero_pp_rank_58_mp_rank_00_optim_states.pt 7: [2023-03-17 12:12:05,914] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step50000 is ready now! 7: [2023-03-17 12:12:05,914] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step50000 is ready now! 7: [2023-03-17 12:12:05,914] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step50000/bf16_zero_pp_rank_62_mp_rank_00_optim_states.pt. 7: [2023-03-17 12:12:05,914] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step50000/bf16_zero_pp_rank_62_mp_rank_00_optim_states.pt 7: [2023-03-17 12:12:05,914] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step50000/bf16_zero_pp_rank_59_mp_rank_00_optim_states.pt. 7: [2023-03-17 12:12:05,914] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step50000/bf16_zero_pp_rank_60_mp_rank_00_optim_states.pt. 7: [2023-03-17 12:12:05,914] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step50000/bf16_zero_pp_rank_63_mp_rank_00_optim_states.pt. 7: [2023-03-17 12:12:05,914] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step50000 is ready now! 7: [2023-03-17 12:12:05,914] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step50000/bf16_zero_pp_rank_59_mp_rank_00_optim_states.pt 7: [2023-03-17 12:12:05,914] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step50000/bf16_zero_pp_rank_60_mp_rank_00_optim_states.pt 7: [2023-03-17 12:12:05,914] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step50000/bf16_zero_pp_rank_63_mp_rank_00_optim_states.pt 7: [2023-03-17 12:12:05,915] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step50000 is ready now! 7: [2023-03-17 12:12:05,915] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step50000 is ready now! 7: [2023-03-17 12:12:05,915] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step50000 is ready now! 2: [2023-03-17 12:12:05,915] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step50000/bf16_zero_pp_rank_18_mp_rank_00_optim_states.pt. 5: [2023-03-17 12:12:05,915] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step50000/bf16_zero_pp_rank_45_mp_rank_00_optim_states.pt. 5: [2023-03-17 12:12:05,915] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step50000/bf16_zero_pp_rank_42_mp_rank_00_optim_states.pt. 5: [2023-03-17 12:12:05,915] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step50000/bf16_zero_pp_rank_44_mp_rank_00_optim_states.pt. 5: [2023-03-17 12:12:05,915] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step50000/bf16_zero_pp_rank_41_mp_rank_00_optim_states.pt. 5: [2023-03-17 12:12:05,915] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step50000/bf16_zero_pp_rank_46_mp_rank_00_optim_states.pt. 5: [2023-03-17 12:12:05,915] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step50000/bf16_zero_pp_rank_47_mp_rank_00_optim_states.pt. 5: [2023-03-17 12:12:05,915] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step50000/bf16_zero_pp_rank_43_mp_rank_00_optim_states.pt. 5: [2023-03-17 12:12:05,915] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step50000/bf16_zero_pp_rank_45_mp_rank_00_optim_states.pt 3: [2023-03-17 12:12:05,915] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step50000/bf16_zero_pp_rank_25_mp_rank_00_optim_states.pt. 3: [2023-03-17 12:12:05,915] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step50000/bf16_zero_pp_rank_28_mp_rank_00_optim_states.pt. 3: [2023-03-17 12:12:05,915] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step50000/bf16_zero_pp_rank_27_mp_rank_00_optim_states.pt. 3: [2023-03-17 12:12:05,915] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step50000/bf16_zero_pp_rank_29_mp_rank_00_optim_states.pt. 3: [2023-03-17 12:12:05,915] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step50000/bf16_zero_pp_rank_24_mp_rank_00_optim_states.pt. 5: [2023-03-17 12:12:05,915] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step50000/bf16_zero_pp_rank_42_mp_rank_00_optim_states.pt 5: [2023-03-17 12:12:05,915] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step50000/bf16_zero_pp_rank_44_mp_rank_00_optim_states.pt 5: [2023-03-17 12:12:05,915] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step50000/bf16_zero_pp_rank_41_mp_rank_00_optim_states.pt 5: [2023-03-17 12:12:05,915] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step50000/bf16_zero_pp_rank_46_mp_rank_00_optim_states.pt 5: [2023-03-17 12:12:05,915] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step50000/bf16_zero_pp_rank_47_mp_rank_00_optim_states.pt 1: [2023-03-17 12:12:05,915] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step50000/bf16_zero_pp_rank_13_mp_rank_00_optim_states.pt. 1: [2023-03-17 12:12:05,915] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step50000/bf16_zero_pp_rank_14_mp_rank_00_optim_states.pt. 1: [2023-03-17 12:12:05,915] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step50000/bf16_zero_pp_rank_11_mp_rank_00_optim_states.pt. 3: [2023-03-17 12:12:05,915] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step50000/bf16_zero_pp_rank_30_mp_rank_00_optim_states.pt. 3: [2023-03-17 12:12:05,915] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step50000/bf16_zero_pp_rank_31_mp_rank_00_optim_states.pt. 3: [2023-03-17 12:12:05,915] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step50000/bf16_zero_pp_rank_26_mp_rank_00_optim_states.pt. 5: [2023-03-17 12:12:05,915] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step50000/bf16_zero_pp_rank_40_mp_rank_00_optim_states.pt. 3: [2023-03-17 12:12:05,915] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step50000/bf16_zero_pp_rank_28_mp_rank_00_optim_states.pt 3: [2023-03-17 12:12:05,915] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step50000/bf16_zero_pp_rank_25_mp_rank_00_optim_states.pt 3: [2023-03-17 12:12:05,915] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step50000/bf16_zero_pp_rank_27_mp_rank_00_optim_states.pt 5: [2023-03-17 12:12:05,915] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step50000 is ready now! 3: [2023-03-17 12:12:05,915] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step50000/bf16_zero_pp_rank_29_mp_rank_00_optim_states.pt 3: [2023-03-17 12:12:05,915] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step50000/bf16_zero_pp_rank_24_mp_rank_00_optim_states.pt 5: [2023-03-17 12:12:05,915] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step50000/bf16_zero_pp_rank_43_mp_rank_00_optim_states.pt 1: [2023-03-17 12:12:05,915] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step50000/bf16_zero_pp_rank_12_mp_rank_00_optim_states.pt. 1: [2023-03-17 12:12:05,915] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step50000/bf16_zero_pp_rank_15_mp_rank_00_optim_states.pt. 1: [2023-03-17 12:12:05,915] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step50000/bf16_zero_pp_rank_8_mp_rank_00_optim_states.pt. 1: [2023-03-17 12:12:05,915] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step50000/bf16_zero_pp_rank_10_mp_rank_00_optim_states.pt. 1: [2023-03-17 12:12:05,915] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step50000/bf16_zero_pp_rank_9_mp_rank_00_optim_states.pt. 3: [2023-03-17 12:12:05,915] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step50000/bf16_zero_pp_rank_26_mp_rank_00_optim_states.pt 3: [2023-03-17 12:12:05,915] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step50000/bf16_zero_pp_rank_30_mp_rank_00_optim_states.pt 3: [2023-03-17 12:12:05,915] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step50000/bf16_zero_pp_rank_31_mp_rank_00_optim_states.pt 5: [2023-03-17 12:12:05,915] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step50000 is ready now! 5: [2023-03-17 12:12:05,915] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step50000 is ready now! 1: [2023-03-17 12:12:05,915] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step50000/bf16_zero_pp_rank_13_mp_rank_00_optim_states.pt 3: [2023-03-17 12:12:05,915] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step50000 is ready now! 5: [2023-03-17 12:12:05,915] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step50000 is ready now! 5: [2023-03-17 12:12:05,915] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step50000 is ready now! 5: [2023-03-17 12:12:05,915] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step50000 is ready now! 1: [2023-03-17 12:12:05,915] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step50000/bf16_zero_pp_rank_11_mp_rank_00_optim_states.pt 1: [2023-03-17 12:12:05,915] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step50000/bf16_zero_pp_rank_14_mp_rank_00_optim_states.pt 3: [2023-03-17 12:12:05,915] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step50000 is ready now! 3: [2023-03-17 12:12:05,915] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step50000 is ready now! 5: [2023-03-17 12:12:05,915] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step50000/bf16_zero_pp_rank_40_mp_rank_00_optim_states.pt 1: [2023-03-17 12:12:05,915] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step50000/bf16_zero_pp_rank_12_mp_rank_00_optim_states.pt 1: [2023-03-17 12:12:05,915] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step50000/bf16_zero_pp_rank_8_mp_rank_00_optim_states.pt 1: [2023-03-17 12:12:05,915] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step50000/bf16_zero_pp_rank_10_mp_rank_00_optim_states.pt 1: [2023-03-17 12:12:05,915] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step50000/bf16_zero_pp_rank_15_mp_rank_00_optim_states.pt 3: [2023-03-17 12:12:05,915] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step50000 is ready now! 3: [2023-03-17 12:12:05,915] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step50000 is ready now! 3: [2023-03-17 12:12:05,915] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step50000 is ready now! 3: [2023-03-17 12:12:05,915] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step50000 is ready now! 3: [2023-03-17 12:12:05,915] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step50000 is ready now! 2: [2023-03-17 12:12:05,915] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step50000/bf16_zero_pp_rank_19_mp_rank_00_optim_states.pt. 2: [2023-03-17 12:12:05,915] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step50000/bf16_zero_pp_rank_23_mp_rank_00_optim_states.pt. 2: [2023-03-17 12:12:05,915] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step50000/bf16_zero_pp_rank_22_mp_rank_00_optim_states.pt. 2: [2023-03-17 12:12:05,915] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step50000/bf16_zero_pp_rank_18_mp_rank_00_optim_states.pt 4: [2023-03-17 12:12:05,915] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step50000/bf16_zero_pp_rank_37_mp_rank_00_optim_states.pt. 4: [2023-03-17 12:12:05,915] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step50000/bf16_zero_pp_rank_35_mp_rank_00_optim_states.pt. 4: [2023-03-17 12:12:05,915] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step50000/bf16_zero_pp_rank_39_mp_rank_00_optim_states.pt. 4: [2023-03-17 12:12:05,915] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step50000/bf16_zero_pp_rank_38_mp_rank_00_optim_states.pt. 4: [2023-03-17 12:12:05,915] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step50000/bf16_zero_pp_rank_36_mp_rank_00_optim_states.pt. 5: [2023-03-17 12:12:05,915] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step50000 is ready now! 1: [2023-03-17 12:12:05,915] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step50000/bf16_zero_pp_rank_9_mp_rank_00_optim_states.pt 1: [2023-03-17 12:12:05,915] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step50000 is ready now! 4: [2023-03-17 12:12:05,915] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step50000/bf16_zero_pp_rank_33_mp_rank_00_optim_states.pt. 4: [2023-03-17 12:12:05,915] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step50000/bf16_zero_pp_rank_34_mp_rank_00_optim_states.pt. 4: [2023-03-17 12:12:05,915] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step50000/bf16_zero_pp_rank_32_mp_rank_00_optim_states.pt. 2: [2023-03-17 12:12:05,915] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step50000/bf16_zero_pp_rank_17_mp_rank_00_optim_states.pt. 2: [2023-03-17 12:12:05,915] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step50000/bf16_zero_pp_rank_16_mp_rank_00_optim_states.pt. 2: [2023-03-17 12:12:05,915] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step50000/bf16_zero_pp_rank_21_mp_rank_00_optim_states.pt. 2: [2023-03-17 12:12:05,915] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step50000/bf16_zero_pp_rank_20_mp_rank_00_optim_states.pt. 2: [2023-03-17 12:12:05,915] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step50000/bf16_zero_pp_rank_19_mp_rank_00_optim_states.pt 2: [2023-03-17 12:12:05,915] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step50000 is ready now! 4: [2023-03-17 12:12:05,915] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step50000/bf16_zero_pp_rank_37_mp_rank_00_optim_states.pt 5: [2023-03-17 12:12:05,915] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step50000 is ready now! 1: [2023-03-17 12:12:05,915] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step50000 is ready now! 1: [2023-03-17 12:12:05,915] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step50000 is ready now! 1: [2023-03-17 12:12:05,915] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step50000 is ready now! 1: [2023-03-17 12:12:05,915] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step50000 is ready now! 2: [2023-03-17 12:12:05,915] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step50000/bf16_zero_pp_rank_17_mp_rank_00_optim_states.pt 2: [2023-03-17 12:12:05,915] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step50000/bf16_zero_pp_rank_23_mp_rank_00_optim_states.pt 2: [2023-03-17 12:12:05,915] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step50000/bf16_zero_pp_rank_22_mp_rank_00_optim_states.pt 2: [2023-03-17 12:12:05,915] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step50000/bf16_zero_pp_rank_16_mp_rank_00_optim_states.pt 2: [2023-03-17 12:12:05,915] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step50000/bf16_zero_pp_rank_21_mp_rank_00_optim_states.pt 4: [2023-03-17 12:12:05,915] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step50000/bf16_zero_pp_rank_35_mp_rank_00_optim_states.pt 4: [2023-03-17 12:12:05,915] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step50000/bf16_zero_pp_rank_38_mp_rank_00_optim_states.pt 4: [2023-03-17 12:12:05,915] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step50000/bf16_zero_pp_rank_36_mp_rank_00_optim_states.pt 4: [2023-03-17 12:12:05,915] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step50000/bf16_zero_pp_rank_39_mp_rank_00_optim_states.pt 1: [2023-03-17 12:12:05,915] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step50000 is ready now! 1: [2023-03-17 12:12:05,915] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step50000 is ready now! 1: [2023-03-17 12:12:05,915] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step50000 is ready now! 2: [2023-03-17 12:12:05,915] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step50000/bf16_zero_pp_rank_20_mp_rank_00_optim_states.pt 2: [2023-03-17 12:12:05,915] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step50000 is ready now! 2: [2023-03-17 12:12:05,915] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step50000 is ready now! 2: [2023-03-17 12:12:05,915] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step50000 is ready now! 2: [2023-03-17 12:12:05,915] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step50000 is ready now! 2: [2023-03-17 12:12:05,915] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step50000 is ready now! 2: [2023-03-17 12:12:05,915] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step50000 is ready now! 4: [2023-03-17 12:12:05,915] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step50000/bf16_zero_pp_rank_32_mp_rank_00_optim_states.pt 4: [2023-03-17 12:12:05,915] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step50000/bf16_zero_pp_rank_33_mp_rank_00_optim_states.pt 4: [2023-03-17 12:12:05,915] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step50000/bf16_zero_pp_rank_34_mp_rank_00_optim_states.pt 2: [2023-03-17 12:12:05,915] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step50000 is ready now! 4: [2023-03-17 12:12:05,915] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step50000 is ready now! 4: [2023-03-17 12:12:05,915] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step50000 is ready now! 4: [2023-03-17 12:12:05,915] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step50000 is ready now! 4: [2023-03-17 12:12:05,915] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step50000 is ready now! 4: [2023-03-17 12:12:05,915] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step50000 is ready now! 4: [2023-03-17 12:12:05,915] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step50000 is ready now! 4: [2023-03-17 12:12:05,915] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step50000 is ready now! 4: [2023-03-17 12:12:05,915] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step50000 is ready now! 6: [2023-03-17 12:12:06,072] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step50000/bf16_zero_pp_rank_55_mp_rank_00_optim_states.pt. 6: [2023-03-17 12:12:06,073] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step50000/bf16_zero_pp_rank_55_mp_rank_00_optim_states.pt 6: [2023-03-17 12:12:06,073] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step50000 is ready now! 6: [2023-03-17 12:12:06,073] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step50000/bf16_zero_pp_rank_51_mp_rank_00_optim_states.pt. 6: [2023-03-17 12:12:06,073] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step50000/bf16_zero_pp_rank_51_mp_rank_00_optim_states.pt 6: [2023-03-17 12:12:06,073] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step50000 is ready now! 6: [2023-03-17 12:12:06,074] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step50000/bf16_zero_pp_rank_54_mp_rank_00_optim_states.pt. 6: [2023-03-17 12:12:06,074] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step50000/bf16_zero_pp_rank_50_mp_rank_00_optim_states.pt. 6: [2023-03-17 12:12:06,074] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step50000/bf16_zero_pp_rank_54_mp_rank_00_optim_states.pt 6: [2023-03-17 12:12:06,074] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step50000/bf16_zero_pp_rank_50_mp_rank_00_optim_states.pt 6: [2023-03-17 12:12:06,074] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step50000 is ready now! 6: [2023-03-17 12:12:06,074] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step50000 is ready now! 6: [2023-03-17 12:12:06,074] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step50000/bf16_zero_pp_rank_53_mp_rank_00_optim_states.pt. 6: [2023-03-17 12:12:06,074] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step50000/bf16_zero_pp_rank_49_mp_rank_00_optim_states.pt. 6: [2023-03-17 12:12:06,074] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step50000/bf16_zero_pp_rank_48_mp_rank_00_optim_states.pt. 6: [2023-03-17 12:12:06,074] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step50000/bf16_zero_pp_rank_53_mp_rank_00_optim_states.pt 6: [2023-03-17 12:12:06,074] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step50000/bf16_zero_pp_rank_49_mp_rank_00_optim_states.pt 6: [2023-03-17 12:12:06,074] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step50000/bf16_zero_pp_rank_52_mp_rank_00_optim_states.pt. 6: [2023-03-17 12:12:06,074] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step50000 is ready now! 6: [2023-03-17 12:12:06,074] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step50000/bf16_zero_pp_rank_48_mp_rank_00_optim_states.pt 6: [2023-03-17 12:12:06,074] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step50000 is ready now! 6: [2023-03-17 12:12:06,074] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step50000/bf16_zero_pp_rank_52_mp_rank_00_optim_states.pt 6: [2023-03-17 12:12:06,074] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step50000 is ready now! 6: [2023-03-17 12:12:06,074] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step50000 is ready now! 0: successfully saved checkpoint at iteration 50000 to checkpoints_44m174b100m 7: time (ms) | save-checkpoint: 432.00 7: iteration 50100/ 331103 | consumed samples: 12825600 | consumed tokens: 26266828800 | elapsed time per iteration (s): 0.17 | learning rate: 1.911E-04 | global batch size: 256 | lm loss: 3.765435E+00 | grad norm: 0.318 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1521.230 | TFLOPs: 23.86 | 7: iteration 50200/ 331103 | consumed samples: 12851200 | consumed tokens: 26319257600 | elapsed time per iteration (s): 0.16 | learning rate: 1.911E-04 | global batch size: 256 | lm loss: 3.763443E+00 | grad norm: 0.294 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1578.656 | TFLOPs: 24.76 | 7: iteration 50300/ 331103 | consumed samples: 12876800 | consumed tokens: 26371686400 | elapsed time per iteration (s): 0.16 | learning rate: 1.910E-04 | global batch size: 256 | lm loss: 3.768454E+00 | grad norm: 0.297 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1560.933 | TFLOPs: 24.48 | 7: iteration 50400/ 331103 | consumed samples: 12902400 | consumed tokens: 26424115200 | elapsed time per iteration (s): 0.17 | learning rate: 1.910E-04 | global batch size: 256 | lm loss: 3.765332E+00 | grad norm: 0.316 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1544.054 | TFLOPs: 24.21 | 7: iteration 50500/ 331103 | consumed samples: 12928000 | consumed tokens: 26476544000 | elapsed time per iteration (s): 0.16 | learning rate: 1.910E-04 | global batch size: 256 | lm loss: 3.766075E+00 | grad norm: 0.339 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1553.397 | TFLOPs: 24.36 | 7: iteration 50600/ 331103 | consumed samples: 12953600 | consumed tokens: 26528972800 | elapsed time per iteration (s): 0.17 | learning rate: 1.909E-04 | global batch size: 256 | lm loss: 3.765295E+00 | grad norm: 0.317 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1539.874 | TFLOPs: 24.15 | 7: iteration 50700/ 331103 | consumed samples: 12979200 | consumed tokens: 26581401600 | elapsed time per iteration (s): 0.16 | learning rate: 1.909E-04 | global batch size: 256 | lm loss: 3.765760E+00 | grad norm: 0.315 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1561.710 | TFLOPs: 24.49 | 7: iteration 50800/ 331103 | consumed samples: 13004800 | consumed tokens: 26633830400 | elapsed time per iteration (s): 0.16 | learning rate: 1.908E-04 | global batch size: 256 | lm loss: 3.764472E+00 | grad norm: 0.315 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1562.697 | TFLOPs: 24.51 | 7: iteration 50900/ 331103 | consumed samples: 13030400 | consumed tokens: 26686259200 | elapsed time per iteration (s): 0.16 | learning rate: 1.908E-04 | global batch size: 256 | lm loss: 3.764390E+00 | grad norm: 0.377 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1575.717 | TFLOPs: 24.71 | 7: iteration 51000/ 331103 | consumed samples: 13056000 | consumed tokens: 26738688000 | elapsed time per iteration (s): 0.16 | learning rate: 1.908E-04 | global batch size: 256 | lm loss: 3.760943E+00 | grad norm: 0.310 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1570.806 | TFLOPs: 24.63 | 7: iteration 51100/ 331103 | consumed samples: 13081600 | consumed tokens: 26791116800 | elapsed time per iteration (s): 0.16 | learning rate: 1.907E-04 | global batch size: 256 | lm loss: 3.764482E+00 | grad norm: 0.277 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1558.350 | TFLOPs: 24.44 | 7: iteration 51200/ 331103 | consumed samples: 13107200 | consumed tokens: 26843545600 | elapsed time per iteration (s): 0.16 | learning rate: 1.907E-04 | global batch size: 256 | lm loss: 3.762949E+00 | grad norm: 0.312 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1558.518 | TFLOPs: 24.44 | 7: iteration 51300/ 331103 | consumed samples: 13132800 | consumed tokens: 26895974400 | elapsed time per iteration (s): 0.17 | learning rate: 1.906E-04 | global batch size: 256 | lm loss: 3.767167E+00 | grad norm: 0.330 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1544.965 | TFLOPs: 24.23 | 7: iteration 51400/ 331103 | consumed samples: 13158400 | consumed tokens: 26948403200 | elapsed time per iteration (s): 0.16 | learning rate: 1.906E-04 | global batch size: 256 | lm loss: 3.765579E+00 | grad norm: 0.292 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1577.421 | TFLOPs: 24.74 | 7: iteration 51500/ 331103 | consumed samples: 13184000 | consumed tokens: 27000832000 | elapsed time per iteration (s): 0.17 | learning rate: 1.906E-04 | global batch size: 256 | lm loss: 3.762720E+00 | grad norm: 0.323 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1532.794 | TFLOPs: 24.04 | 7: iteration 51600/ 331103 | consumed samples: 13209600 | consumed tokens: 27053260800 | elapsed time per iteration (s): 0.17 | learning rate: 1.905E-04 | global batch size: 256 | lm loss: 3.767187E+00 | grad norm: 0.307 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1545.431 | TFLOPs: 24.24 | 7: iteration 51700/ 331103 | consumed samples: 13235200 | consumed tokens: 27105689600 | elapsed time per iteration (s): 0.16 | learning rate: 1.905E-04 | global batch size: 256 | lm loss: 3.765324E+00 | grad norm: 0.301 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1562.218 | TFLOPs: 24.50 | 7: iteration 51800/ 331103 | consumed samples: 13260800 | consumed tokens: 27158118400 | elapsed time per iteration (s): 0.16 | learning rate: 1.905E-04 | global batch size: 256 | lm loss: 3.760990E+00 | grad norm: 0.318 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1572.635 | TFLOPs: 24.66 | 7: iteration 51900/ 331103 | consumed samples: 13286400 | consumed tokens: 27210547200 | elapsed time per iteration (s): 0.17 | learning rate: 1.904E-04 | global batch size: 256 | lm loss: 3.757321E+00 | grad norm: 0.366 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1539.346 | TFLOPs: 24.14 | 0: [2023-03-17 12:17:34,761] [INFO] [logging.py:68:log_dist] [Rank 0] step=52000, skipped=0, lr=[0.00019037767990441178, 0.00019037767990441178, 0.00019037767990441178], mom=[(0.9, 0.999), (0.9, 0.999), (0.9, 0.999)] 7: iteration 52000/ 331103 | consumed samples: 13312000 | consumed tokens: 27262976000 | elapsed time per iteration (s): 0.17 | learning rate: 1.904E-04 | global batch size: 256 | lm loss: 3.760308E+00 | grad norm: 0.327 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1543.861 | TFLOPs: 24.21 | 0: steps: 52000 loss: 3.7678 iter time (s): 0.163 samples/sec: 1569.475 7: iteration 52100/ 331103 | consumed samples: 13337600 | consumed tokens: 27315404800 | elapsed time per iteration (s): 0.16 | learning rate: 1.903E-04 | global batch size: 256 | lm loss: 3.759990E+00 | grad norm: 0.293 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1570.914 | TFLOPs: 24.64 | 7: iteration 52200/ 331103 | consumed samples: 13363200 | consumed tokens: 27367833600 | elapsed time per iteration (s): 0.17 | learning rate: 1.903E-04 | global batch size: 256 | lm loss: 3.759121E+00 | grad norm: 0.316 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1544.930 | TFLOPs: 24.23 | 7: iteration 52300/ 331103 | consumed samples: 13388800 | consumed tokens: 27420262400 | elapsed time per iteration (s): 0.16 | learning rate: 1.903E-04 | global batch size: 256 | lm loss: 3.759849E+00 | grad norm: 0.293 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1583.293 | TFLOPs: 24.83 | 7: iteration 52400/ 331103 | consumed samples: 13414400 | consumed tokens: 27472691200 | elapsed time per iteration (s): 0.16 | learning rate: 1.902E-04 | global batch size: 256 | lm loss: 3.761682E+00 | grad norm: 0.309 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1600.087 | TFLOPs: 25.09 | 7: iteration 52500/ 331103 | consumed samples: 13440000 | consumed tokens: 27525120000 | elapsed time per iteration (s): 0.16 | learning rate: 1.902E-04 | global batch size: 256 | lm loss: 3.761273E+00 | grad norm: 0.341 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1592.713 | TFLOPs: 24.98 | 7: iteration 52600/ 331103 | consumed samples: 13465600 | consumed tokens: 27577548800 | elapsed time per iteration (s): 0.16 | learning rate: 1.901E-04 | global batch size: 256 | lm loss: 3.760022E+00 | grad norm: 0.286 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1612.006 | TFLOPs: 25.28 | 7: iteration 52700/ 331103 | consumed samples: 13491200 | consumed tokens: 27629977600 | elapsed time per iteration (s): 0.18 | learning rate: 1.901E-04 | global batch size: 256 | lm loss: 3.764297E+00 | grad norm: 0.292 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1417.071 | TFLOPs: 22.22 | 7: iteration 52800/ 331103 | consumed samples: 13516800 | consumed tokens: 27682406400 | elapsed time per iteration (s): 0.16 | learning rate: 1.901E-04 | global batch size: 256 | lm loss: 3.760389E+00 | grad norm: 0.334 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1600.590 | TFLOPs: 25.10 | 7: iteration 52900/ 331103 | consumed samples: 13542400 | consumed tokens: 27734835200 | elapsed time per iteration (s): 0.16 | learning rate: 1.900E-04 | global batch size: 256 | lm loss: 3.761339E+00 | grad norm: 0.334 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1598.324 | TFLOPs: 25.07 | 7: iteration 53000/ 331103 | consumed samples: 13568000 | consumed tokens: 27787264000 | elapsed time per iteration (s): 0.16 | learning rate: 1.900E-04 | global batch size: 256 | lm loss: 3.763008E+00 | grad norm: 0.306 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1626.614 | TFLOPs: 25.51 | 7: iteration 53100/ 331103 | consumed samples: 13593600 | consumed tokens: 27839692800 | elapsed time per iteration (s): 0.16 | learning rate: 1.899E-04 | global batch size: 256 | lm loss: 3.760238E+00 | grad norm: 0.289 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1586.225 | TFLOPs: 24.88 | 7: iteration 53200/ 331103 | consumed samples: 13619200 | consumed tokens: 27892121600 | elapsed time per iteration (s): 0.16 | learning rate: 1.899E-04 | global batch size: 256 | lm loss: 3.762482E+00 | grad norm: 0.336 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1604.393 | TFLOPs: 25.16 | 7: iteration 53300/ 331103 | consumed samples: 13644800 | consumed tokens: 27944550400 | elapsed time per iteration (s): 0.16 | learning rate: 1.899E-04 | global batch size: 256 | lm loss: 3.755524E+00 | grad norm: 0.331 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1589.867 | TFLOPs: 24.93 | 7: iteration 53400/ 331103 | consumed samples: 13670400 | consumed tokens: 27996979200 | elapsed time per iteration (s): 0.16 | learning rate: 1.898E-04 | global batch size: 256 | lm loss: 3.762346E+00 | grad norm: 0.300 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1588.541 | TFLOPs: 24.91 | 7: iteration 53500/ 331103 | consumed samples: 13696000 | consumed tokens: 28049408000 | elapsed time per iteration (s): 0.16 | learning rate: 1.898E-04 | global batch size: 256 | lm loss: 3.756360E+00 | grad norm: 0.293 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1585.782 | TFLOPs: 24.87 | 7: iteration 53600/ 331103 | consumed samples: 13721600 | consumed tokens: 28101836800 | elapsed time per iteration (s): 0.16 | learning rate: 1.897E-04 | global batch size: 256 | lm loss: 3.759799E+00 | grad norm: 0.311 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1593.805 | TFLOPs: 24.99 | 7: iteration 53700/ 331103 | consumed samples: 13747200 | consumed tokens: 28154265600 | elapsed time per iteration (s): 0.16 | learning rate: 1.897E-04 | global batch size: 256 | lm loss: 3.759144E+00 | grad norm: 0.264 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1617.816 | TFLOPs: 25.37 | 7: iteration 53800/ 331103 | consumed samples: 13772800 | consumed tokens: 28206694400 | elapsed time per iteration (s): 0.16 | learning rate: 1.897E-04 | global batch size: 256 | lm loss: 3.758044E+00 | grad norm: 0.324 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1607.913 | TFLOPs: 25.22 | 7: iteration 53900/ 331103 | consumed samples: 13798400 | consumed tokens: 28259123200 | elapsed time per iteration (s): 0.16 | learning rate: 1.896E-04 | global batch size: 256 | lm loss: 3.754161E+00 | grad norm: 0.293 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1624.515 | TFLOPs: 25.48 | 0: [2023-03-17 12:22:57,392] [INFO] [logging.py:68:log_dist] [Rank 0] step=54000, skipped=0, lr=[0.0001895868453195527, 0.0001895868453195527, 0.0001895868453195527], mom=[(0.9, 0.999), (0.9, 0.999), (0.9, 0.999)] 7: iteration 54000/ 331103 | consumed samples: 13824000 | consumed tokens: 28311552000 | elapsed time per iteration (s): 0.16 | learning rate: 1.896E-04 | global batch size: 256 | lm loss: 3.759552E+00 | grad norm: 0.301 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1619.756 | TFLOPs: 25.40 | 0: steps: 54000 loss: 3.7976 iter time (s): 0.160 samples/sec: 1601.451 7: iteration 54100/ 331103 | consumed samples: 13849600 | consumed tokens: 28363980800 | elapsed time per iteration (s): 0.16 | learning rate: 1.895E-04 | global batch size: 256 | lm loss: 3.754800E+00 | grad norm: 0.302 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1593.966 | TFLOPs: 25.00 | 7: iteration 54200/ 331103 | consumed samples: 13875200 | consumed tokens: 28416409600 | elapsed time per iteration (s): 0.16 | learning rate: 1.895E-04 | global batch size: 256 | lm loss: 3.757044E+00 | grad norm: 0.365 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1613.757 | TFLOPs: 25.31 | 7: iteration 54300/ 331103 | consumed samples: 13900800 | consumed tokens: 28468838400 | elapsed time per iteration (s): 0.16 | learning rate: 1.895E-04 | global batch size: 256 | lm loss: 3.750348E+00 | grad norm: 0.280 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1609.321 | TFLOPs: 25.24 | 7: iteration 54400/ 331103 | consumed samples: 13926400 | consumed tokens: 28521267200 | elapsed time per iteration (s): 0.16 | learning rate: 1.894E-04 | global batch size: 256 | lm loss: 3.757743E+00 | grad norm: 0.310 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1602.886 | TFLOPs: 25.14 | 7: iteration 54500/ 331103 | consumed samples: 13952000 | consumed tokens: 28573696000 | elapsed time per iteration (s): 0.16 | learning rate: 1.894E-04 | global batch size: 256 | lm loss: 3.762189E+00 | grad norm: 0.331 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1601.337 | TFLOPs: 25.11 | 7: iteration 54600/ 331103 | consumed samples: 13977600 | consumed tokens: 28626124800 | elapsed time per iteration (s): 0.16 | learning rate: 1.893E-04 | global batch size: 256 | lm loss: 3.756522E+00 | grad norm: 0.358 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1605.690 | TFLOPs: 25.18 | 7: iteration 54700/ 331103 | consumed samples: 14003200 | consumed tokens: 28678553600 | elapsed time per iteration (s): 0.16 | learning rate: 1.893E-04 | global batch size: 256 | lm loss: 3.757452E+00 | grad norm: 0.290 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1599.847 | TFLOPs: 25.09 | 7: iteration 54800/ 331103 | consumed samples: 14028800 | consumed tokens: 28730982400 | elapsed time per iteration (s): 0.16 | learning rate: 1.893E-04 | global batch size: 256 | lm loss: 3.751515E+00 | grad norm: 0.316 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1616.924 | TFLOPs: 25.36 | 7: iteration 54900/ 331103 | consumed samples: 14054400 | consumed tokens: 28783411200 | elapsed time per iteration (s): 0.16 | learning rate: 1.892E-04 | global batch size: 256 | lm loss: 3.757508E+00 | grad norm: 0.322 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1607.286 | TFLOPs: 25.21 | 7: iteration 55000/ 331103 | consumed samples: 14080000 | consumed tokens: 28835840000 | elapsed time per iteration (s): 0.16 | learning rate: 1.892E-04 | global batch size: 256 | lm loss: 3.759962E+00 | grad norm: 0.316 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1603.100 | TFLOPs: 25.14 | 7: iteration 55100/ 331103 | consumed samples: 14105600 | consumed tokens: 28888268800 | elapsed time per iteration (s): 0.16 | learning rate: 1.891E-04 | global batch size: 256 | lm loss: 3.754931E+00 | grad norm: 0.299 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1623.262 | TFLOPs: 25.46 | 7: iteration 55200/ 331103 | consumed samples: 14131200 | consumed tokens: 28940697600 | elapsed time per iteration (s): 0.16 | learning rate: 1.891E-04 | global batch size: 256 | lm loss: 3.752804E+00 | grad norm: 0.355 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1610.166 | TFLOPs: 25.25 | 7: iteration 55300/ 331103 | consumed samples: 14156800 | consumed tokens: 28993126400 | elapsed time per iteration (s): 0.16 | learning rate: 1.891E-04 | global batch size: 256 | lm loss: 3.754986E+00 | grad norm: 0.324 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1574.773 | TFLOPs: 24.70 | 7: iteration 55400/ 331103 | consumed samples: 14182400 | consumed tokens: 29045555200 | elapsed time per iteration (s): 0.16 | learning rate: 1.890E-04 | global batch size: 256 | lm loss: 3.754115E+00 | grad norm: 0.297 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1580.510 | TFLOPs: 24.79 | 7: iteration 55500/ 331103 | consumed samples: 14208000 | consumed tokens: 29097984000 | elapsed time per iteration (s): 0.16 | learning rate: 1.890E-04 | global batch size: 256 | lm loss: 3.755041E+00 | grad norm: 0.328 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1627.343 | TFLOPs: 25.52 | 7: iteration 55600/ 331103 | consumed samples: 14233600 | consumed tokens: 29150412800 | elapsed time per iteration (s): 0.16 | learning rate: 1.889E-04 | global batch size: 256 | lm loss: 3.755871E+00 | grad norm: 0.319 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1595.995 | TFLOPs: 25.03 | 7: iteration 55700/ 331103 | consumed samples: 14259200 | consumed tokens: 29202841600 | elapsed time per iteration (s): 0.16 | learning rate: 1.889E-04 | global batch size: 256 | lm loss: 3.756833E+00 | grad norm: 0.351 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1614.319 | TFLOPs: 25.32 | 7: iteration 55800/ 331103 | consumed samples: 14284800 | consumed tokens: 29255270400 | elapsed time per iteration (s): 0.16 | learning rate: 1.889E-04 | global batch size: 256 | lm loss: 3.753950E+00 | grad norm: 0.286 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1616.422 | TFLOPs: 25.35 | 7: iteration 55900/ 331103 | consumed samples: 14310400 | consumed tokens: 29307699200 | elapsed time per iteration (s): 0.16 | learning rate: 1.888E-04 | global batch size: 256 | lm loss: 3.756063E+00 | grad norm: 0.304 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1602.516 | TFLOPs: 25.13 | 0: [2023-03-17 12:28:16,158] [INFO] [logging.py:68:log_dist] [Rank 0] step=56000, skipped=0, lr=[0.00018876676993476922, 0.00018876676993476922, 0.00018876676993476922], mom=[(0.9, 0.999), (0.9, 0.999), (0.9, 0.999)] 7: iteration 56000/ 331103 | consumed samples: 14336000 | consumed tokens: 29360128000 | elapsed time per iteration (s): 0.16 | learning rate: 1.888E-04 | global batch size: 256 | lm loss: 3.751686E+00 | grad norm: 0.318 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1626.617 | TFLOPs: 25.51 | 0: steps: 56000 loss: 3.7332 iter time (s): 0.158 samples/sec: 1621.352 7: iteration 56100/ 331103 | consumed samples: 14361600 | consumed tokens: 29412556800 | elapsed time per iteration (s): 0.16 | learning rate: 1.887E-04 | global batch size: 256 | lm loss: 3.754645E+00 | grad norm: 0.297 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1626.985 | TFLOPs: 25.52 | 7: iteration 56200/ 331103 | consumed samples: 14387200 | consumed tokens: 29464985600 | elapsed time per iteration (s): 0.16 | learning rate: 1.887E-04 | global batch size: 256 | lm loss: 3.749066E+00 | grad norm: 0.292 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1558.673 | TFLOPs: 24.44 | 7: iteration 56300/ 331103 | consumed samples: 14412800 | consumed tokens: 29517414400 | elapsed time per iteration (s): 0.17 | learning rate: 1.886E-04 | global batch size: 256 | lm loss: 3.755583E+00 | grad norm: 0.347 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1508.524 | TFLOPs: 23.66 | 7: iteration 56400/ 331103 | consumed samples: 14438400 | consumed tokens: 29569843200 | elapsed time per iteration (s): 0.16 | learning rate: 1.886E-04 | global batch size: 256 | lm loss: 3.754828E+00 | grad norm: 0.317 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1565.444 | TFLOPs: 24.55 | 7: iteration 56500/ 331103 | consumed samples: 14464000 | consumed tokens: 29622272000 | elapsed time per iteration (s): 0.16 | learning rate: 1.886E-04 | global batch size: 256 | lm loss: 3.754421E+00 | grad norm: 0.341 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1601.835 | TFLOPs: 25.12 | 7: iteration 56600/ 331103 | consumed samples: 14489600 | consumed tokens: 29674700800 | elapsed time per iteration (s): 0.16 | learning rate: 1.885E-04 | global batch size: 256 | lm loss: 3.751732E+00 | grad norm: 0.304 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1580.249 | TFLOPs: 24.78 | 7: iteration 56700/ 331103 | consumed samples: 14515200 | consumed tokens: 29727129600 | elapsed time per iteration (s): 0.16 | learning rate: 1.885E-04 | global batch size: 256 | lm loss: 3.749176E+00 | grad norm: 0.289 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1573.194 | TFLOPs: 24.67 | 7: iteration 56800/ 331103 | consumed samples: 14540800 | consumed tokens: 29779558400 | elapsed time per iteration (s): 0.16 | learning rate: 1.884E-04 | global batch size: 256 | lm loss: 3.748829E+00 | grad norm: 0.313 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1591.676 | TFLOPs: 24.96 | 7: iteration 56900/ 331103 | consumed samples: 14566400 | consumed tokens: 29831987200 | elapsed time per iteration (s): 0.17 | learning rate: 1.884E-04 | global batch size: 256 | lm loss: 3.754515E+00 | grad norm: 0.320 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1537.695 | TFLOPs: 24.11 | 7: iteration 57000/ 331103 | consumed samples: 14592000 | consumed tokens: 29884416000 | elapsed time per iteration (s): 0.16 | learning rate: 1.883E-04 | global batch size: 256 | lm loss: 3.752103E+00 | grad norm: 0.331 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1612.702 | TFLOPs: 25.29 | 7: iteration 57100/ 331103 | consumed samples: 14617600 | consumed tokens: 29936844800 | elapsed time per iteration (s): 0.16 | learning rate: 1.883E-04 | global batch size: 256 | lm loss: 3.752546E+00 | grad norm: 0.308 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1603.956 | TFLOPs: 25.15 | 7: iteration 57200/ 331103 | consumed samples: 14643200 | consumed tokens: 29989273600 | elapsed time per iteration (s): 0.16 | learning rate: 1.883E-04 | global batch size: 256 | lm loss: 3.746419E+00 | grad norm: 0.307 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1616.192 | TFLOPs: 25.35 | 7: iteration 57300/ 331103 | consumed samples: 14668800 | consumed tokens: 30041702400 | elapsed time per iteration (s): 0.19 | learning rate: 1.882E-04 | global batch size: 256 | lm loss: 3.750322E+00 | grad norm: 0.372 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1374.752 | TFLOPs: 21.56 | 7: iteration 57400/ 331103 | consumed samples: 14694400 | consumed tokens: 30094131200 | elapsed time per iteration (s): 0.16 | learning rate: 1.882E-04 | global batch size: 256 | lm loss: 3.751958E+00 | grad norm: 0.291 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1622.534 | TFLOPs: 25.45 | 7: iteration 57500/ 331103 | consumed samples: 14720000 | consumed tokens: 30146560000 | elapsed time per iteration (s): 0.16 | learning rate: 1.881E-04 | global batch size: 256 | lm loss: 3.756983E+00 | grad norm: 0.304 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1621.443 | TFLOPs: 25.43 | 7: iteration 57600/ 331103 | consumed samples: 14745600 | consumed tokens: 30198988800 | elapsed time per iteration (s): 0.16 | learning rate: 1.881E-04 | global batch size: 256 | lm loss: 3.753039E+00 | grad norm: 0.320 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1566.302 | TFLOPs: 24.56 | 7: iteration 57700/ 331103 | consumed samples: 14771200 | consumed tokens: 30251417600 | elapsed time per iteration (s): 0.16 | learning rate: 1.880E-04 | global batch size: 256 | lm loss: 3.749078E+00 | grad norm: 0.278 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1618.854 | TFLOPs: 25.39 | 7: iteration 57800/ 331103 | consumed samples: 14796800 | consumed tokens: 30303846400 | elapsed time per iteration (s): 0.16 | learning rate: 1.880E-04 | global batch size: 256 | lm loss: 3.753367E+00 | grad norm: 0.323 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1582.533 | TFLOPs: 24.82 | 7: iteration 57900/ 331103 | consumed samples: 14822400 | consumed tokens: 30356275200 | elapsed time per iteration (s): 0.16 | learning rate: 1.880E-04 | global batch size: 256 | lm loss: 3.749456E+00 | grad norm: 0.354 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1608.813 | TFLOPs: 25.23 | 0: [2023-03-17 12:33:41,417] [INFO] [logging.py:68:log_dist] [Rank 0] step=58000, skipped=0, lr=[0.00018791775505186758, 0.00018791775505186758, 0.00018791775505186758], mom=[(0.9, 0.999), (0.9, 0.999), (0.9, 0.999)] 7: iteration 58000/ 331103 | consumed samples: 14848000 | consumed tokens: 30408704000 | elapsed time per iteration (s): 0.16 | learning rate: 1.879E-04 | global batch size: 256 | lm loss: 3.746758E+00 | grad norm: 0.325 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1553.339 | TFLOPs: 24.36 | 0: steps: 58000 loss: 3.7474 iter time (s): 0.161 samples/sec: 1588.367 7: iteration 58100/ 331103 | consumed samples: 14873600 | consumed tokens: 30461132800 | elapsed time per iteration (s): 0.17 | learning rate: 1.879E-04 | global batch size: 256 | lm loss: 3.752967E+00 | grad norm: 0.354 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1549.953 | TFLOPs: 24.31 | 7: iteration 58200/ 331103 | consumed samples: 14899200 | consumed tokens: 30513561600 | elapsed time per iteration (s): 0.17 | learning rate: 1.878E-04 | global batch size: 256 | lm loss: 3.751679E+00 | grad norm: 0.307 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1530.219 | TFLOPs: 24.00 | 7: iteration 58300/ 331103 | consumed samples: 14924800 | consumed tokens: 30565990400 | elapsed time per iteration (s): 0.17 | learning rate: 1.878E-04 | global batch size: 256 | lm loss: 3.754006E+00 | grad norm: 0.329 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1514.093 | TFLOPs: 23.74 | 7: iteration 58400/ 331103 | consumed samples: 14950400 | consumed tokens: 30618419200 | elapsed time per iteration (s): 0.16 | learning rate: 1.877E-04 | global batch size: 256 | lm loss: 3.749315E+00 | grad norm: 0.275 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1587.798 | TFLOPs: 24.90 | 7: iteration 58500/ 331103 | consumed samples: 14976000 | consumed tokens: 30670848000 | elapsed time per iteration (s): 0.16 | learning rate: 1.877E-04 | global batch size: 256 | lm loss: 3.750500E+00 | grad norm: 0.342 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1587.234 | TFLOPs: 24.89 | 7: iteration 58600/ 331103 | consumed samples: 15001600 | consumed tokens: 30723276800 | elapsed time per iteration (s): 0.16 | learning rate: 1.877E-04 | global batch size: 256 | lm loss: 3.748807E+00 | grad norm: 0.310 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1580.984 | TFLOPs: 24.79 | 7: iteration 58700/ 331103 | consumed samples: 15027200 | consumed tokens: 30775705600 | elapsed time per iteration (s): 0.16 | learning rate: 1.876E-04 | global batch size: 256 | lm loss: 3.749374E+00 | grad norm: 0.288 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1585.518 | TFLOPs: 24.86 | 7: iteration 58800/ 331103 | consumed samples: 15052800 | consumed tokens: 30828134400 | elapsed time per iteration (s): 0.16 | learning rate: 1.876E-04 | global batch size: 256 | lm loss: 3.749685E+00 | grad norm: 0.313 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1564.552 | TFLOPs: 24.54 | 7: iteration 58900/ 331103 | consumed samples: 15078400 | consumed tokens: 30880563200 | elapsed time per iteration (s): 0.16 | learning rate: 1.875E-04 | global batch size: 256 | lm loss: 3.747552E+00 | grad norm: 0.315 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1577.761 | TFLOPs: 24.74 | 7: iteration 59000/ 331103 | consumed samples: 15104000 | consumed tokens: 30932992000 | elapsed time per iteration (s): 0.16 | learning rate: 1.875E-04 | global batch size: 256 | lm loss: 3.745411E+00 | grad norm: 0.327 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1590.354 | TFLOPs: 24.94 | 7: iteration 59100/ 331103 | consumed samples: 15129600 | consumed tokens: 30985420800 | elapsed time per iteration (s): 0.16 | learning rate: 1.874E-04 | global batch size: 256 | lm loss: 3.745911E+00 | grad norm: 0.351 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1599.956 | TFLOPs: 25.09 | 7: iteration 59200/ 331103 | consumed samples: 15155200 | consumed tokens: 31037849600 | elapsed time per iteration (s): 0.16 | learning rate: 1.874E-04 | global batch size: 256 | lm loss: 3.753093E+00 | grad norm: 0.307 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1605.929 | TFLOPs: 25.18 | 7: iteration 59300/ 331103 | consumed samples: 15180800 | consumed tokens: 31090278400 | elapsed time per iteration (s): 0.16 | learning rate: 1.874E-04 | global batch size: 256 | lm loss: 3.748009E+00 | grad norm: 0.327 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1611.527 | TFLOPs: 25.27 | 7: iteration 59400/ 331103 | consumed samples: 15206400 | consumed tokens: 31142707200 | elapsed time per iteration (s): 0.16 | learning rate: 1.873E-04 | global batch size: 256 | lm loss: 3.741735E+00 | grad norm: 0.313 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1607.405 | TFLOPs: 25.21 | 7: iteration 59500/ 331103 | consumed samples: 15232000 | consumed tokens: 31195136000 | elapsed time per iteration (s): 0.16 | learning rate: 1.873E-04 | global batch size: 256 | lm loss: 3.746602E+00 | grad norm: 0.286 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1572.301 | TFLOPs: 24.66 | 7: iteration 59600/ 331103 | consumed samples: 15257600 | consumed tokens: 31247564800 | elapsed time per iteration (s): 0.16 | learning rate: 1.872E-04 | global batch size: 256 | lm loss: 3.749232E+00 | grad norm: 0.290 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1599.928 | TFLOPs: 25.09 | 7: iteration 59700/ 331103 | consumed samples: 15283200 | consumed tokens: 31299993600 | elapsed time per iteration (s): 0.16 | learning rate: 1.872E-04 | global batch size: 256 | lm loss: 3.748999E+00 | grad norm: 0.326 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1586.951 | TFLOPs: 24.89 | 7: iteration 59800/ 331103 | consumed samples: 15308800 | consumed tokens: 31352422400 | elapsed time per iteration (s): 0.16 | learning rate: 1.871E-04 | global batch size: 256 | lm loss: 3.749010E+00 | grad norm: 0.310 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1619.657 | TFLOPs: 25.40 | 7: iteration 59900/ 331103 | consumed samples: 15334400 | consumed tokens: 31404851200 | elapsed time per iteration (s): 0.16 | learning rate: 1.871E-04 | global batch size: 256 | lm loss: 3.749825E+00 | grad norm: 0.306 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1564.652 | TFLOPs: 24.54 | 0: [2023-03-17 12:39:04,955] [INFO] [logging.py:68:log_dist] [Rank 0] step=60000, skipped=0, lr=[0.0001870401126052413, 0.0001870401126052413, 0.0001870401126052413], mom=[(0.9, 0.999), (0.9, 0.999), (0.9, 0.999)] 7: iteration 60000/ 331103 | consumed samples: 15360000 | consumed tokens: 31457280000 | elapsed time per iteration (s): 0.16 | learning rate: 1.870E-04 | global batch size: 256 | lm loss: 3.751461E+00 | grad norm: 0.311 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1622.875 | TFLOPs: 25.45 | 0: steps: 60000 loss: 3.7601 iter time (s): 0.160 samples/sec: 1597.426 7: ------------------------------------------------------------------------------------------------ 7: validation loss at iteration 60000 | lm loss value: 3.899973E+00 | lm loss PPL: 4.940110E+01 | 7: ------------------------------------------------------------------------------------------------ 0: saving checkpoint at iteration 60000 to checkpoints_44m174b100m 0: [2023-03-17 12:39:05,059] [INFO] [logging.py:68:log_dist] [Rank 0] [Torch] Checkpoint global_step60000 is begin to save! 0: [2023-03-17 12:39:05,064] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step60000/layer_01-model_00-model_states.pt... 0: [2023-03-17 12:39:05,150] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step60000/layer_01-model_00-model_states.pt. 0: [2023-03-17 12:39:05,150] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step60000/layer_03-model_00-model_states.pt... 0: [2023-03-17 12:39:05,161] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step60000/layer_03-model_00-model_states.pt. 0: [2023-03-17 12:39:05,161] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step60000/layer_04-model_00-model_states.pt... 0: [2023-03-17 12:39:05,169] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step60000/layer_04-model_00-model_states.pt. 0: [2023-03-17 12:39:05,169] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step60000/layer_05-model_00-model_states.pt... 0: [2023-03-17 12:39:05,177] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step60000/layer_05-model_00-model_states.pt. 0: [2023-03-17 12:39:05,177] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step60000/layer_06-model_00-model_states.pt... 0: [2023-03-17 12:39:05,185] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step60000/layer_06-model_00-model_states.pt. 0: [2023-03-17 12:39:05,186] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step60000/layer_07-model_00-model_states.pt... 0: [2023-03-17 12:39:05,193] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step60000/layer_07-model_00-model_states.pt. 0: [2023-03-17 12:39:05,194] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step60000/layer_08-model_00-model_states.pt... 0: [2023-03-17 12:39:05,201] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step60000/layer_08-model_00-model_states.pt. 0: [2023-03-17 12:39:05,202] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step60000/layer_09-model_00-model_states.pt... 0: [2023-03-17 12:39:05,209] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step60000/layer_09-model_00-model_states.pt. 0: [2023-03-17 12:39:05,210] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step60000/layer_10-model_00-model_states.pt... 0: [2023-03-17 12:39:05,217] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step60000/layer_10-model_00-model_states.pt. 0: [2023-03-17 12:39:05,217] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step60000/layer_12-model_00-model_states.pt... 0: [2023-03-17 12:39:05,218] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step60000/layer_12-model_00-model_states.pt. 0: [2023-03-17 12:39:05,219] [INFO] [logging.py:68:log_dist] [Rank 0] Saving model checkpoint: checkpoints_44m174b100m/global_step60000/mp_rank_00_model_states.pt 0: [2023-03-17 12:39:05,219] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step60000/mp_rank_00_model_states.pt... 0: [2023-03-17 12:39:05,222] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step60000/mp_rank_00_model_states.pt. 0: [2023-03-17 12:39:05,238] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step60000/bf16_zero_pp_rank_0_mp_rank_00_optim_states.pt... 0: [2023-03-17 12:39:05,238] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step60000/bf16_zero_pp_rank_7_mp_rank_00_optim_states.pt... 0: [2023-03-17 12:39:05,238] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step60000/bf16_zero_pp_rank_4_mp_rank_00_optim_states.pt... 0: [2023-03-17 12:39:05,238] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step60000/bf16_zero_pp_rank_2_mp_rank_00_optim_states.pt... 0: [2023-03-17 12:39:05,238] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step60000/bf16_zero_pp_rank_1_mp_rank_00_optim_states.pt... 0: [2023-03-17 12:39:05,238] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step60000/bf16_zero_pp_rank_3_mp_rank_00_optim_states.pt... 0: [2023-03-17 12:39:05,238] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step60000/bf16_zero_pp_rank_5_mp_rank_00_optim_states.pt... 7: [2023-03-17 12:39:05,238] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step60000/bf16_zero_pp_rank_63_mp_rank_00_optim_states.pt... 7: [2023-03-17 12:39:05,238] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step60000/bf16_zero_pp_rank_59_mp_rank_00_optim_states.pt... 7: [2023-03-17 12:39:05,238] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step60000/bf16_zero_pp_rank_62_mp_rank_00_optim_states.pt... 7: [2023-03-17 12:39:05,238] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step60000/bf16_zero_pp_rank_61_mp_rank_00_optim_states.pt... 7: [2023-03-17 12:39:05,238] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step60000/bf16_zero_pp_rank_56_mp_rank_00_optim_states.pt... 0: [2023-03-17 12:39:05,238] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step60000/bf16_zero_pp_rank_6_mp_rank_00_optim_states.pt... 3: [2023-03-17 12:39:05,238] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step60000/bf16_zero_pp_rank_30_mp_rank_00_optim_states.pt... 3: [2023-03-17 12:39:05,238] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step60000/bf16_zero_pp_rank_24_mp_rank_00_optim_states.pt... 3: [2023-03-17 12:39:05,238] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step60000/bf16_zero_pp_rank_31_mp_rank_00_optim_states.pt... 3: [2023-03-17 12:39:05,238] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step60000/bf16_zero_pp_rank_25_mp_rank_00_optim_states.pt... 3: [2023-03-17 12:39:05,238] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step60000/bf16_zero_pp_rank_28_mp_rank_00_optim_states.pt... 2: [2023-03-17 12:39:05,238] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step60000/bf16_zero_pp_rank_20_mp_rank_00_optim_states.pt... 2: [2023-03-17 12:39:05,238] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step60000/bf16_zero_pp_rank_22_mp_rank_00_optim_states.pt... 2: [2023-03-17 12:39:05,238] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step60000/bf16_zero_pp_rank_16_mp_rank_00_optim_states.pt... 2: [2023-03-17 12:39:05,238] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step60000/bf16_zero_pp_rank_18_mp_rank_00_optim_states.pt... 2: [2023-03-17 12:39:05,238] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step60000/bf16_zero_pp_rank_23_mp_rank_00_optim_states.pt... 4: [2023-03-17 12:39:05,238] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step60000/bf16_zero_pp_rank_37_mp_rank_00_optim_states.pt... 4: [2023-03-17 12:39:05,238] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step60000/bf16_zero_pp_rank_38_mp_rank_00_optim_states.pt... 4: [2023-03-17 12:39:05,238] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step60000/bf16_zero_pp_rank_36_mp_rank_00_optim_states.pt... 4: [2023-03-17 12:39:05,238] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step60000/bf16_zero_pp_rank_32_mp_rank_00_optim_states.pt... 4: [2023-03-17 12:39:05,238] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step60000/bf16_zero_pp_rank_34_mp_rank_00_optim_states.pt... 6: [2023-03-17 12:39:05,238] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step60000/bf16_zero_pp_rank_49_mp_rank_00_optim_states.pt... 6: [2023-03-17 12:39:05,238] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step60000/bf16_zero_pp_rank_48_mp_rank_00_optim_states.pt... 6: [2023-03-17 12:39:05,238] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step60000/bf16_zero_pp_rank_55_mp_rank_00_optim_states.pt... 6: [2023-03-17 12:39:05,238] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step60000/bf16_zero_pp_rank_52_mp_rank_00_optim_states.pt... 6: [2023-03-17 12:39:05,238] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step60000/bf16_zero_pp_rank_51_mp_rank_00_optim_states.pt... 7: [2023-03-17 12:39:05,238] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step60000/bf16_zero_pp_rank_60_mp_rank_00_optim_states.pt... 7: [2023-03-17 12:39:05,238] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step60000/bf16_zero_pp_rank_58_mp_rank_00_optim_states.pt... 7: [2023-03-17 12:39:05,238] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step60000/bf16_zero_pp_rank_57_mp_rank_00_optim_states.pt... 5: [2023-03-17 12:39:05,238] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step60000/bf16_zero_pp_rank_40_mp_rank_00_optim_states.pt... 5: [2023-03-17 12:39:05,238] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step60000/bf16_zero_pp_rank_46_mp_rank_00_optim_states.pt... 5: [2023-03-17 12:39:05,238] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step60000/bf16_zero_pp_rank_45_mp_rank_00_optim_states.pt... 5: [2023-03-17 12:39:05,238] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step60000/bf16_zero_pp_rank_47_mp_rank_00_optim_states.pt... 5: [2023-03-17 12:39:05,238] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step60000/bf16_zero_pp_rank_41_mp_rank_00_optim_states.pt... 1: [2023-03-17 12:39:05,238] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step60000/bf16_zero_pp_rank_8_mp_rank_00_optim_states.pt... 1: [2023-03-17 12:39:05,238] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step60000/bf16_zero_pp_rank_13_mp_rank_00_optim_states.pt... 1: [2023-03-17 12:39:05,238] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step60000/bf16_zero_pp_rank_10_mp_rank_00_optim_states.pt... 1: [2023-03-17 12:39:05,238] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step60000/bf16_zero_pp_rank_11_mp_rank_00_optim_states.pt... 1: [2023-03-17 12:39:05,238] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step60000/bf16_zero_pp_rank_15_mp_rank_00_optim_states.pt... 3: [2023-03-17 12:39:05,238] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step60000/bf16_zero_pp_rank_29_mp_rank_00_optim_states.pt... 3: [2023-03-17 12:39:05,238] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step60000/bf16_zero_pp_rank_26_mp_rank_00_optim_states.pt... 3: [2023-03-17 12:39:05,238] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step60000/bf16_zero_pp_rank_27_mp_rank_00_optim_states.pt... 2: [2023-03-17 12:39:05,238] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step60000/bf16_zero_pp_rank_17_mp_rank_00_optim_states.pt... 2: [2023-03-17 12:39:05,238] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step60000/bf16_zero_pp_rank_19_mp_rank_00_optim_states.pt... 2: [2023-03-17 12:39:05,238] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step60000/bf16_zero_pp_rank_21_mp_rank_00_optim_states.pt... 4: [2023-03-17 12:39:05,238] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step60000/bf16_zero_pp_rank_39_mp_rank_00_optim_states.pt... 4: [2023-03-17 12:39:05,238] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step60000/bf16_zero_pp_rank_33_mp_rank_00_optim_states.pt... 4: [2023-03-17 12:39:05,238] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step60000/bf16_zero_pp_rank_35_mp_rank_00_optim_states.pt... 6: [2023-03-17 12:39:05,238] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step60000/bf16_zero_pp_rank_50_mp_rank_00_optim_states.pt... 6: [2023-03-17 12:39:05,238] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step60000/bf16_zero_pp_rank_54_mp_rank_00_optim_states.pt... 6: [2023-03-17 12:39:05,238] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step60000/bf16_zero_pp_rank_53_mp_rank_00_optim_states.pt... 5: [2023-03-17 12:39:05,238] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step60000/bf16_zero_pp_rank_42_mp_rank_00_optim_states.pt... 5: [2023-03-17 12:39:05,238] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step60000/bf16_zero_pp_rank_44_mp_rank_00_optim_states.pt... 5: [2023-03-17 12:39:05,238] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step60000/bf16_zero_pp_rank_43_mp_rank_00_optim_states.pt... 1: [2023-03-17 12:39:05,238] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step60000/bf16_zero_pp_rank_14_mp_rank_00_optim_states.pt... 1: [2023-03-17 12:39:05,238] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step60000/bf16_zero_pp_rank_9_mp_rank_00_optim_states.pt... 1: [2023-03-17 12:39:05,238] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step60000/bf16_zero_pp_rank_12_mp_rank_00_optim_states.pt... 0: [2023-03-17 12:39:05,250] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step60000/bf16_zero_pp_rank_0_mp_rank_00_optim_states.pt. 7: [2023-03-17 12:39:05,252] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step60000/bf16_zero_pp_rank_63_mp_rank_00_optim_states.pt. 7: [2023-03-17 12:39:05,252] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step60000/bf16_zero_pp_rank_63_mp_rank_00_optim_states.pt 7: [2023-03-17 12:39:05,252] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step60000 is ready now! 0: [2023-03-17 12:39:05,252] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step60000/bf16_zero_pp_rank_4_mp_rank_00_optim_states.pt. 0: [2023-03-17 12:39:05,252] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step60000/bf16_zero_pp_rank_4_mp_rank_00_optim_states.pt 0: [2023-03-17 12:39:05,252] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step60000 is ready now! 0: [2023-03-17 12:39:05,253] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step60000/bf16_zero_pp_rank_0_mp_rank_00_optim_states.pt 0: [2023-03-17 12:39:05,253] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step60000 is ready now! 1: [2023-03-17 12:39:05,253] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step60000/bf16_zero_pp_rank_13_mp_rank_00_optim_states.pt. 1: [2023-03-17 12:39:05,253] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step60000/bf16_zero_pp_rank_13_mp_rank_00_optim_states.pt 1: [2023-03-17 12:39:05,253] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step60000 is ready now! 1: [2023-03-17 12:39:05,253] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step60000/bf16_zero_pp_rank_8_mp_rank_00_optim_states.pt. 6: [2023-03-17 12:39:05,253] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step60000/bf16_zero_pp_rank_55_mp_rank_00_optim_states.pt. 6: [2023-03-17 12:39:05,253] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step60000/bf16_zero_pp_rank_55_mp_rank_00_optim_states.pt 1: [2023-03-17 12:39:05,253] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step60000/bf16_zero_pp_rank_8_mp_rank_00_optim_states.pt 6: [2023-03-17 12:39:05,253] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step60000 is ready now! 1: [2023-03-17 12:39:05,253] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step60000 is ready now! 6: [2023-03-17 12:39:05,253] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step60000/bf16_zero_pp_rank_49_mp_rank_00_optim_states.pt. 6: [2023-03-17 12:39:05,253] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step60000/bf16_zero_pp_rank_49_mp_rank_00_optim_states.pt 4: [2023-03-17 12:39:05,253] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step60000/bf16_zero_pp_rank_37_mp_rank_00_optim_states.pt. 4: [2023-03-17 12:39:05,253] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step60000/bf16_zero_pp_rank_38_mp_rank_00_optim_states.pt. 6: [2023-03-17 12:39:05,253] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step60000 is ready now! 4: [2023-03-17 12:39:05,253] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step60000/bf16_zero_pp_rank_37_mp_rank_00_optim_states.pt 4: [2023-03-17 12:39:05,253] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step60000/bf16_zero_pp_rank_38_mp_rank_00_optim_states.pt 4: [2023-03-17 12:39:05,253] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step60000 is ready now! 4: [2023-03-17 12:39:05,253] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step60000 is ready now! 0: [2023-03-17 12:39:05,254] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step60000/bf16_zero_pp_rank_6_mp_rank_00_optim_states.pt. 0: [2023-03-17 12:39:05,254] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step60000/bf16_zero_pp_rank_6_mp_rank_00_optim_states.pt 0: [2023-03-17 12:39:05,254] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step60000 is ready now! 7: [2023-03-17 12:39:05,254] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step60000/bf16_zero_pp_rank_59_mp_rank_00_optim_states.pt. 7: [2023-03-17 12:39:05,254] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step60000/bf16_zero_pp_rank_59_mp_rank_00_optim_states.pt 7: [2023-03-17 12:39:05,254] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step60000 is ready now! 6: [2023-03-17 12:39:05,254] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step60000/bf16_zero_pp_rank_48_mp_rank_00_optim_states.pt. 6: [2023-03-17 12:39:05,254] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step60000/bf16_zero_pp_rank_48_mp_rank_00_optim_states.pt 6: [2023-03-17 12:39:05,254] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step60000 is ready now! 1: [2023-03-17 12:39:05,254] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step60000/bf16_zero_pp_rank_11_mp_rank_00_optim_states.pt. 0: [2023-03-17 12:39:05,254] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step60000/bf16_zero_pp_rank_5_mp_rank_00_optim_states.pt. 1: [2023-03-17 12:39:05,254] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step60000/bf16_zero_pp_rank_11_mp_rank_00_optim_states.pt 0: [2023-03-17 12:39:05,255] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step60000/bf16_zero_pp_rank_5_mp_rank_00_optim_states.pt 1: [2023-03-17 12:39:05,255] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step60000 is ready now! 0: [2023-03-17 12:39:05,255] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step60000 is ready now! 4: [2023-03-17 12:39:05,255] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step60000/bf16_zero_pp_rank_34_mp_rank_00_optim_states.pt. 4: [2023-03-17 12:39:05,255] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step60000/bf16_zero_pp_rank_34_mp_rank_00_optim_states.pt 7: [2023-03-17 12:39:05,255] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step60000/bf16_zero_pp_rank_61_mp_rank_00_optim_states.pt. 4: [2023-03-17 12:39:05,256] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step60000 is ready now! 7: [2023-03-17 12:39:05,256] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step60000/bf16_zero_pp_rank_61_mp_rank_00_optim_states.pt 7: [2023-03-17 12:39:05,256] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step60000 is ready now! 0: [2023-03-17 12:39:05,256] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step60000/bf16_zero_pp_rank_1_mp_rank_00_optim_states.pt. 0: [2023-03-17 12:39:05,256] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step60000/bf16_zero_pp_rank_1_mp_rank_00_optim_states.pt 0: [2023-03-17 12:39:05,256] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step60000 is ready now! 6: [2023-03-17 12:39:05,256] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step60000/bf16_zero_pp_rank_52_mp_rank_00_optim_states.pt. 6: [2023-03-17 12:39:05,256] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step60000/bf16_zero_pp_rank_52_mp_rank_00_optim_states.pt 6: [2023-03-17 12:39:05,256] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step60000 is ready now! 1: [2023-03-17 12:39:05,256] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step60000/bf16_zero_pp_rank_15_mp_rank_00_optim_states.pt. 1: [2023-03-17 12:39:05,256] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step60000/bf16_zero_pp_rank_15_mp_rank_00_optim_states.pt 1: [2023-03-17 12:39:05,256] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step60000 is ready now! 7: [2023-03-17 12:39:05,257] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step60000/bf16_zero_pp_rank_62_mp_rank_00_optim_states.pt. 5: [2023-03-17 12:39:05,257] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step60000/bf16_zero_pp_rank_46_mp_rank_00_optim_states.pt. 5: [2023-03-17 12:39:05,257] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step60000/bf16_zero_pp_rank_40_mp_rank_00_optim_states.pt. 5: [2023-03-17 12:39:05,257] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step60000/bf16_zero_pp_rank_45_mp_rank_00_optim_states.pt. 7: [2023-03-17 12:39:05,257] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step60000/bf16_zero_pp_rank_62_mp_rank_00_optim_states.pt 7: [2023-03-17 12:39:05,257] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step60000 is ready now! 5: [2023-03-17 12:39:05,257] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step60000/bf16_zero_pp_rank_40_mp_rank_00_optim_states.pt 5: [2023-03-17 12:39:05,257] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step60000/bf16_zero_pp_rank_46_mp_rank_00_optim_states.pt 5: [2023-03-17 12:39:05,257] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step60000/bf16_zero_pp_rank_45_mp_rank_00_optim_states.pt 5: [2023-03-17 12:39:05,257] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step60000/bf16_zero_pp_rank_47_mp_rank_00_optim_states.pt. 5: [2023-03-17 12:39:05,257] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step60000 is ready now! 5: [2023-03-17 12:39:05,257] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step60000 is ready now! 5: [2023-03-17 12:39:05,257] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step60000 is ready now! 5: [2023-03-17 12:39:05,257] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step60000/bf16_zero_pp_rank_47_mp_rank_00_optim_states.pt 5: [2023-03-17 12:39:05,257] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step60000 is ready now! 1: [2023-03-17 12:39:05,257] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step60000/bf16_zero_pp_rank_10_mp_rank_00_optim_states.pt. 4: [2023-03-17 12:39:05,257] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step60000/bf16_zero_pp_rank_32_mp_rank_00_optim_states.pt. 1: [2023-03-17 12:39:05,257] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step60000/bf16_zero_pp_rank_10_mp_rank_00_optim_states.pt 4: [2023-03-17 12:39:05,257] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step60000/bf16_zero_pp_rank_32_mp_rank_00_optim_states.pt 1: [2023-03-17 12:39:05,257] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step60000 is ready now! 4: [2023-03-17 12:39:05,257] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step60000 is ready now! 0: [2023-03-17 12:39:05,257] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step60000/bf16_zero_pp_rank_2_mp_rank_00_optim_states.pt. 6: [2023-03-17 12:39:05,257] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step60000/bf16_zero_pp_rank_51_mp_rank_00_optim_states.pt. 6: [2023-03-17 12:39:05,257] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step60000/bf16_zero_pp_rank_51_mp_rank_00_optim_states.pt 0: [2023-03-17 12:39:05,257] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step60000/bf16_zero_pp_rank_2_mp_rank_00_optim_states.pt 0: [2023-03-17 12:39:05,257] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step60000 is ready now! 6: [2023-03-17 12:39:05,257] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step60000 is ready now! 7: [2023-03-17 12:39:05,257] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step60000/bf16_zero_pp_rank_56_mp_rank_00_optim_states.pt. 7: [2023-03-17 12:39:05,258] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step60000/bf16_zero_pp_rank_56_mp_rank_00_optim_states.pt 7: [2023-03-17 12:39:05,258] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step60000 is ready now! 5: [2023-03-17 12:39:05,258] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step60000/bf16_zero_pp_rank_41_mp_rank_00_optim_states.pt. 2: [2023-03-17 12:39:05,258] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step60000/bf16_zero_pp_rank_19_mp_rank_00_optim_states.pt. 5: [2023-03-17 12:39:05,258] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step60000/bf16_zero_pp_rank_41_mp_rank_00_optim_states.pt 5: [2023-03-17 12:39:05,258] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step60000 is ready now! 2: [2023-03-17 12:39:05,258] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step60000/bf16_zero_pp_rank_22_mp_rank_00_optim_states.pt. 2: [2023-03-17 12:39:05,258] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step60000/bf16_zero_pp_rank_19_mp_rank_00_optim_states.pt 2: [2023-03-17 12:39:05,258] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step60000/bf16_zero_pp_rank_22_mp_rank_00_optim_states.pt 2: [2023-03-17 12:39:05,258] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step60000 is ready now! 2: [2023-03-17 12:39:05,258] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step60000 is ready now! 1: [2023-03-17 12:39:05,258] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step60000/bf16_zero_pp_rank_14_mp_rank_00_optim_states.pt. 1: [2023-03-17 12:39:05,258] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step60000/bf16_zero_pp_rank_14_mp_rank_00_optim_states.pt 1: [2023-03-17 12:39:05,258] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step60000 is ready now! 2: [2023-03-17 12:39:05,258] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step60000/bf16_zero_pp_rank_20_mp_rank_00_optim_states.pt. 2: [2023-03-17 12:39:05,258] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step60000/bf16_zero_pp_rank_20_mp_rank_00_optim_states.pt 0: [2023-03-17 12:39:05,258] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step60000/bf16_zero_pp_rank_3_mp_rank_00_optim_states.pt. 2: [2023-03-17 12:39:05,258] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step60000 is ready now! 0: [2023-03-17 12:39:05,258] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step60000/bf16_zero_pp_rank_3_mp_rank_00_optim_states.pt 0: [2023-03-17 12:39:05,259] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step60000 is ready now! 4: [2023-03-17 12:39:05,259] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step60000/bf16_zero_pp_rank_39_mp_rank_00_optim_states.pt. 4: [2023-03-17 12:39:05,259] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step60000/bf16_zero_pp_rank_39_mp_rank_00_optim_states.pt 2: [2023-03-17 12:39:05,259] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step60000/bf16_zero_pp_rank_23_mp_rank_00_optim_states.pt. 4: [2023-03-17 12:39:05,259] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step60000 is ready now! 6: [2023-03-17 12:39:05,259] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step60000/bf16_zero_pp_rank_54_mp_rank_00_optim_states.pt. 3: [2023-03-17 12:39:05,259] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step60000/bf16_zero_pp_rank_25_mp_rank_00_optim_states.pt. 3: [2023-03-17 12:39:05,259] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step60000/bf16_zero_pp_rank_25_mp_rank_00_optim_states.pt 2: [2023-03-17 12:39:05,259] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step60000/bf16_zero_pp_rank_23_mp_rank_00_optim_states.pt 6: [2023-03-17 12:39:05,259] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step60000/bf16_zero_pp_rank_54_mp_rank_00_optim_states.pt 3: [2023-03-17 12:39:05,259] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step60000/bf16_zero_pp_rank_29_mp_rank_00_optim_states.pt. 7: [2023-03-17 12:39:05,259] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step60000/bf16_zero_pp_rank_58_mp_rank_00_optim_states.pt. 3: [2023-03-17 12:39:05,259] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step60000 is ready now! 2: [2023-03-17 12:39:05,259] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step60000 is ready now! 6: [2023-03-17 12:39:05,259] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step60000 is ready now! 3: [2023-03-17 12:39:05,259] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step60000/bf16_zero_pp_rank_29_mp_rank_00_optim_states.pt 7: [2023-03-17 12:39:05,259] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step60000/bf16_zero_pp_rank_58_mp_rank_00_optim_states.pt 3: [2023-03-17 12:39:05,259] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step60000 is ready now! 3: [2023-03-17 12:39:05,259] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step60000/bf16_zero_pp_rank_24_mp_rank_00_optim_states.pt. 7: [2023-03-17 12:39:05,259] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step60000 is ready now! 3: [2023-03-17 12:39:05,259] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step60000/bf16_zero_pp_rank_26_mp_rank_00_optim_states.pt. 3: [2023-03-17 12:39:05,259] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step60000/bf16_zero_pp_rank_24_mp_rank_00_optim_states.pt 3: [2023-03-17 12:39:05,259] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step60000/bf16_zero_pp_rank_26_mp_rank_00_optim_states.pt 3: [2023-03-17 12:39:05,259] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step60000 is ready now! 3: [2023-03-17 12:39:05,259] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step60000 is ready now! 3: [2023-03-17 12:39:05,259] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step60000/bf16_zero_pp_rank_27_mp_rank_00_optim_states.pt. 3: [2023-03-17 12:39:05,259] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step60000/bf16_zero_pp_rank_27_mp_rank_00_optim_states.pt 5: [2023-03-17 12:39:05,259] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step60000/bf16_zero_pp_rank_42_mp_rank_00_optim_states.pt. 3: [2023-03-17 12:39:05,259] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step60000 is ready now! 5: [2023-03-17 12:39:05,259] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step60000/bf16_zero_pp_rank_42_mp_rank_00_optim_states.pt 5: [2023-03-17 12:39:05,259] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step60000 is ready now! 2: [2023-03-17 12:39:05,259] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step60000/bf16_zero_pp_rank_17_mp_rank_00_optim_states.pt. 2: [2023-03-17 12:39:05,259] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step60000/bf16_zero_pp_rank_17_mp_rank_00_optim_states.pt 2: [2023-03-17 12:39:05,259] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step60000 is ready now! 1: [2023-03-17 12:39:05,260] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step60000/bf16_zero_pp_rank_9_mp_rank_00_optim_states.pt. 4: [2023-03-17 12:39:05,260] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step60000/bf16_zero_pp_rank_36_mp_rank_00_optim_states.pt. 1: [2023-03-17 12:39:05,260] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step60000/bf16_zero_pp_rank_9_mp_rank_00_optim_states.pt 4: [2023-03-17 12:39:05,260] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step60000/bf16_zero_pp_rank_36_mp_rank_00_optim_states.pt 1: [2023-03-17 12:39:05,260] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step60000 is ready now! 6: [2023-03-17 12:39:05,260] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step60000/bf16_zero_pp_rank_53_mp_rank_00_optim_states.pt. 4: [2023-03-17 12:39:05,260] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step60000 is ready now! 6: [2023-03-17 12:39:05,260] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step60000/bf16_zero_pp_rank_53_mp_rank_00_optim_states.pt 0: [2023-03-17 12:39:05,260] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step60000/bf16_zero_pp_rank_7_mp_rank_00_optim_states.pt. 6: [2023-03-17 12:39:05,260] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step60000 is ready now! 0: [2023-03-17 12:39:05,260] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step60000/bf16_zero_pp_rank_7_mp_rank_00_optim_states.pt 2: [2023-03-17 12:39:05,260] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step60000/bf16_zero_pp_rank_16_mp_rank_00_optim_states.pt. 0: [2023-03-17 12:39:05,260] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step60000 is ready now! 2: [2023-03-17 12:39:05,260] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step60000/bf16_zero_pp_rank_16_mp_rank_00_optim_states.pt 2: [2023-03-17 12:39:05,260] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step60000 is ready now! 6: [2023-03-17 12:39:05,260] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step60000/bf16_zero_pp_rank_50_mp_rank_00_optim_states.pt. 6: [2023-03-17 12:39:05,260] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step60000/bf16_zero_pp_rank_50_mp_rank_00_optim_states.pt 6: [2023-03-17 12:39:05,260] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step60000 is ready now! 3: [2023-03-17 12:39:05,260] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step60000/bf16_zero_pp_rank_31_mp_rank_00_optim_states.pt. 3: [2023-03-17 12:39:05,260] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step60000/bf16_zero_pp_rank_31_mp_rank_00_optim_states.pt 7: [2023-03-17 12:39:05,260] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step60000/bf16_zero_pp_rank_57_mp_rank_00_optim_states.pt. 3: [2023-03-17 12:39:05,260] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step60000 is ready now! 5: [2023-03-17 12:39:05,260] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step60000/bf16_zero_pp_rank_43_mp_rank_00_optim_states.pt. 7: [2023-03-17 12:39:05,261] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step60000/bf16_zero_pp_rank_57_mp_rank_00_optim_states.pt 7: [2023-03-17 12:39:05,261] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step60000 is ready now! 5: [2023-03-17 12:39:05,261] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step60000/bf16_zero_pp_rank_43_mp_rank_00_optim_states.pt 5: [2023-03-17 12:39:05,261] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step60000 is ready now! 2: [2023-03-17 12:39:05,261] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step60000/bf16_zero_pp_rank_21_mp_rank_00_optim_states.pt. 4: [2023-03-17 12:39:05,261] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step60000/bf16_zero_pp_rank_33_mp_rank_00_optim_states.pt. 5: [2023-03-17 12:39:05,261] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step60000/bf16_zero_pp_rank_44_mp_rank_00_optim_states.pt. 2: [2023-03-17 12:39:05,261] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step60000/bf16_zero_pp_rank_21_mp_rank_00_optim_states.pt 4: [2023-03-17 12:39:05,261] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step60000/bf16_zero_pp_rank_33_mp_rank_00_optim_states.pt 2: [2023-03-17 12:39:05,261] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step60000 is ready now! 4: [2023-03-17 12:39:05,261] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step60000 is ready now! 5: [2023-03-17 12:39:05,261] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step60000/bf16_zero_pp_rank_44_mp_rank_00_optim_states.pt 5: [2023-03-17 12:39:05,261] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step60000 is ready now! 3: [2023-03-17 12:39:05,261] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step60000/bf16_zero_pp_rank_28_mp_rank_00_optim_states.pt. 3: [2023-03-17 12:39:05,261] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step60000/bf16_zero_pp_rank_28_mp_rank_00_optim_states.pt 3: [2023-03-17 12:39:05,261] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step60000/bf16_zero_pp_rank_30_mp_rank_00_optim_states.pt. 3: [2023-03-17 12:39:05,261] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step60000 is ready now! 3: [2023-03-17 12:39:05,261] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step60000/bf16_zero_pp_rank_30_mp_rank_00_optim_states.pt 1: [2023-03-17 12:39:05,261] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step60000/bf16_zero_pp_rank_12_mp_rank_00_optim_states.pt. 3: [2023-03-17 12:39:05,261] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step60000 is ready now! 4: [2023-03-17 12:39:05,261] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step60000/bf16_zero_pp_rank_35_mp_rank_00_optim_states.pt. 1: [2023-03-17 12:39:05,261] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step60000/bf16_zero_pp_rank_12_mp_rank_00_optim_states.pt 1: [2023-03-17 12:39:05,261] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step60000 is ready now! 4: [2023-03-17 12:39:05,261] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step60000/bf16_zero_pp_rank_35_mp_rank_00_optim_states.pt 4: [2023-03-17 12:39:05,261] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step60000 is ready now! 7: [2023-03-17 12:39:05,261] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step60000/bf16_zero_pp_rank_60_mp_rank_00_optim_states.pt. 7: [2023-03-17 12:39:05,261] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step60000/bf16_zero_pp_rank_60_mp_rank_00_optim_states.pt 7: [2023-03-17 12:39:05,261] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step60000 is ready now! 2: [2023-03-17 12:39:05,262] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step60000/bf16_zero_pp_rank_18_mp_rank_00_optim_states.pt. 2: [2023-03-17 12:39:05,262] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step60000/bf16_zero_pp_rank_18_mp_rank_00_optim_states.pt 2: [2023-03-17 12:39:05,262] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step60000 is ready now! 0: successfully saved checkpoint at iteration 60000 to checkpoints_44m174b100m 7: time (ms) | save-checkpoint: 238.60 7: iteration 60100/ 331103 | consumed samples: 15385600 | consumed tokens: 31509708800 | elapsed time per iteration (s): 0.16 | learning rate: 1.870E-04 | global batch size: 256 | lm loss: 3.751571E+00 | grad norm: 0.301 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1601.750 | TFLOPs: 25.12 | 7: iteration 60200/ 331103 | consumed samples: 15411200 | consumed tokens: 31562137600 | elapsed time per iteration (s): 0.16 | learning rate: 1.870E-04 | global batch size: 256 | lm loss: 3.748264E+00 | grad norm: 0.321 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1633.125 | TFLOPs: 25.61 | 7: iteration 60300/ 331103 | consumed samples: 15436800 | consumed tokens: 31614566400 | elapsed time per iteration (s): 0.16 | learning rate: 1.869E-04 | global batch size: 256 | lm loss: 3.748080E+00 | grad norm: 0.321 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1610.412 | TFLOPs: 25.26 | 7: iteration 60400/ 331103 | consumed samples: 15462400 | consumed tokens: 31666995200 | elapsed time per iteration (s): 0.16 | learning rate: 1.869E-04 | global batch size: 256 | lm loss: 3.747017E+00 | grad norm: 0.323 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1621.475 | TFLOPs: 25.43 | 7: iteration 60500/ 331103 | consumed samples: 15488000 | consumed tokens: 31719424000 | elapsed time per iteration (s): 0.16 | learning rate: 1.868E-04 | global batch size: 256 | lm loss: 3.743781E+00 | grad norm: 0.306 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1621.674 | TFLOPs: 25.43 | 7: iteration 60600/ 331103 | consumed samples: 15513600 | consumed tokens: 31771852800 | elapsed time per iteration (s): 0.16 | learning rate: 1.868E-04 | global batch size: 256 | lm loss: 3.748738E+00 | grad norm: 0.345 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1618.495 | TFLOPs: 25.38 | 7: iteration 60700/ 331103 | consumed samples: 15539200 | consumed tokens: 31824281600 | elapsed time per iteration (s): 0.16 | learning rate: 1.867E-04 | global batch size: 256 | lm loss: 3.746914E+00 | grad norm: 0.326 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1635.306 | TFLOPs: 25.65 | 7: iteration 60800/ 331103 | consumed samples: 15564800 | consumed tokens: 31876710400 | elapsed time per iteration (s): 0.16 | learning rate: 1.867E-04 | global batch size: 256 | lm loss: 3.746856E+00 | grad norm: 0.308 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1632.066 | TFLOPs: 25.59 | 7: iteration 60900/ 331103 | consumed samples: 15590400 | consumed tokens: 31929139200 | elapsed time per iteration (s): 0.15 | learning rate: 1.866E-04 | global batch size: 256 | lm loss: 3.744678E+00 | grad norm: 0.318 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1658.013 | TFLOPs: 26.00 | 7: iteration 61000/ 331103 | consumed samples: 15616000 | consumed tokens: 31981568000 | elapsed time per iteration (s): 0.16 | learning rate: 1.866E-04 | global batch size: 256 | lm loss: 3.746567E+00 | grad norm: 0.369 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1636.628 | TFLOPs: 25.67 | 7: iteration 61100/ 331103 | consumed samples: 15641600 | consumed tokens: 32033996800 | elapsed time per iteration (s): 0.15 | learning rate: 1.865E-04 | global batch size: 256 | lm loss: 3.747996E+00 | grad norm: 0.313 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1659.564 | TFLOPs: 26.03 | 7: iteration 61200/ 331103 | consumed samples: 15667200 | consumed tokens: 32086425600 | elapsed time per iteration (s): 0.16 | learning rate: 1.865E-04 | global batch size: 256 | lm loss: 3.747166E+00 | grad norm: 0.295 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1645.650 | TFLOPs: 25.81 | 7: iteration 61300/ 331103 | consumed samples: 15692800 | consumed tokens: 32138854400 | elapsed time per iteration (s): 0.16 | learning rate: 1.865E-04 | global batch size: 256 | lm loss: 3.744050E+00 | grad norm: 0.314 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1643.370 | TFLOPs: 25.77 | 7: iteration 61400/ 331103 | consumed samples: 15718400 | consumed tokens: 32191283200 | elapsed time per iteration (s): 0.16 | learning rate: 1.864E-04 | global batch size: 256 | lm loss: 3.740903E+00 | grad norm: 0.297 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1648.211 | TFLOPs: 25.85 | 7: iteration 61500/ 331103 | consumed samples: 15744000 | consumed tokens: 32243712000 | elapsed time per iteration (s): 0.16 | learning rate: 1.864E-04 | global batch size: 256 | lm loss: 3.743353E+00 | grad norm: 0.317 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1649.693 | TFLOPs: 25.87 | 7: iteration 61600/ 331103 | consumed samples: 15769600 | consumed tokens: 32296140800 | elapsed time per iteration (s): 0.16 | learning rate: 1.863E-04 | global batch size: 256 | lm loss: 3.744095E+00 | grad norm: 0.294 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1644.342 | TFLOPs: 25.79 | 7: iteration 61700/ 331103 | consumed samples: 15795200 | consumed tokens: 32348569600 | elapsed time per iteration (s): 0.15 | learning rate: 1.863E-04 | global batch size: 256 | lm loss: 3.743158E+00 | grad norm: 0.312 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1653.967 | TFLOPs: 25.94 | 7: iteration 61800/ 331103 | consumed samples: 15820800 | consumed tokens: 32400998400 | elapsed time per iteration (s): 0.16 | learning rate: 1.862E-04 | global batch size: 256 | lm loss: 3.740496E+00 | grad norm: 0.349 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1605.549 | TFLOPs: 25.18 | 7: iteration 61900/ 331103 | consumed samples: 15846400 | consumed tokens: 32453427200 | elapsed time per iteration (s): 0.16 | learning rate: 1.862E-04 | global batch size: 256 | lm loss: 3.746513E+00 | grad norm: 0.312 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1613.185 | TFLOPs: 25.30 | 0: [2023-03-17 12:44:18,683] [INFO] [logging.py:68:log_dist] [Rank 0] step=62000, skipped=0, lr=[0.00018613416504726404, 0.00018613416504726404, 0.00018613416504726404], mom=[(0.9, 0.999), (0.9, 0.999), (0.9, 0.999)] 7: iteration 62000/ 331103 | consumed samples: 15872000 | consumed tokens: 32505856000 | elapsed time per iteration (s): 0.16 | learning rate: 1.861E-04 | global batch size: 256 | lm loss: 3.745346E+00 | grad norm: 0.336 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1611.171 | TFLOPs: 25.27 | 0: steps: 62000 loss: 3.7384 iter time (s): 0.156 samples/sec: 1642.935 7: iteration 62100/ 331103 | consumed samples: 15897600 | consumed tokens: 32558284800 | elapsed time per iteration (s): 0.16 | learning rate: 1.861E-04 | global batch size: 256 | lm loss: 3.743319E+00 | grad norm: 0.348 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1627.123 | TFLOPs: 25.52 | 7: iteration 62200/ 331103 | consumed samples: 15923200 | consumed tokens: 32610713600 | elapsed time per iteration (s): 0.16 | learning rate: 1.860E-04 | global batch size: 256 | lm loss: 3.738315E+00 | grad norm: 0.296 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1607.971 | TFLOPs: 25.22 | 7: iteration 62300/ 331103 | consumed samples: 15948800 | consumed tokens: 32663142400 | elapsed time per iteration (s): 0.16 | learning rate: 1.860E-04 | global batch size: 256 | lm loss: 3.745619E+00 | grad norm: 0.293 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1606.972 | TFLOPs: 25.20 | 7: iteration 62400/ 331103 | consumed samples: 15974400 | consumed tokens: 32715571200 | elapsed time per iteration (s): 0.17 | learning rate: 1.859E-04 | global batch size: 256 | lm loss: 3.744590E+00 | grad norm: 0.363 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1533.783 | TFLOPs: 24.05 | 7: iteration 62500/ 331103 | consumed samples: 16000000 | consumed tokens: 32768000000 | elapsed time per iteration (s): 0.16 | learning rate: 1.859E-04 | global batch size: 256 | lm loss: 3.743530E+00 | grad norm: 0.295 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1638.006 | TFLOPs: 25.69 | 7: iteration 62600/ 331103 | consumed samples: 16025600 | consumed tokens: 32820428800 | elapsed time per iteration (s): 0.16 | learning rate: 1.859E-04 | global batch size: 256 | lm loss: 3.743231E+00 | grad norm: 0.322 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1613.055 | TFLOPs: 25.30 | 7: iteration 62700/ 331103 | consumed samples: 16051200 | consumed tokens: 32872857600 | elapsed time per iteration (s): 0.16 | learning rate: 1.858E-04 | global batch size: 256 | lm loss: 3.740670E+00 | grad norm: 0.301 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1633.495 | TFLOPs: 25.62 | 7: iteration 62800/ 331103 | consumed samples: 16076800 | consumed tokens: 32925286400 | elapsed time per iteration (s): 0.16 | learning rate: 1.858E-04 | global batch size: 256 | lm loss: 3.739045E+00 | grad norm: 0.297 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1631.746 | TFLOPs: 25.59 | 7: iteration 62900/ 331103 | consumed samples: 16102400 | consumed tokens: 32977715200 | elapsed time per iteration (s): 0.16 | learning rate: 1.857E-04 | global batch size: 256 | lm loss: 3.742017E+00 | grad norm: 0.301 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1618.652 | TFLOPs: 25.38 | 7: iteration 63000/ 331103 | consumed samples: 16128000 | consumed tokens: 33030144000 | elapsed time per iteration (s): 0.16 | learning rate: 1.857E-04 | global batch size: 256 | lm loss: 3.737035E+00 | grad norm: 0.340 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1639.436 | TFLOPs: 25.71 | 7: iteration 63100/ 331103 | consumed samples: 16153600 | consumed tokens: 33082572800 | elapsed time per iteration (s): 0.16 | learning rate: 1.856E-04 | global batch size: 256 | lm loss: 3.738165E+00 | grad norm: 0.298 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1607.878 | TFLOPs: 25.22 | 7: iteration 63200/ 331103 | consumed samples: 16179200 | consumed tokens: 33135001600 | elapsed time per iteration (s): 0.16 | learning rate: 1.856E-04 | global batch size: 256 | lm loss: 3.741089E+00 | grad norm: 0.330 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1621.846 | TFLOPs: 25.43 | 7: iteration 63300/ 331103 | consumed samples: 16204800 | consumed tokens: 33187430400 | elapsed time per iteration (s): 0.16 | learning rate: 1.855E-04 | global batch size: 256 | lm loss: 3.744488E+00 | grad norm: 0.327 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1638.127 | TFLOPs: 25.69 | 7: iteration 63400/ 331103 | consumed samples: 16230400 | consumed tokens: 33239859200 | elapsed time per iteration (s): 0.16 | learning rate: 1.855E-04 | global batch size: 256 | lm loss: 3.745763E+00 | grad norm: 0.342 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1628.146 | TFLOPs: 25.53 | 7: iteration 63500/ 331103 | consumed samples: 16256000 | consumed tokens: 33292288000 | elapsed time per iteration (s): 0.16 | learning rate: 1.854E-04 | global batch size: 256 | lm loss: 3.745801E+00 | grad norm: 0.316 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1632.059 | TFLOPs: 25.59 | 7: iteration 63600/ 331103 | consumed samples: 16281600 | consumed tokens: 33344716800 | elapsed time per iteration (s): 0.16 | learning rate: 1.854E-04 | global batch size: 256 | lm loss: 3.742593E+00 | grad norm: 0.320 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1644.139 | TFLOPs: 25.78 | 7: iteration 63700/ 331103 | consumed samples: 16307200 | consumed tokens: 33397145600 | elapsed time per iteration (s): 0.16 | learning rate: 1.853E-04 | global batch size: 256 | lm loss: 3.742950E+00 | grad norm: 0.307 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1644.389 | TFLOPs: 25.79 | 7: iteration 63800/ 331103 | consumed samples: 16332800 | consumed tokens: 33449574400 | elapsed time per iteration (s): 0.16 | learning rate: 1.853E-04 | global batch size: 256 | lm loss: 3.742401E+00 | grad norm: 0.340 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1643.095 | TFLOPs: 25.77 | 7: iteration 63900/ 331103 | consumed samples: 16358400 | consumed tokens: 33502003200 | elapsed time per iteration (s): 0.16 | learning rate: 1.852E-04 | global batch size: 256 | lm loss: 3.741279E+00 | grad norm: 0.328 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1637.211 | TFLOPs: 25.68 | 0: [2023-03-17 12:49:34,015] [INFO] [logging.py:68:log_dist] [Rank 0] step=64000, skipped=0, lr=[0.00018520024522981825, 0.00018520024522981825, 0.00018520024522981825], mom=[(0.9, 0.999), (0.9, 0.999), (0.9, 0.999)] 7: iteration 64000/ 331103 | consumed samples: 16384000 | consumed tokens: 33554432000 | elapsed time per iteration (s): 0.16 | learning rate: 1.852E-04 | global batch size: 256 | lm loss: 3.741445E+00 | grad norm: 0.334 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1633.919 | TFLOPs: 25.62 | 0: steps: 64000 loss: 3.7403 iter time (s): 0.157 samples/sec: 1632.834 7: iteration 64100/ 331103 | consumed samples: 16409600 | consumed tokens: 33606860800 | elapsed time per iteration (s): 0.16 | learning rate: 1.852E-04 | global batch size: 256 | lm loss: 3.739183E+00 | grad norm: 0.340 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1630.679 | TFLOPs: 25.57 | 7: iteration 64200/ 331103 | consumed samples: 16435200 | consumed tokens: 33659289600 | elapsed time per iteration (s): 0.16 | learning rate: 1.851E-04 | global batch size: 256 | lm loss: 3.739704E+00 | grad norm: 0.348 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1617.490 | TFLOPs: 25.37 | 7: iteration 64300/ 331103 | consumed samples: 16460800 | consumed tokens: 33711718400 | elapsed time per iteration (s): 0.16 | learning rate: 1.851E-04 | global batch size: 256 | lm loss: 3.740244E+00 | grad norm: 0.312 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1638.729 | TFLOPs: 25.70 | 7: iteration 64400/ 331103 | consumed samples: 16486400 | consumed tokens: 33764147200 | elapsed time per iteration (s): 0.16 | learning rate: 1.850E-04 | global batch size: 256 | lm loss: 3.745016E+00 | grad norm: 0.302 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1636.488 | TFLOPs: 25.66 | 7: iteration 64500/ 331103 | consumed samples: 16512000 | consumed tokens: 33816576000 | elapsed time per iteration (s): 0.16 | learning rate: 1.850E-04 | global batch size: 256 | lm loss: 3.744743E+00 | grad norm: 0.301 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1574.427 | TFLOPs: 24.69 | 7: iteration 64600/ 331103 | consumed samples: 16537600 | consumed tokens: 33869004800 | elapsed time per iteration (s): 0.16 | learning rate: 1.849E-04 | global batch size: 256 | lm loss: 3.740063E+00 | grad norm: 0.379 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1619.350 | TFLOPs: 25.40 | 7: iteration 64700/ 331103 | consumed samples: 16563200 | consumed tokens: 33921433600 | elapsed time per iteration (s): 0.16 | learning rate: 1.849E-04 | global batch size: 256 | lm loss: 3.739382E+00 | grad norm: 0.311 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1639.153 | TFLOPs: 25.71 | 7: iteration 64800/ 331103 | consumed samples: 16588800 | consumed tokens: 33973862400 | elapsed time per iteration (s): 0.16 | learning rate: 1.848E-04 | global batch size: 256 | lm loss: 3.739363E+00 | grad norm: 0.310 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1601.730 | TFLOPs: 25.12 | 7: iteration 64900/ 331103 | consumed samples: 16614400 | consumed tokens: 34026291200 | elapsed time per iteration (s): 0.16 | learning rate: 1.848E-04 | global batch size: 256 | lm loss: 3.739000E+00 | grad norm: 0.292 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1644.775 | TFLOPs: 25.79 | 7: iteration 65000/ 331103 | consumed samples: 16640000 | consumed tokens: 34078720000 | elapsed time per iteration (s): 0.16 | learning rate: 1.847E-04 | global batch size: 256 | lm loss: 3.735702E+00 | grad norm: 0.331 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1632.133 | TFLOPs: 25.60 | 7: iteration 65100/ 331103 | consumed samples: 16665600 | consumed tokens: 34131148800 | elapsed time per iteration (s): 0.16 | learning rate: 1.847E-04 | global batch size: 256 | lm loss: 3.740927E+00 | grad norm: 0.369 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1622.672 | TFLOPs: 25.45 | 7: iteration 65200/ 331103 | consumed samples: 16691200 | consumed tokens: 34183577600 | elapsed time per iteration (s): 0.16 | learning rate: 1.846E-04 | global batch size: 256 | lm loss: 3.742384E+00 | grad norm: 0.303 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1584.941 | TFLOPs: 24.86 | 7: iteration 65300/ 331103 | consumed samples: 16716800 | consumed tokens: 34236006400 | elapsed time per iteration (s): 0.16 | learning rate: 1.846E-04 | global batch size: 256 | lm loss: 3.738287E+00 | grad norm: 0.283 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1614.118 | TFLOPs: 25.31 | 7: iteration 65400/ 331103 | consumed samples: 16742400 | consumed tokens: 34288435200 | elapsed time per iteration (s): 0.16 | learning rate: 1.845E-04 | global batch size: 256 | lm loss: 3.742853E+00 | grad norm: 0.318 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1618.982 | TFLOPs: 25.39 | 7: iteration 65500/ 331103 | consumed samples: 16768000 | consumed tokens: 34340864000 | elapsed time per iteration (s): 0.16 | learning rate: 1.845E-04 | global batch size: 256 | lm loss: 3.738067E+00 | grad norm: 0.325 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1623.217 | TFLOPs: 25.46 | 7: iteration 65600/ 331103 | consumed samples: 16793600 | consumed tokens: 34393292800 | elapsed time per iteration (s): 0.16 | learning rate: 1.844E-04 | global batch size: 256 | lm loss: 3.740530E+00 | grad norm: 0.305 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1632.186 | TFLOPs: 25.60 | 7: iteration 65700/ 331103 | consumed samples: 16819200 | consumed tokens: 34445721600 | elapsed time per iteration (s): 0.16 | learning rate: 1.844E-04 | global batch size: 256 | lm loss: 3.734239E+00 | grad norm: 0.328 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1643.022 | TFLOPs: 25.77 | 7: iteration 65800/ 331103 | consumed samples: 16844800 | consumed tokens: 34498150400 | elapsed time per iteration (s): 0.16 | learning rate: 1.843E-04 | global batch size: 256 | lm loss: 3.734785E+00 | grad norm: 0.306 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1640.339 | TFLOPs: 25.72 | 7: iteration 65900/ 331103 | consumed samples: 16870400 | consumed tokens: 34550579200 | elapsed time per iteration (s): 0.16 | learning rate: 1.843E-04 | global batch size: 256 | lm loss: 3.735208E+00 | grad norm: 0.310 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1634.200 | TFLOPs: 25.63 | 0: [2023-03-17 12:54:49,189] [INFO] [logging.py:68:log_dist] [Rank 0] step=66000, skipped=0, lr=[0.000184238696282003, 0.000184238696282003, 0.000184238696282003], mom=[(0.9, 0.999), (0.9, 0.999), (0.9, 0.999)] 7: iteration 66000/ 331103 | consumed samples: 16896000 | consumed tokens: 34603008000 | elapsed time per iteration (s): 0.16 | learning rate: 1.842E-04 | global batch size: 256 | lm loss: 3.738531E+00 | grad norm: 0.340 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1645.848 | TFLOPs: 25.81 | 0: steps: 66000 loss: 3.7194 iter time (s): 0.157 samples/sec: 1633.590 7: iteration 66100/ 331103 | consumed samples: 16921600 | consumed tokens: 34655436800 | elapsed time per iteration (s): 0.16 | learning rate: 1.842E-04 | global batch size: 256 | lm loss: 3.737396E+00 | grad norm: 0.298 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1629.386 | TFLOPs: 25.55 | 7: iteration 66200/ 331103 | consumed samples: 16947200 | consumed tokens: 34707865600 | elapsed time per iteration (s): 0.16 | learning rate: 1.841E-04 | global batch size: 256 | lm loss: 3.736670E+00 | grad norm: 0.294 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1632.861 | TFLOPs: 25.61 | 7: iteration 66300/ 331103 | consumed samples: 16972800 | consumed tokens: 34760294400 | elapsed time per iteration (s): 0.16 | learning rate: 1.841E-04 | global batch size: 256 | lm loss: 3.736246E+00 | grad norm: 0.330 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1640.045 | TFLOPs: 25.72 | 7: iteration 66400/ 331103 | consumed samples: 16998400 | consumed tokens: 34812723200 | elapsed time per iteration (s): 0.17 | learning rate: 1.840E-04 | global batch size: 256 | lm loss: 3.732681E+00 | grad norm: 0.312 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1509.478 | TFLOPs: 23.67 | 7: iteration 66500/ 331103 | consumed samples: 17024000 | consumed tokens: 34865152000 | elapsed time per iteration (s): 0.16 | learning rate: 1.840E-04 | global batch size: 256 | lm loss: 3.736069E+00 | grad norm: 0.372 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1632.149 | TFLOPs: 25.60 | 7: iteration 66600/ 331103 | consumed samples: 17049600 | consumed tokens: 34917580800 | elapsed time per iteration (s): 0.16 | learning rate: 1.839E-04 | global batch size: 256 | lm loss: 3.732558E+00 | grad norm: 0.399 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1608.507 | TFLOPs: 25.23 | 7: iteration 66700/ 331103 | consumed samples: 17075200 | consumed tokens: 34970009600 | elapsed time per iteration (s): 0.15 | learning rate: 1.839E-04 | global batch size: 256 | lm loss: 3.732531E+00 | grad norm: 0.317 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1657.059 | TFLOPs: 25.99 | 7: iteration 66800/ 331103 | consumed samples: 17100800 | consumed tokens: 35022438400 | elapsed time per iteration (s): 0.15 | learning rate: 1.838E-04 | global batch size: 256 | lm loss: 3.736628E+00 | grad norm: 0.321 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1660.415 | TFLOPs: 26.04 | 7: iteration 66900/ 331103 | consumed samples: 17126400 | consumed tokens: 35074867200 | elapsed time per iteration (s): 0.16 | learning rate: 1.838E-04 | global batch size: 256 | lm loss: 3.737112E+00 | grad norm: 0.329 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1645.596 | TFLOPs: 25.81 | 7: iteration 67000/ 331103 | consumed samples: 17152000 | consumed tokens: 35127296000 | elapsed time per iteration (s): 0.17 | learning rate: 1.837E-04 | global batch size: 256 | lm loss: 3.740139E+00 | grad norm: 0.330 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1498.152 | TFLOPs: 23.49 | 7: iteration 67100/ 331103 | consumed samples: 17177600 | consumed tokens: 35179724800 | elapsed time per iteration (s): 0.16 | learning rate: 1.837E-04 | global batch size: 256 | lm loss: 3.733953E+00 | grad norm: 0.305 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1638.379 | TFLOPs: 25.69 | 7: iteration 67200/ 331103 | consumed samples: 17203200 | consumed tokens: 35232153600 | elapsed time per iteration (s): 0.16 | learning rate: 1.836E-04 | global batch size: 256 | lm loss: 3.734289E+00 | grad norm: 0.300 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1634.431 | TFLOPs: 25.63 | 7: iteration 67300/ 331103 | consumed samples: 17228800 | consumed tokens: 35284582400 | elapsed time per iteration (s): 0.16 | learning rate: 1.836E-04 | global batch size: 256 | lm loss: 3.737652E+00 | grad norm: 0.285 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1629.083 | TFLOPs: 25.55 | 7: iteration 67400/ 331103 | consumed samples: 17254400 | consumed tokens: 35337011200 | elapsed time per iteration (s): 0.16 | learning rate: 1.835E-04 | global batch size: 256 | lm loss: 3.733696E+00 | grad norm: 0.337 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1639.662 | TFLOPs: 25.71 | 7: iteration 67500/ 331103 | consumed samples: 17280000 | consumed tokens: 35389440000 | elapsed time per iteration (s): 0.16 | learning rate: 1.835E-04 | global batch size: 256 | lm loss: 3.735668E+00 | grad norm: 0.297 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1638.129 | TFLOPs: 25.69 | 7: iteration 67600/ 331103 | consumed samples: 17305600 | consumed tokens: 35441868800 | elapsed time per iteration (s): 0.16 | learning rate: 1.834E-04 | global batch size: 256 | lm loss: 3.737586E+00 | grad norm: 0.290 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1597.970 | TFLOPs: 25.06 | 7: iteration 67700/ 331103 | consumed samples: 17331200 | consumed tokens: 35494297600 | elapsed time per iteration (s): 0.16 | learning rate: 1.834E-04 | global batch size: 256 | lm loss: 3.738094E+00 | grad norm: 0.319 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1640.799 | TFLOPs: 25.73 | 7: iteration 67800/ 331103 | consumed samples: 17356800 | consumed tokens: 35546726400 | elapsed time per iteration (s): 0.16 | learning rate: 1.833E-04 | global batch size: 256 | lm loss: 3.734102E+00 | grad norm: 0.279 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1645.611 | TFLOPs: 25.81 | 7: iteration 67900/ 331103 | consumed samples: 17382400 | consumed tokens: 35599155200 | elapsed time per iteration (s): 0.16 | learning rate: 1.833E-04 | global batch size: 256 | lm loss: 3.736590E+00 | grad norm: 0.300 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1628.821 | TFLOPs: 25.54 | 0: [2023-03-17 13:00:05,100] [INFO] [logging.py:68:log_dist] [Rank 0] step=68000, skipped=0, lr=[0.00018324987148406557, 0.00018324987148406557, 0.00018324987148406557], mom=[(0.9, 0.999), (0.9, 0.999), (0.9, 0.999)] 7: iteration 68000/ 331103 | consumed samples: 17408000 | consumed tokens: 35651584000 | elapsed time per iteration (s): 0.16 | learning rate: 1.832E-04 | global batch size: 256 | lm loss: 3.733132E+00 | grad norm: 0.327 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1630.311 | TFLOPs: 25.57 | 0: steps: 68000 loss: 3.7506 iter time (s): 0.157 samples/sec: 1629.719 7: iteration 68100/ 331103 | consumed samples: 17433600 | consumed tokens: 35704012800 | elapsed time per iteration (s): 0.16 | learning rate: 1.832E-04 | global batch size: 256 | lm loss: 3.734255E+00 | grad norm: 0.319 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1647.649 | TFLOPs: 25.84 | 7: iteration 68200/ 331103 | consumed samples: 17459200 | consumed tokens: 35756441600 | elapsed time per iteration (s): 0.16 | learning rate: 1.831E-04 | global batch size: 256 | lm loss: 3.734486E+00 | grad norm: 0.330 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1648.125 | TFLOPs: 25.85 | 7: iteration 68300/ 331103 | consumed samples: 17484800 | consumed tokens: 35808870400 | elapsed time per iteration (s): 0.16 | learning rate: 1.831E-04 | global batch size: 256 | lm loss: 3.727669E+00 | grad norm: 0.314 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1648.200 | TFLOPs: 25.85 | 7: iteration 68400/ 331103 | consumed samples: 17510400 | consumed tokens: 35861299200 | elapsed time per iteration (s): 0.16 | learning rate: 1.830E-04 | global batch size: 256 | lm loss: 3.736339E+00 | grad norm: 0.305 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1651.225 | TFLOPs: 25.90 | 7: iteration 68500/ 331103 | consumed samples: 17536000 | consumed tokens: 35913728000 | elapsed time per iteration (s): 0.16 | learning rate: 1.830E-04 | global batch size: 256 | lm loss: 3.731342E+00 | grad norm: 0.328 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1636.146 | TFLOPs: 25.66 | 7: iteration 68600/ 331103 | consumed samples: 17561600 | consumed tokens: 35966156800 | elapsed time per iteration (s): 0.16 | learning rate: 1.829E-04 | global batch size: 256 | lm loss: 3.735800E+00 | grad norm: 0.303 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1646.117 | TFLOPs: 25.82 | 7: iteration 68700/ 331103 | consumed samples: 17587200 | consumed tokens: 36018585600 | elapsed time per iteration (s): 0.15 | learning rate: 1.829E-04 | global batch size: 256 | lm loss: 3.733223E+00 | grad norm: 0.303 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1652.128 | TFLOPs: 25.91 | 7: iteration 68800/ 331103 | consumed samples: 17612800 | consumed tokens: 36071014400 | elapsed time per iteration (s): 0.16 | learning rate: 1.828E-04 | global batch size: 256 | lm loss: 3.730764E+00 | grad norm: 0.328 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1646.429 | TFLOPs: 25.82 | 7: iteration 68900/ 331103 | consumed samples: 17638400 | consumed tokens: 36123443200 | elapsed time per iteration (s): 0.16 | learning rate: 1.828E-04 | global batch size: 256 | lm loss: 3.733431E+00 | grad norm: 0.298 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1639.215 | TFLOPs: 25.71 | 7: iteration 69000/ 331103 | consumed samples: 17664000 | consumed tokens: 36175872000 | elapsed time per iteration (s): 0.15 | learning rate: 1.827E-04 | global batch size: 256 | lm loss: 3.735154E+00 | grad norm: 0.297 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1657.064 | TFLOPs: 25.99 | 7: iteration 69100/ 331103 | consumed samples: 17689600 | consumed tokens: 36228300800 | elapsed time per iteration (s): 0.16 | learning rate: 1.827E-04 | global batch size: 256 | lm loss: 3.731518E+00 | grad norm: 0.310 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1649.120 | TFLOPs: 25.86 | 7: iteration 69200/ 331103 | consumed samples: 17715200 | consumed tokens: 36280729600 | elapsed time per iteration (s): 0.16 | learning rate: 1.826E-04 | global batch size: 256 | lm loss: 3.730866E+00 | grad norm: 0.350 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1641.502 | TFLOPs: 25.74 | 7: iteration 69300/ 331103 | consumed samples: 17740800 | consumed tokens: 36333158400 | elapsed time per iteration (s): 0.16 | learning rate: 1.826E-04 | global batch size: 256 | lm loss: 3.734276E+00 | grad norm: 0.301 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1637.599 | TFLOPs: 25.68 | 7: iteration 69400/ 331103 | consumed samples: 17766400 | consumed tokens: 36385587200 | elapsed time per iteration (s): 0.16 | learning rate: 1.825E-04 | global batch size: 256 | lm loss: 3.731709E+00 | grad norm: 0.310 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1643.944 | TFLOPs: 25.78 | 7: iteration 69500/ 331103 | consumed samples: 17792000 | consumed tokens: 36438016000 | elapsed time per iteration (s): 0.16 | learning rate: 1.825E-04 | global batch size: 256 | lm loss: 3.731118E+00 | grad norm: 0.323 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1648.398 | TFLOPs: 25.85 | 7: iteration 69600/ 331103 | consumed samples: 17817600 | consumed tokens: 36490444800 | elapsed time per iteration (s): 0.15 | learning rate: 1.824E-04 | global batch size: 256 | lm loss: 3.728232E+00 | grad norm: 0.310 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1661.669 | TFLOPs: 26.06 | 7: iteration 69700/ 331103 | consumed samples: 17843200 | consumed tokens: 36542873600 | elapsed time per iteration (s): 0.16 | learning rate: 1.824E-04 | global batch size: 256 | lm loss: 3.732003E+00 | grad norm: 0.348 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1649.086 | TFLOPs: 25.86 | 7: iteration 69800/ 331103 | consumed samples: 17868800 | consumed tokens: 36595302400 | elapsed time per iteration (s): 0.15 | learning rate: 1.823E-04 | global batch size: 256 | lm loss: 3.734347E+00 | grad norm: 0.322 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1653.170 | TFLOPs: 25.93 | 7: iteration 69900/ 331103 | consumed samples: 17894400 | consumed tokens: 36647731200 | elapsed time per iteration (s): 0.16 | learning rate: 1.823E-04 | global batch size: 256 | lm loss: 3.729883E+00 | grad norm: 0.287 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1629.705 | TFLOPs: 25.56 | 0: [2023-03-17 13:05:15,959] [INFO] [logging.py:68:log_dist] [Rank 0] step=70000, skipped=0, lr=[0.00018223413413760394, 0.00018223413413760394, 0.00018223413413760394], mom=[(0.9, 0.999), (0.9, 0.999), (0.9, 0.999)] 7: iteration 70000/ 331103 | consumed samples: 17920000 | consumed tokens: 36700160000 | elapsed time per iteration (s): 0.15 | learning rate: 1.822E-04 | global batch size: 256 | lm loss: 3.729741E+00 | grad norm: 0.331 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1655.198 | TFLOPs: 25.96 | 0: steps: 70000 loss: 3.7403 iter time (s): 0.155 samples/sec: 1656.331 7: ------------------------------------------------------------------------------------------------ 7: validation loss at iteration 70000 | lm loss value: 3.899822E+00 | lm loss PPL: 4.939364E+01 | 7: ------------------------------------------------------------------------------------------------ 0: saving checkpoint at iteration 70000 to checkpoints_44m174b100m 0: [2023-03-17 13:05:16,032] [INFO] [logging.py:68:log_dist] [Rank 0] [Torch] Checkpoint global_step70000 is begin to save! 0: [2023-03-17 13:05:16,036] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step70000/layer_01-model_00-model_states.pt... 0: [2023-03-17 13:05:16,101] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step70000/layer_01-model_00-model_states.pt. 0: [2023-03-17 13:05:16,101] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step70000/layer_03-model_00-model_states.pt... 0: [2023-03-17 13:05:16,110] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step70000/layer_03-model_00-model_states.pt. 0: [2023-03-17 13:05:16,110] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step70000/layer_04-model_00-model_states.pt... 0: [2023-03-17 13:05:16,118] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step70000/layer_04-model_00-model_states.pt. 0: [2023-03-17 13:05:16,118] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step70000/layer_05-model_00-model_states.pt... 0: [2023-03-17 13:05:16,126] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step70000/layer_05-model_00-model_states.pt. 0: [2023-03-17 13:05:16,126] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step70000/layer_06-model_00-model_states.pt... 0: [2023-03-17 13:05:16,134] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step70000/layer_06-model_00-model_states.pt. 0: [2023-03-17 13:05:16,134] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step70000/layer_07-model_00-model_states.pt... 0: [2023-03-17 13:05:16,142] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step70000/layer_07-model_00-model_states.pt. 0: [2023-03-17 13:05:16,142] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step70000/layer_08-model_00-model_states.pt... 0: [2023-03-17 13:05:16,150] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step70000/layer_08-model_00-model_states.pt. 0: [2023-03-17 13:05:16,150] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step70000/layer_09-model_00-model_states.pt... 0: [2023-03-17 13:05:16,157] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step70000/layer_09-model_00-model_states.pt. 0: [2023-03-17 13:05:16,158] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step70000/layer_10-model_00-model_states.pt... 0: [2023-03-17 13:05:16,165] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step70000/layer_10-model_00-model_states.pt. 0: [2023-03-17 13:05:16,165] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step70000/layer_12-model_00-model_states.pt... 0: [2023-03-17 13:05:16,166] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step70000/layer_12-model_00-model_states.pt. 0: [2023-03-17 13:05:16,167] [INFO] [logging.py:68:log_dist] [Rank 0] Saving model checkpoint: checkpoints_44m174b100m/global_step70000/mp_rank_00_model_states.pt 0: [2023-03-17 13:05:16,167] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step70000/mp_rank_00_model_states.pt... 0: [2023-03-17 13:05:16,169] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step70000/mp_rank_00_model_states.pt. 0: [2023-03-17 13:05:16,185] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step70000/bf16_zero_pp_rank_0_mp_rank_00_optim_states.pt... 0: [2023-03-17 13:05:16,185] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step70000/bf16_zero_pp_rank_1_mp_rank_00_optim_states.pt... 0: [2023-03-17 13:05:16,185] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step70000/bf16_zero_pp_rank_7_mp_rank_00_optim_states.pt... 0: [2023-03-17 13:05:16,185] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step70000/bf16_zero_pp_rank_4_mp_rank_00_optim_states.pt... 0: [2023-03-17 13:05:16,185] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step70000/bf16_zero_pp_rank_6_mp_rank_00_optim_states.pt... 3: [2023-03-17 13:05:16,185] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step70000/bf16_zero_pp_rank_29_mp_rank_00_optim_states.pt... 3: [2023-03-17 13:05:16,185] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step70000/bf16_zero_pp_rank_24_mp_rank_00_optim_states.pt... 3: [2023-03-17 13:05:16,185] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step70000/bf16_zero_pp_rank_25_mp_rank_00_optim_states.pt... 3: [2023-03-17 13:05:16,185] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step70000/bf16_zero_pp_rank_30_mp_rank_00_optim_states.pt... 3: [2023-03-17 13:05:16,185] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step70000/bf16_zero_pp_rank_31_mp_rank_00_optim_states.pt... 2: [2023-03-17 13:05:16,185] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step70000/bf16_zero_pp_rank_16_mp_rank_00_optim_states.pt... 2: [2023-03-17 13:05:16,185] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step70000/bf16_zero_pp_rank_17_mp_rank_00_optim_states.pt... 2: [2023-03-17 13:05:16,185] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step70000/bf16_zero_pp_rank_22_mp_rank_00_optim_states.pt... 2: [2023-03-17 13:05:16,185] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step70000/bf16_zero_pp_rank_23_mp_rank_00_optim_states.pt... 2: [2023-03-17 13:05:16,185] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step70000/bf16_zero_pp_rank_19_mp_rank_00_optim_states.pt... 4: [2023-03-17 13:05:16,185] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step70000/bf16_zero_pp_rank_33_mp_rank_00_optim_states.pt... 4: [2023-03-17 13:05:16,185] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step70000/bf16_zero_pp_rank_37_mp_rank_00_optim_states.pt... 4: [2023-03-17 13:05:16,185] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step70000/bf16_zero_pp_rank_34_mp_rank_00_optim_states.pt... 4: [2023-03-17 13:05:16,185] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step70000/bf16_zero_pp_rank_32_mp_rank_00_optim_states.pt... 4: [2023-03-17 13:05:16,185] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step70000/bf16_zero_pp_rank_36_mp_rank_00_optim_states.pt... 7: [2023-03-17 13:05:16,185] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step70000/bf16_zero_pp_rank_56_mp_rank_00_optim_states.pt... 7: [2023-03-17 13:05:16,185] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step70000/bf16_zero_pp_rank_63_mp_rank_00_optim_states.pt... 7: [2023-03-17 13:05:16,185] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step70000/bf16_zero_pp_rank_57_mp_rank_00_optim_states.pt... 7: [2023-03-17 13:05:16,185] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step70000/bf16_zero_pp_rank_60_mp_rank_00_optim_states.pt... 7: [2023-03-17 13:05:16,185] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step70000/bf16_zero_pp_rank_58_mp_rank_00_optim_states.pt... 5: [2023-03-17 13:05:16,185] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step70000/bf16_zero_pp_rank_40_mp_rank_00_optim_states.pt... 5: [2023-03-17 13:05:16,185] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step70000/bf16_zero_pp_rank_47_mp_rank_00_optim_states.pt... 5: [2023-03-17 13:05:16,185] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step70000/bf16_zero_pp_rank_46_mp_rank_00_optim_states.pt... 5: [2023-03-17 13:05:16,185] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step70000/bf16_zero_pp_rank_44_mp_rank_00_optim_states.pt... 5: [2023-03-17 13:05:16,185] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step70000/bf16_zero_pp_rank_45_mp_rank_00_optim_states.pt... 0: [2023-03-17 13:05:16,185] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step70000/bf16_zero_pp_rank_3_mp_rank_00_optim_states.pt... 0: [2023-03-17 13:05:16,185] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step70000/bf16_zero_pp_rank_2_mp_rank_00_optim_states.pt... 2: [2023-03-17 13:05:16,185] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step70000/bf16_zero_pp_rank_20_mp_rank_00_optim_states.pt... 2: [2023-03-17 13:05:16,185] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step70000/bf16_zero_pp_rank_21_mp_rank_00_optim_states.pt... 6: [2023-03-17 13:05:16,185] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step70000/bf16_zero_pp_rank_48_mp_rank_00_optim_states.pt... 6: [2023-03-17 13:05:16,185] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step70000/bf16_zero_pp_rank_54_mp_rank_00_optim_states.pt... 6: [2023-03-17 13:05:16,185] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step70000/bf16_zero_pp_rank_55_mp_rank_00_optim_states.pt... 6: [2023-03-17 13:05:16,185] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step70000/bf16_zero_pp_rank_52_mp_rank_00_optim_states.pt... 6: [2023-03-17 13:05:16,185] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step70000/bf16_zero_pp_rank_50_mp_rank_00_optim_states.pt... 0: [2023-03-17 13:05:16,185] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step70000/bf16_zero_pp_rank_5_mp_rank_00_optim_states.pt... 2: [2023-03-17 13:05:16,185] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step70000/bf16_zero_pp_rank_18_mp_rank_00_optim_states.pt... 4: [2023-03-17 13:05:16,185] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step70000/bf16_zero_pp_rank_38_mp_rank_00_optim_states.pt... 4: [2023-03-17 13:05:16,185] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step70000/bf16_zero_pp_rank_35_mp_rank_00_optim_states.pt... 4: [2023-03-17 13:05:16,185] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step70000/bf16_zero_pp_rank_39_mp_rank_00_optim_states.pt... 6: [2023-03-17 13:05:16,185] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step70000/bf16_zero_pp_rank_49_mp_rank_00_optim_states.pt... 6: [2023-03-17 13:05:16,185] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step70000/bf16_zero_pp_rank_53_mp_rank_00_optim_states.pt... 6: [2023-03-17 13:05:16,185] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step70000/bf16_zero_pp_rank_51_mp_rank_00_optim_states.pt... 7: [2023-03-17 13:05:16,185] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step70000/bf16_zero_pp_rank_59_mp_rank_00_optim_states.pt... 7: [2023-03-17 13:05:16,185] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step70000/bf16_zero_pp_rank_62_mp_rank_00_optim_states.pt... 7: [2023-03-17 13:05:16,185] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step70000/bf16_zero_pp_rank_61_mp_rank_00_optim_states.pt... 5: [2023-03-17 13:05:16,185] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step70000/bf16_zero_pp_rank_42_mp_rank_00_optim_states.pt... 5: [2023-03-17 13:05:16,185] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step70000/bf16_zero_pp_rank_43_mp_rank_00_optim_states.pt... 5: [2023-03-17 13:05:16,185] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step70000/bf16_zero_pp_rank_41_mp_rank_00_optim_states.pt... 3: [2023-03-17 13:05:16,185] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step70000/bf16_zero_pp_rank_26_mp_rank_00_optim_states.pt... 3: [2023-03-17 13:05:16,185] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step70000/bf16_zero_pp_rank_28_mp_rank_00_optim_states.pt... 3: [2023-03-17 13:05:16,185] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step70000/bf16_zero_pp_rank_27_mp_rank_00_optim_states.pt... 1: [2023-03-17 13:05:16,185] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step70000/bf16_zero_pp_rank_13_mp_rank_00_optim_states.pt... 1: [2023-03-17 13:05:16,185] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step70000/bf16_zero_pp_rank_8_mp_rank_00_optim_states.pt... 1: [2023-03-17 13:05:16,185] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step70000/bf16_zero_pp_rank_9_mp_rank_00_optim_states.pt... 1: [2023-03-17 13:05:16,185] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step70000/bf16_zero_pp_rank_15_mp_rank_00_optim_states.pt... 1: [2023-03-17 13:05:16,185] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step70000/bf16_zero_pp_rank_14_mp_rank_00_optim_states.pt... 1: [2023-03-17 13:05:16,185] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step70000/bf16_zero_pp_rank_12_mp_rank_00_optim_states.pt... 1: [2023-03-17 13:05:16,185] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step70000/bf16_zero_pp_rank_11_mp_rank_00_optim_states.pt... 1: [2023-03-17 13:05:16,185] [INFO] [torch_checkpoint_engine.py:15:save] [Torch] Saving checkpoints_44m174b100m/global_step70000/bf16_zero_pp_rank_10_mp_rank_00_optim_states.pt... 0: [2023-03-17 13:05:16,197] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step70000/bf16_zero_pp_rank_0_mp_rank_00_optim_states.pt. 0: [2023-03-17 13:05:16,198] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step70000/bf16_zero_pp_rank_0_mp_rank_00_optim_states.pt 0: [2023-03-17 13:05:16,198] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step70000 is ready now! 0: [2023-03-17 13:05:16,199] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step70000/bf16_zero_pp_rank_1_mp_rank_00_optim_states.pt. 0: [2023-03-17 13:05:16,199] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step70000/bf16_zero_pp_rank_1_mp_rank_00_optim_states.pt 5: [2023-03-17 13:05:16,199] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step70000/bf16_zero_pp_rank_40_mp_rank_00_optim_states.pt. 6: [2023-03-17 13:05:16,199] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step70000/bf16_zero_pp_rank_48_mp_rank_00_optim_states.pt. 0: [2023-03-17 13:05:16,199] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step70000 is ready now! 6: [2023-03-17 13:05:16,199] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step70000/bf16_zero_pp_rank_48_mp_rank_00_optim_states.pt 5: [2023-03-17 13:05:16,199] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step70000/bf16_zero_pp_rank_40_mp_rank_00_optim_states.pt 6: [2023-03-17 13:05:16,199] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step70000 is ready now! 5: [2023-03-17 13:05:16,199] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step70000 is ready now! 1: [2023-03-17 13:05:16,199] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step70000/bf16_zero_pp_rank_13_mp_rank_00_optim_states.pt. 1: [2023-03-17 13:05:16,199] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step70000/bf16_zero_pp_rank_13_mp_rank_00_optim_states.pt 2: [2023-03-17 13:05:16,199] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step70000/bf16_zero_pp_rank_16_mp_rank_00_optim_states.pt. 1: [2023-03-17 13:05:16,199] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step70000 is ready now! 2: [2023-03-17 13:05:16,199] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step70000/bf16_zero_pp_rank_16_mp_rank_00_optim_states.pt 2: [2023-03-17 13:05:16,199] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step70000 is ready now! 7: [2023-03-17 13:05:16,199] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step70000/bf16_zero_pp_rank_56_mp_rank_00_optim_states.pt. 7: [2023-03-17 13:05:16,200] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step70000/bf16_zero_pp_rank_56_mp_rank_00_optim_states.pt 7: [2023-03-17 13:05:16,200] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step70000 is ready now! 6: [2023-03-17 13:05:16,200] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step70000/bf16_zero_pp_rank_54_mp_rank_00_optim_states.pt. 2: [2023-03-17 13:05:16,200] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step70000/bf16_zero_pp_rank_23_mp_rank_00_optim_states.pt. 6: [2023-03-17 13:05:16,200] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step70000/bf16_zero_pp_rank_54_mp_rank_00_optim_states.pt 6: [2023-03-17 13:05:16,200] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step70000 is ready now! 2: [2023-03-17 13:05:16,200] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step70000/bf16_zero_pp_rank_23_mp_rank_00_optim_states.pt 5: [2023-03-17 13:05:16,200] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step70000/bf16_zero_pp_rank_46_mp_rank_00_optim_states.pt. 2: [2023-03-17 13:05:16,200] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step70000 is ready now! 5: [2023-03-17 13:05:16,200] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step70000/bf16_zero_pp_rank_46_mp_rank_00_optim_states.pt 4: [2023-03-17 13:05:16,200] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step70000/bf16_zero_pp_rank_37_mp_rank_00_optim_states.pt. 5: [2023-03-17 13:05:16,200] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step70000 is ready now! 4: [2023-03-17 13:05:16,201] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step70000/bf16_zero_pp_rank_37_mp_rank_00_optim_states.pt 4: [2023-03-17 13:05:16,201] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step70000 is ready now! 4: [2023-03-17 13:05:16,201] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step70000/bf16_zero_pp_rank_36_mp_rank_00_optim_states.pt. 7: [2023-03-17 13:05:16,201] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step70000/bf16_zero_pp_rank_58_mp_rank_00_optim_states.pt. 1: [2023-03-17 13:05:16,201] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step70000/bf16_zero_pp_rank_9_mp_rank_00_optim_states.pt. 4: [2023-03-17 13:05:16,201] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step70000/bf16_zero_pp_rank_36_mp_rank_00_optim_states.pt 7: [2023-03-17 13:05:16,201] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step70000/bf16_zero_pp_rank_58_mp_rank_00_optim_states.pt 1: [2023-03-17 13:05:16,201] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step70000/bf16_zero_pp_rank_9_mp_rank_00_optim_states.pt 1: [2023-03-17 13:05:16,201] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step70000 is ready now! 4: [2023-03-17 13:05:16,201] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step70000 is ready now! 7: [2023-03-17 13:05:16,201] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step70000 is ready now! 0: [2023-03-17 13:05:16,201] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step70000/bf16_zero_pp_rank_5_mp_rank_00_optim_states.pt. 0: [2023-03-17 13:05:16,201] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step70000/bf16_zero_pp_rank_5_mp_rank_00_optim_states.pt 0: [2023-03-17 13:05:16,201] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step70000 is ready now! 5: [2023-03-17 13:05:16,202] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step70000/bf16_zero_pp_rank_47_mp_rank_00_optim_states.pt. 5: [2023-03-17 13:05:16,202] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step70000/bf16_zero_pp_rank_47_mp_rank_00_optim_states.pt 5: [2023-03-17 13:05:16,202] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step70000 is ready now! 1: [2023-03-17 13:05:16,202] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step70000/bf16_zero_pp_rank_8_mp_rank_00_optim_states.pt. 1: [2023-03-17 13:05:16,202] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step70000/bf16_zero_pp_rank_8_mp_rank_00_optim_states.pt 1: [2023-03-17 13:05:16,202] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step70000 is ready now! 2: [2023-03-17 13:05:16,202] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step70000/bf16_zero_pp_rank_17_mp_rank_00_optim_states.pt. 6: [2023-03-17 13:05:16,202] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step70000/bf16_zero_pp_rank_52_mp_rank_00_optim_states.pt. 6: [2023-03-17 13:05:16,202] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step70000/bf16_zero_pp_rank_52_mp_rank_00_optim_states.pt 2: [2023-03-17 13:05:16,202] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step70000/bf16_zero_pp_rank_17_mp_rank_00_optim_states.pt 2: [2023-03-17 13:05:16,202] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step70000 is ready now! 6: [2023-03-17 13:05:16,202] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step70000 is ready now! 7: [2023-03-17 13:05:16,203] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step70000/bf16_zero_pp_rank_63_mp_rank_00_optim_states.pt. 7: [2023-03-17 13:05:16,203] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step70000/bf16_zero_pp_rank_63_mp_rank_00_optim_states.pt 7: [2023-03-17 13:05:16,203] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step70000 is ready now! 0: [2023-03-17 13:05:16,203] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step70000/bf16_zero_pp_rank_3_mp_rank_00_optim_states.pt. 0: [2023-03-17 13:05:16,203] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step70000/bf16_zero_pp_rank_3_mp_rank_00_optim_states.pt 0: [2023-03-17 13:05:16,203] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step70000 is ready now! 6: [2023-03-17 13:05:16,203] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step70000/bf16_zero_pp_rank_55_mp_rank_00_optim_states.pt. 6: [2023-03-17 13:05:16,203] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step70000/bf16_zero_pp_rank_55_mp_rank_00_optim_states.pt 6: [2023-03-17 13:05:16,203] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step70000 is ready now! 2: [2023-03-17 13:05:16,203] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step70000/bf16_zero_pp_rank_22_mp_rank_00_optim_states.pt. 2: [2023-03-17 13:05:16,203] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step70000/bf16_zero_pp_rank_22_mp_rank_00_optim_states.pt 2: [2023-03-17 13:05:16,203] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step70000 is ready now! 5: [2023-03-17 13:05:16,204] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step70000/bf16_zero_pp_rank_44_mp_rank_00_optim_states.pt. 4: [2023-03-17 13:05:16,204] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step70000/bf16_zero_pp_rank_38_mp_rank_00_optim_states.pt. 5: [2023-03-17 13:05:16,204] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step70000/bf16_zero_pp_rank_44_mp_rank_00_optim_states.pt 4: [2023-03-17 13:05:16,204] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step70000/bf16_zero_pp_rank_38_mp_rank_00_optim_states.pt 5: [2023-03-17 13:05:16,204] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step70000 is ready now! 4: [2023-03-17 13:05:16,204] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step70000 is ready now! 0: [2023-03-17 13:05:16,204] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step70000/bf16_zero_pp_rank_2_mp_rank_00_optim_states.pt. 0: [2023-03-17 13:05:16,204] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step70000/bf16_zero_pp_rank_2_mp_rank_00_optim_states.pt 0: [2023-03-17 13:05:16,204] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step70000 is ready now! 4: [2023-03-17 13:05:16,204] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step70000/bf16_zero_pp_rank_33_mp_rank_00_optim_states.pt. 7: [2023-03-17 13:05:16,204] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step70000/bf16_zero_pp_rank_60_mp_rank_00_optim_states.pt. 5: [2023-03-17 13:05:16,204] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step70000/bf16_zero_pp_rank_45_mp_rank_00_optim_states.pt. 4: [2023-03-17 13:05:16,205] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step70000/bf16_zero_pp_rank_33_mp_rank_00_optim_states.pt 7: [2023-03-17 13:05:16,205] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step70000/bf16_zero_pp_rank_60_mp_rank_00_optim_states.pt 5: [2023-03-17 13:05:16,205] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step70000/bf16_zero_pp_rank_45_mp_rank_00_optim_states.pt 4: [2023-03-17 13:05:16,205] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step70000 is ready now! 5: [2023-03-17 13:05:16,205] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step70000 is ready now! 7: [2023-03-17 13:05:16,205] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step70000 is ready now! 1: [2023-03-17 13:05:16,205] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step70000/bf16_zero_pp_rank_15_mp_rank_00_optim_states.pt. 1: [2023-03-17 13:05:16,205] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step70000/bf16_zero_pp_rank_15_mp_rank_00_optim_states.pt 2: [2023-03-17 13:05:16,205] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step70000/bf16_zero_pp_rank_19_mp_rank_00_optim_states.pt. 1: [2023-03-17 13:05:16,205] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step70000 is ready now! 2: [2023-03-17 13:05:16,205] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step70000/bf16_zero_pp_rank_19_mp_rank_00_optim_states.pt 2: [2023-03-17 13:05:16,205] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step70000 is ready now! 6: [2023-03-17 13:05:16,205] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step70000/bf16_zero_pp_rank_51_mp_rank_00_optim_states.pt. 6: [2023-03-17 13:05:16,205] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step70000/bf16_zero_pp_rank_51_mp_rank_00_optim_states.pt 6: [2023-03-17 13:05:16,205] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step70000 is ready now! 1: [2023-03-17 13:05:16,206] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step70000/bf16_zero_pp_rank_12_mp_rank_00_optim_states.pt. 1: [2023-03-17 13:05:16,206] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step70000/bf16_zero_pp_rank_12_mp_rank_00_optim_states.pt 1: [2023-03-17 13:05:16,206] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step70000 is ready now! 7: [2023-03-17 13:05:16,206] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step70000/bf16_zero_pp_rank_57_mp_rank_00_optim_states.pt. 7: [2023-03-17 13:05:16,206] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step70000/bf16_zero_pp_rank_57_mp_rank_00_optim_states.pt 0: [2023-03-17 13:05:16,206] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step70000/bf16_zero_pp_rank_7_mp_rank_00_optim_states.pt. 7: [2023-03-17 13:05:16,206] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step70000 is ready now! 0: [2023-03-17 13:05:16,206] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step70000/bf16_zero_pp_rank_7_mp_rank_00_optim_states.pt 0: [2023-03-17 13:05:16,206] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step70000 is ready now! 2: [2023-03-17 13:05:16,206] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step70000/bf16_zero_pp_rank_20_mp_rank_00_optim_states.pt. 2: [2023-03-17 13:05:16,206] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step70000/bf16_zero_pp_rank_20_mp_rank_00_optim_states.pt 2: [2023-03-17 13:05:16,206] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step70000 is ready now! 5: [2023-03-17 13:05:16,206] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step70000/bf16_zero_pp_rank_42_mp_rank_00_optim_states.pt. 5: [2023-03-17 13:05:16,206] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step70000/bf16_zero_pp_rank_42_mp_rank_00_optim_states.pt 5: [2023-03-17 13:05:16,206] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step70000 is ready now! 6: [2023-03-17 13:05:16,206] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step70000/bf16_zero_pp_rank_50_mp_rank_00_optim_states.pt. 6: [2023-03-17 13:05:16,206] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step70000/bf16_zero_pp_rank_50_mp_rank_00_optim_states.pt 6: [2023-03-17 13:05:16,206] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step70000 is ready now! 4: [2023-03-17 13:05:16,207] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step70000/bf16_zero_pp_rank_34_mp_rank_00_optim_states.pt. 1: [2023-03-17 13:05:16,207] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step70000/bf16_zero_pp_rank_14_mp_rank_00_optim_states.pt. 4: [2023-03-17 13:05:16,207] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step70000/bf16_zero_pp_rank_34_mp_rank_00_optim_states.pt 4: [2023-03-17 13:05:16,207] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step70000 is ready now! 1: [2023-03-17 13:05:16,207] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step70000/bf16_zero_pp_rank_14_mp_rank_00_optim_states.pt 1: [2023-03-17 13:05:16,207] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step70000 is ready now! 7: [2023-03-17 13:05:16,207] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step70000/bf16_zero_pp_rank_59_mp_rank_00_optim_states.pt. 7: [2023-03-17 13:05:16,207] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step70000/bf16_zero_pp_rank_59_mp_rank_00_optim_states.pt 7: [2023-03-17 13:05:16,207] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step70000 is ready now! 2: [2023-03-17 13:05:16,207] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step70000/bf16_zero_pp_rank_21_mp_rank_00_optim_states.pt. 2: [2023-03-17 13:05:16,207] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step70000/bf16_zero_pp_rank_21_mp_rank_00_optim_states.pt 0: [2023-03-17 13:05:16,207] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step70000/bf16_zero_pp_rank_6_mp_rank_00_optim_states.pt. 2: [2023-03-17 13:05:16,207] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step70000 is ready now! 0: [2023-03-17 13:05:16,207] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step70000/bf16_zero_pp_rank_6_mp_rank_00_optim_states.pt 0: [2023-03-17 13:05:16,207] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step70000 is ready now! 5: [2023-03-17 13:05:16,207] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step70000/bf16_zero_pp_rank_43_mp_rank_00_optim_states.pt. 5: [2023-03-17 13:05:16,208] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step70000/bf16_zero_pp_rank_43_mp_rank_00_optim_states.pt 5: [2023-03-17 13:05:16,208] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step70000 is ready now! 6: [2023-03-17 13:05:16,208] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step70000/bf16_zero_pp_rank_53_mp_rank_00_optim_states.pt. 6: [2023-03-17 13:05:16,208] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step70000/bf16_zero_pp_rank_53_mp_rank_00_optim_states.pt 6: [2023-03-17 13:05:16,208] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step70000 is ready now! 1: [2023-03-17 13:05:16,208] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step70000/bf16_zero_pp_rank_11_mp_rank_00_optim_states.pt. 1: [2023-03-17 13:05:16,208] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step70000/bf16_zero_pp_rank_11_mp_rank_00_optim_states.pt 1: [2023-03-17 13:05:16,208] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step70000 is ready now! 4: [2023-03-17 13:05:16,208] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step70000/bf16_zero_pp_rank_35_mp_rank_00_optim_states.pt. 0: [2023-03-17 13:05:16,208] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step70000/bf16_zero_pp_rank_4_mp_rank_00_optim_states.pt. 0: [2023-03-17 13:05:16,208] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step70000/bf16_zero_pp_rank_4_mp_rank_00_optim_states.pt 4: [2023-03-17 13:05:16,208] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step70000/bf16_zero_pp_rank_35_mp_rank_00_optim_states.pt 0: [2023-03-17 13:05:16,208] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step70000 is ready now! 4: [2023-03-17 13:05:16,208] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step70000 is ready now! 7: [2023-03-17 13:05:16,208] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step70000/bf16_zero_pp_rank_62_mp_rank_00_optim_states.pt. 2: [2023-03-17 13:05:16,208] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step70000/bf16_zero_pp_rank_18_mp_rank_00_optim_states.pt. 4: [2023-03-17 13:05:16,208] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step70000/bf16_zero_pp_rank_32_mp_rank_00_optim_states.pt. 7: [2023-03-17 13:05:16,208] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step70000/bf16_zero_pp_rank_62_mp_rank_00_optim_states.pt 2: [2023-03-17 13:05:16,208] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step70000/bf16_zero_pp_rank_18_mp_rank_00_optim_states.pt 4: [2023-03-17 13:05:16,208] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step70000/bf16_zero_pp_rank_32_mp_rank_00_optim_states.pt 7: [2023-03-17 13:05:16,208] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step70000 is ready now! 2: [2023-03-17 13:05:16,209] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step70000 is ready now! 4: [2023-03-17 13:05:16,209] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step70000 is ready now! 6: [2023-03-17 13:05:16,209] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step70000/bf16_zero_pp_rank_49_mp_rank_00_optim_states.pt. 6: [2023-03-17 13:05:16,209] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step70000/bf16_zero_pp_rank_49_mp_rank_00_optim_states.pt 7: [2023-03-17 13:05:16,209] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step70000/bf16_zero_pp_rank_61_mp_rank_00_optim_states.pt. 6: [2023-03-17 13:05:16,209] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step70000 is ready now! 7: [2023-03-17 13:05:16,209] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step70000/bf16_zero_pp_rank_61_mp_rank_00_optim_states.pt 4: [2023-03-17 13:05:16,209] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step70000/bf16_zero_pp_rank_39_mp_rank_00_optim_states.pt. 7: [2023-03-17 13:05:16,209] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step70000 is ready now! 4: [2023-03-17 13:05:16,209] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step70000/bf16_zero_pp_rank_39_mp_rank_00_optim_states.pt 4: [2023-03-17 13:05:16,209] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step70000 is ready now! 1: [2023-03-17 13:05:16,209] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step70000/bf16_zero_pp_rank_10_mp_rank_00_optim_states.pt. 1: [2023-03-17 13:05:16,209] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step70000/bf16_zero_pp_rank_10_mp_rank_00_optim_states.pt 1: [2023-03-17 13:05:16,209] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step70000 is ready now! 5: [2023-03-17 13:05:16,212] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step70000/bf16_zero_pp_rank_41_mp_rank_00_optim_states.pt. 5: [2023-03-17 13:05:16,212] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step70000/bf16_zero_pp_rank_41_mp_rank_00_optim_states.pt 5: [2023-03-17 13:05:16,212] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step70000 is ready now! 3: [2023-03-17 13:05:16,221] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step70000/bf16_zero_pp_rank_31_mp_rank_00_optim_states.pt. 3: [2023-03-17 13:05:16,221] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step70000/bf16_zero_pp_rank_31_mp_rank_00_optim_states.pt 3: [2023-03-17 13:05:16,221] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step70000 is ready now! 3: [2023-03-17 13:05:16,221] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step70000/bf16_zero_pp_rank_24_mp_rank_00_optim_states.pt. 3: [2023-03-17 13:05:16,221] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step70000/bf16_zero_pp_rank_24_mp_rank_00_optim_states.pt 3: [2023-03-17 13:05:16,221] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step70000 is ready now! 3: [2023-03-17 13:05:16,221] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step70000/bf16_zero_pp_rank_28_mp_rank_00_optim_states.pt. 3: [2023-03-17 13:05:16,222] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step70000/bf16_zero_pp_rank_28_mp_rank_00_optim_states.pt 3: [2023-03-17 13:05:16,222] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step70000 is ready now! 3: [2023-03-17 13:05:16,222] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step70000/bf16_zero_pp_rank_29_mp_rank_00_optim_states.pt. 3: [2023-03-17 13:05:16,222] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step70000/bf16_zero_pp_rank_29_mp_rank_00_optim_states.pt 3: [2023-03-17 13:05:16,222] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step70000 is ready now! 3: [2023-03-17 13:05:16,222] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step70000/bf16_zero_pp_rank_27_mp_rank_00_optim_states.pt. 3: [2023-03-17 13:05:16,222] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step70000/bf16_zero_pp_rank_27_mp_rank_00_optim_states.pt 3: [2023-03-17 13:05:16,222] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step70000 is ready now! 3: [2023-03-17 13:05:16,222] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step70000/bf16_zero_pp_rank_30_mp_rank_00_optim_states.pt. 3: [2023-03-17 13:05:16,222] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step70000/bf16_zero_pp_rank_25_mp_rank_00_optim_states.pt. 3: [2023-03-17 13:05:16,222] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step70000/bf16_zero_pp_rank_30_mp_rank_00_optim_states.pt 3: [2023-03-17 13:05:16,222] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step70000 is ready now! 3: [2023-03-17 13:05:16,222] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step70000/bf16_zero_pp_rank_25_mp_rank_00_optim_states.pt 3: [2023-03-17 13:05:16,222] [INFO] [torch_checkpoint_engine.py:17:save] [Torch] Saved checkpoints_44m174b100m/global_step70000/bf16_zero_pp_rank_26_mp_rank_00_optim_states.pt. 3: [2023-03-17 13:05:16,222] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step70000 is ready now! 3: [2023-03-17 13:05:16,222] [INFO] [engine.py:3213:_save_zero_checkpoint] bf16_zero checkpoint saved checkpoints_44m174b100m/global_step70000/bf16_zero_pp_rank_26_mp_rank_00_optim_states.pt 3: [2023-03-17 13:05:16,222] [INFO] [torch_checkpoint_engine.py:27:commit] [Torch] Checkpoint global_step70000 is ready now! 0: successfully saved checkpoint at iteration 70000 to checkpoints_44m174b100m 7: time (ms) | save-checkpoint: 195.38 7: iteration 70100/ 331103 | consumed samples: 17945600 | consumed tokens: 36752588800 | elapsed time per iteration (s): 0.16 | learning rate: 1.822E-04 | global batch size: 256 | lm loss: 3.731691E+00 | grad norm: 0.325 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1618.480 | TFLOPs: 25.38 | 7: iteration 70200/ 331103 | consumed samples: 17971200 | consumed tokens: 36805017600 | elapsed time per iteration (s): 0.15 | learning rate: 1.821E-04 | global batch size: 256 | lm loss: 3.729866E+00 | grad norm: 0.358 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1653.881 | TFLOPs: 25.94 | 7: iteration 70300/ 331103 | consumed samples: 17996800 | consumed tokens: 36857446400 | elapsed time per iteration (s): 0.15 | learning rate: 1.821E-04 | global batch size: 256 | lm loss: 3.731951E+00 | grad norm: 0.318 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1651.980 | TFLOPs: 25.91 | 7: iteration 70400/ 331103 | consumed samples: 18022400 | consumed tokens: 36909875200 | elapsed time per iteration (s): 0.16 | learning rate: 1.820E-04 | global batch size: 256 | lm loss: 3.733721E+00 | grad norm: 0.292 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1637.929 | TFLOPs: 25.69 | 7: iteration 70500/ 331103 | consumed samples: 18048000 | consumed tokens: 36962304000 | elapsed time per iteration (s): 0.15 | learning rate: 1.820E-04 | global batch size: 256 | lm loss: 3.727597E+00 | grad norm: 0.309 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1652.850 | TFLOPs: 25.92 | 7: iteration 70600/ 331103 | consumed samples: 18073600 | consumed tokens: 37014732800 | elapsed time per iteration (s): 0.16 | learning rate: 1.819E-04 | global batch size: 256 | lm loss: 3.728631E+00 | grad norm: 0.339 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1611.716 | TFLOPs: 25.28 | 7: iteration 70700/ 331103 | consumed samples: 18099200 | consumed tokens: 37067161600 | elapsed time per iteration (s): 0.15 | learning rate: 1.819E-04 | global batch size: 256 | lm loss: 3.730232E+00 | grad norm: 0.323 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1652.260 | TFLOPs: 25.91 | 7: iteration 70800/ 331103 | consumed samples: 18124800 | consumed tokens: 37119590400 | elapsed time per iteration (s): 0.15 | learning rate: 1.818E-04 | global batch size: 256 | lm loss: 3.730753E+00 | grad norm: 0.324 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1661.240 | TFLOPs: 26.05 | 7: iteration 70900/ 331103 | consumed samples: 18150400 | consumed tokens: 37172019200 | elapsed time per iteration (s): 0.16 | learning rate: 1.818E-04 | global batch size: 256 | lm loss: 3.726313E+00 | grad norm: 0.296 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1645.265 | TFLOPs: 25.80 | 7: iteration 71000/ 331103 | consumed samples: 18176000 | consumed tokens: 37224448000 | elapsed time per iteration (s): 0.16 | learning rate: 1.817E-04 | global batch size: 256 | lm loss: 3.724183E+00 | grad norm: 0.279 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1629.198 | TFLOPs: 25.55 | 7: iteration 71100/ 331103 | consumed samples: 18201600 | consumed tokens: 37276876800 | elapsed time per iteration (s): 0.16 | learning rate: 1.817E-04 | global batch size: 256 | lm loss: 3.728578E+00 | grad norm: 0.306 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1636.985 | TFLOPs: 25.67 | 7: iteration 71200/ 331103 | consumed samples: 18227200 | consumed tokens: 37329305600 | elapsed time per iteration (s): 0.17 | learning rate: 1.816E-04 | global batch size: 256 | lm loss: 3.731612E+00 | grad norm: 0.299 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1505.585 | TFLOPs: 23.61 | 7: iteration 71300/ 331103 | consumed samples: 18252800 | consumed tokens: 37381734400 | elapsed time per iteration (s): 0.16 | learning rate: 1.816E-04 | global batch size: 256 | lm loss: 3.732166E+00 | grad norm: 0.338 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1647.055 | TFLOPs: 25.83 | 7: iteration 71400/ 331103 | consumed samples: 18278400 | consumed tokens: 37434163200 | elapsed time per iteration (s): 0.15 | learning rate: 1.815E-04 | global batch size: 256 | lm loss: 3.731955E+00 | grad norm: 0.321 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1657.827 | TFLOPs: 26.00 | 7: iteration 71500/ 331103 | consumed samples: 18304000 | consumed tokens: 37486592000 | elapsed time per iteration (s): 0.15 | learning rate: 1.815E-04 | global batch size: 256 | lm loss: 3.730786E+00 | grad norm: 0.296 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1658.167 | TFLOPs: 26.00 | 7: iteration 71600/ 331103 | consumed samples: 18329600 | consumed tokens: 37539020800 | elapsed time per iteration (s): 0.16 | learning rate: 1.814E-04 | global batch size: 256 | lm loss: 3.730386E+00 | grad norm: 0.333 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1638.425 | TFLOPs: 25.69 | 7: iteration 71700/ 331103 | consumed samples: 18355200 | consumed tokens: 37591449600 | elapsed time per iteration (s): 0.15 | learning rate: 1.813E-04 | global batch size: 256 | lm loss: 3.725450E+00 | grad norm: 0.360 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1656.874 | TFLOPs: 25.98 | 7: iteration 71800/ 331103 | consumed samples: 18380800 | consumed tokens: 37643878400 | elapsed time per iteration (s): 0.16 | learning rate: 1.813E-04 | global batch size: 256 | lm loss: 3.727208E+00 | grad norm: 0.343 | num zeros: 0.0 | number of skipped iterations: 0 | number of nan iterations: 0 | samples per second: 1634.283 | TFLOPs: 25.63 |