06:44:38.992 [I] Running on: 59541e1c2271                                                         (1728:train.py:196)
INFO:2026-11-09 06:44:39,482:jax._src.xla_bridge:925: Unable to initialize backend 'rocm': module 'jaxlib.xla_extension' has no attribute 'GpuAllocatorConfig'
06:44:39.482 [I] Unable to initialize backend 'rocm': module 'jaxlib.xla_extension' has no attribute 'GpuAllocatorConfig' (1728:xla_bridge.py:925)
INFO:2026-11-09 06:44:39,483:jax._src.xla_bridge:925: Unable to initialize backend 'tpu': INTERNAL: Failed to open libtpu.so: libtpu.so: cannot open shared object file: No such file or directory
06:44:39.483 [I] Unable to initialize backend 'tpu': INTERNAL: Failed to open libtpu.so: libtpu.so: cannot open shared object file: No such file or directory (1728:xla_bridge.py:925)
06:44:40.530 [I] Created BasePyTreeCheckpointHandler: use_ocdbt=True, use_zarr3=False, pytree_metadata_options=PyTreeMetadataOptions(support_rich_types=False), array_metadata_store=<orbax.checkpoint._src.metadata.array_metadata_store.Store object at 0x7bc1755da910> (1728:base_pytree_checkpoint_handler.py:334)
06:44:40.530 [I] Created BasePyTreeCheckpointHandler: use_ocdbt=True, use_zarr3=False, pytree_metadata_options=PyTreeMetadataOptions(support_rich_types=False), array_metadata_store=<orbax.checkpoint._src.metadata.array_metadata_store.Store object at 0x7bc1755da910> (1728:base_pytree_checkpoint_handler.py:334)
06:44:40.530 [I] [thread=MainThread] Failed to get flag value for EXPERIMENTAL_ORBAX_USE_DISTRIBUTED_PROCESS_ID. (1728:multihost.py:390)
06:44:40.531 [I] [process=0][thread=MainThread] CheckpointManager init: checkpointers=None, item_names=None, item_handlers={'assets': <openpi.training.checkpoints.CallbackHandler object at 0x7bbfd9a0cbd0>, 'train_state': <orbax.checkpoint._src.handlers.pytree_checkpoint_handler.PyTreeCheckpointHandler object at 0x7bbfd9386110>, 'params': <orbax.checkpoint._src.handlers.pytree_checkpoint_handler.PyTreeCheckpointHandler object at 0x7bbfd9449890>}, handler_registry=None (1728:checkpoint_manager.py:620)
06:44:40.531 [I] Deferred registration for item: "assets". Adding handler `<openpi.training.checkpoints.CallbackHandler object at 0x7bbfd9a0cbd0>` for item "assets" and save args `<class 'openpi.training.checkpoints.CallbackSave'>` and restore args `<class 'openpi.training.checkpoints.CallbackRestore'>` to `_handler_registry`. (1728:composite_checkpoint_handler.py:234)
06:44:40.531 [I] Deferred registration for item: "train_state". Adding handler `<orbax.checkpoint._src.handlers.pytree_checkpoint_handler.PyTreeCheckpointHandler object at 0x7bbfd9386110>` for item "train_state" and save args `<class 'orbax.checkpoint._src.handlers.pytree_checkpoint_handler.PyTreeSaveArgs'>` and restore args `<class 'orbax.checkpoint._src.handlers.pytree_checkpoint_handler.PyTreeRestoreArgs'>` to `_handler_registry`. (1728:composite_checkpoint_handler.py:234)
06:44:40.531 [I] Deferred registration for item: "params". Adding handler `<orbax.checkpoint._src.handlers.pytree_checkpoint_handler.PyTreeCheckpointHandler object at 0x7bbfd9449890>` for item "params" and save args `<class 'orbax.checkpoint._src.handlers.pytree_checkpoint_handler.PyTreeSaveArgs'>` and restore args `<class 'orbax.checkpoint._src.handlers.pytree_checkpoint_handler.PyTreeRestoreArgs'>` to `_handler_registry`. (1728:composite_checkpoint_handler.py:234)
06:44:40.531 [I] Deferred registration for item: "metrics". Adding handler `<orbax.checkpoint._src.handlers.json_checkpoint_handler.JsonCheckpointHandler object at 0x7bbfd9467750>` for item "metrics" and save args `<class 'orbax.checkpoint._src.handlers.json_checkpoint_handler.JsonSaveArgs'>` and restore args `<class 'orbax.checkpoint._src.handlers.json_checkpoint_handler.JsonRestoreArgs'>` to `_handler_registry`. (1728:composite_checkpoint_handler.py:234)
06:44:40.531 [I] Initialized registry DefaultCheckpointHandlerRegistry({('assets', <class 'openpi.training.checkpoints.CallbackSave'>): <openpi.training.checkpoints.CallbackHandler object at 0x7bbfd9a0cbd0>, ('assets', <class 'openpi.training.checkpoints.CallbackRestore'>): <openpi.training.checkpoints.CallbackHandler object at 0x7bbfd9a0cbd0>, ('train_state', <class 'orbax.checkpoint._src.handlers.pytree_checkpoint_handler.PyTreeSaveArgs'>): <orbax.checkpoint._src.handlers.pytree_checkpoint_handler.PyTreeCheckpointHandler object at 0x7bbfd9386110>, ('train_state', <class 'orbax.checkpoint._src.handlers.pytree_checkpoint_handler.PyTreeRestoreArgs'>): <orbax.checkpoint._src.handlers.pytree_checkpoint_handler.PyTreeCheckpointHandler object at 0x7bbfd9386110>, ('params', <class 'orbax.checkpoint._src.handlers.pytree_checkpoint_handler.PyTreeSaveArgs'>): <orbax.checkpoint._src.handlers.pytree_checkpoint_handler.PyTreeCheckpointHandler object at 0x7bbfd9449890>, ('params', <class 'orbax.checkpoint._src.handlers.pytree_checkpoint_handler.PyTreeRestoreArgs'>): <orbax.checkpoint._src.handlers.pytree_checkpoint_handler.PyTreeCheckpointHandler object at 0x7bbfd9449890>, ('metrics', <class 'orbax.checkpoint._src.handlers.json_checkpoint_handler.JsonSaveArgs'>): <orbax.checkpoint._src.handlers.json_checkpoint_handler.JsonCheckpointHandler object at 0x7bbfd9467750>, ('metrics', <class 'orbax.checkpoint._src.handlers.json_checkpoint_handler.JsonRestoreArgs'>): <orbax.checkpoint._src.handlers.json_checkpoint_handler.JsonCheckpointHandler object at 0x7bbfd9467750>}). (1728:composite_checkpoint_handler.py:502)
06:44:40.532 [I] orbax-checkpoint version: 0.11.13                                                (1728:abstract_checkpointer.py:35)
06:44:40.532 [I] [process=0][thread=MainThread] Using barrier_sync_fn: <function get_barrier_sync_fn.<locals>.<lambda> at 0x7bbfd97e3ba0> timeout: 7200 secs and primary_host=0 for async checkpoint writes (1728:async_checkpointer.py:170)
06:44:40.532 [I] Found 0 checkpoint steps in /workspace/openpi/checkpoints/pi05_mecharm_deep_even10_lora/deep_even10_v1 (1728:checkpoint_manager.py:1701)
06:44:40.532 [I] [process=0][thread=MainThread] CheckpointManager created,  primary_host=0, CheckpointManagerOptions=CheckpointManagerOptions(save_interval_steps=1, max_to_keep=1, keep_time_interval=None, keep_period=1000, should_keep_fn=None, best_fn=None, best_mode='max', keep_checkpoints_without_metrics=True, step_prefix=None, step_format_fixed_length=None, step_name_format=None, create=False, cleanup_tmp_directories=False, save_on_steps=frozenset(), single_host_load_and_broadcast=False, todelete_subdir=None, enable_background_delete=False, read_only=False, enable_async_checkpointing=True, async_options=AsyncOptions(timeout_secs=7200, barrier_sync_fn=None, post_finalization_callback=None, create_directories_asynchronously=True), multiprocessing_options=MultiprocessingOptions(primary_host=0, active_processes=None, barrier_sync_key_prefix=None), should_save_fn=None, file_options=FileOptions(path_permission_mode=None), save_root_metadata=True, temporary_path_class=None, save_decision_policy=None, prevent_write_metrics=False), root_directory=/workspace/openpi/checkpoints/pi05_mecharm_deep_even10_lora/deep_even10_v1: <orbax.checkpoint.checkpoint_manager.CheckpointManager object at 0x7bbfd94597d0> (1728:checkpoint_manager.py:801)
06:44:40.639 [I] Loaded norm stats from /workspace/openpi/assets/pi05_mecharm_deep_even10_lora/local/mecharm_redblock_box_deep_even10_v1 (1728:config.py:197)
06:44:40.640 [I] data_config: DataConfig(repo_id='local/mecharm_redblock_box_deep_even10_v1', asset_id='local/mecharm_redblock_box_deep_even10_v1', norm_stats={'state': NormStats(mean=array([  2.10633135,  46.03452682,   4.41121578,   3.17700362,
       -50.12554169, -31.57465935,   0.4332771 ]), std=array([32.54282379, 20.76770592, 11.63037872,  4.70920372, 20.20290184,
        3.30348682,  0.49552804]), q01=array([-61.32834032,   1.48998002, -33.83168156,  -4.05277558,
       -75.5969573 , -40.60079463,   0.        ]), q99=array([ 44.55244775,  87.87266739,  30.3009803 ,  16.34375506,
        -0.26367661, -26.27586341,   0.9998    ])), 'actions': NormStats(mean=array([ 0.26515433,  0.08391801, -0.02417421,  0.03403188, -0.15184271,
       -0.04009151,  0.43327701]), std=array([9.13575172, 9.12926865, 3.51926446, 1.44124055, 9.11519718,
       0.78290892, 0.49552804]), q01=array([-21.1357602 , -22.60392705, -11.01575097,  -4.75467478,
       -25.78580854,  -2.68701   ,   0.        ]), q99=array([27.83729834, 23.28896594,  9.81135861,  3.74563593, 25.75257956,
        2.84948795,  0.9998    ]))}, repack_transforms=Group(inputs=[RepackTransform(structure={'observation/image': 'image', 'observation/state': 'state', 'actions': 'actions', 'prompt': 'prompt'})], outputs=()), data_transforms=Group(inputs=(MecharmInputs(model_type=<ModelType.PI05: 'pi05'>), DeltaActions(mask=(True, True, True, True, True, True, False))), outputs=(AbsoluteActions(mask=(True, True, True, True, True, True, False)), MecharmOutputs())), model_transforms=Group(inputs=[InjectDefaultPrompt(prompt=None), ResizeImages(height=224, width=224), TokenizePrompt(tokenizer=<openpi.models.tokenizer.PaligemmaTokenizer object at 0x7bbfd96459d0>, discrete_state_input=True), PadStatesAndActions(model_action_dim=32)], outputs=()), use_quantile_norm=True, action_sequence_keys=('actions',), prompt_from_task=True, rlds_data_dir=None, action_space=None, datasets=()) (1728:data_loader.py:243)
06:44:40.922 [I] local_batch_size: 32                                                             (1728:data_loader.py:324)
06:44:56.278 [I] Initialized data loader:
[0].images['base_0_rgb']: (32, 224, 224, 3)@float32
[0].images['left_wrist_0_rgb']: (32, 224, 224, 3)@float32
[0].images['right_wrist_0_rgb']: (32, 224, 224, 3)@float32
[0].image_masks['base_0_rgb']: (32,)@bool
[0].image_masks['left_wrist_0_rgb']: (32,)@bool
[0].image_masks['right_wrist_0_rgb']: (32,)@bool
[0].state: (32, 32)@float32
[0].tokenized_prompt: (32, 200)@int32
[0].tokenized_prompt_mask: (32, 200)@bool
[1]: (32, 10, 32)@float32 (1728:train.py:227)
06:44:57.542 [I] Downloading gs://openpi-assets/checkpoints/pi05_base/params to /root/.cache/openpi/openpi-assets/checkpoints/pi05_base/params (1728:download.py:86)
06:44:57.542 [W] gsutil not found, falling back to gcsfs. This may fail if GCP credentials are not configured correctly. (1728:download.py:111)
06:45:24.528 [I] Progress on: -/12441721931 rate:- remaining:? elapsed:00:00 postfix:-            (1728:tqdm_logging.py:145)
06:45:34.571 [I] Progress on: 5.99GiB/11.6GiB rate:648,602,439.3iB/s remaining:00:09 elapsed:00:10 postfix:- (1728:tqdm_logging.py:145)
06:45:44.611 [I] Progress on: 11.4GiB/11.6GiB rate:434,047,087.2iB/s remaining:00:00 elapsed:00:20 postfix:- (1728:tqdm_logging.py:145)
06:45:46.615 [I] Progress on: 11.6GiB/11.6GiB rate:371,273,818.2iB/s remaining:00:00 elapsed:00:22 postfix:- (1728:tqdm_logging.py:145)
06:45:46.622 [I] Created BasePyTreeCheckpointHandler: use_ocdbt=True, use_zarr3=False, pytree_metadata_options=PyTreeMetadataOptions(support_rich_types=False), array_metadata_store=<orbax.checkpoint._src.metadata.array_metadata_store.Store object at 0x7bc1755da910> (1728:base_pytree_checkpoint_handler.py:334)
06:45:46.676 [I] Restoring checkpoint from /root/.cache/openpi/openpi-assets/checkpoints/pi05_base/params. (1728:checkpointer.py:298)
06:45:54.128 [I] [process=0] /jax/checkpoint/read/bytes_per_sec: 1.7 GiB/s (total bytes: 12.5 GiB) (time elapsed: 7 seconds) (per-host) (1728:base_pytree_checkpoint_handler.py:114)
06:45:54.129 [I] Finished restoring checkpoint in 7.45 seconds from /root/.cache/openpi/openpi-assets/checkpoints/pi05_base/params. (1728:checkpointer.py:309)
06:46:11.201 [I] Initialized train state:
['PaliGemma']['img']['Transformer']['encoder_norm']['bias'].value: (1152,)@float32
['PaliGemma']['img']['Transformer']['encoder_norm']['scale'].value: (1152,)@float32
['PaliGemma']['img']['Transformer']['encoderblock']['LayerNorm_0']['bias'].value: (27, 1152)@float32
['PaliGemma']['img']['Transformer']['encoderblock']['LayerNorm_0']['scale'].value: (27, 1152)@float32
['PaliGemma']['img']['Transformer']['encoderblock']['LayerNorm_1']['bias'].value: (27, 1152)@float32
['PaliGemma']['img']['Transformer']['encoderblock']['LayerNorm_1']['scale'].value: (27, 1152)@float32
['PaliGemma']['img']['Transformer']['encoderblock']['MlpBlock_0']['Dense_0']['bias'].value: (27, 4304)@float32
['PaliGemma']['img']['Transformer']['encoderblock']['MlpBlock_0']['Dense_0']['kernel'].value: (27, 1152, 4304)@float32
['PaliGemma']['img']['Transformer']['encoderblock']['MlpBlock_0']['Dense_1']['bias'].value: (27, 1152)@float32
['PaliGemma']['img']['Transformer']['encoderblock']['MlpBlock_0']['Dense_1']['kernel'].value: (27, 4304, 1152)@float32
['PaliGemma']['img']['Transformer']['encoderblock']['MultiHeadDotProductAttention_0']['key']['bias'].value: (27, 16, 72)@float32
['PaliGemma']['img']['Transformer']['encoderblock']['MultiHeadDotProductAttention_0']['key']['kernel'].value: (27, 1152, 16, 72)@float32
['PaliGemma']['img']['Transformer']['encoderblock']['MultiHeadDotProductAttention_0']['out']['bias'].value: (27, 1152)@float32
['PaliGemma']['img']['Transformer']['encoderblock']['MultiHeadDotProductAttention_0']['out']['kernel'].value: (27, 16, 72, 1152)@float32
['PaliGemma']['img']['Transformer']['encoderblock']['MultiHeadDotProductAttention_0']['query']['bias'].value: (27, 16, 72)@float32
['PaliGemma']['img']['Transformer']['encoderblock']['MultiHeadDotProductAttention_0']['query']['kernel'].value: (27, 1152, 16, 72)@float32
['PaliGemma']['img']['Transformer']['encoderblock']['MultiHeadDotProductAttention_0']['value']['bias'].value: (27, 16, 72)@float32
['PaliGemma']['img']['Transformer']['encoderblock']['MultiHeadDotProductAttention_0']['value']['kernel'].value: (27, 1152, 16, 72)@float32
['PaliGemma']['img']['embedding']['bias'].value: (1152,)@float32
['PaliGemma']['img']['embedding']['kernel'].value: (14, 14, 3, 1152)@float32
['PaliGemma']['img']['head']['bias'].value: (2048,)@float32
['PaliGemma']['img']['head']['kernel'].value: (1152, 2048)@float32
['PaliGemma']['img']['pos_embedding'].value: (1, 256, 1152)@float32
['PaliGemma']['llm']['embedder']['input_embedding'].value: (257152, 2048)@bfloat16
['PaliGemma']['llm']['final_norm']['scale'].value: (2048,)@bfloat16
['PaliGemma']['llm']['final_norm_1']['Dense_0']['bias'].value: (3072,)@bfloat16
['PaliGemma']['llm']['final_norm_1']['Dense_0']['kernel'].value: (1024, 3072)@bfloat16
['PaliGemma']['llm']['layers']['attn']['attn_vec_einsum']['lora_a'].value: (18, 8, 256, 16)@float32
['PaliGemma']['llm']['layers']['attn']['attn_vec_einsum']['lora_b'].value: (18, 8, 16, 2048)@float32
['PaliGemma']['llm']['layers']['attn']['attn_vec_einsum']['w'].value: (18, 8, 256, 2048)@bfloat16
['PaliGemma']['llm']['layers']['attn']['attn_vec_einsum_1']['lora_a'].value: (18, 8, 256, 32)@float32
['PaliGemma']['llm']['layers']['attn']['attn_vec_einsum_1']['lora_b'].value: (18, 8, 32, 1024)@float32
['PaliGemma']['llm']['layers']['attn']['attn_vec_einsum_1']['w'].value: (18, 8, 256, 1024)@bfloat16
['PaliGemma']['llm']['layers']['attn']['kv_einsum']['lora_a'].value: (18, 2, 1, 2048, 16)@float32
['PaliGemma']['llm']['layers']['attn']['kv_einsum']['lora_b'].value: (18, 2, 1, 16, 256)@float32
['PaliGemma']['llm']['layers']['attn']['kv_einsum']['w'].value: (18, 2, 1, 2048, 256)@bfloat16
['PaliGemma']['llm']['layers']['attn']['kv_einsum_1']['lora_a'].value: (18, 2, 1, 1024, 32)@float32
['PaliGemma']['llm']['layers']['attn']['kv_einsum_1']['lora_b'].value: (18, 2, 1, 32, 256)@float32
['PaliGemma']['llm']['layers']['attn']['kv_einsum_1']['w'].value: (18, 2, 1, 1024, 256)@bfloat16
['PaliGemma']['llm']['layers']['attn']['q_einsum']['lora_a'].value: (18, 8, 2048, 16)@float32
['PaliGemma']['llm']['layers']['attn']['q_einsum']['lora_b'].value: (18, 8, 16, 256)@float32
['PaliGemma']['llm']['layers']['attn']['q_einsum']['w'].value: (18, 8, 2048, 256)@bfloat16
['PaliGemma']['llm']['layers']['attn']['q_einsum_1']['lora_a'].value: (18, 8, 1024, 32)@float32
['PaliGemma']['llm']['layers']['attn']['q_einsum_1']['lora_b'].value: (18, 8, 32, 256)@float32
['PaliGemma']['llm']['layers']['attn']['q_einsum_1']['w'].value: (18, 8, 1024, 256)@bfloat16
['PaliGemma']['llm']['layers']['mlp']['gating_einsum'].value: (18, 2, 2048, 16384)@bfloat16
['PaliGemma']['llm']['layers']['mlp']['gating_einsum_lora_a'].value: (18, 2, 2048, 16)@float32
['PaliGemma']['llm']['layers']['mlp']['gating_einsum_lora_b'].value: (18, 2, 16, 16384)@float32
['PaliGemma']['llm']['layers']['mlp']['linear'].value: (18, 16384, 2048)@bfloat16
['PaliGemma']['llm']['layers']['mlp']['linear_lora_a'].value: (18, 16384, 16)@float32
['PaliGemma']['llm']['layers']['mlp']['linear_lora_b'].value: (18, 16, 2048)@float32
['PaliGemma']['llm']['layers']['mlp_1']['gating_einsum'].value: (18, 2, 1024, 4096)@bfloat16
['PaliGemma']['llm']['layers']['mlp_1']['gating_einsum_lora_a'].value: (18, 2, 1024, 32)@float32
['PaliGemma']['llm']['layers']['mlp_1']['gating_einsum_lora_b'].value: (18, 2, 32, 4096)@float32
['PaliGemma']['llm']['layers']['mlp_1']['linear'].value: (18, 4096, 1024)@bfloat16
['PaliGemma']['llm']['layers']['mlp_1']['linear_lora_a'].value: (18, 4096, 32)@float32
['PaliGemma']['llm']['layers']['mlp_1']['linear_lora_b'].value: (18, 32, 1024)@float32
['PaliGemma']['llm']['layers']['pre_attention_norm']['scale'].value: (18, 2048)@bfloat16
['PaliGemma']['llm']['layers']['pre_attention_norm_1']['Dense_0']['bias'].value: (18, 3072)@bfloat16
['PaliGemma']['llm']['layers']['pre_attention_norm_1']['Dense_0']['kernel'].value: (18, 1024, 3072)@bfloat16
['PaliGemma']['llm']['layers']['pre_ffw_norm']['scale'].value: (18, 2048)@bfloat16
['PaliGemma']['llm']['layers']['pre_ffw_norm_1']['Dense_0']['bias'].value: (18, 3072)@bfloat16
['PaliGemma']['llm']['layers']['pre_ffw_norm_1']['Dense_0']['kernel'].value: (18, 1024, 3072)@bfloat16
['action_in_proj']['bias'].value: (1024,)@float32
['action_in_proj']['kernel'].value: (32, 1024)@float32
['action_out_proj']['bias'].value: (32,)@float32
['action_out_proj']['kernel'].value: (1024, 32)@float32
['time_mlp_in']['bias'].value: (1024,)@float32
['time_mlp_in']['kernel'].value: (1024, 1024)@float32
['time_mlp_out']['bias'].value: (1024,)@float32
['time_mlp_out']['kernel'].value: (1024, 1024)@float32 (1728:train.py:238)
06:46:11.211 [I] Progress on: -/3000 rate:- remaining:? elapsed:00:00 postfix:-                   (1728:tqdm_logging.py:145)


Step 0: grad_norm=0.8543, loss=0.1015, param_norm=1803.7705
06:50:45.728 [I] Progress on: -/3000 rate:- remaining:? elapsed:04:34 postfix:-                   (1728:tqdm_logging.py:145)
06:51:00.084 [I] Progress on: 4.00it/3.00kit rate:41.9s/it remaining:34:53:50 elapsed:04:48 postfix:- (1728:tqdm_logging.py:145)
06:51:14.071 [I] Progress on: 6.00it/3.00kit rate:21.7s/it remaining:18:04:40 elapsed:05:02 postfix:- (1728:tqdm_logging.py:145)
06:51:28.067 [I] Progress on: 8.00it/3.00kit rate:13.8s/it remaining:11:26:14 elapsed:05:16 postfix:- (1728:tqdm_logging.py:145)
06:51:42.099 [I] Progress on: 10.0it/3.00kit rate:10.2s/it remaining:8:29:21 elapsed:05:30 postfix:- (1728:tqdm_logging.py:145)
06:51:56.145 [I] Progress on: 12.0it/3.00kit rate:8.6s/it remaining:7:06:36 elapsed:05:44 postfix:- (1728:tqdm_logging.py:145)
06:52:10.225 [I] Progress on: 14.0it/3.00kit rate:7.8s/it remaining:6:27:18 elapsed:05:59 postfix:- (1728:tqdm_logging.py:145)
06:52:24.316 [I] Progress on: 16.0it/3.00kit rate:7.4s/it remaining:6:08:18 elapsed:06:13 postfix:- (1728:tqdm_logging.py:145)
06:52:38.390 [I] Progress on: 18.0it/3.00kit rate:7.2s/it remaining:5:58:41 elapsed:06:27 postfix:- (1728:tqdm_logging.py:145)
06:52:52.460 [I] Progress on: 20.0it/3.00kit rate:7.1s/it remaining:5:53:51 elapsed:06:41 postfix:- (1728:tqdm_logging.py:145)
06:53:06.544 [I] Progress on: 22.0it/3.00kit rate:7.1s/it remaining:5:51:31 elapsed:06:55 postfix:- (1728:tqdm_logging.py:145)
06:53:20.649 [I] Progress on: 24.0it/3.00kit rate:7.1s/it remaining:5:50:31 elapsed:07:09 postfix:- (1728:tqdm_logging.py:145)
06:53:34.755 [I] Progress on: 26.0it/3.00kit rate:7.1s/it remaining:5:49:56 elapsed:07:23 postfix:- (1728:tqdm_logging.py:145)
06:53:48.854 [I] Progress on: 28.0it/3.00kit rate:7.1s/it remaining:5:49:24 elapsed:07:37 postfix:- (1728:tqdm_logging.py:145)
06:54:02.968 [I] Progress on: 30.0it/3.00kit rate:7.1s/it remaining:5:49:16 elapsed:07:51 postfix:- (1728:tqdm_logging.py:145)
06:54:17.068 [I] Progress on: 32.0it/3.00kit rate:7.1s/it remaining:5:48:52 elapsed:08:05 postfix:- (1728:tqdm_logging.py:145)
06:54:31.173 [I] Progress on: 34.0it/3.00kit rate:7.1s/it remaining:5:48:36 elapsed:08:19 postfix:- (1728:tqdm_logging.py:145)
06:54:45.294 [I] Progress on: 36.0it/3.00kit rate:7.1s/it remaining:5:48:34 elapsed:08:34 postfix:- (1728:tqdm_logging.py:145)
06:54:59.380 [I] Progress on: 38.0it/3.00kit rate:7.0s/it remaining:5:47:59 elapsed:08:48 postfix:- (1728:tqdm_logging.py:145)
06:55:13.482 [I] Progress on: 40.0it/3.00kit rate:7.0s/it remaining:5:47:47 elapsed:09:02 postfix:- (1728:tqdm_logging.py:145)
06:55:27.611 [I] Progress on: 42.0it/3.00kit rate:7.1s/it remaining:5:47:54 elapsed:09:16 postfix:- (1728:tqdm_logging.py:145)
06:55:41.739 [I] Progress on: 44.0it/3.00kit rate:7.1s/it remaining:5:47:50 elapsed:09:30 postfix:- (1728:tqdm_logging.py:145)
06:55:55.877 [I] Progress on: 46.0it/3.00kit rate:7.1s/it remaining:5:47:45 elapsed:09:44 postfix:- (1728:tqdm_logging.py:145)
06:56:10.005 [I] Progress on: 48.0it/3.00kit rate:7.1s/it remaining:5:47:29 elapsed:09:58 postfix:- (1728:tqdm_logging.py:145)
06:56:24.151 [I] Progress on: 50.0it/3.00kit rate:7.1s/it remaining:5:47:30 elapsed:10:12 postfix:- (1728:tqdm_logging.py:145)


Step 50: grad_norm=0.3387, loss=0.0611, param_norm=1803.7704
06:56:38.069 [I] Progress on: 50.0it/3.00kit rate:7.1s/it remaining:5:47:30 elapsed:10:26 postfix:- (1728:tqdm_logging.py:145)
06:56:52.642 [I] Progress on: 54.0it/3.00kit rate:6.8s/it remaining:5:33:48 elapsed:10:41 postfix:- (1728:tqdm_logging.py:145)
06:57:06.824 [I] Progress on: 56.0it/3.00kit rate:6.9s/it remaining:5:40:55 elapsed:10:55 postfix:- (1728:tqdm_logging.py:145)
06:57:20.958 [I] Progress on: 58.0it/3.00kit rate:7.0s/it remaining:5:43:41 elapsed:11:09 postfix:- (1728:tqdm_logging.py:145)
06:57:35.129 [I] Progress on: 60.0it/3.00kit rate:7.0s/it remaining:5:45:19 elapsed:11:23 postfix:- (1728:tqdm_logging.py:145)
06:57:49.290 [I] Progress on: 62.0it/3.00kit rate:7.1s/it remaining:5:45:56 elapsed:11:38 postfix:- (1728:tqdm_logging.py:145)
06:58:03.442 [I] Progress on: 64.0it/3.00kit rate:7.1s/it remaining:5:46:00 elapsed:11:52 postfix:- (1728:tqdm_logging.py:145)
06:58:17.617 [I] Progress on: 66.0it/3.00kit rate:7.1s/it remaining:5:46:07 elapsed:12:06 postfix:- (1728:tqdm_logging.py:145)
06:58:31.789 [I] Progress on: 68.0it/3.00kit rate:7.1s/it remaining:5:46:04 elapsed:12:20 postfix:- (1728:tqdm_logging.py:145)
06:58:45.965 [I] Progress on: 70.0it/3.00kit rate:7.1s/it remaining:5:45:58 elapsed:12:34 postfix:- (1728:tqdm_logging.py:145)
06:59:00.128 [I] Progress on: 72.0it/3.00kit rate:7.1s/it remaining:5:45:38 elapsed:12:48 postfix:- (1728:tqdm_logging.py:145)
06:59:14.293 [I] Progress on: 74.0it/3.00kit rate:7.1s/it remaining:5:45:25 elapsed:13:03 postfix:- (1728:tqdm_logging.py:145)
06:59:28.466 [I] Progress on: 76.0it/3.00kit rate:7.1s/it remaining:5:45:17 elapsed:13:17 postfix:- (1728:tqdm_logging.py:145)
06:59:42.649 [I] Progress on: 78.0it/3.00kit rate:7.1s/it remaining:5:45:11 elapsed:13:31 postfix:- (1728:tqdm_logging.py:145)
06:59:56.836 [I] Progress on: 80.0it/3.00kit rate:7.1s/it remaining:5:45:04 elapsed:13:45 postfix:- (1728:tqdm_logging.py:145)
07:00:11.028 [I] Progress on: 82.0it/3.00kit rate:7.1s/it remaining:5:45:00 elapsed:13:59 postfix:- (1728:tqdm_logging.py:145)
07:00:25.183 [I] Progress on: 84.0it/3.00kit rate:7.1s/it remaining:5:44:24 elapsed:14:13 postfix:- (1728:tqdm_logging.py:145)
07:00:39.355 [I] Progress on: 86.0it/3.00kit rate:7.1s/it remaining:5:44:07 elapsed:14:28 postfix:- (1728:tqdm_logging.py:145)
07:00:53.521 [I] Progress on: 88.0it/3.00kit rate:7.1s/it remaining:5:43:49 elapsed:14:42 postfix:- (1728:tqdm_logging.py:145)
07:01:07.697 [I] Progress on: 90.0it/3.00kit rate:7.1s/it remaining:5:43:44 elapsed:14:56 postfix:- (1728:tqdm_logging.py:145)
07:01:21.831 [I] Progress on: 92.0it/3.00kit rate:7.1s/it remaining:5:43:02 elapsed:15:10 postfix:- (1728:tqdm_logging.py:145)
07:01:35.980 [I] Progress on: 94.0it/3.00kit rate:7.1s/it remaining:5:42:41 elapsed:15:24 postfix:- (1728:tqdm_logging.py:145)
07:01:50.132 [I] Progress on: 96.0it/3.00kit rate:7.1s/it remaining:5:42:28 elapsed:15:38 postfix:- (1728:tqdm_logging.py:145)
07:02:04.282 [I] Progress on: 98.0it/3.00kit rate:7.1s/it remaining:5:42:12 elapsed:15:53 postfix:- (1728:tqdm_logging.py:145)
07:02:18.441 [I] Progress on: 100it/3.00kit rate:7.1s/it remaining:5:42:02 elapsed:16:07 postfix:- (1728:tqdm_logging.py:145)


Step 100: grad_norm=0.2040, loss=0.0422, param_norm=1803.7709
07:02:32.238 [I] Progress on: 100it/3.00kit rate:7.1s/it remaining:5:42:02 elapsed:16:21 postfix:- (1728:tqdm_logging.py:145)
07:02:46.850 [I] Progress on: 104it/3.00kit rate:6.8s/it remaining:5:27:58 elapsed:16:35 postfix:- (1728:tqdm_logging.py:145)
07:03:01.010 [I] Progress on: 106it/3.00kit rate:6.9s/it remaining:5:34:43 elapsed:16:49 postfix:- (1728:tqdm_logging.py:145)
07:03:15.144 [I] Progress on: 108it/3.00kit rate:7.0s/it remaining:5:37:39 elapsed:17:03 postfix:- (1728:tqdm_logging.py:145)
07:03:29.301 [I] Progress on: 110it/3.00kit rate:7.0s/it remaining:5:39:17 elapsed:17:18 postfix:- (1728:tqdm_logging.py:145)
07:03:43.490 [I] Progress on: 112it/3.00kit rate:7.1s/it remaining:5:40:18 elapsed:17:32 postfix:- (1728:tqdm_logging.py:145)
07:03:57.660 [I] Progress on: 114it/3.00kit rate:7.1s/it remaining:5:40:03 elapsed:17:46 postfix:- (1728:tqdm_logging.py:145)
07:04:11.855 [I] Progress on: 116it/3.00kit rate:7.1s/it remaining:5:40:32 elapsed:18:00 postfix:- (1728:tqdm_logging.py:145)
07:04:26.003 [I] Progress on: 118it/3.00kit rate:7.1s/it remaining:5:40:04 elapsed:18:14 postfix:- (1728:tqdm_logging.py:145)
07:04:40.162 [I] Progress on: 120it/3.00kit rate:7.1s/it remaining:5:39:50 elapsed:18:28 postfix:- (1728:tqdm_logging.py:145)
07:04:54.353 [I] Progress on: 122it/3.00kit rate:7.1s/it remaining:5:39:59 elapsed:18:43 postfix:- (1728:tqdm_logging.py:145)
07:05:08.513 [I] Progress on: 124it/3.00kit rate:7.1s/it remaining:5:39:32 elapsed:18:57 postfix:- (1728:tqdm_logging.py:145)
07:05:22.665 [I] Progress on: 126it/3.00kit rate:7.1s/it remaining:5:39:09 elapsed:19:11 postfix:- (1728:tqdm_logging.py:145)
07:05:36.821 [I] Progress on: 128it/3.00kit rate:7.1s/it remaining:5:38:51 elapsed:19:25 postfix:- (1728:tqdm_logging.py:145)
07:05:50.992 [I] Progress on: 130it/3.00kit rate:7.1s/it remaining:5:38:47 elapsed:19:39 postfix:- (1728:tqdm_logging.py:145)
07:06:05.175 [I] Progress on: 132it/3.00kit rate:7.1s/it remaining:5:38:45 elapsed:19:53 postfix:- (1728:tqdm_logging.py:145)
07:06:19.345 [I] Progress on: 134it/3.00kit rate:7.1s/it remaining:5:38:28 elapsed:20:08 postfix:- (1728:tqdm_logging.py:145)
07:06:33.524 [I] Progress on: 136it/3.00kit rate:7.1s/it remaining:5:38:19 elapsed:20:22 postfix:- (1728:tqdm_logging.py:145)
07:06:47.687 [I] Progress on: 138it/3.00kit rate:7.1s/it remaining:5:37:56 elapsed:20:36 postfix:- (1728:tqdm_logging.py:145)
07:07:01.867 [I] Progress on: 140it/3.00kit rate:7.1s/it remaining:5:37:51 elapsed:20:50 postfix:- (1728:tqdm_logging.py:145)
07:07:16.031 [I] Progress on: 142it/3.00kit rate:7.1s/it remaining:5:37:30 elapsed:21:04 postfix:- (1728:tqdm_logging.py:145)
07:07:30.191 [I] Progress on: 144it/3.00kit rate:7.1s/it remaining:5:37:07 elapsed:21:18 postfix:- (1728:tqdm_logging.py:145)
07:07:44.352 [I] Progress on: 146it/3.00kit rate:7.1s/it remaining:5:36:50 elapsed:21:33 postfix:- (1728:tqdm_logging.py:145)
07:07:58.514 [I] Progress on: 148it/3.00kit rate:7.1s/it remaining:5:36:33 elapsed:21:47 postfix:- (1728:tqdm_logging.py:145)
07:08:12.671 [I] Progress on: 150it/3.00kit rate:7.1s/it remaining:5:36:15 elapsed:22:01 postfix:- (1728:tqdm_logging.py:145)


Step 150: grad_norm=0.1744, loss=0.0296, param_norm=1803.7731
07:08:26.480 [I] Progress on: 150it/3.00kit rate:7.1s/it remaining:5:36:15 elapsed:22:15 postfix:- (1728:tqdm_logging.py:145)
07:08:41.113 [I] Progress on: 154it/3.00kit rate:6.8s/it remaining:5:22:32 elapsed:22:29 postfix:- (1728:tqdm_logging.py:145)
07:08:55.255 [I] Progress on: 156it/3.00kit rate:6.9s/it remaining:5:28:52 elapsed:22:44 postfix:- (1728:tqdm_logging.py:145)
07:09:09.403 [I] Progress on: 158it/3.00kit rate:7.0s/it remaining:5:31:54 elapsed:22:58 postfix:- (1728:tqdm_logging.py:145)
07:09:23.576 [I] Progress on: 160it/3.00kit rate:7.0s/it remaining:5:33:35 elapsed:23:12 postfix:- (1728:tqdm_logging.py:145)
07:09:37.722 [I] Progress on: 162it/3.00kit rate:7.1s/it remaining:5:34:00 elapsed:23:26 postfix:- (1728:tqdm_logging.py:145)
07:09:51.866 [I] Progress on: 164it/3.00kit rate:7.1s/it remaining:5:34:01 elapsed:23:40 postfix:- (1728:tqdm_logging.py:145)
07:10:06.043 [I] Progress on: 166it/3.00kit rate:7.1s/it remaining:5:34:19 elapsed:23:54 postfix:- (1728:tqdm_logging.py:145)
07:10:20.195 [I] Progress on: 168it/3.00kit rate:7.1s/it remaining:5:34:02 elapsed:24:08 postfix:- (1728:tqdm_logging.py:145)
07:10:34.354 [I] Progress on: 170it/3.00kit rate:7.1s/it remaining:5:33:52 elapsed:24:23 postfix:- (1728:tqdm_logging.py:145)
07:10:48.507 [I] Progress on: 172it/3.00kit rate:7.1s/it remaining:5:33:37 elapsed:24:37 postfix:- (1728:tqdm_logging.py:145)
07:11:02.655 [I] Progress on: 174it/3.00kit rate:7.1s/it remaining:5:33:18 elapsed:24:51 postfix:- (1728:tqdm_logging.py:145)
07:11:16.815 [I] Progress on: 176it/3.00kit rate:7.1s/it remaining:5:33:11 elapsed:25:05 postfix:- (1728:tqdm_logging.py:145)
07:11:30.969 [I] Progress on: 178it/3.00kit rate:7.1s/it remaining:5:32:56 elapsed:25:19 postfix:- (1728:tqdm_logging.py:145)
07:11:45.109 [I] Progress on: 180it/3.00kit rate:7.1s/it remaining:5:32:30 elapsed:25:33 postfix:- (1728:tqdm_logging.py:145)
07:11:59.283 [I] Progress on: 182it/3.00kit rate:7.1s/it remaining:5:32:33 elapsed:25:48 postfix:- (1728:tqdm_logging.py:145)
07:12:13.441 [I] Progress on: 184it/3.00kit rate:7.1s/it remaining:5:32:18 elapsed:26:02 postfix:- (1728:tqdm_logging.py:145)
07:12:27.583 [I] Progress on: 186it/3.00kit rate:7.1s/it remaining:5:31:51 elapsed:26:16 postfix:- (1728:tqdm_logging.py:145)
07:12:41.747 [I] Progress on: 188it/3.00kit rate:7.1s/it remaining:5:31:46 elapsed:26:30 postfix:- (1728:tqdm_logging.py:145)
07:12:55.902 [I] Progress on: 190it/3.00kit rate:7.1s/it remaining:5:31:29 elapsed:26:44 postfix:- (1728:tqdm_logging.py:145)
07:13:10.054 [I] Progress on: 192it/3.00kit rate:7.1s/it remaining:5:31:13 elapsed:26:58 postfix:- (1728:tqdm_logging.py:145)
07:13:24.199 [I] Progress on: 194it/3.00kit rate:7.1s/it remaining:5:30:52 elapsed:27:12 postfix:- (1728:tqdm_logging.py:145)
07:13:38.389 [I] Progress on: 196it/3.00kit rate:7.1s/it remaining:5:31:05 elapsed:27:27 postfix:- (1728:tqdm_logging.py:145)
07:13:52.549 [I] Progress on: 198it/3.00kit rate:7.1s/it remaining:5:30:43 elapsed:27:41 postfix:- (1728:tqdm_logging.py:145)
07:14:06.708 [I] Progress on: 200it/3.00kit rate:7.1s/it remaining:5:30:25 elapsed:27:55 postfix:- (1728:tqdm_logging.py:145)


Step 200: grad_norm=0.1406, loss=0.0213, param_norm=1803.7776
07:14:20.525 [I] Progress on: 200it/3.00kit rate:7.1s/it remaining:5:30:25 elapsed:28:09 postfix:- (1728:tqdm_logging.py:145)
07:14:35.159 [I] Progress on: 204it/3.00kit rate:6.8s/it remaining:5:16:55 elapsed:28:23 postfix:- (1728:tqdm_logging.py:145)
07:14:49.316 [I] Progress on: 206it/3.00kit rate:6.9s/it remaining:5:23:18 elapsed:28:38 postfix:- (1728:tqdm_logging.py:145)
07:15:03.471 [I] Progress on: 208it/3.00kit rate:7.0s/it remaining:5:26:19 elapsed:28:52 postfix:- (1728:tqdm_logging.py:145)
07:15:17.656 [I] Progress on: 210it/3.00kit rate:7.1s/it remaining:5:28:00 elapsed:29:06 postfix:- (1728:tqdm_logging.py:145)
07:15:31.802 [I] Progress on: 212it/3.00kit rate:7.1s/it remaining:5:28:15 elapsed:29:20 postfix:- (1728:tqdm_logging.py:145)
07:15:45.963 [I] Progress on: 214it/3.00kit rate:7.1s/it remaining:5:28:22 elapsed:29:34 postfix:- (1728:tqdm_logging.py:145)
07:16:00.122 [I] Progress on: 216it/3.00kit rate:7.1s/it remaining:5:28:16 elapsed:29:48 postfix:- (1728:tqdm_logging.py:145)
07:16:14.279 [I] Progress on: 218it/3.00kit rate:7.1s/it remaining:5:28:09 elapsed:30:03 postfix:- (1728:tqdm_logging.py:145)
07:16:28.436 [I] Progress on: 220it/3.00kit rate:7.1s/it remaining:5:27:53 elapsed:30:17 postfix:- (1728:tqdm_logging.py:145)
07:16:43.178 [I] Progress on: 222it/3.00kit rate:7.3s/it remaining:5:35:48 elapsed:30:31 postfix:- (1728:tqdm_logging.py:145)
07:16:56.727 [I] Progress on: 224it/3.00kit rate:7.0s/it remaining:5:25:36 elapsed:30:45 postfix:- (1728:tqdm_logging.py:145)
07:17:10.889 [I] Progress on: 226it/3.00kit rate:7.1s/it remaining:5:26:23 elapsed:30:59 postfix:- (1728:tqdm_logging.py:145)
07:17:25.038 [I] Progress on: 228it/3.00kit rate:7.1s/it remaining:5:26:30 elapsed:31:13 postfix:- (1728:tqdm_logging.py:145)
07:17:39.193 [I] Progress on: 230it/3.00kit rate:7.1s/it remaining:5:26:30 elapsed:31:27 postfix:- (1728:tqdm_logging.py:145)
07:17:53.349 [I] Progress on: 232it/3.00kit rate:7.1s/it remaining:5:26:20 elapsed:31:42 postfix:- (1728:tqdm_logging.py:145)
07:18:07.525 [I] Progress on: 234it/3.00kit rate:7.1s/it remaining:5:26:27 elapsed:31:56 postfix:- (1728:tqdm_logging.py:145)
07:18:21.679 [I] Progress on: 236it/3.00kit rate:7.1s/it remaining:5:26:07 elapsed:32:10 postfix:- (1728:tqdm_logging.py:145)
07:18:35.829 [I] Progress on: 238it/3.00kit rate:7.1s/it remaining:5:25:44 elapsed:32:24 postfix:- (1728:tqdm_logging.py:145)
07:18:49.997 [I] Progress on: 240it/3.00kit rate:7.1s/it remaining:5:25:42 elapsed:32:38 postfix:- (1728:tqdm_logging.py:145)
07:19:04.154 [I] Progress on: 242it/3.00kit rate:7.1s/it remaining:5:25:27 elapsed:32:52 postfix:- (1728:tqdm_logging.py:145)
07:19:18.316 [I] Progress on: 244it/3.00kit rate:7.1s/it remaining:5:25:17 elapsed:33:07 postfix:- (1728:tqdm_logging.py:145)
07:19:32.461 [I] Progress on: 246it/3.00kit rate:7.1s/it remaining:5:24:50 elapsed:33:21 postfix:- (1728:tqdm_logging.py:145)
07:19:46.639 [I] Progress on: 248it/3.00kit rate:7.1s/it remaining:5:24:54 elapsed:33:35 postfix:- (1728:tqdm_logging.py:145)
07:20:00.788 [I] Progress on: 250it/3.00kit rate:7.1s/it remaining:5:24:27 elapsed:33:49 postfix:- (1728:tqdm_logging.py:145)


Step 250: grad_norm=0.1348, loss=0.0186, param_norm=1803.7841
07:20:14.582 [I] Progress on: 250it/3.00kit rate:7.1s/it remaining:5:24:27 elapsed:34:03 postfix:- (1728:tqdm_logging.py:145)
07:20:29.164 [I] Progress on: 254it/3.00kit rate:6.8s/it remaining:5:10:47 elapsed:34:17 postfix:- (1728:tqdm_logging.py:145)
07:20:43.332 [I] Progress on: 256it/3.00kit rate:6.9s/it remaining:5:17:21 elapsed:34:32 postfix:- (1728:tqdm_logging.py:145)
07:20:57.496 [I] Progress on: 258it/3.00kit rate:7.0s/it remaining:5:20:26 elapsed:34:46 postfix:- (1728:tqdm_logging.py:145)
07:21:11.651 [I] Progress on: 260it/3.00kit rate:7.0s/it remaining:5:21:44 elapsed:35:00 postfix:- (1728:tqdm_logging.py:145)
07:21:25.802 [I] Progress on: 262it/3.00kit rate:7.1s/it remaining:5:22:13 elapsed:35:14 postfix:- (1728:tqdm_logging.py:145)
07:21:39.965 [I] Progress on: 264it/3.00kit rate:7.1s/it remaining:5:22:29 elapsed:35:28 postfix:- (1728:tqdm_logging.py:145)
07:21:54.120 [I] Progress on: 266it/3.00kit rate:7.1s/it remaining:5:22:25 elapsed:35:42 postfix:- (1728:tqdm_logging.py:145)
07:22:08.255 [I] Progress on: 268it/3.00kit rate:7.1s/it remaining:5:22:00 elapsed:35:57 postfix:- (1728:tqdm_logging.py:145)
07:22:22.434 [I] Progress on: 270it/3.00kit rate:7.1s/it remaining:5:22:11 elapsed:36:11 postfix:- (1728:tqdm_logging.py:145)
07:22:36.592 [I] Progress on: 272it/3.00kit rate:7.1s/it remaining:5:21:57 elapsed:36:25 postfix:- (1728:tqdm_logging.py:145)
07:22:50.749 [I] Progress on: 274it/3.00kit rate:7.1s/it remaining:5:21:42 elapsed:36:39 postfix:- (1728:tqdm_logging.py:145)
07:23:04.874 [I] Progress on: 276it/3.00kit rate:7.1s/it remaining:5:21:04 elapsed:36:53 postfix:- (1728:tqdm_logging.py:145)
07:23:19.012 [I] Progress on: 278it/3.00kit rate:7.1s/it remaining:5:20:47 elapsed:37:07 postfix:- (1728:tqdm_logging.py:145)
07:23:33.132 [I] Progress on: 280it/3.00kit rate:7.1s/it remaining:5:20:20 elapsed:37:21 postfix:- (1728:tqdm_logging.py:145)
07:23:47.302 [I] Progress on: 282it/3.00kit rate:7.1s/it remaining:5:20:32 elapsed:37:36 postfix:- (1728:tqdm_logging.py:145)
07:24:01.457 [I] Progress on: 284it/3.00kit rate:7.1s/it remaining:5:20:22 elapsed:37:50 postfix:- (1728:tqdm_logging.py:145)
07:24:15.592 [I] Progress on: 286it/3.00kit rate:7.1s/it remaining:5:19:55 elapsed:38:04 postfix:- (1728:tqdm_logging.py:145)
07:24:29.761 [I] Progress on: 288it/3.00kit rate:7.1s/it remaining:5:19:57 elapsed:38:18 postfix:- (1728:tqdm_logging.py:145)
07:24:43.906 [I] Progress on: 290it/3.00kit rate:7.1s/it remaining:5:19:34 elapsed:38:32 postfix:- (1728:tqdm_logging.py:145)
07:24:58.058 [I] Progress on: 292it/3.00kit rate:7.1s/it remaining:5:19:22 elapsed:38:46 postfix:- (1728:tqdm_logging.py:145)
07:25:12.197 [I] Progress on: 294it/3.00kit rate:7.1s/it remaining:5:18:59 elapsed:39:00 postfix:- (1728:tqdm_logging.py:145)
07:25:26.336 [I] Progress on: 296it/3.00kit rate:7.1s/it remaining:5:18:41 elapsed:39:15 postfix:- (1728:tqdm_logging.py:145)
07:25:40.482 [I] Progress on: 298it/3.00kit rate:7.1s/it remaining:5:18:32 elapsed:39:29 postfix:- (1728:tqdm_logging.py:145)
07:25:54.626 [I] Progress on: 300it/3.00kit rate:7.1s/it remaining:5:18:16 elapsed:39:43 postfix:- (1728:tqdm_logging.py:145)


Step 300: grad_norm=0.1203, loss=0.0169, param_norm=1803.7925
07:26:08.423 [I] Progress on: 300it/3.00kit rate:7.1s/it remaining:5:18:16 elapsed:39:57 postfix:- (1728:tqdm_logging.py:145)
07:26:23.007 [I] Progress on: 304it/3.00kit rate:6.8s/it remaining:5:05:00 elapsed:40:11 postfix:- (1728:tqdm_logging.py:145)
07:26:37.149 [I] Progress on: 306it/3.00kit rate:6.9s/it remaining:5:11:16 elapsed:40:25 postfix:- (1728:tqdm_logging.py:145)
07:26:51.293 [I] Progress on: 308it/3.00kit rate:7.0s/it remaining:5:14:13 elapsed:40:40 postfix:- (1728:tqdm_logging.py:145)
07:27:05.439 [I] Progress on: 310it/3.00kit rate:7.0s/it remaining:5:15:32 elapsed:40:54 postfix:- (1728:tqdm_logging.py:145)
07:27:19.574 [I] Progress on: 312it/3.00kit rate:7.1s/it remaining:5:15:56 elapsed:41:08 postfix:- (1728:tqdm_logging.py:145)
07:27:33.722 [I] Progress on: 314it/3.00kit rate:7.1s/it remaining:5:16:10 elapsed:41:22 postfix:- (1728:tqdm_logging.py:145)
07:27:47.865 [I] Progress on: 316it/3.00kit rate:7.1s/it remaining:5:16:07 elapsed:41:36 postfix:- (1728:tqdm_logging.py:145)
07:28:02.005 [I] Progress on: 318it/3.00kit rate:7.1s/it remaining:5:15:56 elapsed:41:50 postfix:- (1728:tqdm_logging.py:145)
07:28:16.138 [I] Progress on: 320it/3.00kit rate:7.1s/it remaining:5:15:40 elapsed:42:04 postfix:- (1728:tqdm_logging.py:145)
07:28:30.284 [I] Progress on: 322it/3.00kit rate:7.1s/it remaining:5:15:31 elapsed:42:19 postfix:- (1728:tqdm_logging.py:145)
07:28:44.439 [I] Progress on: 324it/3.00kit rate:7.1s/it remaining:5:15:27 elapsed:42:33 postfix:- (1728:tqdm_logging.py:145)
07:28:58.586 [I] Progress on: 326it/3.00kit rate:7.1s/it remaining:5:15:13 elapsed:42:47 postfix:- (1728:tqdm_logging.py:145)
07:29:12.737 [I] Progress on: 328it/3.00kit rate:7.1s/it remaining:5:15:00 elapsed:43:01 postfix:- (1728:tqdm_logging.py:145)
07:29:26.889 [I] Progress on: 330it/3.00kit rate:7.1s/it remaining:5:14:49 elapsed:43:15 postfix:- (1728:tqdm_logging.py:145)
07:29:41.026 [I] Progress on: 332it/3.00kit rate:7.1s/it remaining:5:14:25 elapsed:43:29 postfix:- (1728:tqdm_logging.py:145)
07:29:55.183 [I] Progress on: 334it/3.00kit rate:7.1s/it remaining:5:14:20 elapsed:43:43 postfix:- (1728:tqdm_logging.py:145)
07:30:09.349 [I] Progress on: 336it/3.00kit rate:7.1s/it remaining:5:14:17 elapsed:43:58 postfix:- (1728:tqdm_logging.py:145)
07:30:23.490 [I] Progress on: 338it/3.00kit rate:7.1s/it remaining:5:13:50 elapsed:44:12 postfix:- (1728:tqdm_logging.py:145)
07:30:37.635 [I] Progress on: 340it/3.00kit rate:7.1s/it remaining:5:13:33 elapsed:44:26 postfix:- (1728:tqdm_logging.py:145)
07:30:51.781 [I] Progress on: 342it/3.00kit rate:7.1s/it remaining:5:13:18 elapsed:44:40 postfix:- (1728:tqdm_logging.py:145)
07:31:05.930 [I] Progress on: 344it/3.00kit rate:7.1s/it remaining:5:13:04 elapsed:44:54 postfix:- (1728:tqdm_logging.py:145)
07:31:20.076 [I] Progress on: 346it/3.00kit rate:7.1s/it remaining:5:12:50 elapsed:45:08 postfix:- (1728:tqdm_logging.py:145)
07:31:34.217 [I] Progress on: 348it/3.00kit rate:7.1s/it remaining:5:12:31 elapsed:45:23 postfix:- (1728:tqdm_logging.py:145)
07:31:48.368 [I] Progress on: 350it/3.00kit rate:7.1s/it remaining:5:12:22 elapsed:45:37 postfix:- (1728:tqdm_logging.py:145)


Step 350: grad_norm=0.0930, loss=0.0143, param_norm=1803.8036
07:32:02.141 [I] Progress on: 350it/3.00kit rate:7.1s/it remaining:5:12:22 elapsed:45:50 postfix:- (1728:tqdm_logging.py:145)
07:32:16.718 [I] Progress on: 354it/3.00kit rate:6.8s/it remaining:4:59:15 elapsed:46:05 postfix:- (1728:tqdm_logging.py:145)
07:32:30.853 [I] Progress on: 356it/3.00kit rate:6.9s/it remaining:5:05:25 elapsed:46:19 postfix:- (1728:tqdm_logging.py:145)
07:32:45.011 [I] Progress on: 358it/3.00kit rate:7.0s/it remaining:5:08:33 elapsed:46:33 postfix:- (1728:tqdm_logging.py:145)
07:32:59.159 [I] Progress on: 360it/3.00kit rate:7.0s/it remaining:5:09:50 elapsed:46:47 postfix:- (1728:tqdm_logging.py:145)
07:33:13.331 [I] Progress on: 362it/3.00kit rate:7.1s/it remaining:5:10:36 elapsed:47:02 postfix:- (1728:tqdm_logging.py:145)
07:33:27.498 [I] Progress on: 364it/3.00kit rate:7.1s/it remaining:5:10:48 elapsed:47:16 postfix:- (1728:tqdm_logging.py:145)
07:33:41.649 [I] Progress on: 366it/3.00kit rate:7.1s/it remaining:5:10:34 elapsed:47:30 postfix:- (1728:tqdm_logging.py:145)
07:33:55.816 [I] Progress on: 368it/3.00kit rate:7.1s/it remaining:5:10:32 elapsed:47:44 postfix:- (1728:tqdm_logging.py:145)
07:34:09.954 [I] Progress on: 370it/3.00kit rate:7.1s/it remaining:5:10:04 elapsed:47:58 postfix:- (1728:tqdm_logging.py:145)
07:34:24.091 [I] Progress on: 372it/3.00kit rate:7.1s/it remaining:5:09:43 elapsed:48:12 postfix:- (1728:tqdm_logging.py:145)
07:34:38.260 [I] Progress on: 374it/3.00kit rate:7.1s/it remaining:5:09:49 elapsed:48:27 postfix:- (1728:tqdm_logging.py:145)
07:34:52.420 [I] Progress on: 376it/3.00kit rate:7.1s/it remaining:5:09:36 elapsed:48:41 postfix:- (1728:tqdm_logging.py:145)
07:35:06.574 [I] Progress on: 378it/3.00kit rate:7.1s/it remaining:5:09:19 elapsed:48:55 postfix:- (1728:tqdm_logging.py:145)
07:35:20.714 [I] Progress on: 380it/3.00kit rate:7.1s/it remaining:5:08:55 elapsed:49:09 postfix:- (1728:tqdm_logging.py:145)
07:35:34.892 [I] Progress on: 382it/3.00kit rate:7.1s/it remaining:5:09:02 elapsed:49:23 postfix:- (1728:tqdm_logging.py:145)
07:35:49.046 [I] Progress on: 384it/3.00kit rate:7.1s/it remaining:5:08:43 elapsed:49:37 postfix:- (1728:tqdm_logging.py:145)
07:36:03.192 [I] Progress on: 386it/3.00kit rate:7.1s/it remaining:5:08:20 elapsed:49:51 postfix:- (1728:tqdm_logging.py:145)
07:36:17.333 [I] Progress on: 388it/3.00kit rate:7.1s/it remaining:5:07:58 elapsed:50:06 postfix:- (1728:tqdm_logging.py:145)
07:36:31.451 [I] Progress on: 390it/3.00kit rate:7.1s/it remaining:5:07:24 elapsed:50:20 postfix:- (1728:tqdm_logging.py:145)
07:36:45.617 [I] Progress on: 392it/3.00kit rate:7.1s/it remaining:5:07:32 elapsed:50:34 postfix:- (1728:tqdm_logging.py:145)
07:36:59.769 [I] Progress on: 394it/3.00kit rate:7.1s/it remaining:5:07:22 elapsed:50:48 postfix:- (1728:tqdm_logging.py:145)
07:37:13.909 [I] Progress on: 396it/3.00kit rate:7.1s/it remaining:5:07:00 elapsed:51:02 postfix:- (1728:tqdm_logging.py:145)
07:37:28.032 [I] Progress on: 398it/3.00kit rate:7.1s/it remaining:5:06:30 elapsed:51:16 postfix:- (1728:tqdm_logging.py:145)
07:37:42.139 [I] Progress on: 400it/3.00kit rate:7.1s/it remaining:5:05:59 elapsed:51:30 postfix:- (1728:tqdm_logging.py:145)


Step 400: grad_norm=0.0826, loss=0.0124, param_norm=1803.8142
07:37:55.913 [I] Progress on: 400it/3.00kit rate:7.1s/it remaining:5:05:59 elapsed:51:44 postfix:- (1728:tqdm_logging.py:145)
07:38:10.465 [I] Progress on: 404it/3.00kit rate:6.8s/it remaining:4:53:13 elapsed:51:59 postfix:- (1728:tqdm_logging.py:145)
07:38:24.613 [I] Progress on: 406it/3.00kit rate:6.9s/it remaining:4:59:29 elapsed:52:13 postfix:- (1728:tqdm_logging.py:145)
07:38:38.765 [I] Progress on: 408it/3.00kit rate:7.0s/it remaining:5:02:34 elapsed:52:27 postfix:- (1728:tqdm_logging.py:145)
07:38:52.895 [I] Progress on: 410it/3.00kit rate:7.0s/it remaining:5:03:37 elapsed:52:41 postfix:- (1728:tqdm_logging.py:145)
07:39:07.034 [I] Progress on: 412it/3.00kit rate:7.1s/it remaining:5:04:08 elapsed:52:55 postfix:- (1728:tqdm_logging.py:145)
07:39:21.185 [I] Progress on: 414it/3.00kit rate:7.1s/it remaining:5:04:26 elapsed:53:09 postfix:- (1728:tqdm_logging.py:145)
07:39:35.340 [I] Progress on: 416it/3.00kit rate:7.1s/it remaining:5:04:28 elapsed:53:24 postfix:- (1728:tqdm_logging.py:145)
07:39:49.495 [I] Progress on: 418it/3.00kit rate:7.1s/it remaining:5:04:23 elapsed:53:38 postfix:- (1728:tqdm_logging.py:145)
07:40:03.639 [I] Progress on: 420it/3.00kit rate:7.1s/it remaining:5:04:04 elapsed:53:52 postfix:- (1728:tqdm_logging.py:145)
07:40:17.783 [I] Progress on: 422it/3.00kit rate:7.1s/it remaining:5:03:49 elapsed:54:06 postfix:- (1728:tqdm_logging.py:145)
07:40:31.947 [I] Progress on: 424it/3.00kit rate:7.1s/it remaining:5:03:47 elapsed:54:20 postfix:- (1728:tqdm_logging.py:145)
07:40:46.097 [I] Progress on: 426it/3.00kit rate:7.1s/it remaining:5:03:31 elapsed:54:34 postfix:- (1728:tqdm_logging.py:145)
07:41:00.260 [I] Progress on: 428it/3.00kit rate:7.1s/it remaining:5:03:23 elapsed:54:49 postfix:- (1728:tqdm_logging.py:145)
07:41:14.403 [I] Progress on: 430it/3.00kit rate:7.1s/it remaining:5:02:58 elapsed:55:03 postfix:- (1728:tqdm_logging.py:145)
07:41:28.561 [I] Progress on: 432it/3.00kit rate:7.1s/it remaining:5:02:48 elapsed:55:17 postfix:- (1728:tqdm_logging.py:145)
07:41:42.704 [I] Progress on: 434it/3.00kit rate:7.1s/it remaining:5:02:30 elapsed:55:31 postfix:- (1728:tqdm_logging.py:145)
07:41:56.859 [I] Progress on: 436it/3.00kit rate:7.1s/it remaining:5:02:19 elapsed:55:45 postfix:- (1728:tqdm_logging.py:145)
07:42:10.992 [I] Progress on: 438it/3.00kit rate:7.1s/it remaining:5:01:53 elapsed:55:59 postfix:- (1728:tqdm_logging.py:145)
07:42:25.132 [I] Progress on: 440it/3.00kit rate:7.1s/it remaining:5:01:38 elapsed:56:13 postfix:- (1728:tqdm_logging.py:145)
07:42:39.273 [I] Progress on: 442it/3.00kit rate:7.1s/it remaining:5:01:24 elapsed:56:28 postfix:- (1728:tqdm_logging.py:145)
07:42:53.984 [I] Progress on: 444it/3.00kit rate:7.2s/it remaining:5:08:26 elapsed:56:42 postfix:- (1728:tqdm_logging.py:145)
07:43:07.598 [I] Progress on: 446it/3.00kit rate:7.0s/it remaining:4:59:58 elapsed:56:56 postfix:- (1728:tqdm_logging.py:145)
07:43:21.753 [I] Progress on: 448it/3.00kit rate:7.1s/it remaining:5:00:32 elapsed:57:10 postfix:- (1728:tqdm_logging.py:145)
07:43:35.909 [I] Progress on: 450it/3.00kit rate:7.1s/it remaining:5:00:43 elapsed:57:24 postfix:- (1728:tqdm_logging.py:145)


Step 450: grad_norm=0.0762, loss=0.0126, param_norm=1803.8242
07:43:49.712 [I] Progress on: 450it/3.00kit rate:7.1s/it remaining:5:00:43 elapsed:57:38 postfix:- (1728:tqdm_logging.py:145)
07:44:04.268 [I] Progress on: 454it/3.00kit rate:6.8s/it remaining:4:47:54 elapsed:57:53 postfix:- (1728:tqdm_logging.py:145)
07:44:18.419 [I] Progress on: 456it/3.00kit rate:6.9s/it remaining:4:53:56 elapsed:58:07 postfix:- (1728:tqdm_logging.py:145)
07:44:32.564 [I] Progress on: 458it/3.00kit rate:7.0s/it remaining:4:56:42 elapsed:58:21 postfix:- (1728:tqdm_logging.py:145)
07:44:46.712 [I] Progress on: 460it/3.00kit rate:7.0s/it remaining:4:57:57 elapsed:58:35 postfix:- (1728:tqdm_logging.py:145)
07:45:00.859 [I] Progress on: 462it/3.00kit rate:7.1s/it remaining:4:58:25 elapsed:58:49 postfix:- (1728:tqdm_logging.py:145)
07:45:15.016 [I] Progress on: 464it/3.00kit rate:7.1s/it remaining:4:58:40 elapsed:59:03 postfix:- (1728:tqdm_logging.py:145)
07:45:29.172 [I] Progress on: 466it/3.00kit rate:7.1s/it remaining:4:58:38 elapsed:59:17 postfix:- (1728:tqdm_logging.py:145)
07:45:43.317 [I] Progress on: 468it/3.00kit rate:7.1s/it remaining:4:58:24 elapsed:59:32 postfix:- (1728:tqdm_logging.py:145)
07:45:57.454 [I] Progress on: 470it/3.00kit rate:7.1s/it remaining:4:58:04 elapsed:59:46 postfix:- (1728:tqdm_logging.py:145)
07:46:11.603 [I] Progress on: 472it/3.00kit rate:7.1s/it remaining:4:57:55 elapsed:1:00:00 postfix:- (1728:tqdm_logging.py:145)
07:46:25.742 [I] Progress on: 474it/3.00kit rate:7.1s/it remaining:4:57:38 elapsed:1:00:14 postfix:- (1728:tqdm_logging.py:145)
07:46:39.897 [I] Progress on: 476it/3.00kit rate:7.1s/it remaining:4:57:36 elapsed:1:00:28 postfix:- (1728:tqdm_logging.py:145)
07:46:54.047 [I] Progress on: 478it/3.00kit rate:7.1s/it remaining:4:57:24 elapsed:1:00:42 postfix:- (1728:tqdm_logging.py:145)
07:47:08.203 [I] Progress on: 480it/3.00kit rate:7.1s/it remaining:4:57:11 elapsed:1:00:56 postfix:- (1728:tqdm_logging.py:145)
07:47:22.366 [I] Progress on: 482it/3.00kit rate:7.1s/it remaining:4:57:06 elapsed:1:01:11 postfix:- (1728:tqdm_logging.py:145)
07:47:36.480 [I] Progress on: 484it/3.00kit rate:7.1s/it remaining:4:56:24 elapsed:1:01:25 postfix:- (1728:tqdm_logging.py:145)
07:47:50.593 [I] Progress on: 486it/3.00kit rate:7.1s/it remaining:4:55:56 elapsed:1:01:39 postfix:- (1728:tqdm_logging.py:145)
07:48:04.710 [I] Progress on: 488it/3.00kit rate:7.1s/it remaining:4:55:38 elapsed:1:01:53 postfix:- (1728:tqdm_logging.py:145)
07:48:18.807 [I] Progress on: 490it/3.00kit rate:7.1s/it remaining:4:55:08 elapsed:1:02:07 postfix:- (1728:tqdm_logging.py:145)
07:48:32.937 [I] Progress on: 492it/3.00kit rate:7.1s/it remaining:4:55:07 elapsed:1:02:21 postfix:- (1728:tqdm_logging.py:145)
07:48:47.054 [I] Progress on: 494it/3.00kit rate:7.1s/it remaining:4:54:51 elapsed:1:02:35 postfix:- (1728:tqdm_logging.py:145)
07:49:01.155 [I] Progress on: 496it/3.00kit rate:7.1s/it remaining:4:54:25 elapsed:1:02:49 postfix:- (1728:tqdm_logging.py:145)
07:49:15.284 [I] Progress on: 498it/3.00kit rate:7.1s/it remaining:4:54:24 elapsed:1:03:04 postfix:- (1728:tqdm_logging.py:145)
07:49:29.417 [I] Progress on: 500it/3.00kit rate:7.1s/it remaining:4:54:17 elapsed:1:03:18 postfix:- (1728:tqdm_logging.py:145)


Step 500: grad_norm=0.0669, loss=0.0113, param_norm=1803.8344
07:49:43.166 [I] Progress on: 500it/3.00kit rate:7.1s/it remaining:4:54:17 elapsed:1:03:31 postfix:- (1728:tqdm_logging.py:145)
07:49:57.687 [I] Progress on: 504it/3.00kit rate:6.8s/it remaining:4:41:33 elapsed:1:03:46 postfix:- (1728:tqdm_logging.py:145)
07:50:11.801 [I] Progress on: 506it/3.00kit rate:6.9s/it remaining:4:47:24 elapsed:1:04:00 postfix:- (1728:tqdm_logging.py:145)
07:50:25.929 [I] Progress on: 508it/3.00kit rate:7.0s/it remaining:4:50:19 elapsed:1:04:14 postfix:- (1728:tqdm_logging.py:145)
07:50:40.057 [I] Progress on: 510it/3.00kit rate:7.0s/it remaining:4:51:36 elapsed:1:04:28 postfix:- (1728:tqdm_logging.py:145)
07:50:54.170 [I] Progress on: 512it/3.00kit rate:7.0s/it remaining:4:51:59 elapsed:1:04:42 postfix:- (1728:tqdm_logging.py:145)
07:51:08.288 [I] Progress on: 514it/3.00kit rate:7.1s/it remaining:4:52:06 elapsed:1:04:57 postfix:- (1728:tqdm_logging.py:145)
07:51:22.408 [I] Progress on: 516it/3.00kit rate:7.1s/it remaining:4:52:04 elapsed:1:05:11 postfix:- (1728:tqdm_logging.py:145)
07:51:36.550 [I] Progress on: 518it/3.00kit rate:7.1s/it remaining:4:52:10 elapsed:1:05:25 postfix:- (1728:tqdm_logging.py:145)
07:51:50.696 [I] Progress on: 520it/3.00kit rate:7.1s/it remaining:4:52:09 elapsed:1:05:39 postfix:- (1728:tqdm_logging.py:145)
07:52:04.850 [I] Progress on: 522it/3.00kit rate:7.1s/it remaining:4:52:05 elapsed:1:05:53 postfix:- (1728:tqdm_logging.py:145)
07:52:19.013 [I] Progress on: 524it/3.00kit rate:7.1s/it remaining:4:52:03 elapsed:1:06:07 postfix:- (1728:tqdm_logging.py:145)
07:52:33.157 [I] Progress on: 526it/3.00kit rate:7.1s/it remaining:4:51:42 elapsed:1:06:21 postfix:- (1728:tqdm_logging.py:145)
07:52:47.290 [I] Progress on: 528it/3.00kit rate:7.1s/it remaining:4:51:19 elapsed:1:06:36 postfix:- (1728:tqdm_logging.py:145)
07:53:01.426 [I] Progress on: 530it/3.00kit rate:7.1s/it remaining:4:51:03 elapsed:1:06:50 postfix:- (1728:tqdm_logging.py:145)
07:53:15.567 [I] Progress on: 532it/3.00kit rate:7.1s/it remaining:4:50:52 elapsed:1:07:04 postfix:- (1728:tqdm_logging.py:145)
07:53:29.713 [I] Progress on: 534it/3.00kit rate:7.1s/it remaining:4:50:40 elapsed:1:07:18 postfix:- (1728:tqdm_logging.py:145)
07:53:43.828 [I] Progress on: 536it/3.00kit rate:7.1s/it remaining:4:50:08 elapsed:1:07:32 postfix:- (1728:tqdm_logging.py:145)
07:53:57.959 [I] Progress on: 538it/3.00kit rate:7.1s/it remaining:4:49:55 elapsed:1:07:46 postfix:- (1728:tqdm_logging.py:145)
07:54:12.070 [I] Progress on: 540it/3.00kit rate:7.1s/it remaining:4:49:28 elapsed:1:08:00 postfix:- (1728:tqdm_logging.py:145)
07:54:26.178 [I] Progress on: 542it/3.00kit rate:7.1s/it remaining:4:49:06 elapsed:1:08:14 postfix:- (1728:tqdm_logging.py:145)
07:54:40.307 [I] Progress on: 544it/3.00kit rate:7.1s/it remaining:4:49:03 elapsed:1:08:29 postfix:- (1728:tqdm_logging.py:145)
07:54:54.422 [I] Progress on: 546it/3.00kit rate:7.1s/it remaining:4:48:45 elapsed:1:08:43 postfix:- (1728:tqdm_logging.py:145)
07:55:08.528 [I] Progress on: 548it/3.00kit rate:7.1s/it remaining:4:48:23 elapsed:1:08:57 postfix:- (1728:tqdm_logging.py:145)
07:55:22.647 [I] Progress on: 550it/3.00kit rate:7.1s/it remaining:4:48:14 elapsed:1:09:11 postfix:- (1728:tqdm_logging.py:145)


Step 550: grad_norm=0.0705, loss=0.0116, param_norm=1803.8448
07:55:36.411 [I] Progress on: 550it/3.00kit rate:7.1s/it remaining:4:48:14 elapsed:1:09:25 postfix:- (1728:tqdm_logging.py:145)
07:55:51.008 [I] Progress on: 554it/3.00kit rate:6.8s/it remaining:4:36:28 elapsed:1:09:39 postfix:- (1728:tqdm_logging.py:145)
07:56:05.123 [I] Progress on: 556it/3.00kit rate:6.9s/it remaining:4:41:51 elapsed:1:09:53 postfix:- (1728:tqdm_logging.py:145)
07:56:19.258 [I] Progress on: 558it/3.00kit rate:7.0s/it remaining:4:44:46 elapsed:1:10:08 postfix:- (1728:tqdm_logging.py:145)
07:56:33.374 [I] Progress on: 560it/3.00kit rate:7.0s/it remaining:4:45:49 elapsed:1:10:22 postfix:- (1728:tqdm_logging.py:145)
07:56:47.506 [I] Progress on: 562it/3.00kit rate:7.0s/it remaining:4:46:23 elapsed:1:10:36 postfix:- (1728:tqdm_logging.py:145)
07:57:01.643 [I] Progress on: 564it/3.00kit rate:7.1s/it remaining:4:46:34 elapsed:1:10:50 postfix:- (1728:tqdm_logging.py:145)
07:57:15.758 [I] Progress on: 566it/3.00kit rate:7.1s/it remaining:4:46:18 elapsed:1:11:04 postfix:- (1728:tqdm_logging.py:145)
07:57:29.871 [I] Progress on: 568it/3.00kit rate:7.1s/it remaining:4:46:02 elapsed:1:11:18 postfix:- (1728:tqdm_logging.py:145)
07:57:43.997 [I] Progress on: 570it/3.00kit rate:7.1s/it remaining:4:45:56 elapsed:1:11:32 postfix:- (1728:tqdm_logging.py:145)
07:57:58.119 [I] Progress on: 572it/3.00kit rate:7.1s/it remaining:4:45:45 elapsed:1:11:46 postfix:- (1728:tqdm_logging.py:145)
07:58:12.261 [I] Progress on: 574it/3.00kit rate:7.1s/it remaining:4:45:43 elapsed:1:12:01 postfix:- (1728:tqdm_logging.py:145)
07:58:26.391 [I] Progress on: 576it/3.00kit rate:7.1s/it remaining:4:45:27 elapsed:1:12:15 postfix:- (1728:tqdm_logging.py:145)
07:58:40.499 [I] Progress on: 578it/3.00kit rate:7.1s/it remaining:4:44:59 elapsed:1:12:29 postfix:- (1728:tqdm_logging.py:145)
07:58:54.650 [I] Progress on: 580it/3.00kit rate:7.1s/it remaining:4:45:02 elapsed:1:12:43 postfix:- (1728:tqdm_logging.py:145)
07:59:08.777 [I] Progress on: 582it/3.00kit rate:7.1s/it remaining:4:44:45 elapsed:1:12:57 postfix:- (1728:tqdm_logging.py:145)
07:59:22.886 [I] Progress on: 584it/3.00kit rate:7.1s/it remaining:4:44:17 elapsed:1:13:11 postfix:- (1728:tqdm_logging.py:145)
07:59:37.028 [I] Progress on: 586it/3.00kit rate:7.1s/it remaining:4:44:17 elapsed:1:13:25 postfix:- (1728:tqdm_logging.py:145)
07:59:51.167 [I] Progress on: 588it/3.00kit rate:7.1s/it remaining:4:44:09 elapsed:1:13:39 postfix:- (1728:tqdm_logging.py:145)
08:00:05.302 [I] Progress on: 590it/3.00kit rate:7.1s/it remaining:4:43:52 elapsed:1:13:54 postfix:- (1728:tqdm_logging.py:145)
08:00:19.464 [I] Progress on: 592it/3.00kit rate:7.1s/it remaining:4:43:53 elapsed:1:14:08 postfix:- (1728:tqdm_logging.py:145)
08:00:33.593 [I] Progress on: 594it/3.00kit rate:7.1s/it remaining:4:43:31 elapsed:1:14:22 postfix:- (1728:tqdm_logging.py:145)
08:00:47.702 [I] Progress on: 596it/3.00kit rate:7.1s/it remaining:4:42:57 elapsed:1:14:36 postfix:- (1728:tqdm_logging.py:145)
08:01:01.844 [I] Progress on: 598it/3.00kit rate:7.1s/it remaining:4:42:56 elapsed:1:14:50 postfix:- (1728:tqdm_logging.py:145)
08:01:15.996 [I] Progress on: 600it/3.00kit rate:7.1s/it remaining:4:42:53 elapsed:1:15:04 postfix:- (1728:tqdm_logging.py:145)


Step 600: grad_norm=0.0696, loss=0.0111, param_norm=1803.8556
08:01:29.757 [I] Progress on: 600it/3.00kit rate:7.1s/it remaining:4:42:53 elapsed:1:15:18 postfix:- (1728:tqdm_logging.py:145)
08:01:44.320 [I] Progress on: 604it/3.00kit rate:6.8s/it remaining:4:30:46 elapsed:1:15:33 postfix:- (1728:tqdm_logging.py:145)
08:01:58.445 [I] Progress on: 606it/3.00kit rate:6.9s/it remaining:4:36:14 elapsed:1:15:47 postfix:- (1728:tqdm_logging.py:145)
08:02:12.601 [I] Progress on: 608it/3.00kit rate:7.0s/it remaining:4:39:07 elapsed:1:16:01 postfix:- (1728:tqdm_logging.py:145)
08:02:26.759 [I] Progress on: 610it/3.00kit rate:7.0s/it remaining:4:40:26 elapsed:1:16:15 postfix:- (1728:tqdm_logging.py:145)
08:02:40.904 [I] Progress on: 612it/3.00kit rate:7.1s/it remaining:4:40:49 elapsed:1:16:29 postfix:- (1728:tqdm_logging.py:145)
08:02:55.063 [I] Progress on: 614it/3.00kit rate:7.1s/it remaining:4:41:02 elapsed:1:16:43 postfix:- (1728:tqdm_logging.py:145)
08:03:09.211 [I] Progress on: 616it/3.00kit rate:7.1s/it remaining:4:40:57 elapsed:1:16:57 postfix:- (1728:tqdm_logging.py:145)
08:03:23.353 [I] Progress on: 618it/3.00kit rate:7.1s/it remaining:4:40:40 elapsed:1:17:12 postfix:- (1728:tqdm_logging.py:145)
08:03:37.511 [I] Progress on: 620it/3.00kit rate:7.1s/it remaining:4:40:34 elapsed:1:17:26 postfix:- (1728:tqdm_logging.py:145)
08:03:51.666 [I] Progress on: 622it/3.00kit rate:7.1s/it remaining:4:40:24 elapsed:1:17:40 postfix:- (1728:tqdm_logging.py:145)
08:04:05.831 [I] Progress on: 624it/3.00kit rate:7.1s/it remaining:4:40:19 elapsed:1:17:54 postfix:- (1728:tqdm_logging.py:145)
08:04:19.981 [I] Progress on: 626it/3.00kit rate:7.1s/it remaining:4:40:00 elapsed:1:18:08 postfix:- (1728:tqdm_logging.py:145)
08:04:34.119 [I] Progress on: 628it/3.00kit rate:7.1s/it remaining:4:39:35 elapsed:1:18:22 postfix:- (1728:tqdm_logging.py:145)
08:04:48.251 [I] Progress on: 630it/3.00kit rate:7.1s/it remaining:4:39:11 elapsed:1:18:37 postfix:- (1728:tqdm_logging.py:145)
08:05:02.380 [I] Progress on: 632it/3.00kit rate:7.1s/it remaining:4:38:53 elapsed:1:18:51 postfix:- (1728:tqdm_logging.py:145)
08:05:16.514 [I] Progress on: 634it/3.00kit rate:7.1s/it remaining:4:38:41 elapsed:1:19:05 postfix:- (1728:tqdm_logging.py:145)
08:05:30.625 [I] Progress on: 636it/3.00kit rate:7.1s/it remaining:4:38:13 elapsed:1:19:19 postfix:- (1728:tqdm_logging.py:145)
08:05:44.746 [I] Progress on: 638it/3.00kit rate:7.1s/it remaining:4:37:58 elapsed:1:19:33 postfix:- (1728:tqdm_logging.py:145)
08:05:58.891 [I] Progress on: 640it/3.00kit rate:7.1s/it remaining:4:37:58 elapsed:1:19:47 postfix:- (1728:tqdm_logging.py:145)
08:06:13.022 [I] Progress on: 642it/3.00kit rate:7.1s/it remaining:4:37:44 elapsed:1:20:01 postfix:- (1728:tqdm_logging.py:145)
08:06:27.158 [I] Progress on: 644it/3.00kit rate:7.1s/it remaining:4:37:30 elapsed:1:20:15 postfix:- (1728:tqdm_logging.py:145)
08:06:41.284 [I] Progress on: 646it/3.00kit rate:7.1s/it remaining:4:37:12 elapsed:1:20:30 postfix:- (1728:tqdm_logging.py:145)
08:06:55.416 [I] Progress on: 648it/3.00kit rate:7.1s/it remaining:4:37:00 elapsed:1:20:44 postfix:- (1728:tqdm_logging.py:145)
08:07:09.528 [I] Progress on: 650it/3.00kit rate:7.1s/it remaining:4:36:33 elapsed:1:20:58 postfix:- (1728:tqdm_logging.py:145)


Step 650: grad_norm=0.0620, loss=0.0100, param_norm=1803.8658
08:07:23.283 [I] Progress on: 650it/3.00kit rate:7.1s/it remaining:4:36:33 elapsed:1:21:12 postfix:- (1728:tqdm_logging.py:145)
08:07:37.836 [I] Progress on: 654it/3.00kit rate:6.8s/it remaining:4:24:50 elapsed:1:21:26 postfix:- (1728:tqdm_logging.py:145)
08:07:51.959 [I] Progress on: 656it/3.00kit rate:6.9s/it remaining:4:30:19 elapsed:1:21:40 postfix:- (1728:tqdm_logging.py:145)
08:08:06.084 [I] Progress on: 658it/3.00kit rate:7.0s/it remaining:4:32:56 elapsed:1:21:54 postfix:- (1728:tqdm_logging.py:145)
08:08:20.206 [I] Progress on: 660it/3.00kit rate:7.0s/it remaining:4:34:03 elapsed:1:22:08 postfix:- (1728:tqdm_logging.py:145)
08:08:34.316 [I] Progress on: 662it/3.00kit rate:7.0s/it remaining:4:34:21 elapsed:1:22:23 postfix:- (1728:tqdm_logging.py:145)
08:08:48.442 [I] Progress on: 664it/3.00kit rate:7.1s/it remaining:4:34:32 elapsed:1:22:37 postfix:- (1728:tqdm_logging.py:145)
08:09:03.055 [I] Progress on: 666it/3.00kit rate:7.2s/it remaining:4:40:14 elapsed:1:22:51 postfix:- (1728:tqdm_logging.py:145)
08:09:16.676 [I] Progress on: 668it/3.00kit rate:7.0s/it remaining:4:33:03 elapsed:1:23:05 postfix:- (1728:tqdm_logging.py:145)
08:09:30.796 [I] Progress on: 670it/3.00kit rate:7.0s/it remaining:4:33:30 elapsed:1:23:19 postfix:- (1728:tqdm_logging.py:145)
08:09:44.914 [I] Progress on: 672it/3.00kit rate:7.1s/it remaining:4:33:37 elapsed:1:23:33 postfix:- (1728:tqdm_logging.py:145)
08:09:59.033 [I] Progress on: 674it/3.00kit rate:7.1s/it remaining:4:33:33 elapsed:1:23:47 postfix:- (1728:tqdm_logging.py:145)
08:10:13.155 [I] Progress on: 676it/3.00kit rate:7.1s/it remaining:4:33:22 elapsed:1:24:01 postfix:- (1728:tqdm_logging.py:145)
08:10:27.263 [I] Progress on: 678it/3.00kit rate:7.1s/it remaining:4:33:04 elapsed:1:24:16 postfix:- (1728:tqdm_logging.py:145)
08:10:41.370 [I] Progress on: 680it/3.00kit rate:7.1s/it remaining:4:32:44 elapsed:1:24:30 postfix:- (1728:tqdm_logging.py:145)
08:10:55.515 [I] Progress on: 682it/3.00kit rate:7.1s/it remaining:4:32:49 elapsed:1:24:44 postfix:- (1728:tqdm_logging.py:145)
08:11:09.609 [I] Progress on: 684it/3.00kit rate:7.1s/it remaining:4:32:16 elapsed:1:24:58 postfix:- (1728:tqdm_logging.py:145)
08:11:23.739 [I] Progress on: 686it/3.00kit rate:7.1s/it remaining:4:32:13 elapsed:1:25:12 postfix:- (1728:tqdm_logging.py:145)
08:11:37.840 [I] Progress on: 688it/3.00kit rate:7.1s/it remaining:4:31:50 elapsed:1:25:26 postfix:- (1728:tqdm_logging.py:145)
08:11:51.969 [I] Progress on: 690it/3.00kit rate:7.1s/it remaining:4:31:48 elapsed:1:25:40 postfix:- (1728:tqdm_logging.py:145)
08:12:06.059 [I] Progress on: 692it/3.00kit rate:7.1s/it remaining:4:31:18 elapsed:1:25:54 postfix:- (1728:tqdm_logging.py:145)
08:12:20.181 [I] Progress on: 694it/3.00kit rate:7.1s/it remaining:4:31:12 elapsed:1:26:08 postfix:- (1728:tqdm_logging.py:145)
08:12:34.295 [I] Progress on: 696it/3.00kit rate:7.1s/it remaining:4:30:58 elapsed:1:26:23 postfix:- (1728:tqdm_logging.py:145)
08:12:48.397 [I] Progress on: 698it/3.00kit rate:7.1s/it remaining:4:30:37 elapsed:1:26:37 postfix:- (1728:tqdm_logging.py:145)
08:13:02.515 [I] Progress on: 700it/3.00kit rate:7.1s/it remaining:4:30:30 elapsed:1:26:51 postfix:- (1728:tqdm_logging.py:145)


Step 700: grad_norm=0.0589, loss=0.0086, param_norm=1803.8752
08:13:16.264 [I] Progress on: 700it/3.00kit rate:7.1s/it remaining:4:30:30 elapsed:1:27:05 postfix:- (1728:tqdm_logging.py:145)
08:13:30.781 [I] Progress on: 704it/3.00kit rate:6.8s/it remaining:4:18:51 elapsed:1:27:19 postfix:- (1728:tqdm_logging.py:145)
08:13:44.901 [I] Progress on: 706it/3.00kit rate:6.9s/it remaining:4:24:22 elapsed:1:27:33 postfix:- (1728:tqdm_logging.py:145)
08:13:59.025 [I] Progress on: 708it/3.00kit rate:7.0s/it remaining:4:26:58 elapsed:1:27:47 postfix:- (1728:tqdm_logging.py:145)
08:14:13.135 [I] Progress on: 710it/3.00kit rate:7.0s/it remaining:4:28:02 elapsed:1:28:01 postfix:- (1728:tqdm_logging.py:145)
08:14:27.260 [I] Progress on: 712it/3.00kit rate:7.0s/it remaining:4:28:33 elapsed:1:28:16 postfix:- (1728:tqdm_logging.py:145)
08:14:41.373 [I] Progress on: 714it/3.00kit rate:7.0s/it remaining:4:28:33 elapsed:1:28:30 postfix:- (1728:tqdm_logging.py:145)
08:14:55.564 [I] Progress on: 716it/3.00kit rate:7.1s/it remaining:4:29:18 elapsed:1:28:44 postfix:- (1728:tqdm_logging.py:145)
08:15:09.643 [I] Progress on: 718it/3.00kit rate:7.1s/it remaining:4:28:21 elapsed:1:28:58 postfix:- (1728:tqdm_logging.py:145)
08:15:23.766 [I] Progress on: 720it/3.00kit rate:7.1s/it remaining:4:28:12 elapsed:1:29:12 postfix:- (1728:tqdm_logging.py:145)
08:15:37.889 [I] Progress on: 722it/3.00kit rate:7.1s/it remaining:4:28:01 elapsed:1:29:26 postfix:- (1728:tqdm_logging.py:145)
08:15:52.004 [I] Progress on: 724it/3.00kit rate:7.1s/it remaining:4:27:44 elapsed:1:29:40 postfix:- (1728:tqdm_logging.py:145)
08:16:06.118 [I] Progress on: 726it/3.00kit rate:7.1s/it remaining:4:27:28 elapsed:1:29:54 postfix:- (1728:tqdm_logging.py:145)
08:16:20.241 [I] Progress on: 728it/3.00kit rate:7.1s/it remaining:4:27:17 elapsed:1:30:09 postfix:- (1728:tqdm_logging.py:145)
08:16:34.364 [I] Progress on: 730it/3.00kit rate:7.1s/it remaining:4:27:05 elapsed:1:30:23 postfix:- (1728:tqdm_logging.py:145)
08:16:48.484 [I] Progress on: 732it/3.00kit rate:7.1s/it remaining:4:26:53 elapsed:1:30:37 postfix:- (1728:tqdm_logging.py:145)
08:17:02.598 [I] Progress on: 734it/3.00kit rate:7.1s/it remaining:4:26:33 elapsed:1:30:51 postfix:- (1728:tqdm_logging.py:145)
08:17:16.728 [I] Progress on: 736it/3.00kit rate:7.1s/it remaining:4:26:27 elapsed:1:31:05 postfix:- (1728:tqdm_logging.py:145)
08:17:30.834 [I] Progress on: 738it/3.00kit rate:7.1s/it remaining:4:26:02 elapsed:1:31:19 postfix:- (1728:tqdm_logging.py:145)
08:17:44.947 [I] Progress on: 740it/3.00kit rate:7.1s/it remaining:4:25:47 elapsed:1:31:33 postfix:- (1728:tqdm_logging.py:145)
08:17:59.066 [I] Progress on: 742it/3.00kit rate:7.1s/it remaining:4:25:34 elapsed:1:31:47 postfix:- (1728:tqdm_logging.py:145)
08:18:13.164 [I] Progress on: 744it/3.00kit rate:7.1s/it remaining:4:25:08 elapsed:1:32:01 postfix:- (1728:tqdm_logging.py:145)
08:18:27.283 [I] Progress on: 746it/3.00kit rate:7.1s/it remaining:4:25:03 elapsed:1:32:16 postfix:- (1728:tqdm_logging.py:145)
08:18:41.398 [I] Progress on: 748it/3.00kit rate:7.1s/it remaining:4:24:50 elapsed:1:32:30 postfix:- (1728:tqdm_logging.py:145)
08:18:55.498 [I] Progress on: 750it/3.00kit rate:7.1s/it remaining:4:24:29 elapsed:1:32:44 postfix:- (1728:tqdm_logging.py:145)


Step 750: grad_norm=0.0588, loss=0.0089, param_norm=1803.8850
08:19:09.264 [I] Progress on: 750it/3.00kit rate:7.1s/it remaining:4:24:29 elapsed:1:32:58 postfix:- (1728:tqdm_logging.py:145)
08:19:23.817 [I] Progress on: 754it/3.00kit rate:6.8s/it remaining:4:13:26 elapsed:1:33:12 postfix:- (1728:tqdm_logging.py:145)
08:19:37.931 [I] Progress on: 756it/3.00kit rate:6.9s/it remaining:4:18:38 elapsed:1:33:26 postfix:- (1728:tqdm_logging.py:145)
08:19:52.055 [I] Progress on: 758it/3.00kit rate:7.0s/it remaining:4:21:11 elapsed:1:33:40 postfix:- (1728:tqdm_logging.py:145)
08:20:06.166 [I] Progress on: 760it/3.00kit rate:7.0s/it remaining:4:22:12 elapsed:1:33:54 postfix:- (1728:tqdm_logging.py:145)
08:20:20.288 [I] Progress on: 762it/3.00kit rate:7.0s/it remaining:4:22:38 elapsed:1:34:09 postfix:- (1728:tqdm_logging.py:145)
08:20:34.397 [I] Progress on: 764it/3.00kit rate:7.0s/it remaining:4:22:38 elapsed:1:34:23 postfix:- (1728:tqdm_logging.py:145)
08:20:48.531 [I] Progress on: 766it/3.00kit rate:7.1s/it remaining:4:22:46 elapsed:1:34:37 postfix:- (1728:tqdm_logging.py:145)
08:21:02.648 [I] Progress on: 768it/3.00kit rate:7.1s/it remaining:4:22:32 elapsed:1:34:51 postfix:- (1728:tqdm_logging.py:145)
08:21:16.759 [I] Progress on: 770it/3.00kit rate:7.1s/it remaining:4:22:14 elapsed:1:35:05 postfix:- (1728:tqdm_logging.py:145)
08:21:30.880 [I] Progress on: 772it/3.00kit rate:7.1s/it remaining:4:22:03 elapsed:1:35:19 postfix:- (1728:tqdm_logging.py:145)
08:21:44.992 [I] Progress on: 774it/3.00kit rate:7.1s/it remaining:4:21:46 elapsed:1:35:33 postfix:- (1728:tqdm_logging.py:145)
08:21:59.125 [I] Progress on: 776it/3.00kit rate:7.1s/it remaining:4:21:44 elapsed:1:35:47 postfix:- (1728:tqdm_logging.py:145)
08:22:13.253 [I] Progress on: 778it/3.00kit rate:7.1s/it remaining:4:21:32 elapsed:1:36:02 postfix:- (1728:tqdm_logging.py:145)
08:22:27.363 [I] Progress on: 780it/3.00kit rate:7.1s/it remaining:4:21:08 elapsed:1:36:16 postfix:- (1728:tqdm_logging.py:145)
08:22:41.470 [I] Progress on: 782it/3.00kit rate:7.1s/it remaining:4:20:47 elapsed:1:36:30 postfix:- (1728:tqdm_logging.py:145)
08:22:55.598 [I] Progress on: 784it/3.00kit rate:7.1s/it remaining:4:20:43 elapsed:1:36:44 postfix:- (1728:tqdm_logging.py:145)
08:23:09.706 [I] Progress on: 786it/3.00kit rate:7.1s/it remaining:4:20:22 elapsed:1:36:58 postfix:- (1728:tqdm_logging.py:145)
08:23:23.833 [I] Progress on: 788it/3.00kit rate:7.1s/it remaining:4:20:14 elapsed:1:37:12 postfix:- (1728:tqdm_logging.py:145)
08:23:37.947 [I] Progress on: 790it/3.00kit rate:7.1s/it remaining:4:19:54 elapsed:1:37:26 postfix:- (1728:tqdm_logging.py:145)
08:23:52.058 [I] Progress on: 792it/3.00kit rate:7.1s/it remaining:4:19:39 elapsed:1:37:40 postfix:- (1728:tqdm_logging.py:145)
08:24:06.173 [I] Progress on: 794it/3.00kit rate:7.1s/it remaining:4:19:26 elapsed:1:37:54 postfix:- (1728:tqdm_logging.py:145)
08:24:20.288 [I] Progress on: 796it/3.00kit rate:7.1s/it remaining:4:19:12 elapsed:1:38:09 postfix:- (1728:tqdm_logging.py:145)
08:24:34.420 [I] Progress on: 798it/3.00kit rate:7.1s/it remaining:4:19:09 elapsed:1:38:23 postfix:- (1728:tqdm_logging.py:145)
08:24:48.534 [I] Progress on: 800it/3.00kit rate:7.1s/it remaining:4:18:49 elapsed:1:38:37 postfix:- (1728:tqdm_logging.py:145)


Step 800: grad_norm=0.0571, loss=0.0083, param_norm=1803.8943
08:25:02.298 [I] Progress on: 800it/3.00kit rate:7.1s/it remaining:4:18:49 elapsed:1:38:51 postfix:- (1728:tqdm_logging.py:145)
08:25:16.857 [I] Progress on: 804it/3.00kit rate:6.8s/it remaining:4:07:56 elapsed:1:39:05 postfix:- (1728:tqdm_logging.py:145)
08:25:30.950 [I] Progress on: 806it/3.00kit rate:6.9s/it remaining:4:12:46 elapsed:1:39:19 postfix:- (1728:tqdm_logging.py:145)
08:25:45.083 [I] Progress on: 808it/3.00kit rate:7.0s/it remaining:4:15:24 elapsed:1:39:33 postfix:- (1728:tqdm_logging.py:145)
08:25:59.196 [I] Progress on: 810it/3.00kit rate:7.0s/it remaining:4:16:21 elapsed:1:39:47 postfix:- (1728:tqdm_logging.py:145)
08:26:13.310 [I] Progress on: 812it/3.00kit rate:7.0s/it remaining:4:16:41 elapsed:1:40:02 postfix:- (1728:tqdm_logging.py:145)
08:26:27.454 [I] Progress on: 814it/3.00kit rate:7.1s/it remaining:4:17:01 elapsed:1:40:16 postfix:- (1728:tqdm_logging.py:145)
08:26:41.575 [I] Progress on: 816it/3.00kit rate:7.1s/it remaining:4:16:54 elapsed:1:40:30 postfix:- (1728:tqdm_logging.py:145)
08:26:55.682 [I] Progress on: 818it/3.00kit rate:7.1s/it remaining:4:16:36 elapsed:1:40:44 postfix:- (1728:tqdm_logging.py:145)
08:27:09.807 [I] Progress on: 820it/3.00kit rate:7.1s/it remaining:4:16:29 elapsed:1:40:58 postfix:- (1728:tqdm_logging.py:145)
08:27:23.920 [I] Progress on: 822it/3.00kit rate:7.1s/it remaining:4:16:11 elapsed:1:41:12 postfix:- (1728:tqdm_logging.py:145)
08:27:38.024 [I] Progress on: 824it/3.00kit rate:7.1s/it remaining:4:15:51 elapsed:1:41:26 postfix:- (1728:tqdm_logging.py:145)
08:27:52.136 [I] Progress on: 826it/3.00kit rate:7.1s/it remaining:4:15:39 elapsed:1:41:40 postfix:- (1728:tqdm_logging.py:145)
08:28:06.266 [I] Progress on: 828it/3.00kit rate:7.1s/it remaining:4:15:37 elapsed:1:41:55 postfix:- (1728:tqdm_logging.py:145)
08:28:20.382 [I] Progress on: 830it/3.00kit rate:7.1s/it remaining:4:15:22 elapsed:1:42:09 postfix:- (1728:tqdm_logging.py:145)
08:28:34.498 [I] Progress on: 832it/3.00kit rate:7.1s/it remaining:4:15:06 elapsed:1:42:23 postfix:- (1728:tqdm_logging.py:145)
08:28:48.607 [I] Progress on: 834it/3.00kit rate:7.1s/it remaining:4:14:47 elapsed:1:42:37 postfix:- (1728:tqdm_logging.py:145)
08:29:02.719 [I] Progress on: 836it/3.00kit rate:7.1s/it remaining:4:14:31 elapsed:1:42:51 postfix:- (1728:tqdm_logging.py:145)
08:29:16.819 [I] Progress on: 838it/3.00kit rate:7.1s/it remaining:4:14:09 elapsed:1:43:05 postfix:- (1728:tqdm_logging.py:145)
08:29:30.963 [I] Progress on: 840it/3.00kit rate:7.1s/it remaining:4:14:18 elapsed:1:43:19 postfix:- (1728:tqdm_logging.py:145)
08:29:45.085 [I] Progress on: 842it/3.00kit rate:7.1s/it remaining:4:14:03 elapsed:1:43:33 postfix:- (1728:tqdm_logging.py:145)
08:29:59.204 [I] Progress on: 844it/3.00kit rate:7.1s/it remaining:4:13:45 elapsed:1:43:47 postfix:- (1728:tqdm_logging.py:145)
08:30:13.329 [I] Progress on: 846it/3.00kit rate:7.1s/it remaining:4:13:31 elapsed:1:44:02 postfix:- (1728:tqdm_logging.py:145)
08:30:27.439 [I] Progress on: 848it/3.00kit rate:7.1s/it remaining:4:13:11 elapsed:1:44:16 postfix:- (1728:tqdm_logging.py:145)
08:30:41.540 [I] Progress on: 850it/3.00kit rate:7.1s/it remaining:4:12:48 elapsed:1:44:30 postfix:- (1728:tqdm_logging.py:145)


Step 850: grad_norm=0.0576, loss=0.0083, param_norm=1803.9048
08:30:55.314 [I] Progress on: 850it/3.00kit rate:7.1s/it remaining:4:12:48 elapsed:1:44:44 postfix:- (1728:tqdm_logging.py:145)
08:31:09.859 [I] Progress on: 854it/3.00kit rate:6.8s/it remaining:4:02:08 elapsed:1:44:58 postfix:- (1728:tqdm_logging.py:145)
08:31:23.974 [I] Progress on: 856it/3.00kit rate:6.9s/it remaining:4:07:08 elapsed:1:45:12 postfix:- (1728:tqdm_logging.py:145)
08:31:38.091 [I] Progress on: 858it/3.00kit rate:7.0s/it remaining:4:09:26 elapsed:1:45:26 postfix:- (1728:tqdm_logging.py:145)
08:31:52.211 [I] Progress on: 860it/3.00kit rate:7.0s/it remaining:4:10:31 elapsed:1:45:40 postfix:- (1728:tqdm_logging.py:145)
08:32:06.331 [I] Progress on: 862it/3.00kit rate:7.0s/it remaining:4:10:56 elapsed:1:45:55 postfix:- (1728:tqdm_logging.py:145)
08:32:20.454 [I] Progress on: 864it/3.00kit rate:7.1s/it remaining:4:11:02 elapsed:1:46:09 postfix:- (1728:tqdm_logging.py:145)
08:32:34.569 [I] Progress on: 866it/3.00kit rate:7.1s/it remaining:4:10:58 elapsed:1:46:23 postfix:- (1728:tqdm_logging.py:145)
08:32:48.667 [I] Progress on: 868it/3.00kit rate:7.1s/it remaining:4:10:36 elapsed:1:46:37 postfix:- (1728:tqdm_logging.py:145)
08:33:02.785 [I] Progress on: 870it/3.00kit rate:7.1s/it remaining:4:10:28 elapsed:1:46:51 postfix:- (1728:tqdm_logging.py:145)
08:33:16.926 [I] Progress on: 872it/3.00kit rate:7.1s/it remaining:4:10:32 elapsed:1:47:05 postfix:- (1728:tqdm_logging.py:145)
08:33:31.063 [I] Progress on: 874it/3.00kit rate:7.1s/it remaining:4:10:21 elapsed:1:47:19 postfix:- (1728:tqdm_logging.py:145)
08:33:45.179 [I] Progress on: 876it/3.00kit rate:7.1s/it remaining:4:10:00 elapsed:1:47:33 postfix:- (1728:tqdm_logging.py:145)
08:33:59.297 [I] Progress on: 878it/3.00kit rate:7.1s/it remaining:4:09:44 elapsed:1:47:48 postfix:- (1728:tqdm_logging.py:145)
08:34:13.395 [I] Progress on: 880it/3.00kit rate:7.1s/it remaining:4:09:17 elapsed:1:48:02 postfix:- (1728:tqdm_logging.py:145)
08:34:27.527 [I] Progress on: 882it/3.00kit rate:7.1s/it remaining:4:09:16 elapsed:1:48:16 postfix:- (1728:tqdm_logging.py:145)
08:34:41.626 [I] Progress on: 884it/3.00kit rate:7.1s/it remaining:4:08:49 elapsed:1:48:30 postfix:- (1728:tqdm_logging.py:145)
08:34:55.748 [I] Progress on: 886it/3.00kit rate:7.1s/it remaining:4:08:40 elapsed:1:48:44 postfix:- (1728:tqdm_logging.py:145)
08:35:10.428 [I] Progress on: 888it/3.00kit rate:7.2s/it remaining:4:14:23 elapsed:1:48:59 postfix:- (1728:tqdm_logging.py:145)
08:35:24.000 [I] Progress on: 890it/3.00kit rate:7.0s/it remaining:4:07:10 elapsed:1:49:12 postfix:- (1728:tqdm_logging.py:145)
08:35:38.093 [I] Progress on: 892it/3.00kit rate:7.0s/it remaining:4:07:20 elapsed:1:49:26 postfix:- (1728:tqdm_logging.py:145)
08:35:52.208 [I] Progress on: 894it/3.00kit rate:7.0s/it remaining:4:07:22 elapsed:1:49:40 postfix:- (1728:tqdm_logging.py:145)
08:36:06.329 [I] Progress on: 896it/3.00kit rate:7.1s/it remaining:4:07:19 elapsed:1:49:55 postfix:- (1728:tqdm_logging.py:145)
08:36:20.461 [I] Progress on: 898it/3.00kit rate:7.1s/it remaining:4:07:18 elapsed:1:50:09 postfix:- (1728:tqdm_logging.py:145)
08:36:34.579 [I] Progress on: 900it/3.00kit rate:7.1s/it remaining:4:07:02 elapsed:1:50:23 postfix:- (1728:tqdm_logging.py:145)


Step 900: grad_norm=0.0554, loss=0.0080, param_norm=1803.9143
08:36:48.331 [I] Progress on: 900it/3.00kit rate:7.1s/it remaining:4:07:02 elapsed:1:50:37 postfix:- (1728:tqdm_logging.py:145)
08:37:02.850 [I] Progress on: 904it/3.00kit rate:6.8s/it remaining:3:56:22 elapsed:1:50:51 postfix:- (1728:tqdm_logging.py:145)
08:37:16.966 [I] Progress on: 906it/3.00kit rate:6.9s/it remaining:4:01:20 elapsed:1:51:05 postfix:- (1728:tqdm_logging.py:145)
08:37:31.089 [I] Progress on: 908it/3.00kit rate:7.0s/it remaining:4:03:43 elapsed:1:51:19 postfix:- (1728:tqdm_logging.py:145)
08:37:45.200 [I] Progress on: 910it/3.00kit rate:7.0s/it remaining:4:04:39 elapsed:1:51:33 postfix:- (1728:tqdm_logging.py:145)
08:37:59.316 [I] Progress on: 912it/3.00kit rate:7.0s/it remaining:4:05:01 elapsed:1:51:48 postfix:- (1728:tqdm_logging.py:145)
08:38:13.433 [I] Progress on: 914it/3.00kit rate:7.0s/it remaining:4:05:05 elapsed:1:52:02 postfix:- (1728:tqdm_logging.py:145)
08:38:27.576 [I] Progress on: 916it/3.00kit rate:7.1s/it remaining:4:05:16 elapsed:1:52:16 postfix:- (1728:tqdm_logging.py:145)
08:38:41.699 [I] Progress on: 918it/3.00kit rate:7.1s/it remaining:4:05:04 elapsed:1:52:30 postfix:- (1728:tqdm_logging.py:145)
08:38:55.823 [I] Progress on: 920it/3.00kit rate:7.1s/it remaining:4:04:50 elapsed:1:52:44 postfix:- (1728:tqdm_logging.py:145)
08:39:09.923 [I] Progress on: 922it/3.00kit rate:7.1s/it remaining:4:04:23 elapsed:1:52:58 postfix:- (1728:tqdm_logging.py:145)
08:39:24.059 [I] Progress on: 924it/3.00kit rate:7.1s/it remaining:4:04:22 elapsed:1:53:12 postfix:- (1728:tqdm_logging.py:145)
08:39:38.170 [I] Progress on: 926it/3.00kit rate:7.1s/it remaining:4:04:01 elapsed:1:53:26 postfix:- (1728:tqdm_logging.py:145)
08:39:52.270 [I] Progress on: 928it/3.00kit rate:7.1s/it remaining:4:03:38 elapsed:1:53:41 postfix:- (1728:tqdm_logging.py:145)
08:40:06.394 [I] Progress on: 930it/3.00kit rate:7.1s/it remaining:4:03:30 elapsed:1:53:55 postfix:- (1728:tqdm_logging.py:145)
08:40:20.519 [I] Progress on: 932it/3.00kit rate:7.1s/it remaining:4:03:20 elapsed:1:54:09 postfix:- (1728:tqdm_logging.py:145)
08:40:34.631 [I] Progress on: 934it/3.00kit rate:7.1s/it remaining:4:03:03 elapsed:1:54:23 postfix:- (1728:tqdm_logging.py:145)
08:40:48.731 [I] Progress on: 936it/3.00kit rate:7.1s/it remaining:4:02:40 elapsed:1:54:37 postfix:- (1728:tqdm_logging.py:145)
08:41:02.859 [I] Progress on: 938it/3.00kit rate:7.1s/it remaining:4:02:39 elapsed:1:54:51 postfix:- (1728:tqdm_logging.py:145)
08:41:16.973 [I] Progress on: 940it/3.00kit rate:7.1s/it remaining:4:02:22 elapsed:1:55:05 postfix:- (1728:tqdm_logging.py:145)
08:41:31.090 [I] Progress on: 942it/3.00kit rate:7.1s/it remaining:4:02:08 elapsed:1:55:19 postfix:- (1728:tqdm_logging.py:145)
08:41:45.206 [I] Progress on: 944it/3.00kit rate:7.1s/it remaining:4:01:54 elapsed:1:55:33 postfix:- (1728:tqdm_logging.py:145)
08:41:59.296 [I] Progress on: 946it/3.00kit rate:7.1s/it remaining:4:01:25 elapsed:1:55:48 postfix:- (1728:tqdm_logging.py:145)
08:42:13.409 [I] Progress on: 948it/3.00kit rate:7.1s/it remaining:4:01:15 elapsed:1:56:02 postfix:- (1728:tqdm_logging.py:145)
08:42:27.537 [I] Progress on: 950it/3.00kit rate:7.1s/it remaining:4:01:11 elapsed:1:56:16 postfix:- (1728:tqdm_logging.py:145)


Step 950: grad_norm=0.0543, loss=0.0075, param_norm=1803.9233
08:42:41.286 [I] Progress on: 950it/3.00kit rate:7.1s/it remaining:4:01:11 elapsed:1:56:30 postfix:- (1728:tqdm_logging.py:145)
08:42:55.799 [I] Progress on: 954it/3.00kit rate:6.8s/it remaining:3:50:40 elapsed:1:56:44 postfix:- (1728:tqdm_logging.py:145)
08:43:09.916 [I] Progress on: 956it/3.00kit rate:6.9s/it remaining:3:55:30 elapsed:1:56:58 postfix:- (1728:tqdm_logging.py:145)
08:43:24.029 [I] Progress on: 958it/3.00kit rate:7.0s/it remaining:3:57:44 elapsed:1:57:12 postfix:- (1728:tqdm_logging.py:145)
08:43:38.162 [I] Progress on: 960it/3.00kit rate:7.0s/it remaining:3:58:55 elapsed:1:57:26 postfix:- (1728:tqdm_logging.py:145)
08:43:52.261 [I] Progress on: 962it/3.00kit rate:7.0s/it remaining:3:59:03 elapsed:1:57:41 postfix:- (1728:tqdm_logging.py:145)
08:44:06.385 [I] Progress on: 964it/3.00kit rate:7.1s/it remaining:3:59:13 elapsed:1:57:55 postfix:- (1728:tqdm_logging.py:145)
08:44:20.504 [I] Progress on: 966it/3.00kit rate:7.1s/it remaining:3:59:11 elapsed:1:58:09 postfix:- (1728:tqdm_logging.py:145)
08:44:34.624 [I] Progress on: 968it/3.00kit rate:7.1s/it remaining:3:59:02 elapsed:1:58:23 postfix:- (1728:tqdm_logging.py:145)
08:44:48.748 [I] Progress on: 970it/3.00kit rate:7.1s/it remaining:3:58:52 elapsed:1:58:37 postfix:- (1728:tqdm_logging.py:145)
08:45:02.879 [I] Progress on: 972it/3.00kit rate:7.1s/it remaining:3:58:44 elapsed:1:58:51 postfix:- (1728:tqdm_logging.py:145)
08:45:16.974 [I] Progress on: 974it/3.00kit rate:7.1s/it remaining:3:58:13 elapsed:1:59:05 postfix:- (1728:tqdm_logging.py:145)
08:45:31.109 [I] Progress on: 976it/3.00kit rate:7.1s/it remaining:3:58:12 elapsed:1:59:19 postfix:- (1728:tqdm_logging.py:145)
08:45:45.234 [I] Progress on: 978it/3.00kit rate:7.1s/it remaining:3:58:02 elapsed:1:59:34 postfix:- (1728:tqdm_logging.py:145)
08:45:59.363 [I] Progress on: 980it/3.00kit rate:7.1s/it remaining:3:57:50 elapsed:1:59:48 postfix:- (1728:tqdm_logging.py:145)
08:46:13.462 [I] Progress on: 982it/3.00kit rate:7.1s/it remaining:3:57:21 elapsed:2:00:02 postfix:- (1728:tqdm_logging.py:145)
08:46:27.578 [I] Progress on: 984it/3.00kit rate:7.1s/it remaining:3:57:07 elapsed:2:00:16 postfix:- (1728:tqdm_logging.py:145)
08:46:41.775 [I] Progress on: 986it/3.00kit rate:7.1s/it remaining:3:57:42 elapsed:2:00:30 postfix:- (1728:tqdm_logging.py:145)
08:46:55.847 [I] Progress on: 988it/3.00kit rate:7.1s/it remaining:3:56:47 elapsed:2:00:44 postfix:- (1728:tqdm_logging.py:145)
08:47:09.943 [I] Progress on: 990it/3.00kit rate:7.1s/it remaining:3:56:22 elapsed:2:00:58 postfix:- (1728:tqdm_logging.py:145)
08:47:24.056 [I] Progress on: 992it/3.00kit rate:7.1s/it remaining:3:56:06 elapsed:2:01:12 postfix:- (1728:tqdm_logging.py:145)
08:47:38.153 [I] Progress on: 994it/3.00kit rate:7.1s/it remaining:3:55:46 elapsed:2:01:26 postfix:- (1728:tqdm_logging.py:145)
08:47:52.270 [I] Progress on: 996it/3.00kit rate:7.1s/it remaining:3:55:36 elapsed:2:01:41 postfix:- (1728:tqdm_logging.py:145)
08:48:06.384 [I] Progress on: 998it/3.00kit rate:7.1s/it remaining:3:55:23 elapsed:2:01:55 postfix:- (1728:tqdm_logging.py:145)
08:48:20.505 [I] Progress on: 1.00kit/3.00kit rate:7.1s/it remaining:3:55:12 elapsed:2:02:09 postfix:- (1728:tqdm_logging.py:145)


Step 1000: grad_norm=0.0518, loss=0.0072, param_norm=1803.9321
08:48:34.253 [I] Progress on: 1.00kit/3.00kit rate:7.1s/it remaining:3:55:12 elapsed:2:02:23 postfix:- (1728:tqdm_logging.py:145)
08:48:34.297 [I] [process=0][thread=MainThread][wait_until_finished] No Save Finalize thread to wait for. Returning. (1728:checkpoint_manager.py:1987)
08:48:34.298 [I] [process=0] Saving checkpoint at step 1000                                       (1728:checkpoint_manager.py:1408)
08:48:34.299 [I] [process=0] Started async saving checkpoint to /workspace/openpi/checkpoints/pi05_mecharm_deep_even10_lora/deep_even10_v1/1000. (1728:async_checkpointer.py:439)
08:48:34.300 [I] Using ThreadSafeKeyValueSignalingClient                                          (1728:signaling_client.py:332)
08:48:34.303 [I] Creating tmp directory /workspace/openpi/checkpoints/pi05_mecharm_deep_even10_lora/deep_even10_v1/1000.orbax-checkpoint-tmp-0 (1728:atomicity.py:144)
08:48:34.319 [I] Wrote Metadata={'item_handlers': None, 'metrics': {}, 'performance_metrics': {}, 'init_timestamp_nsecs': 1794214114317702192, 'commit_timestamp_nsecs': None, 'custom_metadata': {}}, json={"item_handlers": null, "metrics": {}, "performance_metrics": {}, "init_timestamp_nsecs": 1794214114317702192, "commit_timestamp_nsecs": null, "custom_metadata": {}} to /workspace/openpi/checkpoints/pi05_mecharm_deep_even10_lora/deep_even10_v1/1000.orbax-checkpoint-tmp-0/_CHECKPOINT_METADATA (1728:checkpoint.py:186)
08:48:34.321 [I] Creating tmp directory /workspace/openpi/checkpoints/pi05_mecharm_deep_even10_lora/deep_even10_v1/1000.orbax-checkpoint-tmp-0/assets.orbax-checkpoint-tmp-1 (1728:atomicity.py:144)
08:48:34.322 [I] Creating tmp directory /workspace/openpi/checkpoints/pi05_mecharm_deep_even10_lora/deep_even10_v1/1000.orbax-checkpoint-tmp-0/params.orbax-checkpoint-tmp-2 (1728:atomicity.py:144)
08:48:34.323 [I] Creating tmp directory /workspace/openpi/checkpoints/pi05_mecharm_deep_even10_lora/deep_even10_v1/1000.orbax-checkpoint-tmp-0/train_state.orbax-checkpoint-tmp-3 (1728:atomicity.py:144)
08:48:34.343 [I] Transferring arrays to host memory with options: use_replica_parallel=True, enable_pinned_host_transfer=False (1728:replica_slices.py:341)
08:48:40.505 [I] Array name: 'params.PaliGemma.llm.layers.mlp.gating_einsum.value', global shape: (18, 2, 2048, 16384), write shape: (18, 2, 2048, 16384), chosen chunk shape: (18, 2, 2048, 8192) (1728:tensorstore_utils.py:408)
08:48:40.506 [I] Transferring arrays to host memory with options: use_replica_parallel=True, enable_pinned_host_transfer=False (1728:replica_slices.py:341)
08:48:40.565 [I] [process=0][thread=array_type_handler] Wrote 71 array_metadata.ArrayMetadata to /workspace/openpi/checkpoints/pi05_mecharm_deep_even10_lora/deep_even10_v1/1000.orbax-checkpoint-tmp-0/params.orbax-checkpoint-tmp-2/array_metadatas/process_0 (1728:array_metadata_store.py:198)
08:48:44.391 [I] [process=0] /jax/checkpoint/write/blocking_bytes_per_sec: 731.8 MiB/s (total bytes: 7.2 GiB) (time elapsed: 10 seconds) (per-host) (1728:base_pytree_checkpoint_handler.py:114)
08:48:44.396 [I] [process=0] /jax/checkpoint/write/blocking_bytes_per_sec: 353.4 MiB/s (total bytes: 3.5 GiB) (time elapsed: 10 seconds) (per-host) (1728:base_pytree_checkpoint_handler.py:114)
08:48:44.409 [I] [process=0][thread=async_save] Background save thread started.                   (1728:async_checkpointer.py:77)
08:48:44.409 [I] Finished blocking save in 10.11 seconds. Continuing to save asynchronously to /workspace/openpi/checkpoints/pi05_mecharm_deep_even10_lora/deep_even10_v1/1000. (1728:async_checkpointer.py:548)
08:48:44.411 [I] [process=0][thread=MainThread][step=1000] Starting CheckpointManager Save Finalize thread=save_finalize (1728:checkpoint_manager.py:1453)
08:48:44.416 [I] [process=0][thread=save_finalize] Waiting for background save thread=async_save. (1728:async_checkpointer.py:258)
08:48:44.418 [I] {'step': 1000, 'event_type': 'save', 'directory': '/workspace/openpi/checkpoints/pi05_mecharm_deep_even10_lora/deep_even10_v1', 'reached_preemption': False, 'preemption_received_at': None, 'synchronous': False, 'wait_for_prev_start_time': 1794214114.2973533, 'wait_for_prev_duration_secs': 0.0001914501190185547, 'checkpointer_blocking_start_time': 1794214114.2982178, 'checkpointer_blocking_duration_secs': 10.112950325012207, 'get_old_steps_start_time': 1794214124.4111974, 'get_old_steps_duration_secs': 8.106231689453125e-06, 'checkpoint_manager_blocking_start_time': 1794214114.2969413, 'checkpoint_manager_blocking_duration_secs': 10.1212158203125} (1728:standard_logger.py:34)
08:48:44.420 [I] Progress on: 1.00kit/3.00kit rate:12.1s/it remaining:6:43:35 elapsed:2:02:33 postfix:- (1728:tqdm_logging.py:145)
08:48:44.491 [I] [process=0][thread=array_type_handler] Wrote 105 array_metadata.ArrayMetadata to /workspace/openpi/checkpoints/pi05_mecharm_deep_even10_lora/deep_even10_v1/1000.orbax-checkpoint-tmp-0/train_state.orbax-checkpoint-tmp-3/array_metadatas/process_0 (1728:array_metadata_store.py:198)
08:48:45.825 [I] [process=0] /jax/checkpoint/write/bytes_per_sec: 640.7 MiB/s (total bytes: 7.2 GiB) (time elapsed: 11 seconds) (per-host) (1728:base_pytree_checkpoint_handler.py:114)
08:48:47.183 [I] [process=0] /jax/checkpoint/write/bytes_per_sec: 276.9 MiB/s (total bytes: 3.5 GiB) (time elapsed: 12 seconds) (per-host) (1728:base_pytree_checkpoint_handler.py:114)
08:48:47.183 [I] [process=0][thread=async_save] 5 Handler Commit operations completed.            (1728:async_checkpointer.py:87)
08:48:47.184 [I] Read Metadata={'item_handlers': None, 'metrics': {}, 'performance_metrics': {}, 'init_timestamp_nsecs': 1794214114317702192, 'commit_timestamp_nsecs': None, 'custom_metadata': {}} from /workspace/openpi/checkpoints/pi05_mecharm_deep_even10_lora/deep_even10_v1/1000.orbax-checkpoint-tmp-0/_CHECKPOINT_METADATA (1728:checkpoint.py:226)
08:48:47.184 [I] Renaming /workspace/openpi/checkpoints/pi05_mecharm_deep_even10_lora/deep_even10_v1/1000.orbax-checkpoint-tmp-0/assets.orbax-checkpoint-tmp-1 to /workspace/openpi/checkpoints/pi05_mecharm_deep_even10_lora/deep_even10_v1/1000.orbax-checkpoint-tmp-0/assets (1728:atomicity.py:289)
08:48:47.187 [I] Updated Metadata={'item_handlers': {'assets': 'openpi.training.checkpoints.CallbackHandler', 'params': 'orbax.checkpoint._src.handlers.pytree_checkpoint_handler.PyTreeCheckpointHandler', 'train_state': 'orbax.checkpoint._src.handlers.pytree_checkpoint_handler.PyTreeCheckpointHandler'}, 'metrics': {}, 'performance_metrics': {}, 'init_timestamp_nsecs': 1794214114317702192, 'commit_timestamp_nsecs': None, 'custom_metadata': {}} to /workspace/openpi/checkpoints/pi05_mecharm_deep_even10_lora/deep_even10_v1/1000.orbax-checkpoint-tmp-0/_CHECKPOINT_METADATA (1728:checkpoint.py:245)
08:48:47.189 [I] [process=0][thread=async_save] Skipped cross-host ArrayMetadata validation because only one process is found: process_index=0. (1728:array_metadata_store.py:362)
08:48:47.194 [I] Renaming /workspace/openpi/checkpoints/pi05_mecharm_deep_even10_lora/deep_even10_v1/1000.orbax-checkpoint-tmp-0/train_state.orbax-checkpoint-tmp-3 to /workspace/openpi/checkpoints/pi05_mecharm_deep_even10_lora/deep_even10_v1/1000.orbax-checkpoint-tmp-0/train_state (1728:atomicity.py:289)
08:48:47.197 [I] [process=0][thread=async_save] Skipped cross-host ArrayMetadata validation because only one process is found: process_index=0. (1728:array_metadata_store.py:362)
08:48:47.201 [I] Renaming /workspace/openpi/checkpoints/pi05_mecharm_deep_even10_lora/deep_even10_v1/1000.orbax-checkpoint-tmp-0/params.orbax-checkpoint-tmp-2 to /workspace/openpi/checkpoints/pi05_mecharm_deep_even10_lora/deep_even10_v1/1000.orbax-checkpoint-tmp-0/params (1728:atomicity.py:289)
08:48:47.201 [I] Renaming /workspace/openpi/checkpoints/pi05_mecharm_deep_even10_lora/deep_even10_v1/1000.orbax-checkpoint-tmp-0 to /workspace/openpi/checkpoints/pi05_mecharm_deep_even10_lora/deep_even10_v1/1000 (1728:atomicity.py:289)
08:48:47.203 [I] [process=0][thread=async_save] Finished saving checkpoint (finalized tmp dir) to `/workspace/openpi/checkpoints/pi05_mecharm_deep_even10_lora/deep_even10_v1/1000`. (1728:atomicity.py:573)
08:48:47.203 [I] Finished asynchronous save (blocking + background) in 12.91 seconds to /workspace/openpi/checkpoints/pi05_mecharm_deep_even10_lora/deep_even10_v1/1000 (1728:async_checkpointer.py:412)
08:48:47.203 [I] [process=0][thread=async_save] Background save thread done.                      (1728:async_checkpointer.py:143)
08:48:47.203 [I] [process=0][thread=save_finalize] Done with waiting for background save thread=async_save. (1728:async_checkpointer.py:266)
08:48:47.203 [I] [process=0][thread=save_finalize] No errors found in background save thread=async_save. (1728:async_checkpointer.py:276)
08:48:47.204 [I] [process=0][thread=save_finalize][step=1000] CheckpointManager Save Finalize is syncing with other hosts... (1728:checkpoint_manager.py:2095)
08:48:47.204 [I] [process=0][thread=save_finalize][step=1000] CheckpointManager Save Finalize is done on all hosts. (1728:checkpoint_manager.py:2110)
08:48:58.859 [I] Progress on: 1.00kit/3.00kit rate:7.8s/it remaining:4:19:03 elapsed:2:02:47 postfix:- (1728:tqdm_logging.py:145)
08:49:12.953 [I] Progress on: 1.01kit/3.00kit rate:7.4s/it remaining:4:06:17 elapsed:2:03:01 postfix:- (1728:tqdm_logging.py:145)
08:49:27.067 [I] Progress on: 1.01kit/3.00kit rate:7.2s/it remaining:4:00:03 elapsed:2:03:15 postfix:- (1728:tqdm_logging.py:145)
08:49:41.207 [I] Progress on: 1.01kit/3.00kit rate:7.1s/it remaining:3:57:05 elapsed:2:03:29 postfix:- (1728:tqdm_logging.py:145)
08:49:55.309 [I] Progress on: 1.01kit/3.00kit rate:7.1s/it remaining:3:55:13 elapsed:2:03:44 postfix:- (1728:tqdm_logging.py:145)
08:50:09.448 [I] Progress on: 1.01kit/3.00kit rate:7.1s/it remaining:3:54:28 elapsed:2:03:58 postfix:- (1728:tqdm_logging.py:145)
08:50:23.587 [I] Progress on: 1.02kit/3.00kit rate:7.1s/it remaining:3:54:00 elapsed:2:04:12 postfix:- (1728:tqdm_logging.py:145)
08:50:37.713 [I] Progress on: 1.02kit/3.00kit rate:7.1s/it remaining:3:53:29 elapsed:2:04:26 postfix:- (1728:tqdm_logging.py:145)
08:50:51.849 [I] Progress on: 1.02kit/3.00kit rate:7.1s/it remaining:3:53:14 elapsed:2:04:40 postfix:- (1728:tqdm_logging.py:145)
08:51:05.996 [I] Progress on: 1.02kit/3.00kit rate:7.1s/it remaining:3:53:06 elapsed:2:04:54 postfix:- (1728:tqdm_logging.py:145)
08:51:20.152 [I] Progress on: 1.02kit/3.00kit rate:7.1s/it remaining:3:52:59 elapsed:2:05:08 postfix:- (1728:tqdm_logging.py:145)
08:51:34.270 [I] Progress on: 1.03kit/3.00kit rate:7.1s/it remaining:3:52:30 elapsed:2:05:23 postfix:- (1728:tqdm_logging.py:145)
08:51:48.430 [I] Progress on: 1.03kit/3.00kit rate:7.1s/it remaining:3:52:28 elapsed:2:05:37 postfix:- (1728:tqdm_logging.py:145)
08:52:02.572 [I] Progress on: 1.03kit/3.00kit rate:7.1s/it remaining:3:52:13 elapsed:2:05:51 postfix:- (1728:tqdm_logging.py:145)
08:52:16.720 [I] Progress on: 1.03kit/3.00kit rate:7.1s/it remaining:3:52:01 elapsed:2:06:05 postfix:- (1728:tqdm_logging.py:145)
08:52:30.857 [I] Progress on: 1.03kit/3.00kit rate:7.1s/it remaining:3:51:41 elapsed:2:06:19 postfix:- (1728:tqdm_logging.py:145)
08:52:44.976 [I] Progress on: 1.04kit/3.00kit rate:7.1s/it remaining:3:51:14 elapsed:2:06:33 postfix:- (1728:tqdm_logging.py:145)
08:52:59.108 [I] Progress on: 1.04kit/3.00kit rate:7.1s/it remaining:3:50:59 elapsed:2:06:47 postfix:- (1728:tqdm_logging.py:145)
08:53:13.227 [I] Progress on: 1.04kit/3.00kit rate:7.1s/it remaining:3:50:40 elapsed:2:07:02 postfix:- (1728:tqdm_logging.py:145)
08:53:27.356 [I] Progress on: 1.04kit/3.00kit rate:7.1s/it remaining:3:50:27 elapsed:2:07:16 postfix:- (1728:tqdm_logging.py:145)
08:53:41.485 [I] Progress on: 1.04kit/3.00kit rate:7.1s/it remaining:3:50:16 elapsed:2:07:30 postfix:- (1728:tqdm_logging.py:145)
08:53:55.610 [I] Progress on: 1.05kit/3.00kit rate:7.1s/it remaining:3:50:04 elapsed:2:07:44 postfix:- (1728:tqdm_logging.py:145)
08:54:09.711 [I] Progress on: 1.05kit/3.00kit rate:7.1s/it remaining:3:49:35 elapsed:2:07:58 postfix:- (1728:tqdm_logging.py:145)
08:54:23.835 [I] Progress on: 1.05kit/3.00kit rate:7.1s/it remaining:3:49:24 elapsed:2:08:12 postfix:- (1728:tqdm_logging.py:145)


Step 1050: grad_norm=0.0482, loss=0.0069, param_norm=1803.9409
08:54:37.585 [I] Progress on: 1.05kit/3.00kit rate:7.1s/it remaining:3:49:24 elapsed:2:08:26 postfix:- (1728:tqdm_logging.py:145)
08:54:52.105 [I] Progress on: 1.05kit/3.00kit rate:6.8s/it remaining:3:39:28 elapsed:2:08:40 postfix:- (1728:tqdm_logging.py:145)
08:55:06.217 [I] Progress on: 1.06kit/3.00kit rate:6.9s/it remaining:3:44:02 elapsed:2:08:55 postfix:- (1728:tqdm_logging.py:145)
08:55:20.320 [I] Progress on: 1.06kit/3.00kit rate:7.0s/it remaining:3:46:01 elapsed:2:09:09 postfix:- (1728:tqdm_logging.py:145)
08:55:34.467 [I] Progress on: 1.06kit/3.00kit rate:7.0s/it remaining:3:47:14 elapsed:2:09:23 postfix:- (1728:tqdm_logging.py:145)
08:55:48.581 [I] Progress on: 1.06kit/3.00kit rate:7.0s/it remaining:3:47:28 elapsed:2:09:37 postfix:- (1728:tqdm_logging.py:145)
08:56:02.701 [I] Progress on: 1.06kit/3.00kit rate:7.1s/it remaining:3:47:30 elapsed:2:09:51 postfix:- (1728:tqdm_logging.py:145)
08:56:16.818 [I] Progress on: 1.07kit/3.00kit rate:7.1s/it remaining:3:47:24 elapsed:2:10:05 postfix:- (1728:tqdm_logging.py:145)
08:56:30.928 [I] Progress on: 1.07kit/3.00kit rate:7.1s/it remaining:3:47:08 elapsed:2:10:19 postfix:- (1728:tqdm_logging.py:145)
08:56:45.045 [I] Progress on: 1.07kit/3.00kit rate:7.1s/it remaining:3:46:58 elapsed:2:10:33 postfix:- (1728:tqdm_logging.py:145)
08:56:59.173 [I] Progress on: 1.07kit/3.00kit rate:7.1s/it remaining:3:46:51 elapsed:2:10:47 postfix:- (1728:tqdm_logging.py:145)
08:57:13.280 [I] Progress on: 1.07kit/3.00kit rate:7.1s/it remaining:3:46:31 elapsed:2:11:02 postfix:- (1728:tqdm_logging.py:145)
08:57:27.391 [I] Progress on: 1.08kit/3.00kit rate:7.1s/it remaining:3:46:16 elapsed:2:11:16 postfix:- (1728:tqdm_logging.py:145)
08:57:41.513 [I] Progress on: 1.08kit/3.00kit rate:7.1s/it remaining:3:46:03 elapsed:2:11:30 postfix:- (1728:tqdm_logging.py:145)
08:57:55.645 [I] Progress on: 1.08kit/3.00kit rate:7.1s/it remaining:3:45:59 elapsed:2:11:44 postfix:- (1728:tqdm_logging.py:145)
08:58:09.756 [I] Progress on: 1.08kit/3.00kit rate:7.1s/it remaining:3:45:38 elapsed:2:11:58 postfix:- (1728:tqdm_logging.py:145)
08:58:23.870 [I] Progress on: 1.08kit/3.00kit rate:7.1s/it remaining:3:45:21 elapsed:2:12:12 postfix:- (1728:tqdm_logging.py:145)
08:58:37.991 [I] Progress on: 1.09kit/3.00kit rate:7.1s/it remaining:3:45:08 elapsed:2:12:26 postfix:- (1728:tqdm_logging.py:145)
08:58:52.098 [I] Progress on: 1.09kit/3.00kit rate:7.1s/it remaining:3:44:48 elapsed:2:12:40 postfix:- (1728:tqdm_logging.py:145)
08:59:06.226 [I] Progress on: 1.09kit/3.00kit rate:7.1s/it remaining:3:44:41 elapsed:2:12:55 postfix:- (1728:tqdm_logging.py:145)
08:59:20.349 [I] Progress on: 1.09kit/3.00kit rate:7.1s/it remaining:3:44:32 elapsed:2:13:09 postfix:- (1728:tqdm_logging.py:145)
08:59:34.476 [I] Progress on: 1.09kit/3.00kit rate:7.1s/it remaining:3:44:19 elapsed:2:13:23 postfix:- (1728:tqdm_logging.py:145)
08:59:48.606 [I] Progress on: 1.10kit/3.00kit rate:7.1s/it remaining:3:44:10 elapsed:2:13:37 postfix:- (1728:tqdm_logging.py:145)
09:00:02.706 [I] Progress on: 1.10kit/3.00kit rate:7.1s/it remaining:3:43:44 elapsed:2:13:51 postfix:- (1728:tqdm_logging.py:145)
09:00:16.837 [I] Progress on: 1.10kit/3.00kit rate:7.1s/it remaining:3:43:35 elapsed:2:14:05 postfix:- (1728:tqdm_logging.py:145)


Step 1100: grad_norm=0.0518, loss=0.0075, param_norm=1803.9497
09:00:30.606 [I] Progress on: 1.10kit/3.00kit rate:7.1s/it remaining:3:43:35 elapsed:2:14:19 postfix:- (1728:tqdm_logging.py:145)
09:00:45.154 [I] Progress on: 1.10kit/3.00kit rate:6.8s/it remaining:3:33:59 elapsed:2:14:33 postfix:- (1728:tqdm_logging.py:145)
09:00:59.266 [I] Progress on: 1.11kit/3.00kit rate:6.9s/it remaining:3:38:18 elapsed:2:14:48 postfix:- (1728:tqdm_logging.py:145)
09:01:13.390 [I] Progress on: 1.11kit/3.00kit rate:7.0s/it remaining:3:40:25 elapsed:2:15:02 postfix:- (1728:tqdm_logging.py:145)
09:01:28.056 [I] Progress on: 1.11kit/3.00kit rate:7.2s/it remaining:3:46:29 elapsed:2:15:16 postfix:- (1728:tqdm_logging.py:145)
09:01:41.620 [I] Progress on: 1.11kit/3.00kit rate:7.0s/it remaining:3:40:27 elapsed:2:15:30 postfix:- (1728:tqdm_logging.py:145)
09:01:55.736 [I] Progress on: 1.11kit/3.00kit rate:7.0s/it remaining:3:41:04 elapsed:2:15:44 postfix:- (1728:tqdm_logging.py:145)
09:02:09.880 [I] Progress on: 1.12kit/3.00kit rate:7.1s/it remaining:3:41:28 elapsed:2:15:58 postfix:- (1728:tqdm_logging.py:145)
09:02:24.002 [I] Progress on: 1.12kit/3.00kit rate:7.1s/it remaining:3:41:20 elapsed:2:16:12 postfix:- (1728:tqdm_logging.py:145)
09:02:38.145 [I] Progress on: 1.12kit/3.00kit rate:7.1s/it remaining:3:41:20 elapsed:2:16:26 postfix:- (1728:tqdm_logging.py:145)
09:02:52.254 [I] Progress on: 1.12kit/3.00kit rate:7.1s/it remaining:3:40:56 elapsed:2:16:41 postfix:- (1728:tqdm_logging.py:145)
09:03:06.378 [I] Progress on: 1.12kit/3.00kit rate:7.1s/it remaining:3:40:45 elapsed:2:16:55 postfix:- (1728:tqdm_logging.py:145)
09:03:20.480 [I] Progress on: 1.13kit/3.00kit rate:7.1s/it remaining:3:40:20 elapsed:2:17:09 postfix:- (1728:tqdm_logging.py:145)
09:03:34.609 [I] Progress on: 1.13kit/3.00kit rate:7.1s/it remaining:3:40:13 elapsed:2:17:23 postfix:- (1728:tqdm_logging.py:145)
09:03:48.741 [I] Progress on: 1.13kit/3.00kit rate:7.1s/it remaining:3:40:05 elapsed:2:17:37 postfix:- (1728:tqdm_logging.py:145)
09:04:02.869 [I] Progress on: 1.13kit/3.00kit rate:7.1s/it remaining:3:39:52 elapsed:2:17:51 postfix:- (1728:tqdm_logging.py:145)
09:04:16.987 [I] Progress on: 1.13kit/3.00kit rate:7.1s/it remaining:3:39:36 elapsed:2:18:05 postfix:- (1728:tqdm_logging.py:145)
09:04:31.104 [I] Progress on: 1.14kit/3.00kit rate:7.1s/it remaining:3:39:17 elapsed:2:18:19 postfix:- (1728:tqdm_logging.py:145)
09:04:45.220 [I] Progress on: 1.14kit/3.00kit rate:7.1s/it remaining:3:39:01 elapsed:2:18:34 postfix:- (1728:tqdm_logging.py:145)
09:04:59.348 [I] Progress on: 1.14kit/3.00kit rate:7.1s/it remaining:3:38:52 elapsed:2:18:48 postfix:- (1728:tqdm_logging.py:145)
09:05:13.467 [I] Progress on: 1.14kit/3.00kit rate:7.1s/it remaining:3:38:38 elapsed:2:19:02 postfix:- (1728:tqdm_logging.py:145)
09:05:27.597 [I] Progress on: 1.14kit/3.00kit rate:7.1s/it remaining:3:38:29 elapsed:2:19:16 postfix:- (1728:tqdm_logging.py:145)
09:05:41.740 [I] Progress on: 1.15kit/3.00kit rate:7.1s/it remaining:3:38:23 elapsed:2:19:30 postfix:- (1728:tqdm_logging.py:145)
09:05:55.862 [I] Progress on: 1.15kit/3.00kit rate:7.1s/it remaining:3:38:02 elapsed:2:19:44 postfix:- (1728:tqdm_logging.py:145)
09:06:09.970 [I] Progress on: 1.15kit/3.00kit rate:7.1s/it remaining:3:37:38 elapsed:2:19:58 postfix:- (1728:tqdm_logging.py:145)


Step 1150: grad_norm=0.0504, loss=0.0066, param_norm=1803.9578
09:06:23.742 [I] Progress on: 1.15kit/3.00kit rate:7.1s/it remaining:3:37:38 elapsed:2:20:12 postfix:- (1728:tqdm_logging.py:145)
09:06:38.291 [I] Progress on: 1.15kit/3.00kit rate:6.8s/it remaining:3:28:30 elapsed:2:20:27 postfix:- (1728:tqdm_logging.py:145)
09:06:52.416 [I] Progress on: 1.16kit/3.00kit rate:6.9s/it remaining:3:32:45 elapsed:2:20:41 postfix:- (1728:tqdm_logging.py:145)
09:07:06.525 [I] Progress on: 1.16kit/3.00kit rate:7.0s/it remaining:3:34:35 elapsed:2:20:55 postfix:- (1728:tqdm_logging.py:145)
09:07:20.659 [I] Progress on: 1.16kit/3.00kit rate:7.0s/it remaining:3:35:32 elapsed:2:21:09 postfix:- (1728:tqdm_logging.py:145)
09:07:34.792 [I] Progress on: 1.16kit/3.00kit rate:7.0s/it remaining:3:35:53 elapsed:2:21:23 postfix:- (1728:tqdm_logging.py:145)
09:07:48.901 [I] Progress on: 1.16kit/3.00kit rate:7.1s/it remaining:3:35:46 elapsed:2:21:37 postfix:- (1728:tqdm_logging.py:145)
09:08:03.035 [I] Progress on: 1.17kit/3.00kit rate:7.1s/it remaining:3:35:45 elapsed:2:21:51 postfix:- (1728:tqdm_logging.py:145)
09:08:17.160 [I] Progress on: 1.17kit/3.00kit rate:7.1s/it remaining:3:35:33 elapsed:2:22:05 postfix:- (1728:tqdm_logging.py:145)
09:08:31.289 [I] Progress on: 1.17kit/3.00kit rate:7.1s/it remaining:3:35:24 elapsed:2:22:20 postfix:- (1728:tqdm_logging.py:145)
09:08:45.404 [I] Progress on: 1.17kit/3.00kit rate:7.1s/it remaining:3:35:04 elapsed:2:22:34 postfix:- (1728:tqdm_logging.py:145)
09:08:59.527 [I] Progress on: 1.17kit/3.00kit rate:7.1s/it remaining:3:34:52 elapsed:2:22:48 postfix:- (1728:tqdm_logging.py:145)
09:09:13.629 [I] Progress on: 1.18kit/3.00kit rate:7.1s/it remaining:3:34:30 elapsed:2:23:02 postfix:- (1728:tqdm_logging.py:145)
09:09:27.742 [I] Progress on: 1.18kit/3.00kit rate:7.1s/it remaining:3:34:14 elapsed:2:23:16 postfix:- (1728:tqdm_logging.py:145)
09:09:41.859 [I] Progress on: 1.18kit/3.00kit rate:7.1s/it remaining:3:34:02 elapsed:2:23:30 postfix:- (1728:tqdm_logging.py:145)
09:09:55.985 [I] Progress on: 1.18kit/3.00kit rate:7.1s/it remaining:3:33:55 elapsed:2:23:44 postfix:- (1728:tqdm_logging.py:145)
09:10:10.102 [I] Progress on: 1.18kit/3.00kit rate:7.1s/it remaining:3:33:40 elapsed:2:23:58 postfix:- (1728:tqdm_logging.py:145)
09:10:24.194 [I] Progress on: 1.19kit/3.00kit rate:7.1s/it remaining:3:33:13 elapsed:2:24:12 postfix:- (1728:tqdm_logging.py:145)
09:10:38.323 [I] Progress on: 1.19kit/3.00kit rate:7.1s/it remaining:3:33:06 elapsed:2:24:27 postfix:- (1728:tqdm_logging.py:145)
09:10:52.450 [I] Progress on: 1.19kit/3.00kit rate:7.1s/it remaining:3:32:59 elapsed:2:24:41 postfix:- (1728:tqdm_logging.py:145)
09:11:06.556 [I] Progress on: 1.19kit/3.00kit rate:7.1s/it remaining:3:32:37 elapsed:2:24:55 postfix:- (1728:tqdm_logging.py:145)
09:11:20.665 [I] Progress on: 1.19kit/3.00kit rate:7.1s/it remaining:3:32:22 elapsed:2:25:09 postfix:- (1728:tqdm_logging.py:145)
09:11:34.779 [I] Progress on: 1.20kit/3.00kit rate:7.1s/it remaining:3:32:08 elapsed:2:25:23 postfix:- (1728:tqdm_logging.py:145)
09:11:48.905 [I] Progress on: 1.20kit/3.00kit rate:7.1s/it remaining:3:31:59 elapsed:2:25:37 postfix:- (1728:tqdm_logging.py:145)
09:12:03.026 [I] Progress on: 1.20kit/3.00kit rate:7.1s/it remaining:3:31:48 elapsed:2:25:51 postfix:- (1728:tqdm_logging.py:145)


Step 1200: grad_norm=0.0484, loss=0.0066, param_norm=1803.9659
09:12:16.769 [I] Progress on: 1.20kit/3.00kit rate:7.1s/it remaining:3:31:48 elapsed:2:26:05 postfix:- (1728:tqdm_logging.py:145)
09:12:31.292 [I] Progress on: 1.20kit/3.00kit rate:6.8s/it remaining:3:22:32 elapsed:2:26:20 postfix:- (1728:tqdm_logging.py:145)
09:12:45.410 [I] Progress on: 1.21kit/3.00kit rate:6.9s/it remaining:3:26:47 elapsed:2:26:34 postfix:- (1728:tqdm_logging.py:145)
09:12:59.525 [I] Progress on: 1.21kit/3.00kit rate:7.0s/it remaining:3:28:43 elapsed:2:26:48 postfix:- (1728:tqdm_logging.py:145)
09:13:13.643 [I] Progress on: 1.21kit/3.00kit rate:7.0s/it remaining:3:29:35 elapsed:2:27:02 postfix:- (1728:tqdm_logging.py:145)
09:13:27.764 [I] Progress on: 1.21kit/3.00kit rate:7.0s/it remaining:3:29:53 elapsed:2:27:16 postfix:- (1728:tqdm_logging.py:145)
09:13:41.896 [I] Progress on: 1.21kit/3.00kit rate:7.1s/it remaining:3:29:59 elapsed:2:27:30 postfix:- (1728:tqdm_logging.py:145)
09:13:56.011 [I] Progress on: 1.22kit/3.00kit rate:7.1s/it remaining:3:29:48 elapsed:2:27:44 postfix:- (1728:tqdm_logging.py:145)
09:14:10.128 [I] Progress on: 1.22kit/3.00kit rate:7.1s/it remaining:3:29:35 elapsed:2:27:58 postfix:- (1728:tqdm_logging.py:145)
09:14:24.257 [I] Progress on: 1.22kit/3.00kit rate:7.1s/it remaining:3:29:26 elapsed:2:28:13 postfix:- (1728:tqdm_logging.py:145)
09:14:38.378 [I] Progress on: 1.22kit/3.00kit rate:7.1s/it remaining:3:29:12 elapsed:2:28:27 postfix:- (1728:tqdm_logging.py:145)
09:14:52.492 [I] Progress on: 1.22kit/3.00kit rate:7.1s/it remaining:3:28:56 elapsed:2:28:41 postfix:- (1728:tqdm_logging.py:145)
09:15:06.614 [I] Progress on: 1.23kit/3.00kit rate:7.1s/it remaining:3:28:43 elapsed:2:28:55 postfix:- (1728:tqdm_logging.py:145)
09:15:20.756 [I] Progress on: 1.23kit/3.00kit rate:7.1s/it remaining:3:28:36 elapsed:2:29:09 postfix:- (1728:tqdm_logging.py:145)
09:15:34.884 [I] Progress on: 1.23kit/3.00kit rate:7.1s/it remaining:3:28:23 elapsed:2:29:23 postfix:- (1728:tqdm_logging.py:145)
09:15:48.985 [I] Progress on: 1.23kit/3.00kit rate:7.1s/it remaining:3:27:57 elapsed:2:29:37 postfix:- (1728:tqdm_logging.py:145)
09:16:03.095 [I] Progress on: 1.23kit/3.00kit rate:7.1s/it remaining:3:27:40 elapsed:2:29:51 postfix:- (1728:tqdm_logging.py:145)
09:16:17.217 [I] Progress on: 1.24kit/3.00kit rate:7.1s/it remaining:3:27:31 elapsed:2:30:06 postfix:- (1728:tqdm_logging.py:145)
09:16:31.353 [I] Progress on: 1.24kit/3.00kit rate:7.1s/it remaining:3:27:23 elapsed:2:30:20 postfix:- (1728:tqdm_logging.py:145)
09:16:45.482 [I] Progress on: 1.24kit/3.00kit rate:7.1s/it remaining:3:27:13 elapsed:2:30:34 postfix:- (1728:tqdm_logging.py:145)
09:16:59.591 [I] Progress on: 1.24kit/3.00kit rate:7.1s/it remaining:3:26:49 elapsed:2:30:48 postfix:- (1728:tqdm_logging.py:145)
09:17:13.711 [I] Progress on: 1.24kit/3.00kit rate:7.1s/it remaining:3:26:35 elapsed:2:31:02 postfix:- (1728:tqdm_logging.py:145)
09:17:27.834 [I] Progress on: 1.25kit/3.00kit rate:7.1s/it remaining:3:26:25 elapsed:2:31:16 postfix:- (1728:tqdm_logging.py:145)
09:17:41.960 [I] Progress on: 1.25kit/3.00kit rate:7.1s/it remaining:3:26:13 elapsed:2:31:30 postfix:- (1728:tqdm_logging.py:145)
09:17:56.072 [I] Progress on: 1.25kit/3.00kit rate:7.1s/it remaining:3:25:54 elapsed:2:31:44 postfix:- (1728:tqdm_logging.py:145)


Step 1250: grad_norm=0.0472, loss=0.0062, param_norm=1803.9744
09:18:09.830 [I] Progress on: 1.25kit/3.00kit rate:7.1s/it remaining:3:25:54 elapsed:2:31:58 postfix:- (1728:tqdm_logging.py:145)
09:18:24.384 [I] Progress on: 1.25kit/3.00kit rate:6.8s/it remaining:3:17:04 elapsed:2:32:13 postfix:- (1728:tqdm_logging.py:145)
09:18:38.488 [I] Progress on: 1.26kit/3.00kit rate:6.9s/it remaining:3:21:00 elapsed:2:32:27 postfix:- (1728:tqdm_logging.py:145)
09:18:52.604 [I] Progress on: 1.26kit/3.00kit rate:7.0s/it remaining:3:22:52 elapsed:2:32:41 postfix:- (1728:tqdm_logging.py:145)
09:19:06.719 [I] Progress on: 1.26kit/3.00kit rate:7.0s/it remaining:3:23:40 elapsed:2:32:55 postfix:- (1728:tqdm_logging.py:145)
09:19:20.835 [I] Progress on: 1.26kit/3.00kit rate:7.0s/it remaining:3:23:56 elapsed:2:33:09 postfix:- (1728:tqdm_logging.py:145)
09:19:34.966 [I] Progress on: 1.26kit/3.00kit rate:7.1s/it remaining:3:24:03 elapsed:2:33:23 postfix:- (1728:tqdm_logging.py:145)
09:19:49.078 [I] Progress on: 1.27kit/3.00kit rate:7.1s/it remaining:3:23:51 elapsed:2:33:37 postfix:- (1728:tqdm_logging.py:145)
09:20:03.205 [I] Progress on: 1.27kit/3.00kit rate:7.1s/it remaining:3:23:46 elapsed:2:33:51 postfix:- (1728:tqdm_logging.py:145)
09:20:17.312 [I] Progress on: 1.27kit/3.00kit rate:7.1s/it remaining:3:23:28 elapsed:2:34:06 postfix:- (1728:tqdm_logging.py:145)
09:20:31.424 [I] Progress on: 1.27kit/3.00kit rate:7.1s/it remaining:3:23:13 elapsed:2:34:20 postfix:- (1728:tqdm_logging.py:145)
09:20:45.559 [I] Progress on: 1.27kit/3.00kit rate:7.1s/it remaining:3:23:07 elapsed:2:34:34 postfix:- (1728:tqdm_logging.py:145)
09:20:59.674 [I] Progress on: 1.28kit/3.00kit rate:7.1s/it remaining:3:22:51 elapsed:2:34:48 postfix:- (1728:tqdm_logging.py:145)
09:21:13.776 [I] Progress on: 1.28kit/3.00kit rate:7.1s/it remaining:3:22:30 elapsed:2:35:02 postfix:- (1728:tqdm_logging.py:145)
09:21:27.888 [I] Progress on: 1.28kit/3.00kit rate:7.1s/it remaining:3:22:14 elapsed:2:35:16 postfix:- (1728:tqdm_logging.py:145)
09:21:42.011 [I] Progress on: 1.28kit/3.00kit rate:7.1s/it remaining:3:22:04 elapsed:2:35:30 postfix:- (1728:tqdm_logging.py:145)
09:21:56.117 [I] Progress on: 1.28kit/3.00kit rate:7.1s/it remaining:3:21:48 elapsed:2:35:44 postfix:- (1728:tqdm_logging.py:145)
09:22:10.235 [I] Progress on: 1.29kit/3.00kit rate:7.1s/it remaining:3:21:37 elapsed:2:35:59 postfix:- (1728:tqdm_logging.py:145)
09:22:24.338 [I] Progress on: 1.29kit/3.00kit rate:7.1s/it remaining:3:21:17 elapsed:2:36:13 postfix:- (1728:tqdm_logging.py:145)
09:22:38.458 [I] Progress on: 1.29kit/3.00kit rate:7.1s/it remaining:3:21:06 elapsed:2:36:27 postfix:- (1728:tqdm_logging.py:145)
09:22:52.585 [I] Progress on: 1.29kit/3.00kit rate:7.1s/it remaining:3:20:59 elapsed:2:36:41 postfix:- (1728:tqdm_logging.py:145)
09:23:06.694 [I] Progress on: 1.29kit/3.00kit rate:7.1s/it remaining:3:20:41 elapsed:2:36:55 postfix:- (1728:tqdm_logging.py:145)
09:23:20.806 [I] Progress on: 1.30kit/3.00kit rate:7.1s/it remaining:3:20:24 elapsed:2:37:09 postfix:- (1728:tqdm_logging.py:145)
09:23:34.924 [I] Progress on: 1.30kit/3.00kit rate:7.1s/it remaining:3:20:11 elapsed:2:37:23 postfix:- (1728:tqdm_logging.py:145)
09:23:49.041 [I] Progress on: 1.30kit/3.00kit rate:7.1s/it remaining:3:19:58 elapsed:2:37:37 postfix:- (1728:tqdm_logging.py:145)


Step 1300: grad_norm=0.0461, loss=0.0064, param_norm=1803.9818
09:24:02.800 [I] Progress on: 1.30kit/3.00kit rate:7.1s/it remaining:3:19:58 elapsed:2:37:51 postfix:- (1728:tqdm_logging.py:145)
09:24:17.366 [I] Progress on: 1.30kit/3.00kit rate:6.8s/it remaining:3:11:32 elapsed:2:38:06 postfix:- (1728:tqdm_logging.py:145)
09:24:31.466 [I] Progress on: 1.31kit/3.00kit rate:6.9s/it remaining:3:15:14 elapsed:2:38:20 postfix:- (1728:tqdm_logging.py:145)
09:24:45.588 [I] Progress on: 1.31kit/3.00kit rate:7.0s/it remaining:3:17:07 elapsed:2:38:34 postfix:- (1728:tqdm_logging.py:145)
09:24:59.706 [I] Progress on: 1.31kit/3.00kit rate:7.0s/it remaining:3:17:53 elapsed:2:38:48 postfix:- (1728:tqdm_logging.py:145)
09:25:13.815 [I] Progress on: 1.31kit/3.00kit rate:7.0s/it remaining:3:18:04 elapsed:2:39:02 postfix:- (1728:tqdm_logging.py:145)
09:25:27.918 [I] Progress on: 1.31kit/3.00kit rate:7.0s/it remaining:3:17:58 elapsed:2:39:16 postfix:- (1728:tqdm_logging.py:145)
09:25:42.053 [I] Progress on: 1.32kit/3.00kit rate:7.1s/it remaining:3:18:02 elapsed:2:39:30 postfix:- (1728:tqdm_logging.py:145)
09:25:56.158 [I] Progress on: 1.32kit/3.00kit rate:7.1s/it remaining:3:17:43 elapsed:2:39:44 postfix:- (1728:tqdm_logging.py:145)
09:26:10.305 [I] Progress on: 1.32kit/3.00kit rate:7.1s/it remaining:3:17:46 elapsed:2:39:59 postfix:- (1728:tqdm_logging.py:145)
09:26:24.412 [I] Progress on: 1.32kit/3.00kit rate:7.1s/it remaining:3:17:24 elapsed:2:40:13 postfix:- (1728:tqdm_logging.py:145)
09:26:38.522 [I] Progress on: 1.32kit/3.00kit rate:7.1s/it remaining:3:17:08 elapsed:2:40:27 postfix:- (1728:tqdm_logging.py:145)
09:26:52.649 [I] Progress on: 1.33kit/3.00kit rate:7.1s/it remaining:3:16:59 elapsed:2:40:41 postfix:- (1728:tqdm_logging.py:145)
09:27:06.762 [I] Progress on: 1.33kit/3.00kit rate:7.1s/it remaining:3:16:42 elapsed:2:40:55 postfix:- (1728:tqdm_logging.py:145)
09:27:20.861 [I] Progress on: 1.33kit/3.00kit rate:7.1s/it remaining:3:16:20 elapsed:2:41:09 postfix:- (1728:tqdm_logging.py:145)
09:27:35.545 [I] Progress on: 1.33kit/3.00kit rate:7.2s/it remaining:3:20:50 elapsed:2:41:24 postfix:- (1728:tqdm_logging.py:145)
09:27:49.134 [I] Progress on: 1.33kit/3.00kit rate:7.0s/it remaining:3:15:17 elapsed:2:41:37 postfix:- (1728:tqdm_logging.py:145)
09:28:03.241 [I] Progress on: 1.34kit/3.00kit rate:7.0s/it remaining:3:15:26 elapsed:2:41:52 postfix:- (1728:tqdm_logging.py:145)
09:28:17.356 [I] Progress on: 1.34kit/3.00kit rate:7.1s/it remaining:3:15:25 elapsed:2:42:06 postfix:- (1728:tqdm_logging.py:145)
09:28:31.466 [I] Progress on: 1.34kit/3.00kit rate:7.1s/it remaining:3:15:17 elapsed:2:42:20 postfix:- (1728:tqdm_logging.py:145)
09:28:45.588 [I] Progress on: 1.34kit/3.00kit rate:7.1s/it remaining:3:15:10 elapsed:2:42:34 postfix:- (1728:tqdm_logging.py:145)
09:28:59.699 [I] Progress on: 1.34kit/3.00kit rate:7.1s/it remaining:3:14:55 elapsed:2:42:48 postfix:- (1728:tqdm_logging.py:145)
09:29:13.825 [I] Progress on: 1.35kit/3.00kit rate:7.1s/it remaining:3:14:47 elapsed:2:43:02 postfix:- (1728:tqdm_logging.py:145)
09:29:27.934 [I] Progress on: 1.35kit/3.00kit rate:7.1s/it remaining:3:14:29 elapsed:2:43:16 postfix:- (1728:tqdm_logging.py:145)
09:29:42.031 [I] Progress on: 1.35kit/3.00kit rate:7.1s/it remaining:3:14:08 elapsed:2:43:30 postfix:- (1728:tqdm_logging.py:145)


Step 1350: grad_norm=0.0502, loss=0.0063, param_norm=1803.9896
09:29:55.766 [I] Progress on: 1.35kit/3.00kit rate:7.1s/it remaining:3:14:08 elapsed:2:43:44 postfix:- (1728:tqdm_logging.py:145)
09:30:10.284 [I] Progress on: 1.35kit/3.00kit rate:6.8s/it remaining:3:05:38 elapsed:2:43:59 postfix:- (1728:tqdm_logging.py:145)
09:30:24.398 [I] Progress on: 1.36kit/3.00kit rate:6.9s/it remaining:3:09:29 elapsed:2:44:13 postfix:- (1728:tqdm_logging.py:145)
09:30:38.503 [I] Progress on: 1.36kit/3.00kit rate:7.0s/it remaining:3:11:09 elapsed:2:44:27 postfix:- (1728:tqdm_logging.py:145)
09:30:52.618 [I] Progress on: 1.36kit/3.00kit rate:7.0s/it remaining:3:11:55 elapsed:2:44:41 postfix:- (1728:tqdm_logging.py:145)
09:31:06.725 [I] Progress on: 1.36kit/3.00kit rate:7.0s/it remaining:3:12:09 elapsed:2:44:55 postfix:- (1728:tqdm_logging.py:145)
09:31:20.837 [I] Progress on: 1.36kit/3.00kit rate:7.0s/it remaining:3:12:08 elapsed:2:45:09 postfix:- (1728:tqdm_logging.py:145)
09:31:34.962 [I] Progress on: 1.37kit/3.00kit rate:7.1s/it remaining:3:12:07 elapsed:2:45:23 postfix:- (1728:tqdm_logging.py:145)
09:31:49.068 [I] Progress on: 1.37kit/3.00kit rate:7.1s/it remaining:3:11:53 elapsed:2:45:37 postfix:- (1728:tqdm_logging.py:145)
09:32:03.180 [I] Progress on: 1.37kit/3.00kit rate:7.1s/it remaining:3:11:41 elapsed:2:45:51 postfix:- (1728:tqdm_logging.py:145)
09:32:17.280 [I] Progress on: 1.37kit/3.00kit rate:7.1s/it remaining:3:11:21 elapsed:2:46:06 postfix:- (1728:tqdm_logging.py:145)
09:32:31.396 [I] Progress on: 1.37kit/3.00kit rate:7.1s/it remaining:3:11:11 elapsed:2:46:20 postfix:- (1728:tqdm_logging.py:145)
09:32:45.499 [I] Progress on: 1.38kit/3.00kit rate:7.1s/it remaining:3:10:54 elapsed:2:46:34 postfix:- (1728:tqdm_logging.py:145)
09:32:59.610 [I] Progress on: 1.38kit/3.00kit rate:7.1s/it remaining:3:10:43 elapsed:2:46:48 postfix:- (1728:tqdm_logging.py:145)
09:33:13.721 [I] Progress on: 1.38kit/3.00kit rate:7.1s/it remaining:3:10:27 elapsed:2:47:02 postfix:- (1728:tqdm_logging.py:145)
09:33:27.840 [I] Progress on: 1.38kit/3.00kit rate:7.1s/it remaining:3:10:17 elapsed:2:47:16 postfix:- (1728:tqdm_logging.py:145)
09:33:41.953 [I] Progress on: 1.38kit/3.00kit rate:7.1s/it remaining:3:10:03 elapsed:2:47:30 postfix:- (1728:tqdm_logging.py:145)
09:33:56.070 [I] Progress on: 1.39kit/3.00kit rate:7.1s/it remaining:3:09:50 elapsed:2:47:44 postfix:- (1728:tqdm_logging.py:145)
09:34:10.177 [I] Progress on: 1.39kit/3.00kit rate:7.1s/it remaining:3:09:33 elapsed:2:47:58 postfix:- (1728:tqdm_logging.py:145)
09:34:24.319 [I] Progress on: 1.39kit/3.00kit rate:7.1s/it remaining:3:09:32 elapsed:2:48:13 postfix:- (1728:tqdm_logging.py:145)
09:34:38.429 [I] Progress on: 1.39kit/3.00kit rate:7.1s/it remaining:3:09:13 elapsed:2:48:27 postfix:- (1728:tqdm_logging.py:145)
09:34:52.538 [I] Progress on: 1.39kit/3.00kit rate:7.1s/it remaining:3:08:55 elapsed:2:48:41 postfix:- (1728:tqdm_logging.py:145)
09:35:06.668 [I] Progress on: 1.40kit/3.00kit rate:7.1s/it remaining:3:08:48 elapsed:2:48:55 postfix:- (1728:tqdm_logging.py:145)
09:35:20.786 [I] Progress on: 1.40kit/3.00kit rate:7.1s/it remaining:3:08:31 elapsed:2:49:09 postfix:- (1728:tqdm_logging.py:145)
09:35:34.900 [I] Progress on: 1.40kit/3.00kit rate:7.1s/it remaining:3:08:13 elapsed:2:49:23 postfix:- (1728:tqdm_logging.py:145)


Step 1400: grad_norm=0.0499, loss=0.0061, param_norm=1803.9980
09:35:48.655 [I] Progress on: 1.40kit/3.00kit rate:7.1s/it remaining:3:08:13 elapsed:2:49:37 postfix:- (1728:tqdm_logging.py:145)
09:36:03.201 [I] Progress on: 1.40kit/3.00kit rate:6.8s/it remaining:3:00:03 elapsed:2:49:51 postfix:- (1728:tqdm_logging.py:145)
09:36:17.367 [I] Progress on: 1.41kit/3.00kit rate:6.9s/it remaining:3:04:10 elapsed:2:50:06 postfix:- (1728:tqdm_logging.py:145)
09:36:31.423 [I] Progress on: 1.41kit/3.00kit rate:7.0s/it remaining:3:05:13 elapsed:2:50:20 postfix:- (1728:tqdm_logging.py:145)
09:36:45.534 [I] Progress on: 1.41kit/3.00kit rate:7.0s/it remaining:3:05:59 elapsed:2:50:34 postfix:- (1728:tqdm_logging.py:145)
09:36:59.651 [I] Progress on: 1.41kit/3.00kit rate:7.0s/it remaining:3:06:19 elapsed:2:50:48 postfix:- (1728:tqdm_logging.py:145)
09:37:13.752 [I] Progress on: 1.41kit/3.00kit rate:7.0s/it remaining:3:06:14 elapsed:2:51:02 postfix:- (1728:tqdm_logging.py:145)
09:37:27.856 [I] Progress on: 1.42kit/3.00kit rate:7.0s/it remaining:3:06:06 elapsed:2:51:16 postfix:- (1728:tqdm_logging.py:145)
09:37:41.974 [I] Progress on: 1.42kit/3.00kit rate:7.1s/it remaining:3:05:57 elapsed:2:51:30 postfix:- (1728:tqdm_logging.py:145)
09:37:56.078 [I] Progress on: 1.42kit/3.00kit rate:7.0s/it remaining:3:05:38 elapsed:2:51:44 postfix:- (1728:tqdm_logging.py:145)
09:38:10.212 [I] Progress on: 1.42kit/3.00kit rate:7.1s/it remaining:3:05:37 elapsed:2:51:58 postfix:- (1728:tqdm_logging.py:145)
09:38:24.310 [I] Progress on: 1.42kit/3.00kit rate:7.1s/it remaining:3:05:17 elapsed:2:52:13 postfix:- (1728:tqdm_logging.py:145)
09:38:38.452 [I] Progress on: 1.43kit/3.00kit rate:7.1s/it remaining:3:05:10 elapsed:2:52:27 postfix:- (1728:tqdm_logging.py:145)
09:38:52.550 [I] Progress on: 1.43kit/3.00kit rate:7.1s/it remaining:3:04:45 elapsed:2:52:41 postfix:- (1728:tqdm_logging.py:145)
09:39:06.661 [I] Progress on: 1.43kit/3.00kit rate:7.1s/it remaining:3:04:34 elapsed:2:52:55 postfix:- (1728:tqdm_logging.py:145)
09:39:20.780 [I] Progress on: 1.43kit/3.00kit rate:7.1s/it remaining:3:04:26 elapsed:2:53:09 postfix:- (1728:tqdm_logging.py:145)
09:39:34.895 [I] Progress on: 1.43kit/3.00kit rate:7.1s/it remaining:3:04:13 elapsed:2:53:23 postfix:- (1728:tqdm_logging.py:145)
09:39:49.032 [I] Progress on: 1.44kit/3.00kit rate:7.1s/it remaining:3:04:08 elapsed:2:53:37 postfix:- (1728:tqdm_logging.py:145)
09:40:03.134 [I] Progress on: 1.44kit/3.00kit rate:7.1s/it remaining:3:03:43 elapsed:2:53:51 postfix:- (1728:tqdm_logging.py:145)
09:40:17.258 [I] Progress on: 1.44kit/3.00kit rate:7.1s/it remaining:3:03:33 elapsed:2:54:06 postfix:- (1728:tqdm_logging.py:145)
09:40:31.367 [I] Progress on: 1.44kit/3.00kit rate:7.1s/it remaining:3:03:14 elapsed:2:54:20 postfix:- (1728:tqdm_logging.py:145)
09:40:45.479 [I] Progress on: 1.44kit/3.00kit rate:7.1s/it remaining:3:03:01 elapsed:2:54:34 postfix:- (1728:tqdm_logging.py:145)
09:40:59.599 [I] Progress on: 1.45kit/3.00kit rate:7.1s/it remaining:3:02:48 elapsed:2:54:48 postfix:- (1728:tqdm_logging.py:145)
09:41:13.725 [I] Progress on: 1.45kit/3.00kit rate:7.1s/it remaining:3:02:38 elapsed:2:55:02 postfix:- (1728:tqdm_logging.py:145)
09:41:27.838 [I] Progress on: 1.45kit/3.00kit rate:7.1s/it remaining:3:02:22 elapsed:2:55:16 postfix:- (1728:tqdm_logging.py:145)


Step 1450: grad_norm=0.0573, loss=0.0058, param_norm=1804.0057
09:41:41.590 [I] Progress on: 1.45kit/3.00kit rate:7.1s/it remaining:3:02:22 elapsed:2:55:30 postfix:- (1728:tqdm_logging.py:145)
09:41:56.113 [I] Progress on: 1.45kit/3.00kit rate:6.8s/it remaining:2:54:22 elapsed:2:55:44 postfix:- (1728:tqdm_logging.py:145)
09:42:10.247 [I] Progress on: 1.46kit/3.00kit rate:6.9s/it remaining:2:58:05 elapsed:2:55:59 postfix:- (1728:tqdm_logging.py:145)
09:42:24.355 [I] Progress on: 1.46kit/3.00kit rate:7.0s/it remaining:2:59:35 elapsed:2:56:13 postfix:- (1728:tqdm_logging.py:145)
09:42:38.465 [I] Progress on: 1.46kit/3.00kit rate:7.0s/it remaining:3:00:14 elapsed:2:56:27 postfix:- (1728:tqdm_logging.py:145)
09:42:52.604 [I] Progress on: 1.46kit/3.00kit rate:7.0s/it remaining:3:00:37 elapsed:2:56:41 postfix:- (1728:tqdm_logging.py:145)
09:43:06.748 [I] Progress on: 1.46kit/3.00kit rate:7.1s/it remaining:3:00:44 elapsed:2:56:55 postfix:- (1728:tqdm_logging.py:145)
09:43:20.838 [I] Progress on: 1.47kit/3.00kit rate:7.1s/it remaining:3:00:19 elapsed:2:57:09 postfix:- (1728:tqdm_logging.py:145)
09:43:34.945 [I] Progress on: 1.47kit/3.00kit rate:7.1s/it remaining:3:00:06 elapsed:2:57:23 postfix:- (1728:tqdm_logging.py:145)
09:43:49.078 [I] Progress on: 1.47kit/3.00kit rate:7.1s/it remaining:3:00:01 elapsed:2:57:37 postfix:- (1728:tqdm_logging.py:145)
09:44:03.205 [I] Progress on: 1.47kit/3.00kit rate:7.1s/it remaining:2:59:50 elapsed:2:57:51 postfix:- (1728:tqdm_logging.py:145)
09:44:17.327 [I] Progress on: 1.47kit/3.00kit rate:7.1s/it remaining:2:59:36 elapsed:2:58:06 postfix:- (1728:tqdm_logging.py:145)
09:44:31.457 [I] Progress on: 1.48kit/3.00kit rate:7.1s/it remaining:2:59:23 elapsed:2:58:20 postfix:- (1728:tqdm_logging.py:145)
09:44:45.580 [I] Progress on: 1.48kit/3.00kit rate:7.1s/it remaining:2:59:08 elapsed:2:58:34 postfix:- (1728:tqdm_logging.py:145)
09:44:59.717 [I] Progress on: 1.48kit/3.00kit rate:7.1s/it remaining:2:59:00 elapsed:2:58:48 postfix:- (1728:tqdm_logging.py:145)
09:45:13.843 [I] Progress on: 1.48kit/3.00kit rate:7.1s/it remaining:2:58:44 elapsed:2:59:02 postfix:- (1728:tqdm_logging.py:145)
09:45:27.963 [I] Progress on: 1.48kit/3.00kit rate:7.1s/it remaining:2:58:26 elapsed:2:59:16 postfix:- (1728:tqdm_logging.py:145)
09:45:42.081 [I] Progress on: 1.49kit/3.00kit rate:7.1s/it remaining:2:58:10 elapsed:2:59:30 postfix:- (1728:tqdm_logging.py:145)
09:45:56.189 [I] Progress on: 1.49kit/3.00kit rate:7.1s/it remaining:2:57:50 elapsed:2:59:44 postfix:- (1728:tqdm_logging.py:145)
09:46:10.326 [I] Progress on: 1.49kit/3.00kit rate:7.1s/it remaining:2:57:44 elapsed:2:59:59 postfix:- (1728:tqdm_logging.py:145)
09:46:24.444 [I] Progress on: 1.49kit/3.00kit rate:7.1s/it remaining:2:57:28 elapsed:3:00:13 postfix:- (1728:tqdm_logging.py:145)
09:46:38.564 [I] Progress on: 1.49kit/3.00kit rate:7.1s/it remaining:2:57:13 elapsed:3:00:27 postfix:- (1728:tqdm_logging.py:145)
09:46:52.670 [I] Progress on: 1.50kit/3.00kit rate:7.1s/it remaining:2:56:53 elapsed:3:00:41 postfix:- (1728:tqdm_logging.py:145)
09:47:06.792 [I] Progress on: 1.50kit/3.00kit rate:7.1s/it remaining:2:56:42 elapsed:3:00:55 postfix:- (1728:tqdm_logging.py:145)
09:47:20.904 [I] Progress on: 1.50kit/3.00kit rate:7.1s/it remaining:2:56:26 elapsed:3:01:09 postfix:- (1728:tqdm_logging.py:145)


Step 1500: grad_norm=0.0495, loss=0.0058, param_norm=1804.0128
09:47:34.675 [I] Progress on: 1.50kit/3.00kit rate:7.1s/it remaining:2:56:26 elapsed:3:01:23 postfix:- (1728:tqdm_logging.py:145)
09:47:49.222 [I] Progress on: 1.50kit/3.00kit rate:6.8s/it remaining:2:48:51 elapsed:3:01:38 postfix:- (1728:tqdm_logging.py:145)
09:48:03.336 [I] Progress on: 1.51kit/3.00kit rate:6.9s/it remaining:2:52:14 elapsed:3:01:52 postfix:- (1728:tqdm_logging.py:145)
09:48:17.455 [I] Progress on: 1.51kit/3.00kit rate:7.0s/it remaining:2:53:49 elapsed:3:02:06 postfix:- (1728:tqdm_logging.py:145)
09:48:31.556 [I] Progress on: 1.51kit/3.00kit rate:7.0s/it remaining:2:54:21 elapsed:3:02:20 postfix:- (1728:tqdm_logging.py:145)
09:48:45.673 [I] Progress on: 1.51kit/3.00kit rate:7.0s/it remaining:2:54:35 elapsed:3:02:34 postfix:- (1728:tqdm_logging.py:145)
09:48:59.785 [I] Progress on: 1.51kit/3.00kit rate:7.0s/it remaining:2:54:32 elapsed:3:02:48 postfix:- (1728:tqdm_logging.py:145)
09:49:13.891 [I] Progress on: 1.52kit/3.00kit rate:7.0s/it remaining:2:54:22 elapsed:3:03:02 postfix:- (1728:tqdm_logging.py:145)
09:49:27.999 [I] Progress on: 1.52kit/3.00kit rate:7.1s/it remaining:2:54:11 elapsed:3:03:16 postfix:- (1728:tqdm_logging.py:145)
09:49:42.117 [I] Progress on: 1.52kit/3.00kit rate:7.1s/it remaining:2:54:00 elapsed:3:03:30 postfix:- (1728:tqdm_logging.py:145)
09:49:56.238 [I] Progress on: 1.52kit/3.00kit rate:7.1s/it remaining:2:53:50 elapsed:3:03:45 postfix:- (1728:tqdm_logging.py:145)
09:50:10.358 [I] Progress on: 1.52kit/3.00kit rate:7.1s/it remaining:2:53:38 elapsed:3:03:59 postfix:- (1728:tqdm_logging.py:145)
09:50:24.475 [I] Progress on: 1.53kit/3.00kit rate:7.1s/it remaining:2:53:23 elapsed:3:04:13 postfix:- (1728:tqdm_logging.py:145)
09:50:38.585 [I] Progress on: 1.53kit/3.00kit rate:7.1s/it remaining:2:53:06 elapsed:3:04:27 postfix:- (1728:tqdm_logging.py:145)
09:50:52.714 [I] Progress on: 1.53kit/3.00kit rate:7.1s/it remaining:2:52:59 elapsed:3:04:41 postfix:- (1728:tqdm_logging.py:145)
09:51:06.829 [I] Progress on: 1.53kit/3.00kit rate:7.1s/it remaining:2:52:43 elapsed:3:04:55 postfix:- (1728:tqdm_logging.py:145)
09:51:20.949 [I] Progress on: 1.53kit/3.00kit rate:7.1s/it remaining:2:52:28 elapsed:3:05:09 postfix:- (1728:tqdm_logging.py:145)
09:51:35.069 [I] Progress on: 1.54kit/3.00kit rate:7.1s/it remaining:2:52:16 elapsed:3:05:23 postfix:- (1728:tqdm_logging.py:145)
09:51:49.180 [I] Progress on: 1.54kit/3.00kit rate:7.1s/it remaining:2:51:58 elapsed:3:05:37 postfix:- (1728:tqdm_logging.py:145)
09:52:03.293 [I] Progress on: 1.54kit/3.00kit rate:7.1s/it remaining:2:51:43 elapsed:3:05:52 postfix:- (1728:tqdm_logging.py:145)
09:52:17.408 [I] Progress on: 1.54kit/3.00kit rate:7.1s/it remaining:2:51:29 elapsed:3:06:06 postfix:- (1728:tqdm_logging.py:145)
09:52:31.530 [I] Progress on: 1.54kit/3.00kit rate:7.1s/it remaining:2:51:18 elapsed:3:06:20 postfix:- (1728:tqdm_logging.py:145)
09:52:45.643 [I] Progress on: 1.55kit/3.00kit rate:7.1s/it remaining:2:51:02 elapsed:3:06:34 postfix:- (1728:tqdm_logging.py:145)
09:52:59.764 [I] Progress on: 1.55kit/3.00kit rate:7.1s/it remaining:2:50:50 elapsed:3:06:48 postfix:- (1728:tqdm_logging.py:145)
09:53:13.899 [I] Progress on: 1.55kit/3.00kit rate:7.1s/it remaining:2:50:41 elapsed:3:07:02 postfix:- (1728:tqdm_logging.py:145)


Step 1550: grad_norm=0.0472, loss=0.0059, param_norm=1804.0178
09:53:27.647 [I] Progress on: 1.55kit/3.00kit rate:7.1s/it remaining:2:50:41 elapsed:3:07:16 postfix:- (1728:tqdm_logging.py:145)
09:53:42.731 [I] Progress on: 1.55kit/3.00kit rate:6.9s/it remaining:2:47:05 elapsed:3:07:31 postfix:- (1728:tqdm_logging.py:145)
09:53:56.334 [I] Progress on: 1.56kit/3.00kit rate:6.9s/it remaining:2:45:53 elapsed:3:07:45 postfix:- (1728:tqdm_logging.py:145)
09:54:10.453 [I] Progress on: 1.56kit/3.00kit rate:7.0s/it remaining:2:47:47 elapsed:3:07:59 postfix:- (1728:tqdm_logging.py:145)
09:54:24.573 [I] Progress on: 1.56kit/3.00kit rate:7.0s/it remaining:2:48:35 elapsed:3:08:13 postfix:- (1728:tqdm_logging.py:145)
09:54:38.688 [I] Progress on: 1.56kit/3.00kit rate:7.0s/it remaining:2:48:50 elapsed:3:08:27 postfix:- (1728:tqdm_logging.py:145)
09:54:52.791 [I] Progress on: 1.56kit/3.00kit rate:7.1s/it remaining:2:48:46 elapsed:3:08:41 postfix:- (1728:tqdm_logging.py:145)
09:55:06.906 [I] Progress on: 1.57kit/3.00kit rate:7.1s/it remaining:2:48:38 elapsed:3:08:55 postfix:- (1728:tqdm_logging.py:145)
09:55:21.024 [I] Progress on: 1.57kit/3.00kit rate:7.1s/it remaining:2:48:26 elapsed:3:09:09 postfix:- (1728:tqdm_logging.py:145)
09:55:35.136 [I] Progress on: 1.57kit/3.00kit rate:7.1s/it remaining:2:48:11 elapsed:3:09:23 postfix:- (1728:tqdm_logging.py:145)
09:55:49.245 [I] Progress on: 1.57kit/3.00kit rate:7.1s/it remaining:2:47:56 elapsed:3:09:38 postfix:- (1728:tqdm_logging.py:145)
09:56:03.353 [I] Progress on: 1.57kit/3.00kit rate:7.1s/it remaining:2:47:39 elapsed:3:09:52 postfix:- (1728:tqdm_logging.py:145)
09:56:17.467 [I] Progress on: 1.58kit/3.00kit rate:7.1s/it remaining:2:47:25 elapsed:3:10:06 postfix:- (1728:tqdm_logging.py:145)
09:56:31.575 [I] Progress on: 1.58kit/3.00kit rate:7.1s/it remaining:2:47:11 elapsed:3:10:20 postfix:- (1728:tqdm_logging.py:145)
09:56:45.695 [I] Progress on: 1.58kit/3.00kit rate:7.1s/it remaining:2:47:01 elapsed:3:10:34 postfix:- (1728:tqdm_logging.py:145)
09:56:59.801 [I] Progress on: 1.58kit/3.00kit rate:7.1s/it remaining:2:46:44 elapsed:3:10:48 postfix:- (1728:tqdm_logging.py:145)
09:57:13.908 [I] Progress on: 1.58kit/3.00kit rate:7.1s/it remaining:2:46:29 elapsed:3:11:02 postfix:- (1728:tqdm_logging.py:145)
09:57:28.031 [I] Progress on: 1.59kit/3.00kit rate:7.1s/it remaining:2:46:18 elapsed:3:11:16 postfix:- (1728:tqdm_logging.py:145)
09:57:42.162 [I] Progress on: 1.59kit/3.00kit rate:7.1s/it remaining:2:46:11 elapsed:3:11:30 postfix:- (1728:tqdm_logging.py:145)
09:57:56.273 [I] Progress on: 1.59kit/3.00kit rate:7.1s/it remaining:2:45:53 elapsed:3:11:45 postfix:- (1728:tqdm_logging.py:145)
09:58:10.371 [I] Progress on: 1.59kit/3.00kit rate:7.1s/it remaining:2:45:33 elapsed:3:11:59 postfix:- (1728:tqdm_logging.py:145)
09:58:24.500 [I] Progress on: 1.59kit/3.00kit rate:7.1s/it remaining:2:45:25 elapsed:3:12:13 postfix:- (1728:tqdm_logging.py:145)
09:58:38.624 [I] Progress on: 1.60kit/3.00kit rate:7.1s/it remaining:2:45:13 elapsed:3:12:27 postfix:- (1728:tqdm_logging.py:145)
09:58:52.733 [I] Progress on: 1.60kit/3.00kit rate:7.1s/it remaining:2:44:55 elapsed:3:12:41 postfix:- (1728:tqdm_logging.py:145)
09:59:06.852 [I] Progress on: 1.60kit/3.00kit rate:7.1s/it remaining:2:44:42 elapsed:3:12:55 postfix:- (1728:tqdm_logging.py:145)


Step 1600: grad_norm=0.0482, loss=0.0058, param_norm=1804.0236
09:59:20.605 [I] Progress on: 1.60kit/3.00kit rate:7.1s/it remaining:2:44:42 elapsed:3:13:09 postfix:- (1728:tqdm_logging.py:145)
09:59:35.172 [I] Progress on: 1.60kit/3.00kit rate:6.8s/it remaining:2:37:34 elapsed:3:13:23 postfix:- (1728:tqdm_logging.py:145)
09:59:49.290 [I] Progress on: 1.61kit/3.00kit rate:6.9s/it remaining:2:40:44 elapsed:3:13:38 postfix:- (1728:tqdm_logging.py:145)
10:00:03.482 [I] Progress on: 1.61kit/3.00kit rate:7.0s/it remaining:2:42:40 elapsed:3:13:52 postfix:- (1728:tqdm_logging.py:145)
10:00:17.603 [I] Progress on: 1.61kit/3.00kit rate:7.0s/it remaining:2:43:01 elapsed:3:14:06 postfix:- (1728:tqdm_logging.py:145)
10:00:31.728 [I] Progress on: 1.61kit/3.00kit rate:7.0s/it remaining:2:43:04 elapsed:3:14:20 postfix:- (1728:tqdm_logging.py:145)
10:00:45.841 [I] Progress on: 1.61kit/3.00kit rate:7.1s/it remaining:2:42:55 elapsed:3:14:34 postfix:- (1728:tqdm_logging.py:145)
10:00:59.956 [I] Progress on: 1.62kit/3.00kit rate:7.1s/it remaining:2:42:45 elapsed:3:14:48 postfix:- (1728:tqdm_logging.py:145)
10:01:14.060 [I] Progress on: 1.62kit/3.00kit rate:7.1s/it remaining:2:42:27 elapsed:3:15:02 postfix:- (1728:tqdm_logging.py:145)
10:01:28.170 [I] Progress on: 1.62kit/3.00kit rate:7.1s/it remaining:2:42:15 elapsed:3:15:16 postfix:- (1728:tqdm_logging.py:145)
10:01:42.294 [I] Progress on: 1.62kit/3.00kit rate:7.1s/it remaining:2:42:05 elapsed:3:15:31 postfix:- (1728:tqdm_logging.py:145)
10:01:56.405 [I] Progress on: 1.62kit/3.00kit rate:7.1s/it remaining:2:41:50 elapsed:3:15:45 postfix:- (1728:tqdm_logging.py:145)
10:02:10.535 [I] Progress on: 1.63kit/3.00kit rate:7.1s/it remaining:2:41:41 elapsed:3:15:59 postfix:- (1728:tqdm_logging.py:145)
10:02:24.651 [I] Progress on: 1.63kit/3.00kit rate:7.1s/it remaining:2:41:25 elapsed:3:16:13 postfix:- (1728:tqdm_logging.py:145)
10:02:38.767 [I] Progress on: 1.63kit/3.00kit rate:7.1s/it remaining:2:41:11 elapsed:3:16:27 postfix:- (1728:tqdm_logging.py:145)
10:02:52.871 [I] Progress on: 1.63kit/3.00kit rate:7.1s/it remaining:2:40:51 elapsed:3:16:41 postfix:- (1728:tqdm_logging.py:145)
10:03:06.995 [I] Progress on: 1.63kit/3.00kit rate:7.1s/it remaining:2:40:42 elapsed:3:16:55 postfix:- (1728:tqdm_logging.py:145)
10:03:21.112 [I] Progress on: 1.64kit/3.00kit rate:7.1s/it remaining:2:40:28 elapsed:3:17:09 postfix:- (1728:tqdm_logging.py:145)
10:03:35.223 [I] Progress on: 1.64kit/3.00kit rate:7.1s/it remaining:2:40:12 elapsed:3:17:24 postfix:- (1728:tqdm_logging.py:145)
10:03:49.332 [I] Progress on: 1.64kit/3.00kit rate:7.1s/it remaining:2:39:57 elapsed:3:17:38 postfix:- (1728:tqdm_logging.py:145)
10:04:03.451 [I] Progress on: 1.64kit/3.00kit rate:7.1s/it remaining:2:39:44 elapsed:3:17:52 postfix:- (1728:tqdm_logging.py:145)
10:04:17.570 [I] Progress on: 1.64kit/3.00kit rate:7.1s/it remaining:2:39:31 elapsed:3:18:06 postfix:- (1728:tqdm_logging.py:145)
10:04:31.690 [I] Progress on: 1.65kit/3.00kit rate:7.1s/it remaining:2:39:19 elapsed:3:18:20 postfix:- (1728:tqdm_logging.py:145)
10:04:45.827 [I] Progress on: 1.65kit/3.00kit rate:7.1s/it remaining:2:39:11 elapsed:3:18:34 postfix:- (1728:tqdm_logging.py:145)
10:04:59.929 [I] Progress on: 1.65kit/3.00kit rate:7.1s/it remaining:2:38:49 elapsed:3:18:48 postfix:- (1728:tqdm_logging.py:145)


Step 1650: grad_norm=0.0454, loss=0.0058, param_norm=1804.0297
10:05:13.719 [I] Progress on: 1.65kit/3.00kit rate:7.1s/it remaining:2:38:49 elapsed:3:19:02 postfix:- (1728:tqdm_logging.py:145)
10:05:28.276 [I] Progress on: 1.65kit/3.00kit rate:6.8s/it remaining:2:31:58 elapsed:3:19:17 postfix:- (1728:tqdm_logging.py:145)
10:05:42.381 [I] Progress on: 1.66kit/3.00kit rate:6.9s/it remaining:2:34:56 elapsed:3:19:31 postfix:- (1728:tqdm_logging.py:145)
10:05:56.486 [I] Progress on: 1.66kit/3.00kit rate:7.0s/it remaining:2:36:15 elapsed:3:19:45 postfix:- (1728:tqdm_logging.py:145)
10:06:10.598 [I] Progress on: 1.66kit/3.00kit rate:7.0s/it remaining:2:36:48 elapsed:3:19:59 postfix:- (1728:tqdm_logging.py:145)
10:06:24.708 [I] Progress on: 1.66kit/3.00kit rate:7.0s/it remaining:2:36:58 elapsed:3:20:13 postfix:- (1728:tqdm_logging.py:145)
10:06:38.819 [I] Progress on: 1.66kit/3.00kit rate:7.0s/it remaining:2:36:54 elapsed:3:20:27 postfix:- (1728:tqdm_logging.py:145)
10:06:52.933 [I] Progress on: 1.67kit/3.00kit rate:7.1s/it remaining:2:36:44 elapsed:3:20:41 postfix:- (1728:tqdm_logging.py:145)
10:07:07.056 [I] Progress on: 1.67kit/3.00kit rate:7.1s/it remaining:2:36:37 elapsed:3:20:55 postfix:- (1728:tqdm_logging.py:145)
10:07:21.168 [I] Progress on: 1.67kit/3.00kit rate:7.1s/it remaining:2:36:23 elapsed:3:21:09 postfix:- (1728:tqdm_logging.py:145)
10:07:35.283 [I] Progress on: 1.67kit/3.00kit rate:7.1s/it remaining:2:36:10 elapsed:3:21:24 postfix:- (1728:tqdm_logging.py:145)
10:07:49.420 [I] Progress on: 1.67kit/3.00kit rate:7.1s/it remaining:2:36:03 elapsed:3:21:38 postfix:- (1728:tqdm_logging.py:145)
10:08:03.535 [I] Progress on: 1.68kit/3.00kit rate:7.1s/it remaining:2:35:46 elapsed:3:21:52 postfix:- (1728:tqdm_logging.py:145)
10:08:17.659 [I] Progress on: 1.68kit/3.00kit rate:7.1s/it remaining:2:35:34 elapsed:3:22:06 postfix:- (1728:tqdm_logging.py:145)
10:08:31.771 [I] Progress on: 1.68kit/3.00kit rate:7.1s/it remaining:2:35:17 elapsed:3:22:20 postfix:- (1728:tqdm_logging.py:145)
10:08:45.876 [I] Progress on: 1.68kit/3.00kit rate:7.1s/it remaining:2:34:57 elapsed:3:22:34 postfix:- (1728:tqdm_logging.py:145)
10:08:59.987 [I] Progress on: 1.68kit/3.00kit rate:7.1s/it remaining:2:34:44 elapsed:3:22:48 postfix:- (1728:tqdm_logging.py:145)
10:09:14.114 [I] Progress on: 1.69kit/3.00kit rate:7.1s/it remaining:2:34:35 elapsed:3:23:02 postfix:- (1728:tqdm_logging.py:145)
10:09:28.234 [I] Progress on: 1.69kit/3.00kit rate:7.1s/it remaining:2:34:21 elapsed:3:23:17 postfix:- (1728:tqdm_logging.py:145)
10:09:42.343 [I] Progress on: 1.69kit/3.00kit rate:7.1s/it remaining:2:34:04 elapsed:3:23:31 postfix:- (1728:tqdm_logging.py:145)
10:09:56.453 [I] Progress on: 1.69kit/3.00kit rate:7.1s/it remaining:2:33:48 elapsed:3:23:45 postfix:- (1728:tqdm_logging.py:145)
10:10:10.575 [I] Progress on: 1.69kit/3.00kit rate:7.1s/it remaining:2:33:36 elapsed:3:23:59 postfix:- (1728:tqdm_logging.py:145)
10:10:24.695 [I] Progress on: 1.70kit/3.00kit rate:7.1s/it remaining:2:33:23 elapsed:3:24:13 postfix:- (1728:tqdm_logging.py:145)
10:10:38.810 [I] Progress on: 1.70kit/3.00kit rate:7.1s/it remaining:2:33:09 elapsed:3:24:27 postfix:- (1728:tqdm_logging.py:145)
10:10:52.913 [I] Progress on: 1.70kit/3.00kit rate:7.1s/it remaining:2:32:49 elapsed:3:24:41 postfix:- (1728:tqdm_logging.py:145)


Step 1700: grad_norm=0.0460, loss=0.0056, param_norm=1804.0359
10:11:06.654 [I] Progress on: 1.70kit/3.00kit rate:7.1s/it remaining:2:32:49 elapsed:3:24:55 postfix:- (1728:tqdm_logging.py:145)
10:11:21.160 [I] Progress on: 1.70kit/3.00kit rate:6.8s/it remaining:2:26:02 elapsed:3:25:09 postfix:- (1728:tqdm_logging.py:145)
10:11:35.273 [I] Progress on: 1.71kit/3.00kit rate:6.9s/it remaining:2:29:03 elapsed:3:25:24 postfix:- (1728:tqdm_logging.py:145)
10:11:49.378 [I] Progress on: 1.71kit/3.00kit rate:7.0s/it remaining:2:30:23 elapsed:3:25:38 postfix:- (1728:tqdm_logging.py:145)
10:12:03.494 [I] Progress on: 1.71kit/3.00kit rate:7.0s/it remaining:2:30:58 elapsed:3:25:52 postfix:- (1728:tqdm_logging.py:145)
10:12:17.603 [I] Progress on: 1.71kit/3.00kit rate:7.0s/it remaining:2:31:06 elapsed:3:26:06 postfix:- (1728:tqdm_logging.py:145)
10:12:31.712 [I] Progress on: 1.71kit/3.00kit rate:7.0s/it remaining:2:31:02 elapsed:3:26:20 postfix:- (1728:tqdm_logging.py:145)
10:12:45.833 [I] Progress on: 1.72kit/3.00kit rate:7.1s/it remaining:2:30:56 elapsed:3:26:34 postfix:- (1728:tqdm_logging.py:145)
10:12:59.942 [I] Progress on: 1.72kit/3.00kit rate:7.1s/it remaining:2:30:42 elapsed:3:26:48 postfix:- (1728:tqdm_logging.py:145)
10:13:14.066 [I] Progress on: 1.72kit/3.00kit rate:7.1s/it remaining:2:30:34 elapsed:3:27:02 postfix:- (1728:tqdm_logging.py:145)
10:13:28.181 [I] Progress on: 1.72kit/3.00kit rate:7.1s/it remaining:2:30:20 elapsed:3:27:16 postfix:- (1728:tqdm_logging.py:145)
10:13:42.285 [I] Progress on: 1.72kit/3.00kit rate:7.1s/it remaining:2:30:02 elapsed:3:27:31 postfix:- (1728:tqdm_logging.py:145)
10:13:56.409 [I] Progress on: 1.73kit/3.00kit rate:7.1s/it remaining:2:29:52 elapsed:3:27:45 postfix:- (1728:tqdm_logging.py:145)
10:14:10.534 [I] Progress on: 1.73kit/3.00kit rate:7.1s/it remaining:2:29:41 elapsed:3:27:59 postfix:- (1728:tqdm_logging.py:145)
10:14:24.636 [I] Progress on: 1.73kit/3.00kit rate:7.1s/it remaining:2:29:21 elapsed:3:28:13 postfix:- (1728:tqdm_logging.py:145)
10:14:38.745 [I] Progress on: 1.73kit/3.00kit rate:7.1s/it remaining:2:29:06 elapsed:3:28:27 postfix:- (1728:tqdm_logging.py:145)
10:14:52.870 [I] Progress on: 1.73kit/3.00kit rate:7.1s/it remaining:2:28:55 elapsed:3:28:41 postfix:- (1728:tqdm_logging.py:145)
10:15:06.987 [I] Progress on: 1.74kit/3.00kit rate:7.1s/it remaining:2:28:41 elapsed:3:28:55 postfix:- (1728:tqdm_logging.py:145)
10:15:21.108 [I] Progress on: 1.74kit/3.00kit rate:7.1s/it remaining:2:28:28 elapsed:3:29:09 postfix:- (1728:tqdm_logging.py:145)
10:15:35.213 [I] Progress on: 1.74kit/3.00kit rate:7.1s/it remaining:2:28:09 elapsed:3:29:24 postfix:- (1728:tqdm_logging.py:145)
10:15:49.342 [I] Progress on: 1.74kit/3.00kit rate:7.1s/it remaining:2:28:00 elapsed:3:29:38 postfix:- (1728:tqdm_logging.py:145)
10:16:03.449 [I] Progress on: 1.74kit/3.00kit rate:7.1s/it remaining:2:27:42 elapsed:3:29:52 postfix:- (1728:tqdm_logging.py:145)
10:16:17.559 [I] Progress on: 1.75kit/3.00kit rate:7.1s/it remaining:2:27:29 elapsed:3:30:06 postfix:- (1728:tqdm_logging.py:145)
10:16:31.677 [I] Progress on: 1.75kit/3.00kit rate:7.1s/it remaining:2:27:15 elapsed:3:30:20 postfix:- (1728:tqdm_logging.py:145)
10:16:45.784 [I] Progress on: 1.75kit/3.00kit rate:7.1s/it remaining:2:26:59 elapsed:3:30:34 postfix:- (1728:tqdm_logging.py:145)


Step 1750: grad_norm=0.0466, loss=0.0054, param_norm=1804.0421
10:16:59.535 [I] Progress on: 1.75kit/3.00kit rate:7.1s/it remaining:2:26:59 elapsed:3:30:48 postfix:- (1728:tqdm_logging.py:145)
10:17:14.081 [I] Progress on: 1.75kit/3.00kit rate:6.8s/it remaining:2:20:32 elapsed:3:31:02 postfix:- (1728:tqdm_logging.py:145)
10:17:28.198 [I] Progress on: 1.76kit/3.00kit rate:6.9s/it remaining:2:23:24 elapsed:3:31:16 postfix:- (1728:tqdm_logging.py:145)
10:17:42.294 [I] Progress on: 1.76kit/3.00kit rate:7.0s/it remaining:2:24:32 elapsed:3:31:31 postfix:- (1728:tqdm_logging.py:145)
10:17:56.421 [I] Progress on: 1.76kit/3.00kit rate:7.0s/it remaining:2:25:09 elapsed:3:31:45 postfix:- (1728:tqdm_logging.py:145)
10:18:10.527 [I] Progress on: 1.76kit/3.00kit rate:7.0s/it remaining:2:25:14 elapsed:3:31:59 postfix:- (1728:tqdm_logging.py:145)
10:18:24.625 [I] Progress on: 1.76kit/3.00kit rate:7.0s/it remaining:2:25:07 elapsed:3:32:13 postfix:- (1728:tqdm_logging.py:145)
10:18:38.729 [I] Progress on: 1.77kit/3.00kit rate:7.0s/it remaining:2:24:58 elapsed:3:32:27 postfix:- (1728:tqdm_logging.py:145)
10:18:52.825 [I] Progress on: 1.77kit/3.00kit rate:7.0s/it remaining:2:24:43 elapsed:3:32:41 postfix:- (1728:tqdm_logging.py:145)
10:19:06.928 [I] Progress on: 1.77kit/3.00kit rate:7.1s/it remaining:2:24:31 elapsed:3:32:55 postfix:- (1728:tqdm_logging.py:145)
10:19:21.023 [I] Progress on: 1.77kit/3.00kit rate:7.0s/it remaining:2:24:15 elapsed:3:33:09 postfix:- (1728:tqdm_logging.py:145)
10:19:35.121 [I] Progress on: 1.77kit/3.00kit rate:7.0s/it remaining:2:24:01 elapsed:3:33:23 postfix:- (1728:tqdm_logging.py:145)
10:19:49.807 [I] Progress on: 1.78kit/3.00kit rate:7.2s/it remaining:2:27:21 elapsed:3:33:38 postfix:- (1728:tqdm_logging.py:145)
10:20:03.375 [I] Progress on: 1.78kit/3.00kit rate:7.0s/it remaining:2:23:07 elapsed:3:33:52 postfix:- (1728:tqdm_logging.py:145)
10:20:17.480 [I] Progress on: 1.78kit/3.00kit rate:7.0s/it remaining:2:23:12 elapsed:3:34:06 postfix:- (1728:tqdm_logging.py:145)
10:20:31.586 [I] Progress on: 1.78kit/3.00kit rate:7.1s/it remaining:2:23:08 elapsed:3:34:20 postfix:- (1728:tqdm_logging.py:145)
10:20:45.686 [I] Progress on: 1.78kit/3.00kit rate:7.1s/it remaining:2:22:58 elapsed:3:34:34 postfix:- (1728:tqdm_logging.py:145)
10:20:59.802 [I] Progress on: 1.79kit/3.00kit rate:7.1s/it remaining:2:22:50 elapsed:3:34:48 postfix:- (1728:tqdm_logging.py:145)
10:21:13.908 [I] Progress on: 1.79kit/3.00kit rate:7.1s/it remaining:2:22:35 elapsed:3:35:02 postfix:- (1728:tqdm_logging.py:145)
10:21:28.005 [I] Progress on: 1.79kit/3.00kit rate:7.1s/it remaining:2:22:18 elapsed:3:35:16 postfix:- (1728:tqdm_logging.py:145)
10:21:42.113 [I] Progress on: 1.79kit/3.00kit rate:7.1s/it remaining:2:22:06 elapsed:3:35:30 postfix:- (1728:tqdm_logging.py:145)
10:21:56.210 [I] Progress on: 1.79kit/3.00kit rate:7.1s/it remaining:2:21:48 elapsed:3:35:44 postfix:- (1728:tqdm_logging.py:145)
10:22:10.319 [I] Progress on: 1.80kit/3.00kit rate:7.1s/it remaining:2:21:33 elapsed:3:35:59 postfix:- (1728:tqdm_logging.py:145)
10:22:24.411 [I] Progress on: 1.80kit/3.00kit rate:7.1s/it remaining:2:21:14 elapsed:3:36:13 postfix:- (1728:tqdm_logging.py:145)
10:22:38.527 [I] Progress on: 1.80kit/3.00kit rate:7.1s/it remaining:2:21:03 elapsed:3:36:27 postfix:- (1728:tqdm_logging.py:145)


Step 1800: grad_norm=0.0444, loss=0.0056, param_norm=1804.0483
10:22:52.274 [I] Progress on: 1.80kit/3.00kit rate:7.1s/it remaining:2:21:03 elapsed:3:36:41 postfix:- (1728:tqdm_logging.py:145)
10:23:06.806 [I] Progress on: 1.80kit/3.00kit rate:6.8s/it remaining:2:14:50 elapsed:3:36:55 postfix:- (1728:tqdm_logging.py:145)
10:23:20.921 [I] Progress on: 1.81kit/3.00kit rate:6.9s/it remaining:2:17:34 elapsed:3:37:09 postfix:- (1728:tqdm_logging.py:145)
10:23:35.021 [I] Progress on: 1.81kit/3.00kit rate:7.0s/it remaining:2:18:42 elapsed:3:37:23 postfix:- (1728:tqdm_logging.py:145)
10:23:49.141 [I] Progress on: 1.81kit/3.00kit rate:7.0s/it remaining:2:19:15 elapsed:3:37:37 postfix:- (1728:tqdm_logging.py:145)
10:24:03.260 [I] Progress on: 1.81kit/3.00kit rate:7.0s/it remaining:2:19:24 elapsed:3:37:52 postfix:- (1728:tqdm_logging.py:145)
10:24:17.349 [I] Progress on: 1.81kit/3.00kit rate:7.0s/it remaining:2:19:10 elapsed:3:38:06 postfix:- (1728:tqdm_logging.py:145)
10:24:31.472 [I] Progress on: 1.82kit/3.00kit rate:7.1s/it remaining:2:19:08 elapsed:3:38:20 postfix:- (1728:tqdm_logging.py:145)
10:24:45.565 [I] Progress on: 1.82kit/3.00kit rate:7.0s/it remaining:2:18:52 elapsed:3:38:34 postfix:- (1728:tqdm_logging.py:145)
10:24:59.689 [I] Progress on: 1.82kit/3.00kit rate:7.1s/it remaining:2:18:45 elapsed:3:38:48 postfix:- (1728:tqdm_logging.py:145)
10:25:13.791 [I] Progress on: 1.82kit/3.00kit rate:7.1s/it remaining:2:18:29 elapsed:3:39:02 postfix:- (1728:tqdm_logging.py:145)
10:25:27.911 [I] Progress on: 1.82kit/3.00kit rate:7.1s/it remaining:2:18:17 elapsed:3:39:16 postfix:- (1728:tqdm_logging.py:145)
10:25:42.036 [I] Progress on: 1.83kit/3.00kit rate:7.1s/it remaining:2:18:07 elapsed:3:39:30 postfix:- (1728:tqdm_logging.py:145)
10:25:56.150 [I] Progress on: 1.83kit/3.00kit rate:7.1s/it remaining:2:17:53 elapsed:3:39:44 postfix:- (1728:tqdm_logging.py:145)
10:26:10.256 [I] Progress on: 1.83kit/3.00kit rate:7.1s/it remaining:2:17:36 elapsed:3:39:59 postfix:- (1728:tqdm_logging.py:145)
10:26:24.372 [I] Progress on: 1.83kit/3.00kit rate:7.1s/it remaining:2:17:22 elapsed:3:40:13 postfix:- (1728:tqdm_logging.py:145)
10:26:38.478 [I] Progress on: 1.83kit/3.00kit rate:7.1s/it remaining:2:17:06 elapsed:3:40:27 postfix:- (1728:tqdm_logging.py:145)
10:26:52.590 [I] Progress on: 1.84kit/3.00kit rate:7.1s/it remaining:2:16:53 elapsed:3:40:41 postfix:- (1728:tqdm_logging.py:145)
10:27:06.695 [I] Progress on: 1.84kit/3.00kit rate:7.1s/it remaining:2:16:36 elapsed:3:40:55 postfix:- (1728:tqdm_logging.py:145)
10:27:20.815 [I] Progress on: 1.84kit/3.00kit rate:7.1s/it remaining:2:16:25 elapsed:3:41:09 postfix:- (1728:tqdm_logging.py:145)
10:27:34.934 [I] Progress on: 1.84kit/3.00kit rate:7.1s/it remaining:2:16:14 elapsed:3:41:23 postfix:- (1728:tqdm_logging.py:145)
10:27:49.037 [I] Progress on: 1.84kit/3.00kit rate:7.1s/it remaining:2:15:56 elapsed:3:41:37 postfix:- (1728:tqdm_logging.py:145)
10:28:03.159 [I] Progress on: 1.85kit/3.00kit rate:7.1s/it remaining:2:15:46 elapsed:3:41:51 postfix:- (1728:tqdm_logging.py:145)
10:28:17.275 [I] Progress on: 1.85kit/3.00kit rate:7.1s/it remaining:2:15:30 elapsed:3:42:06 postfix:- (1728:tqdm_logging.py:145)
10:28:31.389 [I] Progress on: 1.85kit/3.00kit rate:7.1s/it remaining:2:15:15 elapsed:3:42:20 postfix:- (1728:tqdm_logging.py:145)


Step 1850: grad_norm=0.0454, loss=0.0055, param_norm=1804.0544
10:28:45.136 [I] Progress on: 1.85kit/3.00kit rate:7.1s/it remaining:2:15:15 elapsed:3:42:33 postfix:- (1728:tqdm_logging.py:145)
10:28:59.693 [I] Progress on: 1.85kit/3.00kit rate:6.8s/it remaining:2:09:19 elapsed:3:42:48 postfix:- (1728:tqdm_logging.py:145)
10:29:13.787 [I] Progress on: 1.86kit/3.00kit rate:6.9s/it remaining:2:11:47 elapsed:3:43:02 postfix:- (1728:tqdm_logging.py:145)
10:29:27.891 [I] Progress on: 1.86kit/3.00kit rate:7.0s/it remaining:2:12:55 elapsed:3:43:16 postfix:- (1728:tqdm_logging.py:145)
10:29:42.013 [I] Progress on: 1.86kit/3.00kit rate:7.0s/it remaining:2:13:25 elapsed:3:43:30 postfix:- (1728:tqdm_logging.py:145)
10:29:56.106 [I] Progress on: 1.86kit/3.00kit rate:7.0s/it remaining:2:13:24 elapsed:3:43:44 postfix:- (1728:tqdm_logging.py:145)
10:30:10.336 [I] Progress on: 1.86kit/3.00kit rate:7.1s/it remaining:2:13:50 elapsed:3:43:59 postfix:- (1728:tqdm_logging.py:145)
10:30:24.444 [I] Progress on: 1.87kit/3.00kit rate:7.1s/it remaining:2:13:26 elapsed:3:44:13 postfix:- (1728:tqdm_logging.py:145)
10:30:38.559 [I] Progress on: 1.87kit/3.00kit rate:7.1s/it remaining:2:13:10 elapsed:3:44:27 postfix:- (1728:tqdm_logging.py:145)
10:30:52.679 [I] Progress on: 1.87kit/3.00kit rate:7.1s/it remaining:2:12:56 elapsed:3:44:41 postfix:- (1728:tqdm_logging.py:145)
10:31:06.786 [I] Progress on: 1.87kit/3.00kit rate:7.1s/it remaining:2:12:39 elapsed:3:44:55 postfix:- (1728:tqdm_logging.py:145)
10:31:20.892 [I] Progress on: 1.87kit/3.00kit rate:7.1s/it remaining:2:12:22 elapsed:3:45:09 postfix:- (1728:tqdm_logging.py:145)
10:31:35.005 [I] Progress on: 1.88kit/3.00kit rate:7.1s/it remaining:2:12:10 elapsed:3:45:23 postfix:- (1728:tqdm_logging.py:145)
10:31:49.120 [I] Progress on: 1.88kit/3.00kit rate:7.1s/it remaining:2:11:58 elapsed:3:45:37 postfix:- (1728:tqdm_logging.py:145)
10:32:03.229 [I] Progress on: 1.88kit/3.00kit rate:7.1s/it remaining:2:11:42 elapsed:3:45:52 postfix:- (1728:tqdm_logging.py:145)
10:32:17.330 [I] Progress on: 1.88kit/3.00kit rate:7.1s/it remaining:2:11:25 elapsed:3:46:06 postfix:- (1728:tqdm_logging.py:145)
10:32:31.439 [I] Progress on: 1.88kit/3.00kit rate:7.1s/it remaining:2:11:11 elapsed:3:46:20 postfix:- (1728:tqdm_logging.py:145)
10:32:45.553 [I] Progress on: 1.89kit/3.00kit rate:7.1s/it remaining:2:10:58 elapsed:3:46:34 postfix:- (1728:tqdm_logging.py:145)
10:32:59.660 [I] Progress on: 1.89kit/3.00kit rate:7.1s/it remaining:2:10:43 elapsed:3:46:48 postfix:- (1728:tqdm_logging.py:145)
10:33:13.771 [I] Progress on: 1.89kit/3.00kit rate:7.1s/it remaining:2:10:31 elapsed:3:47:02 postfix:- (1728:tqdm_logging.py:145)
10:33:27.868 [I] Progress on: 1.89kit/3.00kit rate:7.1s/it remaining:2:10:11 elapsed:3:47:16 postfix:- (1728:tqdm_logging.py:145)
10:33:41.971 [I] Progress on: 1.89kit/3.00kit rate:7.1s/it remaining:2:09:57 elapsed:3:47:30 postfix:- (1728:tqdm_logging.py:145)
10:33:56.096 [I] Progress on: 1.90kit/3.00kit rate:7.1s/it remaining:2:09:49 elapsed:3:47:44 postfix:- (1728:tqdm_logging.py:145)
10:34:10.198 [I] Progress on: 1.90kit/3.00kit rate:7.1s/it remaining:2:09:34 elapsed:3:47:58 postfix:- (1728:tqdm_logging.py:145)
10:34:24.304 [I] Progress on: 1.90kit/3.00kit rate:7.1s/it remaining:2:09:18 elapsed:3:48:13 postfix:- (1728:tqdm_logging.py:145)


Step 1900: grad_norm=0.0470, loss=0.0053, param_norm=1804.0609
10:34:38.081 [I] Progress on: 1.90kit/3.00kit rate:7.1s/it remaining:2:09:18 elapsed:3:48:26 postfix:- (1728:tqdm_logging.py:145)
10:34:52.624 [I] Progress on: 1.90kit/3.00kit rate:6.8s/it remaining:2:03:39 elapsed:3:48:41 postfix:- (1728:tqdm_logging.py:145)
10:35:06.732 [I] Progress on: 1.91kit/3.00kit rate:6.9s/it remaining:2:06:03 elapsed:3:48:55 postfix:- (1728:tqdm_logging.py:145)
10:35:20.862 [I] Progress on: 1.91kit/3.00kit rate:7.0s/it remaining:2:07:14 elapsed:3:49:09 postfix:- (1728:tqdm_logging.py:145)
10:35:34.971 [I] Progress on: 1.91kit/3.00kit rate:7.0s/it remaining:2:07:36 elapsed:3:49:23 postfix:- (1728:tqdm_logging.py:145)
10:35:49.078 [I] Progress on: 1.91kit/3.00kit rate:7.0s/it remaining:2:07:38 elapsed:3:49:37 postfix:- (1728:tqdm_logging.py:145)
10:36:03.170 [I] Progress on: 1.91kit/3.00kit rate:7.0s/it remaining:2:07:28 elapsed:3:49:51 postfix:- (1728:tqdm_logging.py:145)
10:36:17.310 [I] Progress on: 1.92kit/3.00kit rate:7.1s/it remaining:2:07:29 elapsed:3:50:06 postfix:- (1728:tqdm_logging.py:145)
10:36:31.479 [I] Progress on: 1.92kit/3.00kit rate:7.1s/it remaining:2:07:34 elapsed:3:50:20 postfix:- (1728:tqdm_logging.py:145)
10:36:45.597 [I] Progress on: 1.92kit/3.00kit rate:7.1s/it remaining:2:07:17 elapsed:3:50:34 postfix:- (1728:tqdm_logging.py:145)
10:36:59.644 [I] Progress on: 1.92kit/3.00kit rate:7.0s/it remaining:2:06:36 elapsed:3:50:48 postfix:- (1728:tqdm_logging.py:145)
10:37:13.765 [I] Progress on: 1.92kit/3.00kit rate:7.1s/it remaining:2:06:29 elapsed:3:51:02 postfix:- (1728:tqdm_logging.py:145)
10:37:27.879 [I] Progress on: 1.93kit/3.00kit rate:7.1s/it remaining:2:06:17 elapsed:3:51:16 postfix:- (1728:tqdm_logging.py:145)
10:37:41.986 [I] Progress on: 1.93kit/3.00kit rate:7.1s/it remaining:2:06:02 elapsed:3:51:30 postfix:- (1728:tqdm_logging.py:145)
10:37:56.101 [I] Progress on: 1.93kit/3.00kit rate:7.1s/it remaining:2:05:49 elapsed:3:51:44 postfix:- (1728:tqdm_logging.py:145)
10:38:10.217 [I] Progress on: 1.93kit/3.00kit rate:7.1s/it remaining:2:05:36 elapsed:3:51:59 postfix:- (1728:tqdm_logging.py:145)
10:38:24.311 [I] Progress on: 1.93kit/3.00kit rate:7.1s/it remaining:2:05:16 elapsed:3:52:13 postfix:- (1728:tqdm_logging.py:145)
10:38:38.404 [I] Progress on: 1.94kit/3.00kit rate:7.0s/it remaining:2:05:00 elapsed:3:52:27 postfix:- (1728:tqdm_logging.py:145)
10:38:52.532 [I] Progress on: 1.94kit/3.00kit rate:7.1s/it remaining:2:04:53 elapsed:3:52:41 postfix:- (1728:tqdm_logging.py:145)
10:39:06.641 [I] Progress on: 1.94kit/3.00kit rate:7.1s/it remaining:2:04:38 elapsed:3:52:55 postfix:- (1728:tqdm_logging.py:145)
10:39:20.755 [I] Progress on: 1.94kit/3.00kit rate:7.1s/it remaining:2:04:25 elapsed:3:53:09 postfix:- (1728:tqdm_logging.py:145)
10:39:34.884 [I] Progress on: 1.94kit/3.00kit rate:7.1s/it remaining:2:04:14 elapsed:3:53:23 postfix:- (1728:tqdm_logging.py:145)
10:39:48.992 [I] Progress on: 1.95kit/3.00kit rate:7.1s/it remaining:2:03:58 elapsed:3:53:37 postfix:- (1728:tqdm_logging.py:145)
10:40:03.096 [I] Progress on: 1.95kit/3.00kit rate:7.1s/it remaining:2:03:41 elapsed:3:53:51 postfix:- (1728:tqdm_logging.py:145)
10:40:17.204 [I] Progress on: 1.95kit/3.00kit rate:7.1s/it remaining:2:03:26 elapsed:3:54:05 postfix:- (1728:tqdm_logging.py:145)


Step 1950: grad_norm=0.0417, loss=0.0053, param_norm=1804.0662
10:40:30.963 [I] Progress on: 1.95kit/3.00kit rate:7.1s/it remaining:2:03:26 elapsed:3:54:19 postfix:- (1728:tqdm_logging.py:145)
10:40:45.506 [I] Progress on: 1.95kit/3.00kit rate:6.8s/it remaining:1:57:59 elapsed:3:54:34 postfix:- (1728:tqdm_logging.py:145)
10:40:59.631 [I] Progress on: 1.96kit/3.00kit rate:6.9s/it remaining:2:00:21 elapsed:3:54:48 postfix:- (1728:tqdm_logging.py:145)
10:41:13.750 [I] Progress on: 1.96kit/3.00kit rate:7.0s/it remaining:2:01:24 elapsed:3:55:02 postfix:- (1728:tqdm_logging.py:145)
10:41:27.858 [I] Progress on: 1.96kit/3.00kit rate:7.0s/it remaining:2:01:44 elapsed:3:55:16 postfix:- (1728:tqdm_logging.py:145)
10:41:41.975 [I] Progress on: 1.96kit/3.00kit rate:7.0s/it remaining:2:01:48 elapsed:3:55:30 postfix:- (1728:tqdm_logging.py:145)
10:41:56.073 [I] Progress on: 1.96kit/3.00kit rate:7.0s/it remaining:2:01:39 elapsed:3:55:44 postfix:- (1728:tqdm_logging.py:145)
10:42:10.176 [I] Progress on: 1.97kit/3.00kit rate:7.0s/it remaining:2:01:27 elapsed:3:55:58 postfix:- (1728:tqdm_logging.py:145)
10:42:24.296 [I] Progress on: 1.97kit/3.00kit rate:7.1s/it remaining:2:01:20 elapsed:3:56:13 postfix:- (1728:tqdm_logging.py:145)
10:42:38.416 [I] Progress on: 1.97kit/3.00kit rate:7.1s/it remaining:2:01:08 elapsed:3:56:27 postfix:- (1728:tqdm_logging.py:145)
10:42:52.527 [I] Progress on: 1.97kit/3.00kit rate:7.1s/it remaining:2:00:53 elapsed:3:56:41 postfix:- (1728:tqdm_logging.py:145)
10:43:06.622 [I] Progress on: 1.97kit/3.00kit rate:7.1s/it remaining:2:00:34 elapsed:3:56:55 postfix:- (1728:tqdm_logging.py:145)
10:43:20.741 [I] Progress on: 1.98kit/3.00kit rate:7.1s/it remaining:2:00:24 elapsed:3:57:09 postfix:- (1728:tqdm_logging.py:145)
10:43:34.857 [I] Progress on: 1.98kit/3.00kit rate:7.1s/it remaining:2:00:10 elapsed:3:57:23 postfix:- (1728:tqdm_logging.py:145)
10:43:48.950 [I] Progress on: 1.98kit/3.00kit rate:7.1s/it remaining:1:59:52 elapsed:3:57:37 postfix:- (1728:tqdm_logging.py:145)
10:44:03.049 [I] Progress on: 1.98kit/3.00kit rate:7.1s/it remaining:1:59:37 elapsed:3:57:51 postfix:- (1728:tqdm_logging.py:145)
10:44:17.168 [I] Progress on: 1.98kit/3.00kit rate:7.1s/it remaining:1:59:27 elapsed:3:58:05 postfix:- (1728:tqdm_logging.py:145)
10:44:31.266 [I] Progress on: 1.99kit/3.00kit rate:7.1s/it remaining:1:59:09 elapsed:3:58:20 postfix:- (1728:tqdm_logging.py:145)
10:44:45.371 [I] Progress on: 1.99kit/3.00kit rate:7.1s/it remaining:1:58:55 elapsed:3:58:34 postfix:- (1728:tqdm_logging.py:145)
10:44:59.496 [I] Progress on: 1.99kit/3.00kit rate:7.1s/it remaining:1:58:47 elapsed:3:58:48 postfix:- (1728:tqdm_logging.py:145)
10:45:13.619 [I] Progress on: 1.99kit/3.00kit rate:7.1s/it remaining:1:58:36 elapsed:3:59:02 postfix:- (1728:tqdm_logging.py:145)
10:45:27.718 [I] Progress on: 1.99kit/3.00kit rate:7.1s/it remaining:1:58:16 elapsed:3:59:16 postfix:- (1728:tqdm_logging.py:145)
10:45:41.844 [I] Progress on: 2.00kit/3.00kit rate:7.1s/it remaining:1:58:06 elapsed:3:59:30 postfix:- (1728:tqdm_logging.py:145)
10:45:56.500 [I] Progress on: 2.00kit/3.00kit rate:7.2s/it remaining:2:00:35 elapsed:3:59:45 postfix:- (1728:tqdm_logging.py:145)
10:46:10.059 [I] Progress on: 2.00kit/3.00kit rate:7.0s/it remaining:1:57:02 elapsed:3:59:58 postfix:- (1728:tqdm_logging.py:145)


Step 2000: grad_norm=0.0434, loss=0.0048, param_norm=1804.0712
10:46:23.852 [I] Progress on: 2.00kit/3.00kit rate:7.0s/it remaining:1:57:02 elapsed:4:00:12 postfix:- (1728:tqdm_logging.py:145)
10:46:23.882 [I] [process=0][thread=MainThread][wait_until_finished] No Save Finalize thread to wait for. Returning. (1728:checkpoint_manager.py:1987)
10:46:23.884 [I] [process=0] Saving checkpoint at step 2000                                       (1728:checkpoint_manager.py:1408)
10:46:23.885 [I] [process=0] Started async saving checkpoint to /workspace/openpi/checkpoints/pi05_mecharm_deep_even10_lora/deep_even10_v1/2000. (1728:async_checkpointer.py:439)
10:46:23.906 [I] Creating tmp directory /workspace/openpi/checkpoints/pi05_mecharm_deep_even10_lora/deep_even10_v1/2000.orbax-checkpoint-tmp-4 (1728:atomicity.py:144)
10:46:23.925 [I] Creating tmp directory /workspace/openpi/checkpoints/pi05_mecharm_deep_even10_lora/deep_even10_v1/2000.orbax-checkpoint-tmp-4/assets.orbax-checkpoint-tmp-5 (1728:atomicity.py:144)
10:46:23.925 [I] Creating tmp directory /workspace/openpi/checkpoints/pi05_mecharm_deep_even10_lora/deep_even10_v1/2000.orbax-checkpoint-tmp-4/params.orbax-checkpoint-tmp-6 (1728:atomicity.py:144)
10:46:23.925 [I] Creating tmp directory /workspace/openpi/checkpoints/pi05_mecharm_deep_even10_lora/deep_even10_v1/2000.orbax-checkpoint-tmp-4/train_state.orbax-checkpoint-tmp-7 (1728:atomicity.py:144)
10:46:23.949 [I] Transferring arrays to host memory with options: use_replica_parallel=True, enable_pinned_host_transfer=False (1728:replica_slices.py:341)
10:46:29.339 [I] Transferring arrays to host memory with options: use_replica_parallel=True, enable_pinned_host_transfer=False (1728:replica_slices.py:341)
10:46:29.347 [I] Array name: 'params.PaliGemma.llm.layers.mlp.gating_einsum.value', global shape: (18, 2, 2048, 16384), write shape: (18, 2, 2048, 16384), chosen chunk shape: (18, 2, 2048, 8192) (1728:tensorstore_utils.py:408)
10:46:29.402 [I] [process=0][thread=array_type_handler] Wrote 71 array_metadata.ArrayMetadata to /workspace/openpi/checkpoints/pi05_mecharm_deep_even10_lora/deep_even10_v1/2000.orbax-checkpoint-tmp-4/params.orbax-checkpoint-tmp-6/array_metadatas/process_0 (1728:array_metadata_store.py:198)
10:46:32.880 [I] [process=0] /jax/checkpoint/write/blocking_bytes_per_sec: 821.2 MiB/s (total bytes: 7.2 GiB) (time elapsed: 8 seconds) (per-host) (1728:base_pytree_checkpoint_handler.py:114)
10:46:32.885 [I] [process=0] /jax/checkpoint/write/blocking_bytes_per_sec: 396.9 MiB/s (total bytes: 3.5 GiB) (time elapsed: 8 seconds) (per-host) (1728:base_pytree_checkpoint_handler.py:114)
10:46:32.890 [I] [process=0][thread=async_save] Background save thread started.                   (1728:async_checkpointer.py:77)
10:46:32.890 [I] Finished blocking save in 9.01 seconds. Continuing to save asynchronously to /workspace/openpi/checkpoints/pi05_mecharm_deep_even10_lora/deep_even10_v1/2000. (1728:async_checkpointer.py:548)
10:46:32.891 [I] Preserving Checkpoint[step=1000 | time=2026-11-09 08:48:44.411183+00:00]: (Reason: on keep_period=1000). (1728:checkpoint_manager.py:1949)
10:46:32.892 [I] [process=0][thread=MainThread][step=2000] Starting CheckpointManager Save Finalize thread=save_finalize (1728:checkpoint_manager.py:1453)
10:46:32.893 [I] [process=0][thread=save_finalize] Waiting for background save thread=async_save. (1728:async_checkpointer.py:258)
10:46:32.894 [I] {'step': 2000, 'event_type': 'save', 'directory': '/workspace/openpi/checkpoints/pi05_mecharm_deep_even10_lora/deep_even10_v1', 'reached_preemption': False, 'preemption_received_at': None, 'synchronous': False, 'wait_for_prev_start_time': 1794221183.882874, 'wait_for_prev_duration_secs': 0.00022172927856445312, 'checkpointer_blocking_start_time': 1794221183.8842478, 'checkpointer_blocking_duration_secs': 9.006876945495605, 'get_old_steps_start_time': 1794221192.8911457, 'get_old_steps_duration_secs': 0.0008509159088134766, 'checkpoint_manager_blocking_start_time': 1794221183.8821158, 'checkpoint_manager_blocking_duration_secs': 9.011928796768188} (1728:standard_logger.py:34)
10:46:32.976 [I] [process=0][thread=array_type_handler] Wrote 105 array_metadata.ArrayMetadata to /workspace/openpi/checkpoints/pi05_mecharm_deep_even10_lora/deep_even10_v1/2000.orbax-checkpoint-tmp-4/train_state.orbax-checkpoint-tmp-7/array_metadatas/process_0 (1728:array_metadata_store.py:198)
10:46:35.022 [I] [process=0] /jax/checkpoint/write/bytes_per_sec: 663.2 MiB/s (total bytes: 7.2 GiB) (time elapsed: 11 seconds) (per-host) (1728:base_pytree_checkpoint_handler.py:114)
10:46:35.991 [I] [process=0] /jax/checkpoint/write/bytes_per_sec: 294.9 MiB/s (total bytes: 3.5 GiB) (time elapsed: 12 seconds) (per-host) (1728:base_pytree_checkpoint_handler.py:114)
10:46:35.991 [I] [process=0][thread=async_save] 5 Handler Commit operations completed.            (1728:async_checkpointer.py:87)
10:46:35.992 [I] Renaming /workspace/openpi/checkpoints/pi05_mecharm_deep_even10_lora/deep_even10_v1/2000.orbax-checkpoint-tmp-4/assets.orbax-checkpoint-tmp-5 to /workspace/openpi/checkpoints/pi05_mecharm_deep_even10_lora/deep_even10_v1/2000.orbax-checkpoint-tmp-4/assets (1728:atomicity.py:289)
10:46:35.997 [I] [process=0][thread=async_save] Skipped cross-host ArrayMetadata validation because only one process is found: process_index=0. (1728:array_metadata_store.py:362)
10:46:36.004 [I] Renaming /workspace/openpi/checkpoints/pi05_mecharm_deep_even10_lora/deep_even10_v1/2000.orbax-checkpoint-tmp-4/train_state.orbax-checkpoint-tmp-7 to /workspace/openpi/checkpoints/pi05_mecharm_deep_even10_lora/deep_even10_v1/2000.orbax-checkpoint-tmp-4/train_state (1728:atomicity.py:289)
10:46:36.008 [I] [process=0][thread=async_save] Skipped cross-host ArrayMetadata validation because only one process is found: process_index=0. (1728:array_metadata_store.py:362)
10:46:36.014 [I] Renaming /workspace/openpi/checkpoints/pi05_mecharm_deep_even10_lora/deep_even10_v1/2000.orbax-checkpoint-tmp-4/params.orbax-checkpoint-tmp-6 to /workspace/openpi/checkpoints/pi05_mecharm_deep_even10_lora/deep_even10_v1/2000.orbax-checkpoint-tmp-4/params (1728:atomicity.py:289)
10:46:36.015 [I] Renaming /workspace/openpi/checkpoints/pi05_mecharm_deep_even10_lora/deep_even10_v1/2000.orbax-checkpoint-tmp-4 to /workspace/openpi/checkpoints/pi05_mecharm_deep_even10_lora/deep_even10_v1/2000 (1728:atomicity.py:289)
10:46:36.018 [I] [process=0][thread=async_save] Finished saving checkpoint (finalized tmp dir) to `/workspace/openpi/checkpoints/pi05_mecharm_deep_even10_lora/deep_even10_v1/2000`. (1728:atomicity.py:573)
10:46:36.018 [I] Finished asynchronous save (blocking + background) in 12.13 seconds to /workspace/openpi/checkpoints/pi05_mecharm_deep_even10_lora/deep_even10_v1/2000 (1728:async_checkpointer.py:412)
10:46:36.018 [I] [process=0][thread=async_save] Background save thread done.                      (1728:async_checkpointer.py:143)
10:46:36.019 [I] [process=0][thread=save_finalize] Done with waiting for background save thread=async_save. (1728:async_checkpointer.py:266)
10:46:36.019 [I] [process=0][thread=save_finalize] No errors found in background save thread=async_save. (1728:async_checkpointer.py:276)
10:46:36.019 [I] [process=0][thread=save_finalize][step=2000] CheckpointManager Save Finalize is syncing with other hosts... (1728:checkpoint_manager.py:2095)
10:46:36.019 [I] [process=0][thread=save_finalize][step=2000] CheckpointManager Save Finalize is done on all hosts. (1728:checkpoint_manager.py:2110)
10:46:40.313 [I] Progress on: 2.00kit/3.00kit rate:7.9s/it remaining:2:11:43 elapsed:4:00:29 postfix:- (1728:tqdm_logging.py:145)
10:46:54.365 [I] Progress on: 2.00kit/3.00kit rate:7.5s/it remaining:2:03:48 elapsed:4:00:43 postfix:- (1728:tqdm_logging.py:145)
10:47:08.437 [I] Progress on: 2.01kit/3.00kit rate:7.2s/it remaining:1:59:52 elapsed:4:00:57 postfix:- (1728:tqdm_logging.py:145)
10:47:22.534 [I] Progress on: 2.01kit/3.00kit rate:7.1s/it remaining:1:57:57 elapsed:4:01:11 postfix:- (1728:tqdm_logging.py:145)
10:47:36.665 [I] Progress on: 2.01kit/3.00kit rate:7.1s/it remaining:1:57:03 elapsed:4:01:25 postfix:- (1728:tqdm_logging.py:145)
10:47:50.782 [I] Progress on: 2.01kit/3.00kit rate:7.1s/it remaining:1:56:27 elapsed:4:01:39 postfix:- (1728:tqdm_logging.py:145)
10:48:04.895 [I] Progress on: 2.02kit/3.00kit rate:7.1s/it remaining:1:56:02 elapsed:4:01:53 postfix:- (1728:tqdm_logging.py:145)
10:48:19.031 [I] Progress on: 2.02kit/3.00kit rate:7.1s/it remaining:1:55:47 elapsed:4:02:07 postfix:- (1728:tqdm_logging.py:145)
10:48:33.145 [I] Progress on: 2.02kit/3.00kit rate:7.1s/it remaining:1:55:28 elapsed:4:02:21 postfix:- (1728:tqdm_logging.py:145)
10:48:47.257 [I] Progress on: 2.02kit/3.00kit rate:7.1s/it remaining:1:55:10 elapsed:4:02:36 postfix:- (1728:tqdm_logging.py:145)
10:49:01.388 [I] Progress on: 2.02kit/3.00kit rate:7.1s/it remaining:1:54:59 elapsed:4:02:50 postfix:- (1728:tqdm_logging.py:145)
10:49:15.512 [I] Progress on: 2.02kit/3.00kit rate:7.1s/it remaining:1:54:44 elapsed:4:03:04 postfix:- (1728:tqdm_logging.py:145)
10:49:29.628 [I] Progress on: 2.03kit/3.00kit rate:7.1s/it remaining:1:54:29 elapsed:4:03:18 postfix:- (1728:tqdm_logging.py:145)
10:49:43.757 [I] Progress on: 2.03kit/3.00kit rate:7.1s/it remaining:1:54:17 elapsed:4:03:32 postfix:- (1728:tqdm_logging.py:145)
10:49:57.890 [I] Progress on: 2.03kit/3.00kit rate:7.1s/it remaining:1:54:05 elapsed:4:03:46 postfix:- (1728:tqdm_logging.py:145)
10:50:12.021 [I] Progress on: 2.03kit/3.00kit rate:7.1s/it remaining:1:53:51 elapsed:4:04:00 postfix:- (1728:tqdm_logging.py:145)
10:50:26.147 [I] Progress on: 2.04kit/3.00kit rate:7.1s/it remaining:1:53:37 elapsed:4:04:14 postfix:- (1728:tqdm_logging.py:145)
10:50:40.286 [I] Progress on: 2.04kit/3.00kit rate:7.1s/it remaining:1:53:25 elapsed:4:04:29 postfix:- (1728:tqdm_logging.py:145)
10:50:54.408 [I] Progress on: 2.04kit/3.00kit rate:7.1s/it remaining:1:53:08 elapsed:4:04:43 postfix:- (1728:tqdm_logging.py:145)
10:51:08.538 [I] Progress on: 2.04kit/3.00kit rate:7.1s/it remaining:1:52:55 elapsed:4:04:57 postfix:- (1728:tqdm_logging.py:145)
10:51:22.660 [I] Progress on: 2.04kit/3.00kit rate:7.1s/it remaining:1:52:39 elapsed:4:05:11 postfix:- (1728:tqdm_logging.py:145)
10:51:36.773 [I] Progress on: 2.04kit/3.00kit rate:7.1s/it remaining:1:52:22 elapsed:4:05:25 postfix:- (1728:tqdm_logging.py:145)
10:51:50.918 [I] Progress on: 2.05kit/3.00kit rate:7.1s/it remaining:1:52:14 elapsed:4:05:39 postfix:- (1728:tqdm_logging.py:145)
10:52:05.033 [I] Progress on: 2.05kit/3.00kit rate:7.1s/it remaining:1:51:55 elapsed:4:05:53 postfix:- (1728:tqdm_logging.py:145)


Step 2050: grad_norm=0.0472, loss=0.0055, param_norm=1804.0768
10:52:25.853 [I] Progress on: 2.05kit/3.00kit rate:7.1s/it remaining:1:51:45 elapsed:4:06:14 postfix:- (1728:tqdm_logging.py:145)
10:52:40.436 [I] Progress on: 2.05kit/3.00kit rate:6.8s/it remaining:1:46:54 elapsed:4:06:29 postfix:- (1728:tqdm_logging.py:145)
10:52:54.553 [I] Progress on: 2.06kit/3.00kit rate:6.9s/it remaining:1:48:53 elapsed:4:06:43 postfix:- (1728:tqdm_logging.py:145)
10:53:08.695 [I] Progress on: 2.06kit/3.00kit rate:7.0s/it remaining:1:49:52 elapsed:4:06:57 postfix:- (1728:tqdm_logging.py:145)
10:53:22.827 [I] Progress on: 2.06kit/3.00kit rate:7.0s/it remaining:1:50:10 elapsed:4:07:11 postfix:- (1728:tqdm_logging.py:145)
10:53:36.965 [I] Progress on: 2.06kit/3.00kit rate:7.1s/it remaining:1:50:13 elapsed:4:07:25 postfix:- (1728:tqdm_logging.py:145)
10:53:51.120 [I] Progress on: 2.06kit/3.00kit rate:7.1s/it remaining:1:50:12 elapsed:4:07:39 postfix:- (1728:tqdm_logging.py:145)
10:54:05.247 [I] Progress on: 2.07kit/3.00kit rate:7.1s/it remaining:1:49:57 elapsed:4:07:54 postfix:- (1728:tqdm_logging.py:145)
10:54:19.371 [I] Progress on: 2.07kit/3.00kit rate:7.1s/it remaining:1:49:42 elapsed:4:08:08 postfix:- (1728:tqdm_logging.py:145)
10:54:33.495 [I] Progress on: 2.07kit/3.00kit rate:7.1s/it remaining:1:49:28 elapsed:4:08:22 postfix:- (1728:tqdm_logging.py:145)
10:54:47.631 [I] Progress on: 2.07kit/3.00kit rate:7.1s/it remaining:1:49:16 elapsed:4:08:36 postfix:- (1728:tqdm_logging.py:145)
10:55:01.764 [I] Progress on: 2.07kit/3.00kit rate:7.1s/it remaining:1:49:02 elapsed:4:08:50 postfix:- (1728:tqdm_logging.py:145)
10:55:15.903 [I] Progress on: 2.08kit/3.00kit rate:7.1s/it remaining:1:48:50 elapsed:4:09:04 postfix:- (1728:tqdm_logging.py:145)
10:55:30.029 [I] Progress on: 2.08kit/3.00kit rate:7.1s/it remaining:1:48:34 elapsed:4:09:18 postfix:- (1728:tqdm_logging.py:145)
10:55:44.152 [I] Progress on: 2.08kit/3.00kit rate:7.1s/it remaining:1:48:18 elapsed:4:09:32 postfix:- (1728:tqdm_logging.py:145)
10:55:58.262 [I] Progress on: 2.08kit/3.00kit rate:7.1s/it remaining:1:48:00 elapsed:4:09:47 postfix:- (1728:tqdm_logging.py:145)
10:56:12.400 [I] Progress on: 2.08kit/3.00kit rate:7.1s/it remaining:1:47:50 elapsed:4:10:01 postfix:- (1728:tqdm_logging.py:145)
10:56:26.530 [I] Progress on: 2.09kit/3.00kit rate:7.1s/it remaining:1:47:37 elapsed:4:10:15 postfix:- (1728:tqdm_logging.py:145)
10:56:40.658 [I] Progress on: 2.09kit/3.00kit rate:7.1s/it remaining:1:47:23 elapsed:4:10:29 postfix:- (1728:tqdm_logging.py:145)
10:56:54.761 [I] Progress on: 2.09kit/3.00kit rate:7.1s/it remaining:1:47:04 elapsed:4:10:43 postfix:- (1728:tqdm_logging.py:145)
10:57:08.864 [I] Progress on: 2.09kit/3.00kit rate:7.1s/it remaining:1:46:46 elapsed:4:10:57 postfix:- (1728:tqdm_logging.py:145)
10:57:22.983 [I] Progress on: 2.09kit/3.00kit rate:7.1s/it remaining:1:46:34 elapsed:4:11:11 postfix:- (1728:tqdm_logging.py:145)
10:57:37.089 [I] Progress on: 2.10kit/3.00kit rate:7.1s/it remaining:1:46:18 elapsed:4:11:25 postfix:- (1728:tqdm_logging.py:145)
10:57:51.210 [I] Progress on: 2.10kit/3.00kit rate:7.1s/it remaining:1:46:06 elapsed:4:11:39 postfix:- (1728:tqdm_logging.py:145)
10:58:05.333 [I] Progress on: 2.10kit/3.00kit rate:7.1s/it remaining:1:45:53 elapsed:4:11:54 postfix:- (1728:tqdm_logging.py:145)


Step 2100: grad_norm=0.0436, loss=0.0047, param_norm=1804.0822
10:58:19.083 [I] Progress on: 2.10kit/3.00kit rate:7.1s/it remaining:1:45:53 elapsed:4:12:07 postfix:- (1728:tqdm_logging.py:145)
10:58:33.636 [I] Progress on: 2.10kit/3.00kit rate:6.8s/it remaining:1:41:06 elapsed:4:12:22 postfix:- (1728:tqdm_logging.py:145)
10:58:47.727 [I] Progress on: 2.11kit/3.00kit rate:6.9s/it remaining:1:42:58 elapsed:4:12:36 postfix:- (1728:tqdm_logging.py:145)
10:59:01.848 [I] Progress on: 2.11kit/3.00kit rate:7.0s/it remaining:1:43:52 elapsed:4:12:50 postfix:- (1728:tqdm_logging.py:145)
10:59:15.947 [I] Progress on: 2.11kit/3.00kit rate:7.0s/it remaining:1:44:06 elapsed:4:13:04 postfix:- (1728:tqdm_logging.py:145)
10:59:30.039 [I] Progress on: 2.11kit/3.00kit rate:7.0s/it remaining:1:44:04 elapsed:4:13:18 postfix:- (1728:tqdm_logging.py:145)
10:59:44.157 [I] Progress on: 2.11kit/3.00kit rate:7.0s/it remaining:1:44:01 elapsed:4:13:32 postfix:- (1728:tqdm_logging.py:145)
10:59:58.262 [I] Progress on: 2.12kit/3.00kit rate:7.0s/it remaining:1:43:50 elapsed:4:13:47 postfix:- (1728:tqdm_logging.py:145)
11:00:12.371 [I] Progress on: 2.12kit/3.00kit rate:7.1s/it remaining:1:43:39 elapsed:4:14:01 postfix:- (1728:tqdm_logging.py:145)
11:00:26.482 [I] Progress on: 2.12kit/3.00kit rate:7.1s/it remaining:1:43:26 elapsed:4:14:15 postfix:- (1728:tqdm_logging.py:145)
11:00:40.583 [I] Progress on: 2.12kit/3.00kit rate:7.1s/it remaining:1:43:11 elapsed:4:14:29 postfix:- (1728:tqdm_logging.py:145)
11:00:54.687 [I] Progress on: 2.12kit/3.00kit rate:7.1s/it remaining:1:42:56 elapsed:4:14:43 postfix:- (1728:tqdm_logging.py:145)
11:01:08.792 [I] Progress on: 2.13kit/3.00kit rate:7.1s/it remaining:1:42:43 elapsed:4:14:57 postfix:- (1728:tqdm_logging.py:145)
11:01:22.897 [I] Progress on: 2.13kit/3.00kit rate:7.1s/it remaining:1:42:28 elapsed:4:15:11 postfix:- (1728:tqdm_logging.py:145)
11:01:37.002 [I] Progress on: 2.13kit/3.00kit rate:7.1s/it remaining:1:42:15 elapsed:4:15:25 postfix:- (1728:tqdm_logging.py:145)
11:01:51.103 [I] Progress on: 2.13kit/3.00kit rate:7.1s/it remaining:1:42:00 elapsed:4:15:39 postfix:- (1728:tqdm_logging.py:145)
11:02:05.215 [I] Progress on: 2.13kit/3.00kit rate:7.1s/it remaining:1:41:47 elapsed:4:15:54 postfix:- (1728:tqdm_logging.py:145)
11:02:19.325 [I] Progress on: 2.14kit/3.00kit rate:7.1s/it remaining:1:41:34 elapsed:4:16:08 postfix:- (1728:tqdm_logging.py:145)
11:02:33.420 [I] Progress on: 2.14kit/3.00kit rate:7.1s/it remaining:1:41:17 elapsed:4:16:22 postfix:- (1728:tqdm_logging.py:145)
11:02:47.516 [I] Progress on: 2.14kit/3.00kit rate:7.0s/it remaining:1:41:02 elapsed:4:16:36 postfix:- (1728:tqdm_logging.py:145)
11:03:01.634 [I] Progress on: 2.14kit/3.00kit rate:7.1s/it remaining:1:40:52 elapsed:4:16:50 postfix:- (1728:tqdm_logging.py:145)
11:03:15.724 [I] Progress on: 2.14kit/3.00kit rate:7.1s/it remaining:1:40:35 elapsed:4:17:04 postfix:- (1728:tqdm_logging.py:145)
11:03:29.818 [I] Progress on: 2.15kit/3.00kit rate:7.0s/it remaining:1:40:19 elapsed:4:17:18 postfix:- (1728:tqdm_logging.py:145)
11:03:43.920 [I] Progress on: 2.15kit/3.00kit rate:7.1s/it remaining:1:40:07 elapsed:4:17:32 postfix:- (1728:tqdm_logging.py:145)
11:03:58.031 [I] Progress on: 2.15kit/3.00kit rate:7.1s/it remaining:1:39:55 elapsed:4:17:46 postfix:- (1728:tqdm_logging.py:145)


Step 2150: grad_norm=0.0452, loss=0.0046, param_norm=1804.0868
11:04:11.794 [I] Progress on: 2.15kit/3.00kit rate:7.1s/it remaining:1:39:55 elapsed:4:18:00 postfix:- (1728:tqdm_logging.py:145)
11:04:26.318 [I] Progress on: 2.15kit/3.00kit rate:6.8s/it remaining:1:35:25 elapsed:4:18:15 postfix:- (1728:tqdm_logging.py:145)
11:04:40.409 [I] Progress on: 2.16kit/3.00kit rate:6.9s/it remaining:1:37:11 elapsed:4:18:29 postfix:- (1728:tqdm_logging.py:145)
11:04:54.530 [I] Progress on: 2.16kit/3.00kit rate:7.0s/it remaining:1:38:03 elapsed:4:18:43 postfix:- (1728:tqdm_logging.py:145)
11:05:08.631 [I] Progress on: 2.16kit/3.00kit rate:7.0s/it remaining:1:38:15 elapsed:4:18:57 postfix:- (1728:tqdm_logging.py:145)
11:05:22.734 [I] Progress on: 2.16kit/3.00kit rate:7.0s/it remaining:1:38:16 elapsed:4:19:11 postfix:- (1728:tqdm_logging.py:145)
11:05:36.847 [I] Progress on: 2.16kit/3.00kit rate:7.0s/it remaining:1:38:11 elapsed:4:19:25 postfix:- (1728:tqdm_logging.py:145)
11:05:50.952 [I] Progress on: 2.17kit/3.00kit rate:7.1s/it remaining:1:38:00 elapsed:4:19:39 postfix:- (1728:tqdm_logging.py:145)
11:06:05.049 [I] Progress on: 2.17kit/3.00kit rate:7.0s/it remaining:1:37:45 elapsed:4:19:53 postfix:- (1728:tqdm_logging.py:145)
11:06:19.145 [I] Progress on: 2.17kit/3.00kit rate:7.0s/it remaining:1:37:31 elapsed:4:20:07 postfix:- (1728:tqdm_logging.py:145)
11:06:33.270 [I] Progress on: 2.17kit/3.00kit rate:7.1s/it remaining:1:37:23 elapsed:4:20:22 postfix:- (1728:tqdm_logging.py:145)
11:06:47.377 [I] Progress on: 2.17kit/3.00kit rate:7.1s/it remaining:1:37:07 elapsed:4:20:36 postfix:- (1728:tqdm_logging.py:145)
11:07:01.484 [I] Progress on: 2.18kit/3.00kit rate:7.1s/it remaining:1:36:52 elapsed:4:20:50 postfix:- (1728:tqdm_logging.py:145)
11:07:15.596 [I] Progress on: 2.18kit/3.00kit rate:7.1s/it remaining:1:36:39 elapsed:4:21:04 postfix:- (1728:tqdm_logging.py:145)
11:07:29.688 [I] Progress on: 2.18kit/3.00kit rate:7.1s/it remaining:1:36:21 elapsed:4:21:18 postfix:- (1728:tqdm_logging.py:145)
11:07:43.792 [I] Progress on: 2.18kit/3.00kit rate:7.1s/it remaining:1:36:08 elapsed:4:21:32 postfix:- (1728:tqdm_logging.py:145)
11:07:57.889 [I] Progress on: 2.18kit/3.00kit rate:7.1s/it remaining:1:35:53 elapsed:4:21:46 postfix:- (1728:tqdm_logging.py:145)
11:08:11.999 [I] Progress on: 2.19kit/3.00kit rate:7.1s/it remaining:1:35:41 elapsed:4:22:00 postfix:- (1728:tqdm_logging.py:145)
11:08:26.099 [I] Progress on: 2.19kit/3.00kit rate:7.1s/it remaining:1:35:26 elapsed:4:22:14 postfix:- (1728:tqdm_logging.py:145)
11:08:40.195 [I] Progress on: 2.19kit/3.00kit rate:7.1s/it remaining:1:35:11 elapsed:4:22:28 postfix:- (1728:tqdm_logging.py:145)
11:08:54.309 [I] Progress on: 2.19kit/3.00kit rate:7.1s/it remaining:1:35:00 elapsed:4:22:43 postfix:- (1728:tqdm_logging.py:145)
11:09:08.433 [I] Progress on: 2.19kit/3.00kit rate:7.1s/it remaining:1:34:49 elapsed:4:22:57 postfix:- (1728:tqdm_logging.py:145)
11:09:22.526 [I] Progress on: 2.20kit/3.00kit rate:7.1s/it remaining:1:34:30 elapsed:4:23:11 postfix:- (1728:tqdm_logging.py:145)
11:09:36.633 [I] Progress on: 2.20kit/3.00kit rate:7.1s/it remaining:1:34:16 elapsed:4:23:25 postfix:- (1728:tqdm_logging.py:145)
11:09:50.743 [I] Progress on: 2.20kit/3.00kit rate:7.1s/it remaining:1:34:03 elapsed:4:23:39 postfix:- (1728:tqdm_logging.py:145)


Step 2200: grad_norm=0.0434, loss=0.0047, param_norm=1804.0909
11:10:04.480 [I] Progress on: 2.20kit/3.00kit rate:7.1s/it remaining:1:34:03 elapsed:4:23:53 postfix:- (1728:tqdm_logging.py:145)
11:10:18.992 [I] Progress on: 2.20kit/3.00kit rate:6.8s/it remaining:1:29:43 elapsed:4:24:07 postfix:- (1728:tqdm_logging.py:145)
11:10:33.108 [I] Progress on: 2.21kit/3.00kit rate:6.9s/it remaining:1:31:29 elapsed:4:24:21 postfix:- (1728:tqdm_logging.py:145)
11:10:47.227 [I] Progress on: 2.21kit/3.00kit rate:7.0s/it remaining:1:32:14 elapsed:4:24:36 postfix:- (1728:tqdm_logging.py:145)
11:11:01.336 [I] Progress on: 2.21kit/3.00kit rate:7.0s/it remaining:1:32:27 elapsed:4:24:50 postfix:- (1728:tqdm_logging.py:145)
11:11:15.439 [I] Progress on: 2.21kit/3.00kit rate:7.0s/it remaining:1:32:25 elapsed:4:25:04 postfix:- (1728:tqdm_logging.py:145)
11:11:29.564 [I] Progress on: 2.21kit/3.00kit rate:7.0s/it remaining:1:32:21 elapsed:4:25:18 postfix:- (1728:tqdm_logging.py:145)
11:11:43.674 [I] Progress on: 2.22kit/3.00kit rate:7.1s/it remaining:1:32:09 elapsed:4:25:32 postfix:- (1728:tqdm_logging.py:145)
11:11:57.794 [I] Progress on: 2.22kit/3.00kit rate:7.1s/it remaining:1:31:58 elapsed:4:25:46 postfix:- (1728:tqdm_logging.py:145)
11:12:12.404 [I] Progress on: 2.22kit/3.00kit rate:7.2s/it remaining:1:33:40 elapsed:4:26:01 postfix:- (1728:tqdm_logging.py:145)
11:12:26.010 [I] Progress on: 2.22kit/3.00kit rate:7.0s/it remaining:1:31:06 elapsed:4:26:14 postfix:- (1728:tqdm_logging.py:145)
11:12:40.129 [I] Progress on: 2.22kit/3.00kit rate:7.0s/it remaining:1:31:08 elapsed:4:26:28 postfix:- (1728:tqdm_logging.py:145)
11:12:54.240 [I] Progress on: 2.23kit/3.00kit rate:7.1s/it remaining:1:30:59 elapsed:4:26:43 postfix:- (1728:tqdm_logging.py:145)
11:13:08.357 [I] Progress on: 2.23kit/3.00kit rate:7.1s/it remaining:1:30:49 elapsed:4:26:57 postfix:- (1728:tqdm_logging.py:145)
11:13:22.472 [I] Progress on: 2.23kit/3.00kit rate:7.1s/it remaining:1:30:38 elapsed:4:27:11 postfix:- (1728:tqdm_logging.py:145)
11:13:36.573 [I] Progress on: 2.23kit/3.00kit rate:7.1s/it remaining:1:30:22 elapsed:4:27:25 postfix:- (1728:tqdm_logging.py:145)
11:13:50.664 [I] Progress on: 2.23kit/3.00kit rate:7.1s/it remaining:1:30:05 elapsed:4:27:39 postfix:- (1728:tqdm_logging.py:145)
11:14:04.779 [I] Progress on: 2.24kit/3.00kit rate:7.1s/it remaining:1:29:54 elapsed:4:27:53 postfix:- (1728:tqdm_logging.py:145)
11:14:18.895 [I] Progress on: 2.24kit/3.00kit rate:7.1s/it remaining:1:29:42 elapsed:4:28:07 postfix:- (1728:tqdm_logging.py:145)
11:14:33.010 [I] Progress on: 2.24kit/3.00kit rate:7.1s/it remaining:1:29:28 elapsed:4:28:21 postfix:- (1728:tqdm_logging.py:145)
11:14:47.111 [I] Progress on: 2.24kit/3.00kit rate:7.1s/it remaining:1:29:11 elapsed:4:28:35 postfix:- (1728:tqdm_logging.py:145)
11:15:01.233 [I] Progress on: 2.24kit/3.00kit rate:7.1s/it remaining:1:29:00 elapsed:4:28:50 postfix:- (1728:tqdm_logging.py:145)
11:15:15.359 [I] Progress on: 2.25kit/3.00kit rate:7.1s/it remaining:1:28:48 elapsed:4:29:04 postfix:- (1728:tqdm_logging.py:145)
11:15:29.454 [I] Progress on: 2.25kit/3.00kit rate:7.1s/it remaining:1:28:28 elapsed:4:29:18 postfix:- (1728:tqdm_logging.py:145)
11:15:43.542 [I] Progress on: 2.25kit/3.00kit rate:7.1s/it remaining:1:28:07 elapsed:4:29:32 postfix:- (1728:tqdm_logging.py:145)


Step 2250: grad_norm=0.0419, loss=0.0046, param_norm=1804.0946
11:15:57.294 [I] Progress on: 2.25kit/3.00kit rate:7.1s/it remaining:1:28:07 elapsed:4:29:46 postfix:- (1728:tqdm_logging.py:145)
11:16:11.826 [I] Progress on: 2.25kit/3.00kit rate:6.8s/it remaining:1:24:06 elapsed:4:30:00 postfix:- (1728:tqdm_logging.py:145)
11:16:25.986 [I] Progress on: 2.26kit/3.00kit rate:6.9s/it remaining:1:25:55 elapsed:4:30:14 postfix:- (1728:tqdm_logging.py:145)
11:16:40.121 [I] Progress on: 2.26kit/3.00kit rate:7.0s/it remaining:1:26:36 elapsed:4:30:28 postfix:- (1728:tqdm_logging.py:145)
11:16:54.166 [I] Progress on: 2.26kit/3.00kit rate:7.0s/it remaining:1:26:29 elapsed:4:30:42 postfix:- (1728:tqdm_logging.py:145)
11:17:08.265 [I] Progress on: 2.26kit/3.00kit rate:7.0s/it remaining:1:26:29 elapsed:4:30:57 postfix:- (1728:tqdm_logging.py:145)
11:17:22.435 [I] Progress on: 2.26kit/3.00kit rate:7.1s/it remaining:1:26:34 elapsed:4:31:11 postfix:- (1728:tqdm_logging.py:145)
11:17:36.480 [I] Progress on: 2.27kit/3.00kit rate:7.0s/it remaining:1:26:08 elapsed:4:31:25 postfix:- (1728:tqdm_logging.py:145)
11:17:50.581 [I] Progress on: 2.27kit/3.00kit rate:7.0s/it remaining:1:25:57 elapsed:4:31:39 postfix:- (1728:tqdm_logging.py:145)
11:18:04.690 [I] Progress on: 2.27kit/3.00kit rate:7.1s/it remaining:1:25:46 elapsed:4:31:53 postfix:- (1728:tqdm_logging.py:145)
11:18:18.817 [I] Progress on: 2.27kit/3.00kit rate:7.1s/it remaining:1:25:37 elapsed:4:32:07 postfix:- (1728:tqdm_logging.py:145)
11:18:32.902 [I] Progress on: 2.27kit/3.00kit rate:7.0s/it remaining:1:25:18 elapsed:4:32:21 postfix:- (1728:tqdm_logging.py:145)
11:18:47.014 [I] Progress on: 2.28kit/3.00kit rate:7.1s/it remaining:1:25:06 elapsed:4:32:35 postfix:- (1728:tqdm_logging.py:145)
11:19:01.141 [I] Progress on: 2.28kit/3.00kit rate:7.1s/it remaining:1:24:56 elapsed:4:32:49 postfix:- (1728:tqdm_logging.py:145)
11:19:15.249 [I] Progress on: 2.28kit/3.00kit rate:7.1s/it remaining:1:24:41 elapsed:4:33:04 postfix:- (1728:tqdm_logging.py:145)
11:19:29.356 [I] Progress on: 2.28kit/3.00kit rate:7.1s/it remaining:1:24:26 elapsed:4:33:18 postfix:- (1728:tqdm_logging.py:145)
11:19:43.447 [I] Progress on: 2.28kit/3.00kit rate:7.1s/it remaining:1:24:08 elapsed:4:33:32 postfix:- (1728:tqdm_logging.py:145)
11:19:57.557 [I] Progress on: 2.29kit/3.00kit rate:7.1s/it remaining:1:23:55 elapsed:4:33:46 postfix:- (1728:tqdm_logging.py:145)
11:20:11.676 [I] Progress on: 2.29kit/3.00kit rate:7.1s/it remaining:1:23:44 elapsed:4:34:00 postfix:- (1728:tqdm_logging.py:145)
11:20:25.796 [I] Progress on: 2.29kit/3.00kit rate:7.1s/it remaining:1:23:30 elapsed:4:34:14 postfix:- (1728:tqdm_logging.py:145)
11:20:39.911 [I] Progress on: 2.29kit/3.00kit rate:7.1s/it remaining:1:23:16 elapsed:4:34:28 postfix:- (1728:tqdm_logging.py:145)
11:20:54.033 [I] Progress on: 2.29kit/3.00kit rate:7.1s/it remaining:1:23:04 elapsed:4:34:42 postfix:- (1728:tqdm_logging.py:145)
11:21:08.141 [I] Progress on: 2.30kit/3.00kit rate:7.1s/it remaining:1:22:48 elapsed:4:34:56 postfix:- (1728:tqdm_logging.py:145)
11:21:22.258 [I] Progress on: 2.30kit/3.00kit rate:7.1s/it remaining:1:22:35 elapsed:4:35:11 postfix:- (1728:tqdm_logging.py:145)
11:21:36.359 [I] Progress on: 2.30kit/3.00kit rate:7.1s/it remaining:1:22:19 elapsed:4:35:25 postfix:- (1728:tqdm_logging.py:145)


Step 2300: grad_norm=0.0421, loss=0.0047, param_norm=1804.0990
11:21:50.098 [I] Progress on: 2.30kit/3.00kit rate:7.1s/it remaining:1:22:19 elapsed:4:35:38 postfix:- (1728:tqdm_logging.py:145)
11:22:04.611 [I] Progress on: 2.30kit/3.00kit rate:6.8s/it remaining:1:18:27 elapsed:4:35:53 postfix:- (1728:tqdm_logging.py:145)
11:22:18.718 [I] Progress on: 2.31kit/3.00kit rate:6.9s/it remaining:1:19:56 elapsed:4:36:07 postfix:- (1728:tqdm_logging.py:145)
11:22:32.840 [I] Progress on: 2.31kit/3.00kit rate:7.0s/it remaining:1:20:35 elapsed:4:36:21 postfix:- (1728:tqdm_logging.py:145)
11:22:46.950 [I] Progress on: 2.31kit/3.00kit rate:7.0s/it remaining:1:20:45 elapsed:4:36:35 postfix:- (1728:tqdm_logging.py:145)
11:23:01.051 [I] Progress on: 2.31kit/3.00kit rate:7.0s/it remaining:1:20:40 elapsed:4:36:49 postfix:- (1728:tqdm_logging.py:145)
11:23:15.171 [I] Progress on: 2.31kit/3.00kit rate:7.0s/it remaining:1:20:35 elapsed:4:37:03 postfix:- (1728:tqdm_logging.py:145)
11:23:29.291 [I] Progress on: 2.32kit/3.00kit rate:7.1s/it remaining:1:20:25 elapsed:4:37:18 postfix:- (1728:tqdm_logging.py:145)
11:23:43.394 [I] Progress on: 2.32kit/3.00kit rate:7.1s/it remaining:1:20:11 elapsed:4:37:32 postfix:- (1728:tqdm_logging.py:145)
11:23:57.505 [I] Progress on: 2.32kit/3.00kit rate:7.1s/it remaining:1:19:57 elapsed:4:37:46 postfix:- (1728:tqdm_logging.py:145)
11:24:11.627 [I] Progress on: 2.32kit/3.00kit rate:7.1s/it remaining:1:19:46 elapsed:4:38:00 postfix:- (1728:tqdm_logging.py:145)
11:24:25.750 [I] Progress on: 2.32kit/3.00kit rate:7.1s/it remaining:1:19:33 elapsed:4:38:14 postfix:- (1728:tqdm_logging.py:145)
11:24:39.857 [I] Progress on: 2.33kit/3.00kit rate:7.1s/it remaining:1:19:16 elapsed:4:38:28 postfix:- (1728:tqdm_logging.py:145)
11:24:53.964 [I] Progress on: 2.33kit/3.00kit rate:7.1s/it remaining:1:19:00 elapsed:4:38:42 postfix:- (1728:tqdm_logging.py:145)
11:25:08.087 [I] Progress on: 2.33kit/3.00kit rate:7.1s/it remaining:1:18:48 elapsed:4:38:56 postfix:- (1728:tqdm_logging.py:145)
11:25:22.214 [I] Progress on: 2.33kit/3.00kit rate:7.1s/it remaining:1:18:36 elapsed:4:39:11 postfix:- (1728:tqdm_logging.py:145)
11:25:36.301 [I] Progress on: 2.33kit/3.00kit rate:7.1s/it remaining:1:18:16 elapsed:4:39:25 postfix:- (1728:tqdm_logging.py:145)
11:25:50.412 [I] Progress on: 2.34kit/3.00kit rate:7.1s/it remaining:1:18:03 elapsed:4:39:39 postfix:- (1728:tqdm_logging.py:145)
11:26:04.522 [I] Progress on: 2.34kit/3.00kit rate:7.1s/it remaining:1:17:49 elapsed:4:39:53 postfix:- (1728:tqdm_logging.py:145)
11:26:18.694 [I] Progress on: 2.34kit/3.00kit rate:7.1s/it remaining:1:17:43 elapsed:4:40:07 postfix:- (1728:tqdm_logging.py:145)
11:26:32.807 [I] Progress on: 2.34kit/3.00kit rate:7.1s/it remaining:1:17:27 elapsed:4:40:21 postfix:- (1728:tqdm_logging.py:145)
11:26:46.892 [I] Progress on: 2.34kit/3.00kit rate:7.1s/it remaining:1:17:07 elapsed:4:40:35 postfix:- (1728:tqdm_logging.py:145)
11:27:00.977 [I] Progress on: 2.35kit/3.00kit rate:7.0s/it remaining:1:16:49 elapsed:4:40:49 postfix:- (1728:tqdm_logging.py:145)
11:27:15.078 [I] Progress on: 2.35kit/3.00kit rate:7.0s/it remaining:1:16:36 elapsed:4:41:03 postfix:- (1728:tqdm_logging.py:145)
11:27:29.201 [I] Progress on: 2.35kit/3.00kit rate:7.1s/it remaining:1:16:25 elapsed:4:41:17 postfix:- (1728:tqdm_logging.py:145)


Step 2350: grad_norm=0.0401, loss=0.0041, param_norm=1804.1027
11:27:42.959 [I] Progress on: 2.35kit/3.00kit rate:7.1s/it remaining:1:16:25 elapsed:4:41:31 postfix:- (1728:tqdm_logging.py:145)
11:27:57.491 [I] Progress on: 2.35kit/3.00kit rate:6.8s/it remaining:1:12:51 elapsed:4:41:46 postfix:- (1728:tqdm_logging.py:145)
11:28:11.606 [I] Progress on: 2.36kit/3.00kit rate:6.9s/it remaining:1:14:12 elapsed:4:42:00 postfix:- (1728:tqdm_logging.py:145)
11:28:25.721 [I] Progress on: 2.36kit/3.00kit rate:7.0s/it remaining:1:14:45 elapsed:4:42:14 postfix:- (1728:tqdm_logging.py:145)
11:28:39.815 [I] Progress on: 2.36kit/3.00kit rate:7.0s/it remaining:1:14:50 elapsed:4:42:28 postfix:- (1728:tqdm_logging.py:145)
11:28:53.934 [I] Progress on: 2.36kit/3.00kit rate:7.0s/it remaining:1:14:50 elapsed:4:42:42 postfix:- (1728:tqdm_logging.py:145)
11:29:08.037 [I] Progress on: 2.36kit/3.00kit rate:7.0s/it remaining:1:14:40 elapsed:4:42:56 postfix:- (1728:tqdm_logging.py:145)
11:29:22.132 [I] Progress on: 2.37kit/3.00kit rate:7.0s/it remaining:1:14:27 elapsed:4:43:10 postfix:- (1728:tqdm_logging.py:145)
11:29:36.249 [I] Progress on: 2.37kit/3.00kit rate:7.1s/it remaining:1:14:17 elapsed:4:43:25 postfix:- (1728:tqdm_logging.py:145)
11:29:50.333 [I] Progress on: 2.37kit/3.00kit rate:7.0s/it remaining:1:13:59 elapsed:4:43:39 postfix:- (1728:tqdm_logging.py:145)
11:30:04.433 [I] Progress on: 2.37kit/3.00kit rate:7.0s/it remaining:1:13:46 elapsed:4:43:53 postfix:- (1728:tqdm_logging.py:145)
11:30:18.554 [I] Progress on: 2.37kit/3.00kit rate:7.1s/it remaining:1:13:35 elapsed:4:44:07 postfix:- (1728:tqdm_logging.py:145)
11:30:32.664 [I] Progress on: 2.38kit/3.00kit rate:7.1s/it remaining:1:13:21 elapsed:4:44:21 postfix:- (1728:tqdm_logging.py:145)
11:30:46.785 [I] Progress on: 2.38kit/3.00kit rate:7.1s/it remaining:1:13:09 elapsed:4:44:35 postfix:- (1728:tqdm_logging.py:145)
11:31:00.878 [I] Progress on: 2.38kit/3.00kit rate:7.1s/it remaining:1:12:52 elapsed:4:44:49 postfix:- (1728:tqdm_logging.py:145)
11:31:14.992 [I] Progress on: 2.38kit/3.00kit rate:7.1s/it remaining:1:12:39 elapsed:4:45:03 postfix:- (1728:tqdm_logging.py:145)
11:31:29.102 [I] Progress on: 2.38kit/3.00kit rate:7.1s/it remaining:1:12:25 elapsed:4:45:17 postfix:- (1728:tqdm_logging.py:145)
11:31:43.210 [I] Progress on: 2.39kit/3.00kit rate:7.1s/it remaining:1:12:11 elapsed:4:45:31 postfix:- (1728:tqdm_logging.py:145)
11:31:57.331 [I] Progress on: 2.39kit/3.00kit rate:7.1s/it remaining:1:11:59 elapsed:4:45:46 postfix:- (1728:tqdm_logging.py:145)
11:32:11.437 [I] Progress on: 2.39kit/3.00kit rate:7.1s/it remaining:1:11:43 elapsed:4:46:00 postfix:- (1728:tqdm_logging.py:145)
11:32:25.535 [I] Progress on: 2.39kit/3.00kit rate:7.1s/it remaining:1:11:27 elapsed:4:46:14 postfix:- (1728:tqdm_logging.py:145)
11:32:39.647 [I] Progress on: 2.39kit/3.00kit rate:7.1s/it remaining:1:11:14 elapsed:4:46:28 postfix:- (1728:tqdm_logging.py:145)
11:32:53.755 [I] Progress on: 2.40kit/3.00kit rate:7.1s/it remaining:1:10:59 elapsed:4:46:42 postfix:- (1728:tqdm_logging.py:145)
11:33:07.857 [I] Progress on: 2.40kit/3.00kit rate:7.1s/it remaining:1:10:45 elapsed:4:46:56 postfix:- (1728:tqdm_logging.py:145)
11:33:21.972 [I] Progress on: 2.40kit/3.00kit rate:7.1s/it remaining:1:10:32 elapsed:4:47:10 postfix:- (1728:tqdm_logging.py:145)


Step 2400: grad_norm=0.0420, loss=0.0044, param_norm=1804.1060
11:33:35.732 [I] Progress on: 2.40kit/3.00kit rate:7.1s/it remaining:1:10:32 elapsed:4:47:24 postfix:- (1728:tqdm_logging.py:145)
11:33:50.260 [I] Progress on: 2.40kit/3.00kit rate:6.8s/it remaining:1:07:11 elapsed:4:47:39 postfix:- (1728:tqdm_logging.py:145)
11:34:04.369 [I] Progress on: 2.41kit/3.00kit rate:6.9s/it remaining:1:08:26 elapsed:4:47:53 postfix:- (1728:tqdm_logging.py:145)
11:34:18.470 [I] Progress on: 2.41kit/3.00kit rate:7.0s/it remaining:1:08:54 elapsed:4:48:07 postfix:- (1728:tqdm_logging.py:145)
11:34:32.574 [I] Progress on: 2.41kit/3.00kit rate:7.0s/it remaining:1:09:00 elapsed:4:48:21 postfix:- (1728:tqdm_logging.py:145)
11:34:46.667 [I] Progress on: 2.41kit/3.00kit rate:7.0s/it remaining:1:08:54 elapsed:4:48:35 postfix:- (1728:tqdm_logging.py:145)
11:35:00.779 [I] Progress on: 2.41kit/3.00kit rate:7.0s/it remaining:1:08:47 elapsed:4:48:49 postfix:- (1728:tqdm_logging.py:145)
11:35:14.862 [I] Progress on: 2.42kit/3.00kit rate:7.0s/it remaining:1:08:32 elapsed:4:49:03 postfix:- (1728:tqdm_logging.py:145)
11:35:28.957 [I] Progress on: 2.42kit/3.00kit rate:7.0s/it remaining:1:08:19 elapsed:4:49:17 postfix:- (1728:tqdm_logging.py:145)
11:35:43.074 [I] Progress on: 2.42kit/3.00kit rate:7.1s/it remaining:1:08:10 elapsed:4:49:31 postfix:- (1728:tqdm_logging.py:145)
11:35:57.177 [I] Progress on: 2.42kit/3.00kit rate:7.1s/it remaining:1:07:56 elapsed:4:49:45 postfix:- (1728:tqdm_logging.py:145)
11:36:11.284 [I] Progress on: 2.42kit/3.00kit rate:7.1s/it remaining:1:07:42 elapsed:4:50:00 postfix:- (1728:tqdm_logging.py:145)
11:36:25.371 [I] Progress on: 2.43kit/3.00kit rate:7.0s/it remaining:1:07:25 elapsed:4:50:14 postfix:- (1728:tqdm_logging.py:145)
11:36:39.493 [I] Progress on: 2.43kit/3.00kit rate:7.1s/it remaining:1:07:15 elapsed:4:50:28 postfix:- (1728:tqdm_logging.py:145)
11:36:53.608 [I] Progress on: 2.43kit/3.00kit rate:7.1s/it remaining:1:07:01 elapsed:4:50:42 postfix:- (1728:tqdm_logging.py:145)
11:37:07.712 [I] Progress on: 2.43kit/3.00kit rate:7.1s/it remaining:1:06:46 elapsed:4:50:56 postfix:- (1728:tqdm_logging.py:145)
11:37:21.824 [I] Progress on: 2.43kit/3.00kit rate:7.1s/it remaining:1:06:33 elapsed:4:51:10 postfix:- (1728:tqdm_logging.py:145)
11:37:35.930 [I] Progress on: 2.44kit/3.00kit rate:7.1s/it remaining:1:06:18 elapsed:4:51:24 postfix:- (1728:tqdm_logging.py:145)
11:37:50.032 [I] Progress on: 2.44kit/3.00kit rate:7.1s/it remaining:1:06:03 elapsed:4:51:38 postfix:- (1728:tqdm_logging.py:145)
11:38:04.133 [I] Progress on: 2.44kit/3.00kit rate:7.1s/it remaining:1:05:49 elapsed:4:51:52 postfix:- (1728:tqdm_logging.py:145)
11:38:18.732 [I] Progress on: 2.44kit/3.00kit rate:7.2s/it remaining:1:06:57 elapsed:4:52:07 postfix:- (1728:tqdm_logging.py:145)
11:38:32.342 [I] Progress on: 2.44kit/3.00kit rate:7.0s/it remaining:1:05:04 elapsed:4:52:21 postfix:- (1728:tqdm_logging.py:145)
11:38:46.439 [I] Progress on: 2.45kit/3.00kit rate:7.0s/it remaining:1:04:57 elapsed:4:52:35 postfix:- (1728:tqdm_logging.py:145)
11:39:00.521 [I] Progress on: 2.45kit/3.00kit rate:7.0s/it remaining:1:04:45 elapsed:4:52:49 postfix:- (1728:tqdm_logging.py:145)
11:39:14.620 [I] Progress on: 2.45kit/3.00kit rate:7.0s/it remaining:1:04:33 elapsed:4:53:03 postfix:- (1728:tqdm_logging.py:145)


Step 2450: grad_norm=0.0408, loss=0.0043, param_norm=1804.1099
11:39:28.369 [I] Progress on: 2.45kit/3.00kit rate:7.0s/it remaining:1:04:33 elapsed:4:53:17 postfix:- (1728:tqdm_logging.py:145)
11:39:42.892 [I] Progress on: 2.45kit/3.00kit rate:6.8s/it remaining:1:01:32 elapsed:4:53:31 postfix:- (1728:tqdm_logging.py:145)
11:39:56.980 [I] Progress on: 2.46kit/3.00kit rate:6.9s/it remaining:1:02:36 elapsed:4:53:45 postfix:- (1728:tqdm_logging.py:145)
11:40:11.086 [I] Progress on: 2.46kit/3.00kit rate:7.0s/it remaining:1:03:03 elapsed:4:53:59 postfix:- (1728:tqdm_logging.py:145)
11:40:25.201 [I] Progress on: 2.46kit/3.00kit rate:7.0s/it remaining:1:03:10 elapsed:4:54:13 postfix:- (1728:tqdm_logging.py:145)
11:40:39.305 [I] Progress on: 2.46kit/3.00kit rate:7.0s/it remaining:1:03:04 elapsed:4:54:28 postfix:- (1728:tqdm_logging.py:145)
11:40:53.414 [I] Progress on: 2.46kit/3.00kit rate:7.0s/it remaining:1:02:56 elapsed:4:54:42 postfix:- (1728:tqdm_logging.py:145)
11:41:07.527 [I] Progress on: 2.47kit/3.00kit rate:7.1s/it remaining:1:02:46 elapsed:4:54:56 postfix:- (1728:tqdm_logging.py:145)
11:41:21.639 [I] Progress on: 2.47kit/3.00kit rate:7.1s/it remaining:1:02:33 elapsed:4:55:10 postfix:- (1728:tqdm_logging.py:145)
11:41:35.737 [I] Progress on: 2.47kit/3.00kit rate:7.1s/it remaining:1:02:17 elapsed:4:55:24 postfix:- (1728:tqdm_logging.py:145)
11:41:49.841 [I] Progress on: 2.47kit/3.00kit rate:7.1s/it remaining:1:02:03 elapsed:4:55:38 postfix:- (1728:tqdm_logging.py:145)
11:42:03.954 [I] Progress on: 2.47kit/3.00kit rate:7.1s/it remaining:1:01:50 elapsed:4:55:52 postfix:- (1728:tqdm_logging.py:145)
11:42:18.071 [I] Progress on: 2.48kit/3.00kit rate:7.1s/it remaining:1:01:37 elapsed:4:56:06 postfix:- (1728:tqdm_logging.py:145)
11:42:32.186 [I] Progress on: 2.48kit/3.00kit rate:7.1s/it remaining:1:01:23 elapsed:4:56:20 postfix:- (1728:tqdm_logging.py:145)
11:42:46.301 [I] Progress on: 2.48kit/3.00kit rate:7.1s/it remaining:1:01:09 elapsed:4:56:35 postfix:- (1728:tqdm_logging.py:145)
11:43:00.412 [I] Progress on: 2.48kit/3.00kit rate:7.1s/it remaining:1:00:55 elapsed:4:56:49 postfix:- (1728:tqdm_logging.py:145)
11:43:14.531 [I] Progress on: 2.48kit/3.00kit rate:7.1s/it remaining:1:00:41 elapsed:4:57:03 postfix:- (1728:tqdm_logging.py:145)
11:43:28.624 [I] Progress on: 2.49kit/3.00kit rate:7.1s/it remaining:1:00:24 elapsed:4:57:17 postfix:- (1728:tqdm_logging.py:145)
11:43:42.734 [I] Progress on: 2.49kit/3.00kit rate:7.1s/it remaining:1:00:11 elapsed:4:57:31 postfix:- (1728:tqdm_logging.py:145)
11:43:56.845 [I] Progress on: 2.49kit/3.00kit rate:7.1s/it remaining:59:57 elapsed:4:57:45 postfix:- (1728:tqdm_logging.py:145)
11:44:10.963 [I] Progress on: 2.49kit/3.00kit rate:7.1s/it remaining:59:45 elapsed:4:57:59 postfix:- (1728:tqdm_logging.py:145)
11:44:25.074 [I] Progress on: 2.49kit/3.00kit rate:7.1s/it remaining:59:30 elapsed:4:58:13 postfix:- (1728:tqdm_logging.py:145)
11:44:39.174 [I] Progress on: 2.50kit/3.00kit rate:7.1s/it remaining:59:14 elapsed:4:58:27 postfix:- (1728:tqdm_logging.py:145)
11:44:53.283 [I] Progress on: 2.50kit/3.00kit rate:7.1s/it remaining:59:01 elapsed:4:58:42 postfix:- (1728:tqdm_logging.py:145)
11:45:07.721 [I] Progress on: 2.50kit/3.00kit rate:7.2s/it remaining:59:36 elapsed:4:58:56 postfix:- (1728:tqdm_logging.py:145)


Step 2500: grad_norm=0.0428, loss=0.0043, param_norm=1804.1134
11:45:21.457 [I] Progress on: 2.50kit/3.00kit rate:7.2s/it remaining:59:36 elapsed:4:59:10 postfix:- (1728:tqdm_logging.py:145)
11:45:35.963 [I] Progress on: 2.50kit/3.00kit rate:6.8s/it remaining:56:05 elapsed:4:59:24 postfix:- (1728:tqdm_logging.py:145)
11:45:50.080 [I] Progress on: 2.51kit/3.00kit rate:6.9s/it remaining:57:00 elapsed:4:59:38 postfix:- (1728:tqdm_logging.py:145)
11:46:04.185 [I] Progress on: 2.51kit/3.00kit rate:7.0s/it remaining:57:18 elapsed:4:59:52 postfix:- (1728:tqdm_logging.py:145)
11:46:18.292 [I] Progress on: 2.51kit/3.00kit rate:7.0s/it remaining:57:20 elapsed:5:00:07 postfix:- (1728:tqdm_logging.py:145)
11:46:32.389 [I] Progress on: 2.51kit/3.00kit rate:7.0s/it remaining:57:13 elapsed:5:00:21 postfix:- (1728:tqdm_logging.py:145)
11:46:46.487 [I] Progress on: 2.51kit/3.00kit rate:7.0s/it remaining:57:02 elapsed:5:00:35 postfix:- (1728:tqdm_logging.py:145)
11:47:00.602 [I] Progress on: 2.52kit/3.00kit rate:7.0s/it remaining:56:51 elapsed:5:00:49 postfix:- (1728:tqdm_logging.py:145)
11:47:14.715 [I] Progress on: 2.52kit/3.00kit rate:7.1s/it remaining:56:38 elapsed:5:01:03 postfix:- (1728:tqdm_logging.py:145)
11:47:28.825 [I] Progress on: 2.52kit/3.00kit rate:7.1s/it remaining:56:25 elapsed:5:01:17 postfix:- (1728:tqdm_logging.py:145)
11:47:42.936 [I] Progress on: 2.52kit/3.00kit rate:7.1s/it remaining:56:12 elapsed:5:01:31 postfix:- (1728:tqdm_logging.py:145)
11:47:57.044 [I] Progress on: 2.52kit/3.00kit rate:7.1s/it remaining:55:57 elapsed:5:01:45 postfix:- (1728:tqdm_logging.py:145)
11:48:11.134 [I] Progress on: 2.53kit/3.00kit rate:7.0s/it remaining:55:41 elapsed:5:01:59 postfix:- (1728:tqdm_logging.py:145)
11:48:25.263 [I] Progress on: 2.53kit/3.00kit rate:7.1s/it remaining:55:31 elapsed:5:02:14 postfix:- (1728:tqdm_logging.py:145)
11:48:39.363 [I] Progress on: 2.53kit/3.00kit rate:7.1s/it remaining:55:15 elapsed:5:02:28 postfix:- (1728:tqdm_logging.py:145)
11:48:53.460 [I] Progress on: 2.53kit/3.00kit rate:7.1s/it remaining:54:59 elapsed:5:02:42 postfix:- (1728:tqdm_logging.py:145)
11:49:07.577 [I] Progress on: 2.53kit/3.00kit rate:7.1s/it remaining:54:47 elapsed:5:02:56 postfix:- (1728:tqdm_logging.py:145)
11:49:21.683 [I] Progress on: 2.54kit/3.00kit rate:7.1s/it remaining:54:33 elapsed:5:03:10 postfix:- (1728:tqdm_logging.py:145)
11:49:35.799 [I] Progress on: 2.54kit/3.00kit rate:7.1s/it remaining:54:20 elapsed:5:03:24 postfix:- (1728:tqdm_logging.py:145)
11:49:49.900 [I] Progress on: 2.54kit/3.00kit rate:7.1s/it remaining:54:04 elapsed:5:03:38 postfix:- (1728:tqdm_logging.py:145)
11:50:04.021 [I] Progress on: 2.54kit/3.00kit rate:7.1s/it remaining:53:52 elapsed:5:03:52 postfix:- (1728:tqdm_logging.py:145)
11:50:18.133 [I] Progress on: 2.54kit/3.00kit rate:7.1s/it remaining:53:37 elapsed:5:04:06 postfix:- (1728:tqdm_logging.py:145)
11:50:32.259 [I] Progress on: 2.55kit/3.00kit rate:7.1s/it remaining:53:24 elapsed:5:04:21 postfix:- (1728:tqdm_logging.py:145)
11:50:46.378 [I] Progress on: 2.55kit/3.00kit rate:7.1s/it remaining:53:11 elapsed:5:04:35 postfix:- (1728:tqdm_logging.py:145)
11:51:00.484 [I] Progress on: 2.55kit/3.00kit rate:7.1s/it remaining:52:55 elapsed:5:04:49 postfix:- (1728:tqdm_logging.py:145)


Step 2550: grad_norm=0.0416, loss=0.0042, param_norm=1804.1166
11:51:14.223 [I] Progress on: 2.55kit/3.00kit rate:7.1s/it remaining:52:55 elapsed:5:05:03 postfix:- (1728:tqdm_logging.py:145)
11:51:28.759 [I] Progress on: 2.55kit/3.00kit rate:6.8s/it remaining:50:17 elapsed:5:05:17 postfix:- (1728:tqdm_logging.py:145)
11:51:42.878 [I] Progress on: 2.56kit/3.00kit rate:6.9s/it remaining:51:10 elapsed:5:05:31 postfix:- (1728:tqdm_logging.py:145)
11:51:56.999 [I] Progress on: 2.56kit/3.00kit rate:7.0s/it remaining:51:29 elapsed:5:05:45 postfix:- (1728:tqdm_logging.py:145)
11:52:11.111 [I] Progress on: 2.56kit/3.00kit rate:7.0s/it remaining:51:30 elapsed:5:05:59 postfix:- (1728:tqdm_logging.py:145)
11:52:25.219 [I] Progress on: 2.56kit/3.00kit rate:7.0s/it remaining:51:22 elapsed:5:06:14 postfix:- (1728:tqdm_logging.py:145)
11:52:39.330 [I] Progress on: 2.56kit/3.00kit rate:7.0s/it remaining:51:12 elapsed:5:06:28 postfix:- (1728:tqdm_logging.py:145)
11:52:53.455 [I] Progress on: 2.57kit/3.00kit rate:7.1s/it remaining:51:01 elapsed:5:06:42 postfix:- (1728:tqdm_logging.py:145)
11:53:07.563 [I] Progress on: 2.57kit/3.00kit rate:7.1s/it remaining:50:47 elapsed:5:06:56 postfix:- (1728:tqdm_logging.py:145)
11:53:21.678 [I] Progress on: 2.57kit/3.00kit rate:7.1s/it remaining:50:33 elapsed:5:07:10 postfix:- (1728:tqdm_logging.py:145)
11:53:35.801 [I] Progress on: 2.57kit/3.00kit rate:7.1s/it remaining:50:21 elapsed:5:07:24 postfix:- (1728:tqdm_logging.py:145)
11:53:49.890 [I] Progress on: 2.57kit/3.00kit rate:7.1s/it remaining:50:04 elapsed:5:07:38 postfix:- (1728:tqdm_logging.py:145)
11:54:03.998 [I] Progress on: 2.58kit/3.00kit rate:7.1s/it remaining:49:50 elapsed:5:07:52 postfix:- (1728:tqdm_logging.py:145)
11:54:18.127 [I] Progress on: 2.58kit/3.00kit rate:7.1s/it remaining:49:38 elapsed:5:08:06 postfix:- (1728:tqdm_logging.py:145)
11:54:32.236 [I] Progress on: 2.58kit/3.00kit rate:7.1s/it remaining:49:23 elapsed:5:08:21 postfix:- (1728:tqdm_logging.py:145)
11:54:46.335 [I] Progress on: 2.58kit/3.00kit rate:7.1s/it remaining:49:07 elapsed:5:08:35 postfix:- (1728:tqdm_logging.py:145)
11:55:00.458 [I] Progress on: 2.58kit/3.00kit rate:7.1s/it remaining:48:55 elapsed:5:08:49 postfix:- (1728:tqdm_logging.py:145)
11:55:14.575 [I] Progress on: 2.59kit/3.00kit rate:7.1s/it remaining:48:41 elapsed:5:09:03 postfix:- (1728:tqdm_logging.py:145)
11:55:28.685 [I] Progress on: 2.59kit/3.00kit rate:7.1s/it remaining:48:27 elapsed:5:09:17 postfix:- (1728:tqdm_logging.py:145)
11:55:42.803 [I] Progress on: 2.59kit/3.00kit rate:7.1s/it remaining:48:13 elapsed:5:09:31 postfix:- (1728:tqdm_logging.py:145)
11:55:56.923 [I] Progress on: 2.59kit/3.00kit rate:7.1s/it remaining:48:00 elapsed:5:09:45 postfix:- (1728:tqdm_logging.py:145)
11:56:11.028 [I] Progress on: 2.59kit/3.00kit rate:7.1s/it remaining:47:44 elapsed:5:09:59 postfix:- (1728:tqdm_logging.py:145)
11:56:25.155 [I] Progress on: 2.60kit/3.00kit rate:7.1s/it remaining:47:32 elapsed:5:10:13 postfix:- (1728:tqdm_logging.py:145)
11:56:39.265 [I] Progress on: 2.60kit/3.00kit rate:7.1s/it remaining:47:17 elapsed:5:10:28 postfix:- (1728:tqdm_logging.py:145)
11:56:53.377 [I] Progress on: 2.60kit/3.00kit rate:7.1s/it remaining:47:02 elapsed:5:10:42 postfix:- (1728:tqdm_logging.py:145)


Step 2600: grad_norm=0.0440, loss=0.0046, param_norm=1804.1196
11:57:07.142 [I] Progress on: 2.60kit/3.00kit rate:7.1s/it remaining:47:02 elapsed:5:10:55 postfix:- (1728:tqdm_logging.py:145)
11:57:21.710 [I] Progress on: 2.60kit/3.00kit rate:6.8s/it remaining:44:43 elapsed:5:11:10 postfix:- (1728:tqdm_logging.py:145)
11:57:35.828 [I] Progress on: 2.61kit/3.00kit rate:6.9s/it remaining:45:27 elapsed:5:11:24 postfix:- (1728:tqdm_logging.py:145)
11:57:49.933 [I] Progress on: 2.61kit/3.00kit rate:7.0s/it remaining:45:39 elapsed:5:11:38 postfix:- (1728:tqdm_logging.py:145)
11:58:04.032 [I] Progress on: 2.61kit/3.00kit rate:7.0s/it remaining:45:37 elapsed:5:11:52 postfix:- (1728:tqdm_logging.py:145)
11:58:18.174 [I] Progress on: 2.61kit/3.00kit rate:7.0s/it remaining:45:34 elapsed:5:12:06 postfix:- (1728:tqdm_logging.py:145)
11:58:32.286 [I] Progress on: 2.61kit/3.00kit rate:7.1s/it remaining:45:21 elapsed:5:12:21 postfix:- (1728:tqdm_logging.py:145)
11:58:46.389 [I] Progress on: 2.62kit/3.00kit rate:7.1s/it remaining:45:08 elapsed:5:12:35 postfix:- (1728:tqdm_logging.py:145)
11:59:00.489 [I] Progress on: 2.62kit/3.00kit rate:7.1s/it remaining:44:53 elapsed:5:12:49 postfix:- (1728:tqdm_logging.py:145)
11:59:14.600 [I] Progress on: 2.62kit/3.00kit rate:7.1s/it remaining:44:40 elapsed:5:13:03 postfix:- (1728:tqdm_logging.py:145)
11:59:28.709 [I] Progress on: 2.62kit/3.00kit rate:7.1s/it remaining:44:26 elapsed:5:13:17 postfix:- (1728:tqdm_logging.py:145)
11:59:42.825 [I] Progress on: 2.62kit/3.00kit rate:7.1s/it remaining:44:12 elapsed:5:13:31 postfix:- (1728:tqdm_logging.py:145)
11:59:56.947 [I] Progress on: 2.63kit/3.00kit rate:7.1s/it remaining:43:59 elapsed:5:13:45 postfix:- (1728:tqdm_logging.py:145)
12:00:11.075 [I] Progress on: 2.63kit/3.00kit rate:7.1s/it remaining:43:47 elapsed:5:13:59 postfix:- (1728:tqdm_logging.py:145)
12:00:25.179 [I] Progress on: 2.63kit/3.00kit rate:7.1s/it remaining:43:31 elapsed:5:14:13 postfix:- (1728:tqdm_logging.py:145)
12:00:39.271 [I] Progress on: 2.63kit/3.00kit rate:7.1s/it remaining:43:14 elapsed:5:14:28 postfix:- (1728:tqdm_logging.py:145)
12:00:53.387 [I] Progress on: 2.63kit/3.00kit rate:7.1s/it remaining:43:01 elapsed:5:14:42 postfix:- (1728:tqdm_logging.py:145)
12:01:07.507 [I] Progress on: 2.64kit/3.00kit rate:7.1s/it remaining:42:49 elapsed:5:14:56 postfix:- (1728:tqdm_logging.py:145)
12:01:21.618 [I] Progress on: 2.64kit/3.00kit rate:7.1s/it remaining:42:34 elapsed:5:15:10 postfix:- (1728:tqdm_logging.py:145)
12:01:35.734 [I] Progress on: 2.64kit/3.00kit rate:7.1s/it remaining:42:20 elapsed:5:15:24 postfix:- (1728:tqdm_logging.py:145)
12:01:49.844 [I] Progress on: 2.64kit/3.00kit rate:7.1s/it remaining:42:06 elapsed:5:15:38 postfix:- (1728:tqdm_logging.py:145)
12:02:03.959 [I] Progress on: 2.64kit/3.00kit rate:7.1s/it remaining:41:52 elapsed:5:15:52 postfix:- (1728:tqdm_logging.py:145)
12:02:18.055 [I] Progress on: 2.65kit/3.00kit rate:7.1s/it remaining:41:36 elapsed:5:16:06 postfix:- (1728:tqdm_logging.py:145)
12:02:32.167 [I] Progress on: 2.65kit/3.00kit rate:7.1s/it remaining:41:22 elapsed:5:16:20 postfix:- (1728:tqdm_logging.py:145)
12:02:46.284 [I] Progress on: 2.65kit/3.00kit rate:7.1s/it remaining:41:09 elapsed:5:16:35 postfix:- (1728:tqdm_logging.py:145)


Step 2650: grad_norm=0.0409, loss=0.0045, param_norm=1804.1228
12:03:00.055 [I] Progress on: 2.65kit/3.00kit rate:7.1s/it remaining:41:09 elapsed:5:16:48 postfix:- (1728:tqdm_logging.py:145)
12:03:14.610 [I] Progress on: 2.65kit/3.00kit rate:6.8s/it remaining:39:03 elapsed:5:17:03 postfix:- (1728:tqdm_logging.py:145)
12:03:28.700 [I] Progress on: 2.66kit/3.00kit rate:6.9s/it remaining:39:37 elapsed:5:17:17 postfix:- (1728:tqdm_logging.py:145)
12:03:42.815 [I] Progress on: 2.66kit/3.00kit rate:7.0s/it remaining:39:49 elapsed:5:17:31 postfix:- (1728:tqdm_logging.py:145)
12:03:56.912 [I] Progress on: 2.66kit/3.00kit rate:7.0s/it remaining:39:46 elapsed:5:17:45 postfix:- (1728:tqdm_logging.py:145)
12:04:11.000 [I] Progress on: 2.66kit/3.00kit rate:7.0s/it remaining:39:36 elapsed:5:17:59 postfix:- (1728:tqdm_logging.py:145)
12:04:25.610 [I] Progress on: 2.66kit/3.00kit rate:7.2s/it remaining:40:17 elapsed:5:18:14 postfix:- (1728:tqdm_logging.py:145)
12:04:39.219 [I] Progress on: 2.67kit/3.00kit rate:7.0s/it remaining:39:05 elapsed:5:18:28 postfix:- (1728:tqdm_logging.py:145)
12:04:53.316 [I] Progress on: 2.67kit/3.00kit rate:7.0s/it remaining:38:57 elapsed:5:18:42 postfix:- (1728:tqdm_logging.py:145)
12:05:07.409 [I] Progress on: 2.67kit/3.00kit rate:7.0s/it remaining:38:45 elapsed:5:18:56 postfix:- (1728:tqdm_logging.py:145)
12:05:21.541 [I] Progress on: 2.67kit/3.00kit rate:7.1s/it remaining:38:35 elapsed:5:19:10 postfix:- (1728:tqdm_logging.py:145)
12:05:35.634 [I] Progress on: 2.67kit/3.00kit rate:7.1s/it remaining:38:19 elapsed:5:19:24 postfix:- (1728:tqdm_logging.py:145)
12:05:49.761 [I] Progress on: 2.68kit/3.00kit rate:7.1s/it remaining:38:06 elapsed:5:19:38 postfix:- (1728:tqdm_logging.py:145)
12:06:03.867 [I] Progress on: 2.68kit/3.00kit rate:7.1s/it remaining:37:51 elapsed:5:19:52 postfix:- (1728:tqdm_logging.py:145)
12:06:17.970 [I] Progress on: 2.68kit/3.00kit rate:7.1s/it remaining:37:37 elapsed:5:20:06 postfix:- (1728:tqdm_logging.py:145)
12:06:32.074 [I] Progress on: 2.68kit/3.00kit rate:7.1s/it remaining:37:22 elapsed:5:20:20 postfix:- (1728:tqdm_logging.py:145)
12:06:46.177 [I] Progress on: 2.68kit/3.00kit rate:7.1s/it remaining:37:08 elapsed:5:20:34 postfix:- (1728:tqdm_logging.py:145)
12:07:00.294 [I] Progress on: 2.69kit/3.00kit rate:7.1s/it remaining:36:55 elapsed:5:20:49 postfix:- (1728:tqdm_logging.py:145)
12:07:14.428 [I] Progress on: 2.69kit/3.00kit rate:7.1s/it remaining:36:42 elapsed:5:21:03 postfix:- (1728:tqdm_logging.py:145)
12:07:28.533 [I] Progress on: 2.69kit/3.00kit rate:7.1s/it remaining:36:27 elapsed:5:21:17 postfix:- (1728:tqdm_logging.py:145)
12:07:42.642 [I] Progress on: 2.69kit/3.00kit rate:7.1s/it remaining:36:13 elapsed:5:21:31 postfix:- (1728:tqdm_logging.py:145)
12:07:56.763 [I] Progress on: 2.69kit/3.00kit rate:7.1s/it remaining:35:59 elapsed:5:21:45 postfix:- (1728:tqdm_logging.py:145)
12:08:10.861 [I] Progress on: 2.70kit/3.00kit rate:7.1s/it remaining:35:44 elapsed:5:21:59 postfix:- (1728:tqdm_logging.py:145)
12:08:24.988 [I] Progress on: 2.70kit/3.00kit rate:7.1s/it remaining:35:31 elapsed:5:22:13 postfix:- (1728:tqdm_logging.py:145)
12:08:39.109 [I] Progress on: 2.70kit/3.00kit rate:7.1s/it remaining:35:17 elapsed:5:22:27 postfix:- (1728:tqdm_logging.py:145)


Step 2700: grad_norm=0.0426, loss=0.0042, param_norm=1804.1262
12:08:52.858 [I] Progress on: 2.70kit/3.00kit rate:7.1s/it remaining:35:17 elapsed:5:22:41 postfix:- (1728:tqdm_logging.py:145)
12:09:07.433 [I] Progress on: 2.70kit/3.00kit rate:6.8s/it remaining:33:25 elapsed:5:22:56 postfix:- (1728:tqdm_logging.py:145)
12:09:21.531 [I] Progress on: 2.71kit/3.00kit rate:6.9s/it remaining:33:53 elapsed:5:23:10 postfix:- (1728:tqdm_logging.py:145)
12:09:35.650 [I] Progress on: 2.71kit/3.00kit rate:7.0s/it remaining:34:00 elapsed:5:23:24 postfix:- (1728:tqdm_logging.py:145)
12:09:49.780 [I] Progress on: 2.71kit/3.00kit rate:7.0s/it remaining:33:58 elapsed:5:23:38 postfix:- (1728:tqdm_logging.py:145)
12:10:03.908 [I] Progress on: 2.71kit/3.00kit rate:7.0s/it remaining:33:49 elapsed:5:23:52 postfix:- (1728:tqdm_logging.py:145)
12:10:18.037 [I] Progress on: 2.71kit/3.00kit rate:7.1s/it remaining:33:37 elapsed:5:24:06 postfix:- (1728:tqdm_logging.py:145)
12:10:32.161 [I] Progress on: 2.72kit/3.00kit rate:7.1s/it remaining:33:24 elapsed:5:24:20 postfix:- (1728:tqdm_logging.py:145)
12:10:46.266 [I] Progress on: 2.72kit/3.00kit rate:7.1s/it remaining:33:09 elapsed:5:24:35 postfix:- (1728:tqdm_logging.py:145)
12:11:00.376 [I] Progress on: 2.72kit/3.00kit rate:7.1s/it remaining:32:55 elapsed:5:24:49 postfix:- (1728:tqdm_logging.py:145)
12:11:14.501 [I] Progress on: 2.72kit/3.00kit rate:7.1s/it remaining:32:42 elapsed:5:25:03 postfix:- (1728:tqdm_logging.py:145)
12:11:28.607 [I] Progress on: 2.72kit/3.00kit rate:7.1s/it remaining:32:27 elapsed:5:25:17 postfix:- (1728:tqdm_logging.py:145)
12:11:42.736 [I] Progress on: 2.73kit/3.00kit rate:7.1s/it remaining:32:14 elapsed:5:25:31 postfix:- (1728:tqdm_logging.py:145)
12:11:56.847 [I] Progress on: 2.73kit/3.00kit rate:7.1s/it remaining:31:59 elapsed:5:25:45 postfix:- (1728:tqdm_logging.py:145)
12:12:10.960 [I] Progress on: 2.73kit/3.00kit rate:7.1s/it remaining:31:45 elapsed:5:25:59 postfix:- (1728:tqdm_logging.py:145)
12:12:25.087 [I] Progress on: 2.73kit/3.00kit rate:7.1s/it remaining:31:32 elapsed:5:26:13 postfix:- (1728:tqdm_logging.py:145)
12:12:39.216 [I] Progress on: 2.73kit/3.00kit rate:7.1s/it remaining:31:18 elapsed:5:26:28 postfix:- (1728:tqdm_logging.py:145)
12:12:53.325 [I] Progress on: 2.74kit/3.00kit rate:7.1s/it remaining:31:03 elapsed:5:26:42 postfix:- (1728:tqdm_logging.py:145)
12:13:07.452 [I] Progress on: 2.74kit/3.00kit rate:7.1s/it remaining:30:48 elapsed:5:26:56 postfix:- (1728:tqdm_logging.py:145)
12:13:21.569 [I] Progress on: 2.74kit/3.00kit rate:7.1s/it remaining:30:33 elapsed:5:27:10 postfix:- (1728:tqdm_logging.py:145)
12:13:35.670 [I] Progress on: 2.74kit/3.00kit rate:7.1s/it remaining:30:18 elapsed:5:27:24 postfix:- (1728:tqdm_logging.py:145)
12:13:49.792 [I] Progress on: 2.74kit/3.00kit rate:7.1s/it remaining:30:06 elapsed:5:27:38 postfix:- (1728:tqdm_logging.py:145)
12:14:03.929 [I] Progress on: 2.75kit/3.00kit rate:7.1s/it remaining:29:53 elapsed:5:27:52 postfix:- (1728:tqdm_logging.py:145)
12:14:18.021 [I] Progress on: 2.75kit/3.00kit rate:7.1s/it remaining:29:37 elapsed:5:28:06 postfix:- (1728:tqdm_logging.py:145)
12:14:32.141 [I] Progress on: 2.75kit/3.00kit rate:7.1s/it remaining:29:24 elapsed:5:28:20 postfix:- (1728:tqdm_logging.py:145)


Step 2750: grad_norm=0.0424, loss=0.0041, param_norm=1804.1294
12:14:45.887 [I] Progress on: 2.75kit/3.00kit rate:7.1s/it remaining:29:24 elapsed:5:28:34 postfix:- (1728:tqdm_logging.py:145)
12:15:00.411 [I] Progress on: 2.75kit/3.00kit rate:6.8s/it remaining:27:44 elapsed:5:28:49 postfix:- (1728:tqdm_logging.py:145)
12:15:14.551 [I] Progress on: 2.76kit/3.00kit rate:6.9s/it remaining:28:08 elapsed:5:29:03 postfix:- (1728:tqdm_logging.py:145)
12:15:28.660 [I] Progress on: 2.76kit/3.00kit rate:7.0s/it remaining:28:11 elapsed:5:29:17 postfix:- (1728:tqdm_logging.py:145)
12:15:42.795 [I] Progress on: 2.76kit/3.00kit rate:7.0s/it remaining:28:06 elapsed:5:29:31 postfix:- (1728:tqdm_logging.py:145)
12:15:56.931 [I] Progress on: 2.76kit/3.00kit rate:7.0s/it remaining:27:57 elapsed:5:29:45 postfix:- (1728:tqdm_logging.py:145)
12:16:11.052 [I] Progress on: 2.76kit/3.00kit rate:7.1s/it remaining:27:44 elapsed:5:29:59 postfix:- (1728:tqdm_logging.py:145)
12:16:25.178 [I] Progress on: 2.77kit/3.00kit rate:7.1s/it remaining:27:31 elapsed:5:30:13 postfix:- (1728:tqdm_logging.py:145)
12:16:39.298 [I] Progress on: 2.77kit/3.00kit rate:7.1s/it remaining:27:17 elapsed:5:30:28 postfix:- (1728:tqdm_logging.py:145)
12:16:53.418 [I] Progress on: 2.77kit/3.00kit rate:7.1s/it remaining:27:03 elapsed:5:30:42 postfix:- (1728:tqdm_logging.py:145)
12:17:07.521 [I] Progress on: 2.77kit/3.00kit rate:7.1s/it remaining:26:48 elapsed:5:30:56 postfix:- (1728:tqdm_logging.py:145)
12:17:21.651 [I] Progress on: 2.77kit/3.00kit rate:7.1s/it remaining:26:35 elapsed:5:31:10 postfix:- (1728:tqdm_logging.py:145)
12:17:35.773 [I] Progress on: 2.78kit/3.00kit rate:7.1s/it remaining:26:21 elapsed:5:31:24 postfix:- (1728:tqdm_logging.py:145)
12:17:49.899 [I] Progress on: 2.78kit/3.00kit rate:7.1s/it remaining:26:07 elapsed:5:31:38 postfix:- (1728:tqdm_logging.py:145)
12:18:04.027 [I] Progress on: 2.78kit/3.00kit rate:7.1s/it remaining:25:53 elapsed:5:31:52 postfix:- (1728:tqdm_logging.py:145)
12:18:18.146 [I] Progress on: 2.78kit/3.00kit rate:7.1s/it remaining:25:39 elapsed:5:32:06 postfix:- (1728:tqdm_logging.py:145)
12:18:32.255 [I] Progress on: 2.78kit/3.00kit rate:7.1s/it remaining:25:24 elapsed:5:32:21 postfix:- (1728:tqdm_logging.py:145)
12:18:46.387 [I] Progress on: 2.79kit/3.00kit rate:7.1s/it remaining:25:10 elapsed:5:32:35 postfix:- (1728:tqdm_logging.py:145)
12:19:00.495 [I] Progress on: 2.79kit/3.00kit rate:7.1s/it remaining:24:56 elapsed:5:32:49 postfix:- (1728:tqdm_logging.py:145)
12:19:14.607 [I] Progress on: 2.79kit/3.00kit rate:7.1s/it remaining:24:41 elapsed:5:33:03 postfix:- (1728:tqdm_logging.py:145)
12:19:28.783 [I] Progress on: 2.79kit/3.00kit rate:7.1s/it remaining:24:31 elapsed:5:33:17 postfix:- (1728:tqdm_logging.py:145)
12:19:42.917 [I] Progress on: 2.79kit/3.00kit rate:7.1s/it remaining:24:17 elapsed:5:33:31 postfix:- (1728:tqdm_logging.py:145)
12:19:56.983 [I] Progress on: 2.80kit/3.00kit rate:7.1s/it remaining:23:59 elapsed:5:33:45 postfix:- (1728:tqdm_logging.py:145)
12:20:11.106 [I] Progress on: 2.80kit/3.00kit rate:7.1s/it remaining:23:45 elapsed:5:33:59 postfix:- (1728:tqdm_logging.py:145)
12:20:25.239 [I] Progress on: 2.80kit/3.00kit rate:7.1s/it remaining:23:31 elapsed:5:34:14 postfix:- (1728:tqdm_logging.py:145)


Step 2800: grad_norm=0.0404, loss=0.0038, param_norm=1804.1315
12:20:38.988 [I] Progress on: 2.80kit/3.00kit rate:7.1s/it remaining:23:31 elapsed:5:34:27 postfix:- (1728:tqdm_logging.py:145)
12:20:53.609 [I] Progress on: 2.80kit/3.00kit rate:6.8s/it remaining:22:08 elapsed:5:34:42 postfix:- (1728:tqdm_logging.py:145)
12:21:07.712 [I] Progress on: 2.81kit/3.00kit rate:6.9s/it remaining:22:22 elapsed:5:34:56 postfix:- (1728:tqdm_logging.py:145)
12:21:21.840 [I] Progress on: 2.81kit/3.00kit rate:7.0s/it remaining:22:22 elapsed:5:35:10 postfix:- (1728:tqdm_logging.py:145)
12:21:35.955 [I] Progress on: 2.81kit/3.00kit rate:7.0s/it remaining:22:14 elapsed:5:35:24 postfix:- (1728:tqdm_logging.py:145)
12:21:50.058 [I] Progress on: 2.81kit/3.00kit rate:7.0s/it remaining:22:03 elapsed:5:35:38 postfix:- (1728:tqdm_logging.py:145)
12:22:04.195 [I] Progress on: 2.81kit/3.00kit rate:7.1s/it remaining:21:52 elapsed:5:35:52 postfix:- (1728:tqdm_logging.py:145)
12:22:18.308 [I] Progress on: 2.82kit/3.00kit rate:7.1s/it remaining:21:38 elapsed:5:36:07 postfix:- (1728:tqdm_logging.py:145)
12:22:32.440 [I] Progress on: 2.82kit/3.00kit rate:7.1s/it remaining:21:25 elapsed:5:36:21 postfix:- (1728:tqdm_logging.py:145)
12:22:46.573 [I] Progress on: 2.82kit/3.00kit rate:7.1s/it remaining:21:11 elapsed:5:36:35 postfix:- (1728:tqdm_logging.py:145)
12:23:00.696 [I] Progress on: 2.82kit/3.00kit rate:7.1s/it remaining:20:56 elapsed:5:36:49 postfix:- (1728:tqdm_logging.py:145)
12:23:14.816 [I] Progress on: 2.82kit/3.00kit rate:7.1s/it remaining:20:42 elapsed:5:37:03 postfix:- (1728:tqdm_logging.py:145)
12:23:28.926 [I] Progress on: 2.83kit/3.00kit rate:7.1s/it remaining:20:28 elapsed:5:37:17 postfix:- (1728:tqdm_logging.py:145)
12:23:43.054 [I] Progress on: 2.83kit/3.00kit rate:7.1s/it remaining:20:14 elapsed:5:37:31 postfix:- (1728:tqdm_logging.py:145)
12:23:57.183 [I] Progress on: 2.83kit/3.00kit rate:7.1s/it remaining:20:00 elapsed:5:37:45 postfix:- (1728:tqdm_logging.py:145)
12:24:11.299 [I] Progress on: 2.83kit/3.00kit rate:7.1s/it remaining:19:46 elapsed:5:38:00 postfix:- (1728:tqdm_logging.py:145)
12:24:25.408 [I] Progress on: 2.83kit/3.00kit rate:7.1s/it remaining:19:31 elapsed:5:38:14 postfix:- (1728:tqdm_logging.py:145)
12:24:39.535 [I] Progress on: 2.84kit/3.00kit rate:7.1s/it remaining:19:17 elapsed:5:38:28 postfix:- (1728:tqdm_logging.py:145)
12:24:53.646 [I] Progress on: 2.84kit/3.00kit rate:7.1s/it remaining:19:03 elapsed:5:38:42 postfix:- (1728:tqdm_logging.py:145)
12:25:07.765 [I] Progress on: 2.84kit/3.00kit rate:7.1s/it remaining:18:49 elapsed:5:38:56 postfix:- (1728:tqdm_logging.py:145)
12:25:21.878 [I] Progress on: 2.84kit/3.00kit rate:7.1s/it remaining:18:35 elapsed:5:39:10 postfix:- (1728:tqdm_logging.py:145)
12:25:36.002 [I] Progress on: 2.84kit/3.00kit rate:7.1s/it remaining:18:21 elapsed:5:39:24 postfix:- (1728:tqdm_logging.py:145)
12:25:50.132 [I] Progress on: 2.85kit/3.00kit rate:7.1s/it remaining:18:07 elapsed:5:39:38 postfix:- (1728:tqdm_logging.py:145)
12:26:04.263 [I] Progress on: 2.85kit/3.00kit rate:7.1s/it remaining:17:53 elapsed:5:39:53 postfix:- (1728:tqdm_logging.py:145)
12:26:18.372 [I] Progress on: 2.85kit/3.00kit rate:7.1s/it remaining:17:38 elapsed:5:40:07 postfix:- (1728:tqdm_logging.py:145)


Step 2850: grad_norm=0.0450, loss=0.0045, param_norm=1804.1337
12:26:32.158 [I] Progress on: 2.85kit/3.00kit rate:7.1s/it remaining:17:38 elapsed:5:40:20 postfix:- (1728:tqdm_logging.py:145)
12:26:46.694 [I] Progress on: 2.85kit/3.00kit rate:6.8s/it remaining:16:28 elapsed:5:40:35 postfix:- (1728:tqdm_logging.py:145)
12:27:00.823 [I] Progress on: 2.86kit/3.00kit rate:6.9s/it remaining:16:36 elapsed:5:40:49 postfix:- (1728:tqdm_logging.py:145)
12:27:14.942 [I] Progress on: 2.86kit/3.00kit rate:7.0s/it remaining:16:32 elapsed:5:41:03 postfix:- (1728:tqdm_logging.py:145)
12:27:29.068 [I] Progress on: 2.86kit/3.00kit rate:7.0s/it remaining:16:24 elapsed:5:41:17 postfix:- (1728:tqdm_logging.py:145)
12:27:43.187 [I] Progress on: 2.86kit/3.00kit rate:7.0s/it remaining:16:11 elapsed:5:41:31 postfix:- (1728:tqdm_logging.py:145)
12:27:57.305 [I] Progress on: 2.86kit/3.00kit rate:7.1s/it remaining:15:58 elapsed:5:41:46 postfix:- (1728:tqdm_logging.py:145)
12:28:11.423 [I] Progress on: 2.87kit/3.00kit rate:7.1s/it remaining:15:45 elapsed:5:42:00 postfix:- (1728:tqdm_logging.py:145)
12:28:25.546 [I] Progress on: 2.87kit/3.00kit rate:7.1s/it remaining:15:31 elapsed:5:42:14 postfix:- (1728:tqdm_logging.py:145)
12:28:39.643 [I] Progress on: 2.87kit/3.00kit rate:7.1s/it remaining:15:16 elapsed:5:42:28 postfix:- (1728:tqdm_logging.py:145)
12:28:53.782 [I] Progress on: 2.87kit/3.00kit rate:7.1s/it remaining:15:03 elapsed:5:42:42 postfix:- (1728:tqdm_logging.py:145)
12:29:07.909 [I] Progress on: 2.87kit/3.00kit rate:7.1s/it remaining:14:49 elapsed:5:42:56 postfix:- (1728:tqdm_logging.py:145)
12:29:22.040 [I] Progress on: 2.88kit/3.00kit rate:7.1s/it remaining:14:35 elapsed:5:43:10 postfix:- (1728:tqdm_logging.py:145)
12:29:36.160 [I] Progress on: 2.88kit/3.00kit rate:7.1s/it remaining:14:21 elapsed:5:43:24 postfix:- (1728:tqdm_logging.py:145)
12:29:50.269 [I] Progress on: 2.88kit/3.00kit rate:7.1s/it remaining:14:07 elapsed:5:43:39 postfix:- (1728:tqdm_logging.py:145)
12:30:04.382 [I] Progress on: 2.88kit/3.00kit rate:7.1s/it remaining:13:52 elapsed:5:43:53 postfix:- (1728:tqdm_logging.py:145)
12:30:18.514 [I] Progress on: 2.88kit/3.00kit rate:7.1s/it remaining:13:39 elapsed:5:44:07 postfix:- (1728:tqdm_logging.py:145)
12:30:33.186 [I] Progress on: 2.89kit/3.00kit rate:7.2s/it remaining:13:43 elapsed:5:44:21 postfix:- (1728:tqdm_logging.py:145)
12:30:46.778 [I] Progress on: 2.89kit/3.00kit rate:7.0s/it remaining:13:07 elapsed:5:44:35 postfix:- (1728:tqdm_logging.py:145)
12:31:00.910 [I] Progress on: 2.89kit/3.00kit rate:7.1s/it remaining:12:56 elapsed:5:44:49 postfix:- (1728:tqdm_logging.py:145)
12:31:15.024 [I] Progress on: 2.89kit/3.00kit rate:7.1s/it remaining:12:42 elapsed:5:45:03 postfix:- (1728:tqdm_logging.py:145)
12:31:29.129 [I] Progress on: 2.89kit/3.00kit rate:7.1s/it remaining:12:28 elapsed:5:45:17 postfix:- (1728:tqdm_logging.py:145)
12:31:43.252 [I] Progress on: 2.90kit/3.00kit rate:7.1s/it remaining:12:14 elapsed:5:45:32 postfix:- (1728:tqdm_logging.py:145)
12:31:57.363 [I] Progress on: 2.90kit/3.00kit rate:7.1s/it remaining:12:00 elapsed:5:45:46 postfix:- (1728:tqdm_logging.py:145)
12:32:11.470 [I] Progress on: 2.90kit/3.00kit rate:7.1s/it remaining:11:45 elapsed:5:46:00 postfix:- (1728:tqdm_logging.py:145)


Step 2900: grad_norm=0.0406, loss=0.0037, param_norm=1804.1359
12:32:25.228 [I] Progress on: 2.90kit/3.00kit rate:7.1s/it remaining:11:45 elapsed:5:46:14 postfix:- (1728:tqdm_logging.py:145)
12:32:39.778 [I] Progress on: 2.90kit/3.00kit rate:6.8s/it remaining:10:50 elapsed:5:46:28 postfix:- (1728:tqdm_logging.py:145)
12:32:53.888 [I] Progress on: 2.91kit/3.00kit rate:6.9s/it remaining:10:50 elapsed:5:46:42 postfix:- (1728:tqdm_logging.py:145)
12:33:08.008 [I] Progress on: 2.91kit/3.00kit rate:7.0s/it remaining:10:43 elapsed:5:46:56 postfix:- (1728:tqdm_logging.py:145)
12:33:22.131 [I] Progress on: 2.91kit/3.00kit rate:7.0s/it remaining:10:32 elapsed:5:47:10 postfix:- (1728:tqdm_logging.py:145)
12:33:36.229 [I] Progress on: 2.91kit/3.00kit rate:7.0s/it remaining:10:19 elapsed:5:47:25 postfix:- (1728:tqdm_logging.py:145)
12:33:50.346 [I] Progress on: 2.91kit/3.00kit rate:7.0s/it remaining:10:06 elapsed:5:47:39 postfix:- (1728:tqdm_logging.py:145)
12:34:04.471 [I] Progress on: 2.92kit/3.00kit rate:7.1s/it remaining:09:52 elapsed:5:47:53 postfix:- (1728:tqdm_logging.py:145)
12:34:18.573 [I] Progress on: 2.92kit/3.00kit rate:7.1s/it remaining:09:38 elapsed:5:48:07 postfix:- (1728:tqdm_logging.py:145)
12:34:32.708 [I] Progress on: 2.92kit/3.00kit rate:7.1s/it remaining:09:24 elapsed:5:48:21 postfix:- (1728:tqdm_logging.py:145)
12:34:46.845 [I] Progress on: 2.92kit/3.00kit rate:7.1s/it remaining:09:11 elapsed:5:48:35 postfix:- (1728:tqdm_logging.py:145)
12:35:00.963 [I] Progress on: 2.92kit/3.00kit rate:7.1s/it remaining:08:56 elapsed:5:48:49 postfix:- (1728:tqdm_logging.py:145)
12:35:15.083 [I] Progress on: 2.93kit/3.00kit rate:7.1s/it remaining:08:42 elapsed:5:49:03 postfix:- (1728:tqdm_logging.py:145)
12:35:29.196 [I] Progress on: 2.93kit/3.00kit rate:7.1s/it remaining:08:28 elapsed:5:49:17 postfix:- (1728:tqdm_logging.py:145)
12:35:43.313 [I] Progress on: 2.93kit/3.00kit rate:7.1s/it remaining:08:14 elapsed:5:49:32 postfix:- (1728:tqdm_logging.py:145)
12:35:57.436 [I] Progress on: 2.93kit/3.00kit rate:7.1s/it remaining:08:00 elapsed:5:49:46 postfix:- (1728:tqdm_logging.py:145)
12:36:11.553 [I] Progress on: 2.93kit/3.00kit rate:7.1s/it remaining:07:45 elapsed:5:50:00 postfix:- (1728:tqdm_logging.py:145)
12:36:25.665 [I] Progress on: 2.94kit/3.00kit rate:7.1s/it remaining:07:31 elapsed:5:50:14 postfix:- (1728:tqdm_logging.py:145)
12:36:39.781 [I] Progress on: 2.94kit/3.00kit rate:7.1s/it remaining:07:17 elapsed:5:50:28 postfix:- (1728:tqdm_logging.py:145)
12:36:53.901 [I] Progress on: 2.94kit/3.00kit rate:7.1s/it remaining:07:03 elapsed:5:50:42 postfix:- (1728:tqdm_logging.py:145)
12:37:08.001 [I] Progress on: 2.94kit/3.00kit rate:7.1s/it remaining:06:49 elapsed:5:50:56 postfix:- (1728:tqdm_logging.py:145)
12:37:22.126 [I] Progress on: 2.94kit/3.00kit rate:7.1s/it remaining:06:35 elapsed:5:51:10 postfix:- (1728:tqdm_logging.py:145)
12:37:36.322 [I] Progress on: 2.95kit/3.00kit rate:7.1s/it remaining:06:22 elapsed:5:51:25 postfix:- (1728:tqdm_logging.py:145)
12:37:50.371 [I] Progress on: 2.95kit/3.00kit rate:7.1s/it remaining:06:06 elapsed:5:51:39 postfix:- (1728:tqdm_logging.py:145)
12:38:04.494 [I] Progress on: 2.95kit/3.00kit rate:7.1s/it remaining:05:52 elapsed:5:51:53 postfix:- (1728:tqdm_logging.py:145)


Step 2950: grad_norm=0.0384, loss=0.0036, param_norm=1804.1384
12:38:18.248 [I] Progress on: 2.95kit/3.00kit rate:7.1s/it remaining:05:52 elapsed:5:52:07 postfix:- (1728:tqdm_logging.py:145)
12:38:32.799 [I] Progress on: 2.95kit/3.00kit rate:6.8s/it remaining:05:11 elapsed:5:52:21 postfix:- (1728:tqdm_logging.py:145)
12:38:46.907 [I] Progress on: 2.96kit/3.00kit rate:6.9s/it remaining:05:04 elapsed:5:52:35 postfix:- (1728:tqdm_logging.py:145)
12:39:01.041 [I] Progress on: 2.96kit/3.00kit rate:7.0s/it remaining:04:53 elapsed:5:52:49 postfix:- (1728:tqdm_logging.py:145)
12:39:15.198 [I] Progress on: 2.96kit/3.00kit rate:7.0s/it remaining:04:41 elapsed:5:53:03 postfix:- (1728:tqdm_logging.py:145)
12:39:29.302 [I] Progress on: 2.96kit/3.00kit rate:7.0s/it remaining:04:27 elapsed:5:53:18 postfix:- (1728:tqdm_logging.py:145)
12:39:43.498 [I] Progress on: 2.96kit/3.00kit rate:7.1s/it remaining:04:14 elapsed:5:53:32 postfix:- (1728:tqdm_logging.py:145)
12:39:57.607 [I] Progress on: 2.97kit/3.00kit rate:7.1s/it remaining:04:00 elapsed:5:53:46 postfix:- (1728:tqdm_logging.py:145)
12:40:11.688 [I] Progress on: 2.97kit/3.00kit rate:7.1s/it remaining:03:45 elapsed:5:54:00 postfix:- (1728:tqdm_logging.py:145)
12:40:25.794 [I] Progress on: 2.97kit/3.00kit rate:7.1s/it remaining:03:31 elapsed:5:54:14 postfix:- (1728:tqdm_logging.py:145)
12:40:39.906 [I] Progress on: 2.97kit/3.00kit rate:7.1s/it remaining:03:17 elapsed:5:54:28 postfix:- (1728:tqdm_logging.py:145)
12:40:54.025 [I] Progress on: 2.97kit/3.00kit rate:7.1s/it remaining:03:03 elapsed:5:54:42 postfix:- (1728:tqdm_logging.py:145)
12:41:08.137 [I] Progress on: 2.98kit/3.00kit rate:7.1s/it remaining:02:49 elapsed:5:54:56 postfix:- (1728:tqdm_logging.py:145)
12:41:22.261 [I] Progress on: 2.98kit/3.00kit rate:7.1s/it remaining:02:35 elapsed:5:55:11 postfix:- (1728:tqdm_logging.py:145)
12:41:36.360 [I] Progress on: 2.98kit/3.00kit rate:7.1s/it remaining:02:21 elapsed:5:55:25 postfix:- (1728:tqdm_logging.py:145)
12:41:50.479 [I] Progress on: 2.98kit/3.00kit rate:7.1s/it remaining:02:07 elapsed:5:55:39 postfix:- (1728:tqdm_logging.py:145)
12:42:04.597 [I] Progress on: 2.98kit/3.00kit rate:7.1s/it remaining:01:52 elapsed:5:55:53 postfix:- (1728:tqdm_logging.py:145)
12:42:18.722 [I] Progress on: 2.99kit/3.00kit rate:7.1s/it remaining:01:38 elapsed:5:56:07 postfix:- (1728:tqdm_logging.py:145)
12:42:32.841 [I] Progress on: 2.99kit/3.00kit rate:7.1s/it remaining:01:24 elapsed:5:56:21 postfix:- (1728:tqdm_logging.py:145)
12:42:46.947 [I] Progress on: 2.99kit/3.00kit rate:7.1s/it remaining:01:10 elapsed:5:56:35 postfix:- (1728:tqdm_logging.py:145)
12:43:01.085 [I] Progress on: 2.99kit/3.00kit rate:7.1s/it remaining:00:56 elapsed:5:56:49 postfix:- (1728:tqdm_logging.py:145)
12:43:15.213 [I] Progress on: 2.99kit/3.00kit rate:7.1s/it remaining:00:42 elapsed:5:57:04 postfix:- (1728:tqdm_logging.py:145)
12:43:29.334 [I] Progress on: 3.00kit/3.00kit rate:7.1s/it remaining:00:28 elapsed:5:57:18 postfix:- (1728:tqdm_logging.py:145)
12:43:43.461 [I] Progress on: 3.00kit/3.00kit rate:7.1s/it remaining:00:14 elapsed:5:57:32 postfix:- (1728:tqdm_logging.py:145)
12:43:57.591 [I] [process=0][thread=MainThread][wait_until_finished] No Save Finalize thread to wait for. Returning. (1728:checkpoint_manager.py:1987)
12:43:57.592 [I] [process=0] Saving checkpoint at step 2999                                       (1728:checkpoint_manager.py:1408)
12:43:57.593 [I] [process=0] Started async saving checkpoint to /workspace/openpi/checkpoints/pi05_mecharm_deep_even10_lora/deep_even10_v1/2999. (1728:async_checkpointer.py:439)
12:43:57.595 [I] Creating tmp directory /workspace/openpi/checkpoints/pi05_mecharm_deep_even10_lora/deep_even10_v1/2999.orbax-checkpoint-tmp-8 (1728:atomicity.py:144)
12:43:57.608 [I] Creating tmp directory /workspace/openpi/checkpoints/pi05_mecharm_deep_even10_lora/deep_even10_v1/2999.orbax-checkpoint-tmp-8/assets.orbax-checkpoint-tmp-9 (1728:atomicity.py:144)
12:43:57.609 [I] Creating tmp directory /workspace/openpi/checkpoints/pi05_mecharm_deep_even10_lora/deep_even10_v1/2999.orbax-checkpoint-tmp-8/params.orbax-checkpoint-tmp-10 (1728:atomicity.py:144)
12:43:57.609 [I] Creating tmp directory /workspace/openpi/checkpoints/pi05_mecharm_deep_even10_lora/deep_even10_v1/2999.orbax-checkpoint-tmp-8/train_state.orbax-checkpoint-tmp-11 (1728:atomicity.py:144)
12:43:57.620 [I] Transferring arrays to host memory with options: use_replica_parallel=True, enable_pinned_host_transfer=False (1728:replica_slices.py:341)
12:44:08.151 [I] Transferring arrays to host memory with options: use_replica_parallel=True, enable_pinned_host_transfer=False (1728:replica_slices.py:341)
12:44:08.154 [I] Array name: 'params.PaliGemma.llm.layers.mlp.gating_einsum.value', global shape: (18, 2, 2048, 16384), write shape: (18, 2, 2048, 16384), chosen chunk shape: (18, 2, 2048, 8192) (1728:tensorstore_utils.py:408)
12:44:08.222 [I] [process=0][thread=array_type_handler] Wrote 71 array_metadata.ArrayMetadata to /workspace/openpi/checkpoints/pi05_mecharm_deep_even10_lora/deep_even10_v1/2999.orbax-checkpoint-tmp-8/params.orbax-checkpoint-tmp-10/array_metadatas/process_0 (1728:array_metadata_store.py:198)
12:44:14.500 [I] [process=0] /jax/checkpoint/write/blocking_bytes_per_sec: 436.7 MiB/s (total bytes: 7.2 GiB) (time elapsed: 16 seconds) (per-host) (1728:base_pytree_checkpoint_handler.py:114)
12:44:14.507 [I] [process=0] /jax/checkpoint/write/blocking_bytes_per_sec: 210.8 MiB/s (total bytes: 3.5 GiB) (time elapsed: 16 seconds) (per-host) (1728:base_pytree_checkpoint_handler.py:114)
12:44:14.515 [I] [process=0][thread=async_save] Background save thread started.                   (1728:async_checkpointer.py:77)
12:44:14.516 [I] Finished blocking save in 16.92 seconds. Continuing to save asynchronously to /workspace/openpi/checkpoints/pi05_mecharm_deep_even10_lora/deep_even10_v1/2999. (1728:async_checkpointer.py:548)
12:44:14.522 [I] Preserving Checkpoint[step=1000 | time=2026-11-09 08:48:44.411183+00:00]: (Reason: on keep_period=1000). (1728:checkpoint_manager.py:1949)
12:44:14.522 [I] Preserving Checkpoint[step=2000 | time=2026-11-09 10:46:32.891133+00:00]: (Reason: on keep_period=1000). (1728:checkpoint_manager.py:1949)
12:44:14.523 [I] [process=0][thread=MainThread][step=2999] Starting CheckpointManager Save Finalize thread=save_finalize (1728:checkpoint_manager.py:1453)
12:44:14.524 [I] [process=0][thread=save_finalize] Waiting for background save thread=async_save. (1728:async_checkpointer.py:258)
12:44:14.524 [I] {'step': 2999, 'event_type': 'save', 'directory': '/workspace/openpi/checkpoints/pi05_mecharm_deep_even10_lora/deep_even10_v1', 'reached_preemption': False, 'preemption_received_at': None, 'synchronous': False, 'wait_for_prev_start_time': 1794228237.5919025, 'wait_for_prev_duration_secs': 0.0002002716064453125, 'checkpointer_blocking_start_time': 1794228237.5925238, 'checkpointer_blocking_duration_secs': 16.929755926132202, 'get_old_steps_start_time': 1794228254.5223072, 'get_old_steps_duration_secs': 0.0012187957763671875, 'checkpoint_manager_blocking_start_time': 1794228237.5915833, 'checkpoint_manager_blocking_duration_secs': 16.933098077774048} (1728:standard_logger.py:34)
12:44:14.525 [I] Progress on: 3.00kit/3.00kit rate:12.1s/it remaining:00:00 elapsed:5:58:03 postfix:- (1728:tqdm_logging.py:145)
12:44:14.526 [I] Progress on: 3.00kit/3.00kit rate:12.1s/it remaining:00:00 elapsed:5:58:03 postfix:- (1728:tqdm_logging.py:145)
12:44:14.528 [I] Waiting for checkpoint manager to finish                                         (1728:train.py:275)
12:44:14.528 [I] [process=0][thread=MainThread][step=2999][wait_until_finished] Waiting for Save Finalize thread (save_finalize) to complete. (1728:checkpoint_manager.py:1998)
12:44:14.532 [I] [process=0] /jax/checkpoint/write/bytes_per_sec: 435.9 MiB/s (total bytes: 7.2 GiB) (time elapsed: 16 seconds) (per-host) (1728:base_pytree_checkpoint_handler.py:114)
12:44:14.616 [I] [process=0][thread=array_type_handler] Wrote 105 array_metadata.ArrayMetadata to /workspace/openpi/checkpoints/pi05_mecharm_deep_even10_lora/deep_even10_v1/2999.orbax-checkpoint-tmp-8/train_state.orbax-checkpoint-tmp-11/array_metadatas/process_0 (1728:array_metadata_store.py:198)
12:44:17.981 [I] [process=0] /jax/checkpoint/write/bytes_per_sec: 174.8 MiB/s (total bytes: 3.5 GiB) (time elapsed: 20 seconds) (per-host) (1728:base_pytree_checkpoint_handler.py:114)
12:44:17.981 [I] [process=0][thread=async_save] 5 Handler Commit operations completed.            (1728:async_checkpointer.py:87)
12:44:17.982 [I] Renaming /workspace/openpi/checkpoints/pi05_mecharm_deep_even10_lora/deep_even10_v1/2999.orbax-checkpoint-tmp-8/assets.orbax-checkpoint-tmp-9 to /workspace/openpi/checkpoints/pi05_mecharm_deep_even10_lora/deep_even10_v1/2999.orbax-checkpoint-tmp-8/assets (1728:atomicity.py:289)
12:44:17.988 [I] [process=0][thread=async_save] Skipped cross-host ArrayMetadata validation because only one process is found: process_index=0. (1728:array_metadata_store.py:362)
12:44:18.001 [I] Renaming /workspace/openpi/checkpoints/pi05_mecharm_deep_even10_lora/deep_even10_v1/2999.orbax-checkpoint-tmp-8/train_state.orbax-checkpoint-tmp-11 to /workspace/openpi/checkpoints/pi05_mecharm_deep_even10_lora/deep_even10_v1/2999.orbax-checkpoint-tmp-8/train_state (1728:atomicity.py:289)
12:44:18.010 [I] [process=0][thread=async_save] Skipped cross-host ArrayMetadata validation because only one process is found: process_index=0. (1728:array_metadata_store.py:362)
12:44:18.021 [I] Renaming /workspace/openpi/checkpoints/pi05_mecharm_deep_even10_lora/deep_even10_v1/2999.orbax-checkpoint-tmp-8/params.orbax-checkpoint-tmp-10 to /workspace/openpi/checkpoints/pi05_mecharm_deep_even10_lora/deep_even10_v1/2999.orbax-checkpoint-tmp-8/params (1728:atomicity.py:289)
12:44:18.022 [I] Renaming /workspace/openpi/checkpoints/pi05_mecharm_deep_even10_lora/deep_even10_v1/2999.orbax-checkpoint-tmp-8 to /workspace/openpi/checkpoints/pi05_mecharm_deep_even10_lora/deep_even10_v1/2999 (1728:atomicity.py:289)
12:44:18.024 [I] [process=0][thread=async_save] Finished saving checkpoint (finalized tmp dir) to `/workspace/openpi/checkpoints/pi05_mecharm_deep_even10_lora/deep_even10_v1/2999`. (1728:atomicity.py:573)
12:44:18.025 [I] Finished asynchronous save (blocking + background) in 20.43 seconds to /workspace/openpi/checkpoints/pi05_mecharm_deep_even10_lora/deep_even10_v1/2999 (1728:async_checkpointer.py:412)
12:44:18.025 [I] [process=0][thread=async_save] Background save thread done.                      (1728:async_checkpointer.py:143)
12:44:18.025 [I] [process=0][thread=save_finalize] Done with waiting for background save thread=async_save. (1728:async_checkpointer.py:266)
12:44:18.025 [I] [process=0][thread=save_finalize] No errors found in background save thread=async_save. (1728:async_checkpointer.py:276)
12:44:18.026 [I] [process=0][thread=save_finalize][step=2999] CheckpointManager Save Finalize is syncing with other hosts... (1728:checkpoint_manager.py:2095)
12:44:18.026 [I] [process=0][thread=save_finalize][step=2999] CheckpointManager Save Finalize is done on all hosts. (1728:checkpoint_manager.py:2110)
12:44:18.026 [I] [process=0][thread=MainThread][step=2999][wait_until_finished] Done waiting for Save Finalize thread (save_finalize) running at step=2999. (1728:checkpoint_manager.py:2010)