10:57:13.096 [I] Running on: a4e3c12d84ee                                                         (1084:train.py:196)
INFO:2026-11-06 10:57:13,626:jax._src.xla_bridge:925: Unable to initialize backend 'rocm': module 'jaxlib.xla_extension' has no attribute 'GpuAllocatorConfig'
10:57:13.626 [I] Unable to initialize backend 'rocm': module 'jaxlib.xla_extension' has no attribute 'GpuAllocatorConfig' (1084:xla_bridge.py:925)
INFO:2026-11-06 10:57:13,631:jax._src.xla_bridge:925: Unable to initialize backend 'tpu': INTERNAL: Failed to open libtpu.so: libtpu.so: cannot open shared object file: No such file or directory
10:57:13.631 [I] Unable to initialize backend 'tpu': INTERNAL: Failed to open libtpu.so: libtpu.so: cannot open shared object file: No such file or directory (1084:xla_bridge.py:925)
10:57:14.790 [I] Created BasePyTreeCheckpointHandler: use_ocdbt=True, use_zarr3=False, pytree_metadata_options=PyTreeMetadataOptions(support_rich_types=False), array_metadata_store=<orbax.checkpoint._src.metadata.array_metadata_store.Store object at 0x7664e0596590> (1084:base_pytree_checkpoint_handler.py:334)
10:57:14.790 [I] Created BasePyTreeCheckpointHandler: use_ocdbt=True, use_zarr3=False, pytree_metadata_options=PyTreeMetadataOptions(support_rich_types=False), array_metadata_store=<orbax.checkpoint._src.metadata.array_metadata_store.Store object at 0x7664e0596590> (1084:base_pytree_checkpoint_handler.py:334)
10:57:14.790 [I] [thread=MainThread] Failed to get flag value for EXPERIMENTAL_ORBAX_USE_DISTRIBUTED_PROCESS_ID. (1084:multihost.py:390)
10:57:14.791 [I] [process=0][thread=MainThread] CheckpointManager init: checkpointers=None, item_names=None, item_handlers={'assets': <openpi.training.checkpoints.CallbackHandler object at 0x7663418e5910>, 'train_state': <orbax.checkpoint._src.handlers.pytree_checkpoint_handler.PyTreeCheckpointHandler object at 0x7663418e58d0>, 'params': <orbax.checkpoint._src.handlers.pytree_checkpoint_handler.PyTreeCheckpointHandler object at 0x766342f15d50>}, handler_registry=None (1084:checkpoint_manager.py:620)
10:57:14.791 [I] Deferred registration for item: "assets". Adding handler `<openpi.training.checkpoints.CallbackHandler object at 0x7663418e5910>` for item "assets" and save args `<class 'openpi.training.checkpoints.CallbackSave'>` and restore args `<class 'openpi.training.checkpoints.CallbackRestore'>` to `_handler_registry`. (1084:composite_checkpoint_handler.py:234)
10:57:14.791 [I] Deferred registration for item: "train_state". Adding handler `<orbax.checkpoint._src.handlers.pytree_checkpoint_handler.PyTreeCheckpointHandler object at 0x7663418e58d0>` for item "train_state" and save args `<class 'orbax.checkpoint._src.handlers.pytree_checkpoint_handler.PyTreeSaveArgs'>` and restore args `<class 'orbax.checkpoint._src.handlers.pytree_checkpoint_handler.PyTreeRestoreArgs'>` to `_handler_registry`. (1084:composite_checkpoint_handler.py:234)
10:57:14.791 [I] Deferred registration for item: "params". Adding handler `<orbax.checkpoint._src.handlers.pytree_checkpoint_handler.PyTreeCheckpointHandler object at 0x766342f15d50>` for item "params" and save args `<class 'orbax.checkpoint._src.handlers.pytree_checkpoint_handler.PyTreeSaveArgs'>` and restore args `<class 'orbax.checkpoint._src.handlers.pytree_checkpoint_handler.PyTreeRestoreArgs'>` to `_handler_registry`. (1084:composite_checkpoint_handler.py:234)
10:57:14.791 [I] Deferred registration for item: "metrics". Adding handler `<orbax.checkpoint._src.handlers.json_checkpoint_handler.JsonCheckpointHandler object at 0x766342fe0910>` for item "metrics" and save args `<class 'orbax.checkpoint._src.handlers.json_checkpoint_handler.JsonSaveArgs'>` and restore args `<class 'orbax.checkpoint._src.handlers.json_checkpoint_handler.JsonRestoreArgs'>` to `_handler_registry`. (1084:composite_checkpoint_handler.py:234)
10:57:14.791 [I] Initialized registry DefaultCheckpointHandlerRegistry({('assets', <class 'openpi.training.checkpoints.CallbackSave'>): <openpi.training.checkpoints.CallbackHandler object at 0x7663418e5910>, ('assets', <class 'openpi.training.checkpoints.CallbackRestore'>): <openpi.training.checkpoints.CallbackHandler object at 0x7663418e5910>, ('train_state', <class 'orbax.checkpoint._src.handlers.pytree_checkpoint_handler.PyTreeSaveArgs'>): <orbax.checkpoint._src.handlers.pytree_checkpoint_handler.PyTreeCheckpointHandler object at 0x7663418e58d0>, ('train_state', <class 'orbax.checkpoint._src.handlers.pytree_checkpoint_handler.PyTreeRestoreArgs'>): <orbax.checkpoint._src.handlers.pytree_checkpoint_handler.PyTreeCheckpointHandler object at 0x7663418e58d0>, ('params', <class 'orbax.checkpoint._src.handlers.pytree_checkpoint_handler.PyTreeSaveArgs'>): <orbax.checkpoint._src.handlers.pytree_checkpoint_handler.PyTreeCheckpointHandler object at 0x766342f15d50>, ('params', <class 'orbax.checkpoint._src.handlers.pytree_checkpoint_handler.PyTreeRestoreArgs'>): <orbax.checkpoint._src.handlers.pytree_checkpoint_handler.PyTreeCheckpointHandler object at 0x766342f15d50>, ('metrics', <class 'orbax.checkpoint._src.handlers.json_checkpoint_handler.JsonSaveArgs'>): <orbax.checkpoint._src.handlers.json_checkpoint_handler.JsonCheckpointHandler object at 0x766342fe0910>, ('metrics', <class 'orbax.checkpoint._src.handlers.json_checkpoint_handler.JsonRestoreArgs'>): <orbax.checkpoint._src.handlers.json_checkpoint_handler.JsonCheckpointHandler object at 0x766342fe0910>}). (1084:composite_checkpoint_handler.py:502)
10:57:14.791 [I] orbax-checkpoint version: 0.11.13                                                (1084:abstract_checkpointer.py:35)
10:57:14.792 [I] [process=0][thread=MainThread] Using barrier_sync_fn: <function get_barrier_sync_fn.<locals>.<lambda> at 0x766341966840> timeout: 7200 secs and primary_host=0 for async checkpoint writes (1084:async_checkpointer.py:170)
10:57:14.792 [I] Found 0 checkpoint steps in /workspace/openpi/checkpoints/pi05_mecharm_trim_corr_v5_lora/trim_corr_v5 (1084:checkpoint_manager.py:1701)
10:57:14.792 [I] [process=0][thread=MainThread] CheckpointManager created,  primary_host=0, CheckpointManagerOptions=CheckpointManagerOptions(save_interval_steps=1, max_to_keep=1, keep_time_interval=None, keep_period=1000, should_keep_fn=None, best_fn=None, best_mode='max', keep_checkpoints_without_metrics=True, step_prefix=None, step_format_fixed_length=None, step_name_format=None, create=False, cleanup_tmp_directories=False, save_on_steps=frozenset(), single_host_load_and_broadcast=False, todelete_subdir=None, enable_background_delete=False, read_only=False, enable_async_checkpointing=True, async_options=AsyncOptions(timeout_secs=7200, barrier_sync_fn=None, post_finalization_callback=None, create_directories_asynchronously=True), multiprocessing_options=MultiprocessingOptions(primary_host=0, active_processes=None, barrier_sync_key_prefix=None), should_save_fn=None, file_options=FileOptions(path_permission_mode=None), save_root_metadata=True, temporary_path_class=None, save_decision_policy=None, prevent_write_metrics=False), root_directory=/workspace/openpi/checkpoints/pi05_mecharm_trim_corr_v5_lora/trim_corr_v5: <orbax.checkpoint.checkpoint_manager.CheckpointManager object at 0x766341be4c10> (1084:checkpoint_manager.py:801)
10:57:14.807 [I] Downloading gs://big_vision/paligemma_tokenizer.model to /root/.cache/openpi/big_vision/paligemma_tokenizer.model (1084:download.py:86)
10:57:15.297 [I] Progress on: -/4264023 rate:- remaining:? elapsed:00:00 postfix:-                (1084:tqdm_logging.py:145)
10:57:16.299 [I] Progress on: 4.07MiB/4.07MiB rate:4,258,692.4iB/s remaining:00:00 elapsed:00:01 postfix:- (1084:tqdm_logging.py:145)
10:57:16.460 [I] Loaded norm stats from /workspace/openpi/assets/pi05_mecharm_trim_corr_v5_lora/local/mecharm_redblock_box_teach_v1_trim2_corr_v5 (1084:config.py:197)
10:57:16.462 [I] data_config: DataConfig(repo_id='local/mecharm_redblock_box_teach_v1_trim2_corr_v5', asset_id='local/mecharm_redblock_box_teach_v1_trim2_corr_v5', norm_stats={'state': NormStats(mean=array([  0.11632019,  47.20463562,   4.25571823,   3.47346711,
       -49.80432129, -32.3266716 ,   0.42522314]), std=array([33.52104568, 21.01031494, 13.69488239,  5.18986511, 21.06582069,
        3.66200471,  0.49437681]), q01=array([-61.62614893,   1.48315994, -34.35241028,  -7.64541887,
       -86.05951349, -40.59999847,   0.        ]), q99=array([ 44.61974199,  87.87268339,  30.92747083,  16.58067341,
        -0.260924  , -26.45196724,   0.9998    ])), 'actions': NormStats(mean=array([ 0.192315  ,  0.027426  , -0.01522407,  0.04785711, -0.08074823,
       -0.05333935,  0.42522326]), std=array([9.28794479, 9.00317574, 4.14188147, 1.60169351, 9.50789356,
       0.9818573 , 0.49437684]), q01=array([-21.17317298, -23.18362675, -14.27432952,  -5.18250646,
       -26.67564539,  -3.16441521,   0.        ]), q99=array([27.84317563, 22.359031  , 11.17634333,  4.02140303, 25.71650509,
        3.24633011,  0.9998    ]))}, repack_transforms=Group(inputs=[RepackTransform(structure={'observation/image': 'image', 'observation/state': 'state', 'actions': 'actions', 'prompt': 'prompt'})], outputs=()), data_transforms=Group(inputs=(MecharmInputs(model_type=<ModelType.PI05: 'pi05'>), DeltaActions(mask=(True, True, True, True, True, True, False))), outputs=(AbsoluteActions(mask=(True, True, True, True, True, True, False)), MecharmOutputs())), model_transforms=Group(inputs=[InjectDefaultPrompt(prompt=None), ResizeImages(height=224, width=224), TokenizePrompt(tokenizer=<openpi.models.tokenizer.PaligemmaTokenizer object at 0x7663426d5950>, discrete_state_input=True), PadStatesAndActions(model_action_dim=32)], outputs=()), use_quantile_norm=True, action_sequence_keys=('actions',), prompt_from_task=True, rlds_data_dir=None, action_space=None, datasets=()) (1084:data_loader.py:243)

Downloading data:   0%|          | 0/30 [00:00<?, ?files/s]
Downloading data: 100%|██████████| 30/30 [00:00<00:00, 63389.98files/s]

Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 496 examples [00:00, 2611.14 examples/s]
Generating train split: 993 examples [00:00, 2406.98 examples/s]
Generating train split: 1371 examples [00:00, 2543.03 examples/s]
Generating train split: 1828 examples [00:00, 2972.41 examples/s]
Generating train split: 2321 examples [00:00, 3385.30 examples/s]
Generating train split: 2837 examples [00:00, 3741.62 examples/s]
Generating train split: 3437 examples [00:01, 4147.41 examples/s]
Generating train split: 4032 examples [00:01, 4238.63 examples/s]
Generating train split: 4301 examples [00:01, 3608.88 examples/s]
10:57:18.011 [I] local_batch_size: 32                                                             (1084:data_loader.py:324)
10:57:34.787 [I] Initialized data loader:
[0].images['base_0_rgb']: (32, 224, 224, 3)@float32
[0].images['left_wrist_0_rgb']: (32, 224, 224, 3)@float32
[0].images['right_wrist_0_rgb']: (32, 224, 224, 3)@float32
[0].image_masks['base_0_rgb']: (32,)@bool
[0].image_masks['left_wrist_0_rgb']: (32,)@bool
[0].image_masks['right_wrist_0_rgb']: (32,)@bool
[0].state: (32, 32)@float32
[0].tokenized_prompt: (32, 200)@int32
[0].tokenized_prompt_mask: (32, 200)@bool
[1]: (32, 10, 32)@float32 (1084:train.py:227)
10:57:36.304 [I] Created BasePyTreeCheckpointHandler: use_ocdbt=True, use_zarr3=False, pytree_metadata_options=PyTreeMetadataOptions(support_rich_types=False), array_metadata_store=<orbax.checkpoint._src.metadata.array_metadata_store.Store object at 0x7664e0596590> (1084:base_pytree_checkpoint_handler.py:334)
10:57:36.336 [I] Restoring checkpoint from /workspace/ckpt/corr_v2_1499/params.                   (1084:checkpointer.py:298)
10:57:41.428 [I] [process=0] /jax/checkpoint/read/bytes_per_sec: 1.4 GiB/s (total bytes: 7.2 GiB) (time elapsed: 5 seconds) (per-host) (1084:base_pytree_checkpoint_handler.py:114)
10:57:41.429 [I] Finished restoring checkpoint in 5.09 seconds from /workspace/ckpt/corr_v2_1499/params. (1084:checkpointer.py:309)
10:57:44.800 [I] Initialized train state:
['PaliGemma']['img']['Transformer']['encoder_norm']['bias'].value: (1152,)@float32
['PaliGemma']['img']['Transformer']['encoder_norm']['scale'].value: (1152,)@float32
['PaliGemma']['img']['Transformer']['encoderblock']['LayerNorm_0']['bias'].value: (27, 1152)@float32
['PaliGemma']['img']['Transformer']['encoderblock']['LayerNorm_0']['scale'].value: (27, 1152)@float32
['PaliGemma']['img']['Transformer']['encoderblock']['LayerNorm_1']['bias'].value: (27, 1152)@float32
['PaliGemma']['img']['Transformer']['encoderblock']['LayerNorm_1']['scale'].value: (27, 1152)@float32
['PaliGemma']['img']['Transformer']['encoderblock']['MlpBlock_0']['Dense_0']['bias'].value: (27, 4304)@float32
['PaliGemma']['img']['Transformer']['encoderblock']['MlpBlock_0']['Dense_0']['kernel'].value: (27, 1152, 4304)@float32
['PaliGemma']['img']['Transformer']['encoderblock']['MlpBlock_0']['Dense_1']['bias'].value: (27, 1152)@float32
['PaliGemma']['img']['Transformer']['encoderblock']['MlpBlock_0']['Dense_1']['kernel'].value: (27, 4304, 1152)@float32
['PaliGemma']['img']['Transformer']['encoderblock']['MultiHeadDotProductAttention_0']['key']['bias'].value: (27, 16, 72)@float32
['PaliGemma']['img']['Transformer']['encoderblock']['MultiHeadDotProductAttention_0']['key']['kernel'].value: (27, 1152, 16, 72)@float32
['PaliGemma']['img']['Transformer']['encoderblock']['MultiHeadDotProductAttention_0']['out']['bias'].value: (27, 1152)@float32
['PaliGemma']['img']['Transformer']['encoderblock']['MultiHeadDotProductAttention_0']['out']['kernel'].value: (27, 16, 72, 1152)@float32
['PaliGemma']['img']['Transformer']['encoderblock']['MultiHeadDotProductAttention_0']['query']['bias'].value: (27, 16, 72)@float32
['PaliGemma']['img']['Transformer']['encoderblock']['MultiHeadDotProductAttention_0']['query']['kernel'].value: (27, 1152, 16, 72)@float32
['PaliGemma']['img']['Transformer']['encoderblock']['MultiHeadDotProductAttention_0']['value']['bias'].value: (27, 16, 72)@float32
['PaliGemma']['img']['Transformer']['encoderblock']['MultiHeadDotProductAttention_0']['value']['kernel'].value: (27, 1152, 16, 72)@float32
['PaliGemma']['img']['embedding']['bias'].value: (1152,)@float32
['PaliGemma']['img']['embedding']['kernel'].value: (14, 14, 3, 1152)@float32
['PaliGemma']['img']['head']['bias'].value: (2048,)@float32
['PaliGemma']['img']['head']['kernel'].value: (1152, 2048)@float32
['PaliGemma']['img']['pos_embedding'].value: (1, 256, 1152)@float32
['PaliGemma']['llm']['embedder']['input_embedding'].value: (257152, 2048)@bfloat16
['PaliGemma']['llm']['final_norm']['scale'].value: (2048,)@bfloat16
['PaliGemma']['llm']['final_norm_1']['Dense_0']['bias'].value: (3072,)@bfloat16
['PaliGemma']['llm']['final_norm_1']['Dense_0']['kernel'].value: (1024, 3072)@bfloat16
['PaliGemma']['llm']['layers']['attn']['attn_vec_einsum']['lora_a'].value: (18, 8, 256, 16)@float32
['PaliGemma']['llm']['layers']['attn']['attn_vec_einsum']['lora_b'].value: (18, 8, 16, 2048)@float32
['PaliGemma']['llm']['layers']['attn']['attn_vec_einsum']['w'].value: (18, 8, 256, 2048)@bfloat16
['PaliGemma']['llm']['layers']['attn']['attn_vec_einsum_1']['lora_a'].value: (18, 8, 256, 32)@float32
['PaliGemma']['llm']['layers']['attn']['attn_vec_einsum_1']['lora_b'].value: (18, 8, 32, 1024)@float32
['PaliGemma']['llm']['layers']['attn']['attn_vec_einsum_1']['w'].value: (18, 8, 256, 1024)@bfloat16
['PaliGemma']['llm']['layers']['attn']['kv_einsum']['lora_a'].value: (18, 2, 1, 2048, 16)@float32
['PaliGemma']['llm']['layers']['attn']['kv_einsum']['lora_b'].value: (18, 2, 1, 16, 256)@float32
['PaliGemma']['llm']['layers']['attn']['kv_einsum']['w'].value: (18, 2, 1, 2048, 256)@bfloat16
['PaliGemma']['llm']['layers']['attn']['kv_einsum_1']['lora_a'].value: (18, 2, 1, 1024, 32)@float32
['PaliGemma']['llm']['layers']['attn']['kv_einsum_1']['lora_b'].value: (18, 2, 1, 32, 256)@float32
['PaliGemma']['llm']['layers']['attn']['kv_einsum_1']['w'].value: (18, 2, 1, 1024, 256)@bfloat16
['PaliGemma']['llm']['layers']['attn']['q_einsum']['lora_a'].value: (18, 8, 2048, 16)@float32
['PaliGemma']['llm']['layers']['attn']['q_einsum']['lora_b'].value: (18, 8, 16, 256)@float32
['PaliGemma']['llm']['layers']['attn']['q_einsum']['w'].value: (18, 8, 2048, 256)@bfloat16
['PaliGemma']['llm']['layers']['attn']['q_einsum_1']['lora_a'].value: (18, 8, 1024, 32)@float32
['PaliGemma']['llm']['layers']['attn']['q_einsum_1']['lora_b'].value: (18, 8, 32, 256)@float32
['PaliGemma']['llm']['layers']['attn']['q_einsum_1']['w'].value: (18, 8, 1024, 256)@bfloat16
['PaliGemma']['llm']['layers']['mlp']['gating_einsum'].value: (18, 2, 2048, 16384)@bfloat16
['PaliGemma']['llm']['layers']['mlp']['gating_einsum_lora_a'].value: (18, 2, 2048, 16)@float32
['PaliGemma']['llm']['layers']['mlp']['gating_einsum_lora_b'].value: (18, 2, 16, 16384)@float32
['PaliGemma']['llm']['layers']['mlp']['linear'].value: (18, 16384, 2048)@bfloat16
['PaliGemma']['llm']['layers']['mlp']['linear_lora_a'].value: (18, 16384, 16)@float32
['PaliGemma']['llm']['layers']['mlp']['linear_lora_b'].value: (18, 16, 2048)@float32
['PaliGemma']['llm']['layers']['mlp_1']['gating_einsum'].value: (18, 2, 1024, 4096)@bfloat16
['PaliGemma']['llm']['layers']['mlp_1']['gating_einsum_lora_a'].value: (18, 2, 1024, 32)@float32
['PaliGemma']['llm']['layers']['mlp_1']['gating_einsum_lora_b'].value: (18, 2, 32, 4096)@float32
['PaliGemma']['llm']['layers']['mlp_1']['linear'].value: (18, 4096, 1024)@bfloat16
['PaliGemma']['llm']['layers']['mlp_1']['linear_lora_a'].value: (18, 4096, 32)@float32
['PaliGemma']['llm']['layers']['mlp_1']['linear_lora_b'].value: (18, 32, 1024)@float32
['PaliGemma']['llm']['layers']['pre_attention_norm']['scale'].value: (18, 2048)@bfloat16
['PaliGemma']['llm']['layers']['pre_attention_norm_1']['Dense_0']['bias'].value: (18, 3072)@bfloat16
['PaliGemma']['llm']['layers']['pre_attention_norm_1']['Dense_0']['kernel'].value: (18, 1024, 3072)@bfloat16
['PaliGemma']['llm']['layers']['pre_ffw_norm']['scale'].value: (18, 2048)@bfloat16
['PaliGemma']['llm']['layers']['pre_ffw_norm_1']['Dense_0']['bias'].value: (18, 3072)@bfloat16
['PaliGemma']['llm']['layers']['pre_ffw_norm_1']['Dense_0']['kernel'].value: (18, 1024, 3072)@bfloat16
['action_in_proj']['bias'].value: (1024,)@float32
['action_in_proj']['kernel'].value: (32, 1024)@float32
['action_out_proj']['bias'].value: (32,)@float32
['action_out_proj']['kernel'].value: (1024, 32)@float32
['time_mlp_in']['bias'].value: (1024,)@float32
['time_mlp_in']['kernel'].value: (1024, 1024)@float32
['time_mlp_out']['bias'].value: (1024,)@float32
['time_mlp_out']['kernel'].value: (1024, 1024)@float32 (1084:train.py:238)
10:57:44.810 [I] Progress on: -/1500 rate:- remaining:? elapsed:00:00 postfix:-                   (1084:tqdm_logging.py:145)


Step 0: grad_norm=0.0816, loss=0.0129, param_norm=1804.2159
11:02:03.514 [I] Progress on: -/1500 rate:- remaining:? elapsed:04:18 postfix:-                   (1084:tqdm_logging.py:145)
11:02:17.889 [I] Progress on: 4.00it/1.50kit rate:39.8s/it remaining:16:32:14 elapsed:04:33 postfix:- (1084:tqdm_logging.py:145)
11:02:31.893 [I] Progress on: 6.00it/1.50kit rate:20.8s/it remaining:8:38:57 elapsed:04:47 postfix:- (1084:tqdm_logging.py:145)
11:02:45.882 [I] Progress on: 8.00it/1.50kit rate:13.3s/it remaining:5:31:55 elapsed:05:01 postfix:- (1084:tqdm_logging.py:145)
11:02:59.918 [I] Progress on: 10.0it/1.50kit rate:10.0s/it remaining:4:09:00 elapsed:05:15 postfix:- (1084:tqdm_logging.py:145)
11:03:13.986 [I] Progress on: 12.0it/1.50kit rate:8.5s/it remaining:3:30:17 elapsed:05:29 postfix:- (1084:tqdm_logging.py:145)
11:03:28.073 [I] Progress on: 14.0it/1.50kit rate:7.7s/it remaining:3:11:46 elapsed:05:43 postfix:- (1084:tqdm_logging.py:145)
11:03:42.158 [I] Progress on: 16.0it/1.50kit rate:7.4s/it remaining:3:02:41 elapsed:05:57 postfix:- (1084:tqdm_logging.py:145)
11:03:56.265 [I] Progress on: 18.0it/1.50kit rate:7.2s/it remaining:2:58:14 elapsed:06:11 postfix:- (1084:tqdm_logging.py:145)
11:04:10.379 [I] Progress on: 20.0it/1.50kit rate:7.1s/it remaining:2:55:59 elapsed:06:25 postfix:- (1084:tqdm_logging.py:145)
11:04:24.487 [I] Progress on: 22.0it/1.50kit rate:7.1s/it remaining:2:54:44 elapsed:06:39 postfix:- (1084:tqdm_logging.py:145)
11:04:38.587 [I] Progress on: 24.0it/1.50kit rate:7.1s/it remaining:2:53:57 elapsed:06:53 postfix:- (1084:tqdm_logging.py:145)
11:04:52.739 [I] Progress on: 26.0it/1.50kit rate:7.1s/it remaining:2:53:47 elapsed:07:07 postfix:- (1084:tqdm_logging.py:145)
11:05:06.872 [I] Progress on: 28.0it/1.50kit rate:7.1s/it remaining:2:53:26 elapsed:07:22 postfix:- (1084:tqdm_logging.py:145)
11:05:21.005 [I] Progress on: 30.0it/1.50kit rate:7.1s/it remaining:2:53:09 elapsed:07:36 postfix:- (1084:tqdm_logging.py:145)
11:05:35.157 [I] Progress on: 32.0it/1.50kit rate:7.1s/it remaining:2:53:01 elapsed:07:50 postfix:- (1084:tqdm_logging.py:145)
11:05:49.273 [I] Progress on: 34.0it/1.50kit rate:7.1s/it remaining:2:52:37 elapsed:08:04 postfix:- (1084:tqdm_logging.py:145)
11:06:03.396 [I] Progress on: 36.0it/1.50kit rate:7.1s/it remaining:2:52:19 elapsed:08:18 postfix:- (1084:tqdm_logging.py:145)
11:06:17.542 [I] Progress on: 38.0it/1.50kit rate:7.1s/it remaining:2:52:10 elapsed:08:32 postfix:- (1084:tqdm_logging.py:145)
11:06:31.671 [I] Progress on: 40.0it/1.50kit rate:7.1s/it remaining:2:51:55 elapsed:08:46 postfix:- (1084:tqdm_logging.py:145)
11:06:45.812 [I] Progress on: 42.0it/1.50kit rate:7.1s/it remaining:2:51:44 elapsed:09:01 postfix:- (1084:tqdm_logging.py:145)
11:06:59.964 [I] Progress on: 44.0it/1.50kit rate:7.1s/it remaining:2:51:37 elapsed:09:15 postfix:- (1084:tqdm_logging.py:145)
11:07:14.094 [I] Progress on: 46.0it/1.50kit rate:7.1s/it remaining:2:51:16 elapsed:09:29 postfix:- (1084:tqdm_logging.py:145)
11:07:28.232 [I] Progress on: 48.0it/1.50kit rate:7.1s/it remaining:2:51:03 elapsed:09:43 postfix:- (1084:tqdm_logging.py:145)
11:07:42.371 [I] Progress on: 50.0it/1.50kit rate:7.1s/it remaining:2:50:49 elapsed:09:57 postfix:- (1084:tqdm_logging.py:145)


Step 50: grad_norm=0.0707, loss=0.0066, param_norm=1804.2159
11:07:56.270 [I] Progress on: 50.0it/1.50kit rate:7.1s/it remaining:2:50:49 elapsed:10:11 postfix:- (1084:tqdm_logging.py:145)
11:08:10.804 [I] Progress on: 54.0it/1.50kit rate:6.8s/it remaining:2:43:26 elapsed:10:25 postfix:- (1084:tqdm_logging.py:145)
11:08:24.902 [I] Progress on: 56.0it/1.50kit rate:6.9s/it remaining:2:46:28 elapsed:10:40 postfix:- (1084:tqdm_logging.py:145)
11:08:39.013 [I] Progress on: 58.0it/1.50kit rate:7.0s/it remaining:2:47:56 elapsed:10:54 postfix:- (1084:tqdm_logging.py:145)
11:08:53.106 [I] Progress on: 60.0it/1.50kit rate:7.0s/it remaining:2:48:25 elapsed:11:08 postfix:- (1084:tqdm_logging.py:145)
11:09:07.212 [I] Progress on: 62.0it/1.50kit rate:7.0s/it remaining:2:48:34 elapsed:11:22 postfix:- (1084:tqdm_logging.py:145)
11:09:21.325 [I] Progress on: 64.0it/1.50kit rate:7.0s/it remaining:2:48:34 elapsed:11:36 postfix:- (1084:tqdm_logging.py:145)
11:09:35.416 [I] Progress on: 66.0it/1.50kit rate:7.0s/it remaining:2:48:18 elapsed:11:50 postfix:- (1084:tqdm_logging.py:145)
11:09:49.510 [I] Progress on: 68.0it/1.50kit rate:7.0s/it remaining:2:48:09 elapsed:12:04 postfix:- (1084:tqdm_logging.py:145)
11:10:03.599 [I] Progress on: 70.0it/1.50kit rate:7.0s/it remaining:2:47:56 elapsed:12:18 postfix:- (1084:tqdm_logging.py:145)
11:10:17.701 [I] Progress on: 72.0it/1.50kit rate:7.1s/it remaining:2:47:47 elapsed:12:32 postfix:- (1084:tqdm_logging.py:145)
11:10:31.802 [I] Progress on: 74.0it/1.50kit rate:7.1s/it remaining:2:47:35 elapsed:12:46 postfix:- (1084:tqdm_logging.py:145)
11:10:45.925 [I] Progress on: 76.0it/1.50kit rate:7.1s/it remaining:2:47:27 elapsed:13:01 postfix:- (1084:tqdm_logging.py:145)
11:11:00.039 [I] Progress on: 78.0it/1.50kit rate:7.1s/it remaining:2:47:12 elapsed:13:15 postfix:- (1084:tqdm_logging.py:145)
11:11:14.153 [I] Progress on: 80.0it/1.50kit rate:7.1s/it remaining:2:47:00 elapsed:13:29 postfix:- (1084:tqdm_logging.py:145)
11:11:28.259 [I] Progress on: 82.0it/1.50kit rate:7.1s/it remaining:2:46:42 elapsed:13:43 postfix:- (1084:tqdm_logging.py:145)
11:11:42.381 [I] Progress on: 84.0it/1.50kit rate:7.1s/it remaining:2:46:32 elapsed:13:57 postfix:- (1084:tqdm_logging.py:145)
11:11:56.511 [I] Progress on: 86.0it/1.50kit rate:7.1s/it remaining:2:46:26 elapsed:14:11 postfix:- (1084:tqdm_logging.py:145)
11:12:10.632 [I] Progress on: 88.0it/1.50kit rate:7.1s/it remaining:2:46:13 elapsed:14:25 postfix:- (1084:tqdm_logging.py:145)
11:12:24.734 [I] Progress on: 90.0it/1.50kit rate:7.1s/it remaining:2:45:52 elapsed:14:39 postfix:- (1084:tqdm_logging.py:145)
11:12:38.823 [I] Progress on: 92.0it/1.50kit rate:7.1s/it remaining:2:45:26 elapsed:14:54 postfix:- (1084:tqdm_logging.py:145)
11:12:52.897 [I] Progress on: 94.0it/1.50kit rate:7.0s/it remaining:2:45:01 elapsed:15:08 postfix:- (1084:tqdm_logging.py:145)
11:13:07.016 [I] Progress on: 96.0it/1.50kit rate:7.0s/it remaining:2:44:57 elapsed:15:22 postfix:- (1084:tqdm_logging.py:145)
11:13:21.157 [I] Progress on: 98.0it/1.50kit rate:7.1s/it remaining:2:44:59 elapsed:15:36 postfix:- (1084:tqdm_logging.py:145)
11:13:35.225 [I] Progress on: 100it/1.50kit rate:7.0s/it remaining:2:44:27 elapsed:15:50 postfix:- (1084:tqdm_logging.py:145)


Step 100: grad_norm=0.0495, loss=0.0048, param_norm=1804.2166
11:13:49.001 [I] Progress on: 100it/1.50kit rate:7.0s/it remaining:2:44:27 elapsed:16:04 postfix:- (1084:tqdm_logging.py:145)
11:14:03.535 [I] Progress on: 104it/1.50kit rate:6.8s/it remaining:2:37:30 elapsed:16:18 postfix:- (1084:tqdm_logging.py:145)
11:14:17.641 [I] Progress on: 106it/1.50kit rate:6.9s/it remaining:2:40:38 elapsed:16:32 postfix:- (1084:tqdm_logging.py:145)
11:14:31.723 [I] Progress on: 108it/1.50kit rate:7.0s/it remaining:2:41:53 elapsed:16:46 postfix:- (1084:tqdm_logging.py:145)
11:14:45.848 [I] Progress on: 110it/1.50kit rate:7.0s/it remaining:2:42:40 elapsed:17:01 postfix:- (1084:tqdm_logging.py:145)
11:14:59.941 [I] Progress on: 112it/1.50kit rate:7.0s/it remaining:2:42:44 elapsed:17:15 postfix:- (1084:tqdm_logging.py:145)
11:15:14.053 [I] Progress on: 114it/1.50kit rate:7.0s/it remaining:2:42:43 elapsed:17:29 postfix:- (1084:tqdm_logging.py:145)
11:15:28.160 [I] Progress on: 116it/1.50kit rate:7.0s/it remaining:2:42:35 elapsed:17:43 postfix:- (1084:tqdm_logging.py:145)
11:15:42.261 [I] Progress on: 118it/1.50kit rate:7.1s/it remaining:2:42:23 elapsed:17:57 postfix:- (1084:tqdm_logging.py:145)
11:15:56.346 [I] Progress on: 120it/1.50kit rate:7.0s/it remaining:2:42:03 elapsed:18:11 postfix:- (1084:tqdm_logging.py:145)
11:16:10.433 [I] Progress on: 122it/1.50kit rate:7.0s/it remaining:2:41:47 elapsed:18:25 postfix:- (1084:tqdm_logging.py:145)
11:16:24.544 [I] Progress on: 124it/1.50kit rate:7.0s/it remaining:2:41:38 elapsed:18:39 postfix:- (1084:tqdm_logging.py:145)
11:16:38.640 [I] Progress on: 126it/1.50kit rate:7.0s/it remaining:2:41:23 elapsed:18:53 postfix:- (1084:tqdm_logging.py:145)
11:16:52.746 [I] Progress on: 128it/1.50kit rate:7.1s/it remaining:2:41:14 elapsed:19:07 postfix:- (1084:tqdm_logging.py:145)
11:17:06.851 [I] Progress on: 130it/1.50kit rate:7.1s/it remaining:2:41:02 elapsed:19:22 postfix:- (1084:tqdm_logging.py:145)
11:17:20.984 [I] Progress on: 132it/1.50kit rate:7.1s/it remaining:2:40:59 elapsed:19:36 postfix:- (1084:tqdm_logging.py:145)
11:17:35.626 [I] Progress on: 134it/1.50kit rate:7.2s/it remaining:2:44:22 elapsed:19:50 postfix:- (1084:tqdm_logging.py:145)
11:17:49.203 [I] Progress on: 136it/1.50kit rate:7.0s/it remaining:2:39:45 elapsed:20:04 postfix:- (1084:tqdm_logging.py:145)
11:18:03.343 [I] Progress on: 138it/1.50kit rate:7.1s/it remaining:2:40:06 elapsed:20:18 postfix:- (1084:tqdm_logging.py:145)
11:18:17.458 [I] Progress on: 140it/1.50kit rate:7.1s/it remaining:2:40:01 elapsed:20:32 postfix:- (1084:tqdm_logging.py:145)
11:18:31.569 [I] Progress on: 142it/1.50kit rate:7.1s/it remaining:2:39:50 elapsed:20:46 postfix:- (1084:tqdm_logging.py:145)
11:18:45.683 [I] Progress on: 144it/1.50kit rate:7.1s/it remaining:2:39:39 elapsed:21:00 postfix:- (1084:tqdm_logging.py:145)
11:18:59.782 [I] Progress on: 146it/1.50kit rate:7.1s/it remaining:2:39:22 elapsed:21:14 postfix:- (1084:tqdm_logging.py:145)
11:19:13.871 [I] Progress on: 148it/1.50kit rate:7.1s/it remaining:2:39:01 elapsed:21:29 postfix:- (1084:tqdm_logging.py:145)
11:19:27.988 [I] Progress on: 150it/1.50kit rate:7.1s/it remaining:2:38:55 elapsed:21:43 postfix:- (1084:tqdm_logging.py:145)


Step 150: grad_norm=0.0530, loss=0.0051, param_norm=1804.2188
11:19:41.734 [I] Progress on: 150it/1.50kit rate:7.1s/it remaining:2:38:55 elapsed:21:56 postfix:- (1084:tqdm_logging.py:145)
11:19:56.272 [I] Progress on: 154it/1.50kit rate:6.8s/it remaining:2:31:51 elapsed:22:11 postfix:- (1084:tqdm_logging.py:145)
11:20:10.357 [I] Progress on: 156it/1.50kit rate:6.9s/it remaining:2:34:45 elapsed:22:25 postfix:- (1084:tqdm_logging.py:145)
11:20:24.445 [I] Progress on: 158it/1.50kit rate:7.0s/it remaining:2:36:04 elapsed:22:39 postfix:- (1084:tqdm_logging.py:145)
11:20:38.529 [I] Progress on: 160it/1.50kit rate:7.0s/it remaining:2:36:31 elapsed:22:53 postfix:- (1084:tqdm_logging.py:145)
11:20:52.619 [I] Progress on: 162it/1.50kit rate:7.0s/it remaining:2:36:41 elapsed:23:07 postfix:- (1084:tqdm_logging.py:145)
11:21:06.723 [I] Progress on: 164it/1.50kit rate:7.0s/it remaining:2:36:45 elapsed:23:21 postfix:- (1084:tqdm_logging.py:145)
11:21:20.821 [I] Progress on: 166it/1.50kit rate:7.0s/it remaining:2:36:36 elapsed:23:36 postfix:- (1084:tqdm_logging.py:145)
11:21:34.923 [I] Progress on: 168it/1.50kit rate:7.0s/it remaining:2:36:26 elapsed:23:50 postfix:- (1084:tqdm_logging.py:145)
11:21:49.016 [I] Progress on: 170it/1.50kit rate:7.0s/it remaining:2:36:14 elapsed:24:04 postfix:- (1084:tqdm_logging.py:145)
11:22:03.086 [I] Progress on: 172it/1.50kit rate:7.0s/it remaining:2:35:51 elapsed:24:18 postfix:- (1084:tqdm_logging.py:145)
11:22:17.171 [I] Progress on: 174it/1.50kit rate:7.0s/it remaining:2:35:36 elapsed:24:32 postfix:- (1084:tqdm_logging.py:145)
11:22:31.260 [I] Progress on: 176it/1.50kit rate:7.0s/it remaining:2:35:24 elapsed:24:46 postfix:- (1084:tqdm_logging.py:145)
11:22:45.334 [I] Progress on: 178it/1.50kit rate:7.0s/it remaining:2:35:06 elapsed:25:00 postfix:- (1084:tqdm_logging.py:145)
11:22:59.410 [I] Progress on: 180it/1.50kit rate:7.0s/it remaining:2:34:51 elapsed:25:14 postfix:- (1084:tqdm_logging.py:145)
11:23:13.504 [I] Progress on: 182it/1.50kit rate:7.0s/it remaining:2:34:40 elapsed:25:28 postfix:- (1084:tqdm_logging.py:145)
11:23:27.603 [I] Progress on: 184it/1.50kit rate:7.0s/it remaining:2:34:31 elapsed:25:42 postfix:- (1084:tqdm_logging.py:145)
11:23:41.685 [I] Progress on: 186it/1.50kit rate:7.0s/it remaining:2:34:15 elapsed:25:56 postfix:- (1084:tqdm_logging.py:145)
11:23:55.770 [I] Progress on: 188it/1.50kit rate:7.0s/it remaining:2:33:59 elapsed:26:10 postfix:- (1084:tqdm_logging.py:145)
11:24:09.868 [I] Progress on: 190it/1.50kit rate:7.0s/it remaining:2:33:49 elapsed:26:25 postfix:- (1084:tqdm_logging.py:145)
11:24:23.972 [I] Progress on: 192it/1.50kit rate:7.0s/it remaining:2:33:40 elapsed:26:39 postfix:- (1084:tqdm_logging.py:145)
11:24:38.060 [I] Progress on: 194it/1.50kit rate:7.0s/it remaining:2:33:24 elapsed:26:53 postfix:- (1084:tqdm_logging.py:145)
11:24:52.160 [I] Progress on: 196it/1.50kit rate:7.0s/it remaining:2:33:11 elapsed:27:07 postfix:- (1084:tqdm_logging.py:145)
11:25:06.260 [I] Progress on: 198it/1.50kit rate:7.0s/it remaining:2:32:59 elapsed:27:21 postfix:- (1084:tqdm_logging.py:145)
11:25:20.375 [I] Progress on: 200it/1.50kit rate:7.1s/it remaining:2:32:48 elapsed:27:35 postfix:- (1084:tqdm_logging.py:145)


Step 200: grad_norm=0.0513, loss=0.0052, param_norm=1804.2212
11:25:34.115 [I] Progress on: 200it/1.50kit rate:7.1s/it remaining:2:32:48 elapsed:27:49 postfix:- (1084:tqdm_logging.py:145)
11:25:48.636 [I] Progress on: 204it/1.50kit rate:6.8s/it remaining:2:26:00 elapsed:28:03 postfix:- (1084:tqdm_logging.py:145)
11:26:02.717 [I] Progress on: 206it/1.50kit rate:6.9s/it remaining:2:28:51 elapsed:28:17 postfix:- (1084:tqdm_logging.py:145)
11:26:16.813 [I] Progress on: 208it/1.50kit rate:7.0s/it remaining:2:30:13 elapsed:28:32 postfix:- (1084:tqdm_logging.py:145)
11:26:30.914 [I] Progress on: 210it/1.50kit rate:7.0s/it remaining:2:30:49 elapsed:28:46 postfix:- (1084:tqdm_logging.py:145)
11:26:44.998 [I] Progress on: 212it/1.50kit rate:7.0s/it remaining:2:30:54 elapsed:29:00 postfix:- (1084:tqdm_logging.py:145)
11:26:59.093 [I] Progress on: 214it/1.50kit rate:7.0s/it remaining:2:30:52 elapsed:29:14 postfix:- (1084:tqdm_logging.py:145)
11:27:13.176 [I] Progress on: 216it/1.50kit rate:7.0s/it remaining:2:30:39 elapsed:29:28 postfix:- (1084:tqdm_logging.py:145)
11:27:27.271 [I] Progress on: 218it/1.50kit rate:7.0s/it remaining:2:30:27 elapsed:29:42 postfix:- (1084:tqdm_logging.py:145)
11:27:41.366 [I] Progress on: 220it/1.50kit rate:7.0s/it remaining:2:30:15 elapsed:29:56 postfix:- (1084:tqdm_logging.py:145)
11:27:55.469 [I] Progress on: 222it/1.50kit rate:7.0s/it remaining:2:30:04 elapsed:30:10 postfix:- (1084:tqdm_logging.py:145)
11:28:09.558 [I] Progress on: 224it/1.50kit rate:7.0s/it remaining:2:29:49 elapsed:30:24 postfix:- (1084:tqdm_logging.py:145)
11:28:23.644 [I] Progress on: 226it/1.50kit rate:7.0s/it remaining:2:29:34 elapsed:30:38 postfix:- (1084:tqdm_logging.py:145)
11:28:37.773 [I] Progress on: 228it/1.50kit rate:7.1s/it remaining:2:29:34 elapsed:30:52 postfix:- (1084:tqdm_logging.py:145)
11:28:51.880 [I] Progress on: 230it/1.50kit rate:7.1s/it remaining:2:29:20 elapsed:31:07 postfix:- (1084:tqdm_logging.py:145)
11:29:05.989 [I] Progress on: 232it/1.50kit rate:7.1s/it remaining:2:29:06 elapsed:31:21 postfix:- (1084:tqdm_logging.py:145)
11:29:20.105 [I] Progress on: 234it/1.50kit rate:7.1s/it remaining:2:28:54 elapsed:31:35 postfix:- (1084:tqdm_logging.py:145)
11:29:34.216 [I] Progress on: 236it/1.50kit rate:7.1s/it remaining:2:28:38 elapsed:31:49 postfix:- (1084:tqdm_logging.py:145)
11:29:48.362 [I] Progress on: 238it/1.50kit rate:7.1s/it remaining:2:28:34 elapsed:32:03 postfix:- (1084:tqdm_logging.py:145)
11:30:02.444 [I] Progress on: 240it/1.50kit rate:7.1s/it remaining:2:28:06 elapsed:32:17 postfix:- (1084:tqdm_logging.py:145)
11:30:16.607 [I] Progress on: 242it/1.50kit rate:7.1s/it remaining:2:28:11 elapsed:32:31 postfix:- (1084:tqdm_logging.py:145)
11:30:30.678 [I] Progress on: 244it/1.50kit rate:7.1s/it remaining:2:27:37 elapsed:32:45 postfix:- (1084:tqdm_logging.py:145)
11:30:44.802 [I] Progress on: 246it/1.50kit rate:7.1s/it remaining:2:27:29 elapsed:32:59 postfix:- (1084:tqdm_logging.py:145)
11:30:58.915 [I] Progress on: 248it/1.50kit rate:7.1s/it remaining:2:27:15 elapsed:33:14 postfix:- (1084:tqdm_logging.py:145)
11:31:12.993 [I] Progress on: 250it/1.50kit rate:7.0s/it remaining:2:26:48 elapsed:33:28 postfix:- (1084:tqdm_logging.py:145)


Step 250: grad_norm=0.0536, loss=0.0045, param_norm=1804.2238
11:31:26.757 [I] Progress on: 250it/1.50kit rate:7.0s/it remaining:2:26:48 elapsed:33:41 postfix:- (1084:tqdm_logging.py:145)
11:31:41.275 [I] Progress on: 254it/1.50kit rate:6.8s/it remaining:2:20:31 elapsed:33:56 postfix:- (1084:tqdm_logging.py:145)
11:31:55.381 [I] Progress on: 256it/1.50kit rate:6.9s/it remaining:2:23:20 elapsed:34:10 postfix:- (1084:tqdm_logging.py:145)
11:32:09.482 [I] Progress on: 258it/1.50kit rate:7.0s/it remaining:2:24:33 elapsed:34:24 postfix:- (1084:tqdm_logging.py:145)
11:32:23.599 [I] Progress on: 260it/1.50kit rate:7.0s/it remaining:2:25:05 elapsed:34:38 postfix:- (1084:tqdm_logging.py:145)
11:32:37.702 [I] Progress on: 262it/1.50kit rate:7.0s/it remaining:2:25:09 elapsed:34:52 postfix:- (1084:tqdm_logging.py:145)
11:32:51.828 [I] Progress on: 264it/1.50kit rate:7.0s/it remaining:2:25:10 elapsed:35:07 postfix:- (1084:tqdm_logging.py:145)
11:33:05.957 [I] Progress on: 266it/1.50kit rate:7.1s/it remaining:2:25:07 elapsed:35:21 postfix:- (1084:tqdm_logging.py:145)
11:33:20.606 [I] Progress on: 268it/1.50kit rate:7.2s/it remaining:2:28:11 elapsed:35:35 postfix:- (1084:tqdm_logging.py:145)
11:33:34.178 [I] Progress on: 270it/1.50kit rate:7.0s/it remaining:2:23:59 elapsed:35:49 postfix:- (1084:tqdm_logging.py:145)
11:33:48.297 [I] Progress on: 272it/1.50kit rate:7.0s/it remaining:2:24:08 elapsed:36:03 postfix:- (1084:tqdm_logging.py:145)
11:34:02.415 [I] Progress on: 274it/1.50kit rate:7.1s/it remaining:2:24:04 elapsed:36:17 postfix:- (1084:tqdm_logging.py:145)
11:34:16.515 [I] Progress on: 276it/1.50kit rate:7.0s/it remaining:2:23:48 elapsed:36:31 postfix:- (1084:tqdm_logging.py:145)
11:34:30.621 [I] Progress on: 278it/1.50kit rate:7.1s/it remaining:2:23:35 elapsed:36:45 postfix:- (1084:tqdm_logging.py:145)
11:34:44.723 [I] Progress on: 280it/1.50kit rate:7.1s/it remaining:2:23:23 elapsed:36:59 postfix:- (1084:tqdm_logging.py:145)
11:34:58.811 [I] Progress on: 282it/1.50kit rate:7.0s/it remaining:2:23:04 elapsed:37:14 postfix:- (1084:tqdm_logging.py:145)
11:35:12.911 [I] Progress on: 284it/1.50kit rate:7.0s/it remaining:2:22:50 elapsed:37:28 postfix:- (1084:tqdm_logging.py:145)
11:35:27.044 [I] Progress on: 286it/1.50kit rate:7.1s/it remaining:2:22:47 elapsed:37:42 postfix:- (1084:tqdm_logging.py:145)
11:35:41.133 [I] Progress on: 288it/1.50kit rate:7.1s/it remaining:2:22:25 elapsed:37:56 postfix:- (1084:tqdm_logging.py:145)
11:35:55.231 [I] Progress on: 290it/1.50kit rate:7.0s/it remaining:2:22:09 elapsed:38:10 postfix:- (1084:tqdm_logging.py:145)
11:36:09.314 [I] Progress on: 292it/1.50kit rate:7.0s/it remaining:2:21:50 elapsed:38:24 postfix:- (1084:tqdm_logging.py:145)
11:36:23.438 [I] Progress on: 294it/1.50kit rate:7.1s/it remaining:2:21:47 elapsed:38:38 postfix:- (1084:tqdm_logging.py:145)
11:36:37.541 [I] Progress on: 296it/1.50kit rate:7.1s/it remaining:2:21:30 elapsed:38:52 postfix:- (1084:tqdm_logging.py:145)
11:36:51.641 [I] Progress on: 298it/1.50kit rate:7.1s/it remaining:2:21:17 elapsed:39:06 postfix:- (1084:tqdm_logging.py:145)
11:37:05.705 [I] Progress on: 300it/1.50kit rate:7.0s/it remaining:2:20:50 elapsed:39:20 postfix:- (1084:tqdm_logging.py:145)


Step 300: grad_norm=0.0467, loss=0.0044, param_norm=1804.2260
11:37:19.438 [I] Progress on: 300it/1.50kit rate:7.0s/it remaining:2:20:50 elapsed:39:34 postfix:- (1084:tqdm_logging.py:145)
11:37:33.920 [I] Progress on: 304it/1.50kit rate:6.8s/it remaining:2:14:34 elapsed:39:49 postfix:- (1084:tqdm_logging.py:145)
11:37:48.022 [I] Progress on: 306it/1.50kit rate:6.9s/it remaining:2:17:26 elapsed:40:03 postfix:- (1084:tqdm_logging.py:145)
11:38:02.136 [I] Progress on: 308it/1.50kit rate:7.0s/it remaining:2:18:46 elapsed:40:17 postfix:- (1084:tqdm_logging.py:145)
11:38:16.206 [I] Progress on: 310it/1.50kit rate:7.0s/it remaining:2:19:04 elapsed:40:31 postfix:- (1084:tqdm_logging.py:145)
11:38:30.280 [I] Progress on: 312it/1.50kit rate:7.0s/it remaining:2:19:07 elapsed:40:45 postfix:- (1084:tqdm_logging.py:145)
11:38:44.366 [I] Progress on: 314it/1.50kit rate:7.0s/it remaining:2:19:04 elapsed:40:59 postfix:- (1084:tqdm_logging.py:145)
11:38:58.446 [I] Progress on: 316it/1.50kit rate:7.0s/it remaining:2:18:51 elapsed:41:13 postfix:- (1084:tqdm_logging.py:145)
11:39:12.551 [I] Progress on: 318it/1.50kit rate:7.0s/it remaining:2:18:47 elapsed:41:27 postfix:- (1084:tqdm_logging.py:145)
11:39:26.644 [I] Progress on: 320it/1.50kit rate:7.0s/it remaining:2:18:33 elapsed:41:41 postfix:- (1084:tqdm_logging.py:145)
11:39:40.764 [I] Progress on: 322it/1.50kit rate:7.1s/it remaining:2:18:28 elapsed:41:55 postfix:- (1084:tqdm_logging.py:145)
11:39:54.858 [I] Progress on: 324it/1.50kit rate:7.1s/it remaining:2:18:12 elapsed:42:10 postfix:- (1084:tqdm_logging.py:145)
11:40:08.962 [I] Progress on: 326it/1.50kit rate:7.1s/it remaining:2:17:58 elapsed:42:24 postfix:- (1084:tqdm_logging.py:145)
11:40:23.083 [I] Progress on: 328it/1.50kit rate:7.1s/it remaining:2:17:51 elapsed:42:38 postfix:- (1084:tqdm_logging.py:145)
11:40:37.180 [I] Progress on: 330it/1.50kit rate:7.1s/it remaining:2:17:31 elapsed:42:52 postfix:- (1084:tqdm_logging.py:145)
11:40:51.281 [I] Progress on: 332it/1.50kit rate:7.1s/it remaining:2:17:17 elapsed:43:06 postfix:- (1084:tqdm_logging.py:145)
11:41:05.370 [I] Progress on: 334it/1.50kit rate:7.0s/it remaining:2:16:56 elapsed:43:20 postfix:- (1084:tqdm_logging.py:145)
11:41:19.496 [I] Progress on: 336it/1.50kit rate:7.1s/it remaining:2:16:50 elapsed:43:34 postfix:- (1084:tqdm_logging.py:145)
11:41:33.559 [I] Progress on: 338it/1.50kit rate:7.0s/it remaining:2:16:24 elapsed:43:48 postfix:- (1084:tqdm_logging.py:145)
11:41:47.663 [I] Progress on: 340it/1.50kit rate:7.0s/it remaining:2:16:15 elapsed:44:02 postfix:- (1084:tqdm_logging.py:145)
11:42:01.799 [I] Progress on: 342it/1.50kit rate:7.1s/it remaining:2:16:12 elapsed:44:16 postfix:- (1084:tqdm_logging.py:145)
11:42:15.946 [I] Progress on: 344it/1.50kit rate:7.1s/it remaining:2:16:07 elapsed:44:31 postfix:- (1084:tqdm_logging.py:145)
11:42:30.024 [I] Progress on: 346it/1.50kit rate:7.1s/it remaining:2:15:36 elapsed:44:45 postfix:- (1084:tqdm_logging.py:145)
11:42:44.100 [I] Progress on: 348it/1.50kit rate:7.0s/it remaining:2:15:16 elapsed:44:59 postfix:- (1084:tqdm_logging.py:145)
11:42:58.219 [I] Progress on: 350it/1.50kit rate:7.1s/it remaining:2:15:11 elapsed:45:13 postfix:- (1084:tqdm_logging.py:145)


Step 350: grad_norm=0.0481, loss=0.0042, param_norm=1804.2288
11:43:11.982 [I] Progress on: 350it/1.50kit rate:7.1s/it remaining:2:15:11 elapsed:45:27 postfix:- (1084:tqdm_logging.py:145)
11:43:26.521 [I] Progress on: 354it/1.50kit rate:6.8s/it remaining:2:09:09 elapsed:45:41 postfix:- (1084:tqdm_logging.py:145)
11:43:40.619 [I] Progress on: 356it/1.50kit rate:6.9s/it remaining:2:11:43 elapsed:45:55 postfix:- (1084:tqdm_logging.py:145)
11:43:54.712 [I] Progress on: 358it/1.50kit rate:7.0s/it remaining:2:12:49 elapsed:46:09 postfix:- (1084:tqdm_logging.py:145)
11:44:08.789 [I] Progress on: 360it/1.50kit rate:7.0s/it remaining:2:13:09 elapsed:46:23 postfix:- (1084:tqdm_logging.py:145)
11:44:22.869 [I] Progress on: 362it/1.50kit rate:7.0s/it remaining:2:13:12 elapsed:46:38 postfix:- (1084:tqdm_logging.py:145)
11:44:36.982 [I] Progress on: 364it/1.50kit rate:7.0s/it remaining:2:13:16 elapsed:46:52 postfix:- (1084:tqdm_logging.py:145)
11:44:51.075 [I] Progress on: 366it/1.50kit rate:7.0s/it remaining:2:13:04 elapsed:47:06 postfix:- (1084:tqdm_logging.py:145)
11:45:05.173 [I] Progress on: 368it/1.50kit rate:7.0s/it remaining:2:12:54 elapsed:47:20 postfix:- (1084:tqdm_logging.py:145)
11:45:19.284 [I] Progress on: 370it/1.50kit rate:7.0s/it remaining:2:12:44 elapsed:47:34 postfix:- (1084:tqdm_logging.py:145)
11:45:33.398 [I] Progress on: 372it/1.50kit rate:7.1s/it remaining:2:12:36 elapsed:47:48 postfix:- (1084:tqdm_logging.py:145)
11:45:47.487 [I] Progress on: 374it/1.50kit rate:7.1s/it remaining:2:12:18 elapsed:48:02 postfix:- (1084:tqdm_logging.py:145)
11:46:01.601 [I] Progress on: 376it/1.50kit rate:7.1s/it remaining:2:12:07 elapsed:48:16 postfix:- (1084:tqdm_logging.py:145)
11:46:15.697 [I] Progress on: 378it/1.50kit rate:7.1s/it remaining:2:11:50 elapsed:48:30 postfix:- (1084:tqdm_logging.py:145)
11:46:29.831 [I] Progress on: 380it/1.50kit rate:7.1s/it remaining:2:11:47 elapsed:48:45 postfix:- (1084:tqdm_logging.py:145)
11:46:43.939 [I] Progress on: 382it/1.50kit rate:7.1s/it remaining:2:11:30 elapsed:48:59 postfix:- (1084:tqdm_logging.py:145)
11:46:58.058 [I] Progress on: 384it/1.50kit rate:7.1s/it remaining:2:11:17 elapsed:49:13 postfix:- (1084:tqdm_logging.py:145)
11:47:12.160 [I] Progress on: 386it/1.50kit rate:7.1s/it remaining:2:10:56 elapsed:49:27 postfix:- (1084:tqdm_logging.py:145)
11:47:26.271 [I] Progress on: 388it/1.50kit rate:7.1s/it remaining:2:10:41 elapsed:49:41 postfix:- (1084:tqdm_logging.py:145)
11:47:40.370 [I] Progress on: 390it/1.50kit rate:7.1s/it remaining:2:10:27 elapsed:49:55 postfix:- (1084:tqdm_logging.py:145)
11:47:54.462 [I] Progress on: 392it/1.50kit rate:7.0s/it remaining:2:10:09 elapsed:50:09 postfix:- (1084:tqdm_logging.py:145)
11:48:08.588 [I] Progress on: 394it/1.50kit rate:7.1s/it remaining:2:10:03 elapsed:50:23 postfix:- (1084:tqdm_logging.py:145)
11:48:22.670 [I] Progress on: 396it/1.50kit rate:7.1s/it remaining:2:09:43 elapsed:50:37 postfix:- (1084:tqdm_logging.py:145)
11:48:36.770 [I] Progress on: 398it/1.50kit rate:7.1s/it remaining:2:09:30 elapsed:50:51 postfix:- (1084:tqdm_logging.py:145)
11:48:50.853 [I] Progress on: 400it/1.50kit rate:7.0s/it remaining:2:09:12 elapsed:51:06 postfix:- (1084:tqdm_logging.py:145)


Step 400: grad_norm=0.0463, loss=0.0039, param_norm=1804.2307
11:49:04.612 [I] Progress on: 400it/1.50kit rate:7.0s/it remaining:2:09:12 elapsed:51:19 postfix:- (1084:tqdm_logging.py:145)
11:49:19.225 [I] Progress on: 404it/1.50kit rate:6.7s/it remaining:2:03:08 elapsed:51:34 postfix:- (1084:tqdm_logging.py:145)
11:49:33.354 [I] Progress on: 406it/1.50kit rate:6.9s/it remaining:2:06:02 elapsed:51:48 postfix:- (1084:tqdm_logging.py:145)
11:49:47.423 [I] Progress on: 408it/1.50kit rate:7.0s/it remaining:2:07:03 elapsed:52:02 postfix:- (1084:tqdm_logging.py:145)
11:50:01.568 [I] Progress on: 410it/1.50kit rate:7.0s/it remaining:2:07:46 elapsed:52:16 postfix:- (1084:tqdm_logging.py:145)
11:50:15.688 [I] Progress on: 412it/1.50kit rate:7.1s/it remaining:2:07:51 elapsed:52:30 postfix:- (1084:tqdm_logging.py:145)
11:50:29.825 [I] Progress on: 414it/1.50kit rate:7.1s/it remaining:2:07:53 elapsed:52:45 postfix:- (1084:tqdm_logging.py:145)
11:50:43.971 [I] Progress on: 416it/1.50kit rate:7.1s/it remaining:2:07:48 elapsed:52:59 postfix:- (1084:tqdm_logging.py:145)
11:50:58.106 [I] Progress on: 418it/1.50kit rate:7.1s/it remaining:2:07:35 elapsed:53:13 postfix:- (1084:tqdm_logging.py:145)
11:51:12.252 [I] Progress on: 420it/1.50kit rate:7.1s/it remaining:2:07:24 elapsed:53:27 postfix:- (1084:tqdm_logging.py:145)
11:51:26.418 [I] Progress on: 422it/1.50kit rate:7.1s/it remaining:2:07:18 elapsed:53:41 postfix:- (1084:tqdm_logging.py:145)
11:51:40.556 [I] Progress on: 424it/1.50kit rate:7.1s/it remaining:2:06:59 elapsed:53:55 postfix:- (1084:tqdm_logging.py:145)
11:51:54.697 [I] Progress on: 426it/1.50kit rate:7.1s/it remaining:2:06:44 elapsed:54:09 postfix:- (1084:tqdm_logging.py:145)
11:52:08.851 [I] Progress on: 428it/1.50kit rate:7.1s/it remaining:2:06:32 elapsed:54:24 postfix:- (1084:tqdm_logging.py:145)
11:52:22.995 [I] Progress on: 430it/1.50kit rate:7.1s/it remaining:2:06:17 elapsed:54:38 postfix:- (1084:tqdm_logging.py:145)
11:52:37.149 [I] Progress on: 432it/1.50kit rate:7.1s/it remaining:2:06:04 elapsed:54:52 postfix:- (1084:tqdm_logging.py:145)
11:52:51.316 [I] Progress on: 434it/1.50kit rate:7.1s/it remaining:2:05:53 elapsed:55:06 postfix:- (1084:tqdm_logging.py:145)
11:53:05.484 [I] Progress on: 436it/1.50kit rate:7.1s/it remaining:2:05:43 elapsed:55:20 postfix:- (1084:tqdm_logging.py:145)
11:53:19.645 [I] Progress on: 438it/1.50kit rate:7.1s/it remaining:2:05:30 elapsed:55:34 postfix:- (1084:tqdm_logging.py:145)
11:53:33.797 [I] Progress on: 440it/1.50kit rate:7.1s/it remaining:2:05:13 elapsed:55:48 postfix:- (1084:tqdm_logging.py:145)
11:53:47.949 [I] Progress on: 442it/1.50kit rate:7.1s/it remaining:2:04:58 elapsed:56:03 postfix:- (1084:tqdm_logging.py:145)
11:54:02.077 [I] Progress on: 444it/1.50kit rate:7.1s/it remaining:2:04:37 elapsed:56:17 postfix:- (1084:tqdm_logging.py:145)
11:54:16.235 [I] Progress on: 446it/1.50kit rate:7.1s/it remaining:2:04:27 elapsed:56:31 postfix:- (1084:tqdm_logging.py:145)
11:54:30.323 [I] Progress on: 448it/1.50kit rate:7.1s/it remaining:2:03:54 elapsed:56:45 postfix:- (1084:tqdm_logging.py:145)
11:54:44.424 [I] Progress on: 450it/1.50kit rate:7.1s/it remaining:2:03:31 elapsed:56:59 postfix:- (1084:tqdm_logging.py:145)


Step 450: grad_norm=0.0525, loss=0.0043, param_norm=1804.2330
11:54:58.194 [I] Progress on: 450it/1.50kit rate:7.1s/it remaining:2:03:31 elapsed:57:13 postfix:- (1084:tqdm_logging.py:145)
11:55:12.776 [I] Progress on: 454it/1.50kit rate:6.8s/it remaining:1:58:10 elapsed:57:27 postfix:- (1084:tqdm_logging.py:145)
11:55:26.897 [I] Progress on: 456it/1.50kit rate:6.9s/it remaining:2:00:26 elapsed:57:42 postfix:- (1084:tqdm_logging.py:145)
11:55:41.012 [I] Progress on: 458it/1.50kit rate:7.0s/it remaining:2:01:23 elapsed:57:56 postfix:- (1084:tqdm_logging.py:145)
11:55:55.125 [I] Progress on: 460it/1.50kit rate:7.0s/it remaining:2:01:44 elapsed:58:10 postfix:- (1084:tqdm_logging.py:145)
11:56:09.239 [I] Progress on: 462it/1.50kit rate:7.0s/it remaining:2:01:47 elapsed:58:24 postfix:- (1084:tqdm_logging.py:145)
11:56:23.352 [I] Progress on: 464it/1.50kit rate:7.0s/it remaining:2:01:41 elapsed:58:38 postfix:- (1084:tqdm_logging.py:145)
11:56:37.468 [I] Progress on: 466it/1.50kit rate:7.1s/it remaining:2:01:32 elapsed:58:52 postfix:- (1084:tqdm_logging.py:145)
11:56:51.603 [I] Progress on: 468it/1.50kit rate:7.1s/it remaining:2:01:26 elapsed:59:06 postfix:- (1084:tqdm_logging.py:145)
11:57:05.699 [I] Progress on: 470it/1.50kit rate:7.1s/it remaining:2:01:06 elapsed:59:20 postfix:- (1084:tqdm_logging.py:145)
11:57:19.801 [I] Progress on: 472it/1.50kit rate:7.1s/it remaining:2:00:50 elapsed:59:34 postfix:- (1084:tqdm_logging.py:145)
11:57:33.931 [I] Progress on: 474it/1.50kit rate:7.1s/it remaining:2:00:42 elapsed:59:49 postfix:- (1084:tqdm_logging.py:145)
11:57:48.051 [I] Progress on: 476it/1.50kit rate:7.1s/it remaining:2:00:28 elapsed:1:00:03 postfix:- (1084:tqdm_logging.py:145)
11:58:02.171 [I] Progress on: 478it/1.50kit rate:7.1s/it remaining:2:00:15 elapsed:1:00:17 postfix:- (1084:tqdm_logging.py:145)
11:58:16.289 [I] Progress on: 480it/1.50kit rate:7.1s/it remaining:2:00:00 elapsed:1:00:31 postfix:- (1084:tqdm_logging.py:145)
11:58:30.412 [I] Progress on: 482it/1.50kit rate:7.1s/it remaining:1:59:48 elapsed:1:00:45 postfix:- (1084:tqdm_logging.py:145)
11:58:44.516 [I] Progress on: 484it/1.50kit rate:7.1s/it remaining:1:59:29 elapsed:1:00:59 postfix:- (1084:tqdm_logging.py:145)
11:58:58.632 [I] Progress on: 486it/1.50kit rate:7.1s/it remaining:1:59:16 elapsed:1:01:13 postfix:- (1084:tqdm_logging.py:145)
11:59:12.755 [I] Progress on: 488it/1.50kit rate:7.1s/it remaining:1:59:04 elapsed:1:01:27 postfix:- (1084:tqdm_logging.py:145)
11:59:26.898 [I] Progress on: 490it/1.50kit rate:7.1s/it remaining:1:58:55 elapsed:1:01:42 postfix:- (1084:tqdm_logging.py:145)
11:59:40.988 [I] Progress on: 492it/1.50kit rate:7.1s/it remaining:1:58:29 elapsed:1:01:56 postfix:- (1084:tqdm_logging.py:145)
11:59:55.111 [I] Progress on: 494it/1.50kit rate:7.1s/it remaining:1:58:19 elapsed:1:02:10 postfix:- (1084:tqdm_logging.py:145)
12:00:09.233 [I] Progress on: 496it/1.50kit rate:7.1s/it remaining:1:58:07 elapsed:1:02:24 postfix:- (1084:tqdm_logging.py:145)
12:00:23.354 [I] Progress on: 498it/1.50kit rate:7.1s/it remaining:1:57:54 elapsed:1:02:38 postfix:- (1084:tqdm_logging.py:145)
12:00:37.453 [I] Progress on: 500it/1.50kit rate:7.1s/it remaining:1:57:35 elapsed:1:02:52 postfix:- (1084:tqdm_logging.py:145)


Step 500: grad_norm=0.0476, loss=0.0041, param_norm=1804.2352
12:00:51.210 [I] Progress on: 500it/1.50kit rate:7.1s/it remaining:1:57:35 elapsed:1:03:06 postfix:- (1084:tqdm_logging.py:145)
12:01:05.722 [I] Progress on: 504it/1.50kit rate:6.8s/it remaining:1:52:18 elapsed:1:03:20 postfix:- (1084:tqdm_logging.py:145)
12:01:19.849 [I] Progress on: 506it/1.50kit rate:6.9s/it remaining:1:54:36 elapsed:1:03:35 postfix:- (1084:tqdm_logging.py:145)
12:01:33.968 [I] Progress on: 508it/1.50kit rate:7.0s/it remaining:1:55:35 elapsed:1:03:49 postfix:- (1084:tqdm_logging.py:145)
12:01:48.068 [I] Progress on: 510it/1.50kit rate:7.0s/it remaining:1:55:51 elapsed:1:04:03 postfix:- (1084:tqdm_logging.py:145)
12:02:02.192 [I] Progress on: 512it/1.50kit rate:7.0s/it remaining:1:55:57 elapsed:1:04:17 postfix:- (1084:tqdm_logging.py:145)
12:02:16.299 [I] Progress on: 514it/1.50kit rate:7.0s/it remaining:1:55:48 elapsed:1:04:31 postfix:- (1084:tqdm_logging.py:145)
12:02:30.418 [I] Progress on: 516it/1.50kit rate:7.1s/it remaining:1:55:40 elapsed:1:04:45 postfix:- (1084:tqdm_logging.py:145)
12:02:44.540 [I] Progress on: 518it/1.50kit rate:7.1s/it remaining:1:55:29 elapsed:1:04:59 postfix:- (1084:tqdm_logging.py:145)
12:02:58.668 [I] Progress on: 520it/1.50kit rate:7.1s/it remaining:1:55:17 elapsed:1:05:13 postfix:- (1084:tqdm_logging.py:145)
12:03:12.811 [I] Progress on: 522it/1.50kit rate:7.1s/it remaining:1:55:09 elapsed:1:05:28 postfix:- (1084:tqdm_logging.py:145)
12:03:26.941 [I] Progress on: 524it/1.50kit rate:7.1s/it remaining:1:54:54 elapsed:1:05:42 postfix:- (1084:tqdm_logging.py:145)
12:03:41.040 [I] Progress on: 526it/1.50kit rate:7.1s/it remaining:1:54:31 elapsed:1:05:56 postfix:- (1084:tqdm_logging.py:145)
12:03:55.141 [I] Progress on: 528it/1.50kit rate:7.1s/it remaining:1:54:15 elapsed:1:06:10 postfix:- (1084:tqdm_logging.py:145)
12:04:09.285 [I] Progress on: 530it/1.50kit rate:7.1s/it remaining:1:54:12 elapsed:1:06:24 postfix:- (1084:tqdm_logging.py:145)
12:04:23.418 [I] Progress on: 532it/1.50kit rate:7.1s/it remaining:1:53:59 elapsed:1:06:38 postfix:- (1084:tqdm_logging.py:145)
12:04:37.507 [I] Progress on: 534it/1.50kit rate:7.1s/it remaining:1:53:34 elapsed:1:06:52 postfix:- (1084:tqdm_logging.py:145)
12:04:52.197 [I] Progress on: 536it/1.50kit rate:7.2s/it remaining:1:56:08 elapsed:1:07:07 postfix:- (1084:tqdm_logging.py:145)
12:05:05.766 [I] Progress on: 538it/1.50kit rate:7.0s/it remaining:1:52:43 elapsed:1:07:20 postfix:- (1084:tqdm_logging.py:145)
12:05:19.880 [I] Progress on: 540it/1.50kit rate:7.0s/it remaining:1:52:46 elapsed:1:07:35 postfix:- (1084:tqdm_logging.py:145)
12:05:33.961 [I] Progress on: 542it/1.50kit rate:7.0s/it remaining:1:52:31 elapsed:1:07:49 postfix:- (1084:tqdm_logging.py:145)
12:05:48.082 [I] Progress on: 544it/1.50kit rate:7.1s/it remaining:1:52:21 elapsed:1:08:03 postfix:- (1084:tqdm_logging.py:145)
12:06:02.191 [I] Progress on: 546it/1.50kit rate:7.1s/it remaining:1:52:09 elapsed:1:08:17 postfix:- (1084:tqdm_logging.py:145)
12:06:16.306 [I] Progress on: 548it/1.50kit rate:7.1s/it remaining:1:51:57 elapsed:1:08:31 postfix:- (1084:tqdm_logging.py:145)
12:06:30.430 [I] Progress on: 550it/1.50kit rate:7.1s/it remaining:1:51:47 elapsed:1:08:45 postfix:- (1084:tqdm_logging.py:145)


Step 550: grad_norm=0.0493, loss=0.0043, param_norm=1804.2368
12:06:44.180 [I] Progress on: 550it/1.50kit rate:7.1s/it remaining:1:51:47 elapsed:1:08:59 postfix:- (1084:tqdm_logging.py:145)
12:06:58.746 [I] Progress on: 554it/1.50kit rate:6.8s/it remaining:1:46:49 elapsed:1:09:13 postfix:- (1084:tqdm_logging.py:145)
12:07:12.855 [I] Progress on: 556it/1.50kit rate:6.9s/it remaining:1:48:51 elapsed:1:09:28 postfix:- (1084:tqdm_logging.py:145)
12:07:26.968 [I] Progress on: 558it/1.50kit rate:7.0s/it remaining:1:49:44 elapsed:1:09:42 postfix:- (1084:tqdm_logging.py:145)
12:07:41.080 [I] Progress on: 560it/1.50kit rate:7.0s/it remaining:1:50:03 elapsed:1:09:56 postfix:- (1084:tqdm_logging.py:145)
12:07:55.196 [I] Progress on: 562it/1.50kit rate:7.0s/it remaining:1:50:05 elapsed:1:10:10 postfix:- (1084:tqdm_logging.py:145)
12:08:09.311 [I] Progress on: 564it/1.50kit rate:7.0s/it remaining:1:49:57 elapsed:1:10:24 postfix:- (1084:tqdm_logging.py:145)
12:08:23.430 [I] Progress on: 566it/1.50kit rate:7.1s/it remaining:1:49:46 elapsed:1:10:38 postfix:- (1084:tqdm_logging.py:145)
12:08:37.545 [I] Progress on: 568it/1.50kit rate:7.1s/it remaining:1:49:34 elapsed:1:10:52 postfix:- (1084:tqdm_logging.py:145)
12:08:51.665 [I] Progress on: 570it/1.50kit rate:7.1s/it remaining:1:49:22 elapsed:1:11:06 postfix:- (1084:tqdm_logging.py:145)
12:09:05.772 [I] Progress on: 572it/1.50kit rate:7.1s/it remaining:1:49:07 elapsed:1:11:20 postfix:- (1084:tqdm_logging.py:145)
12:09:19.892 [I] Progress on: 574it/1.50kit rate:7.1s/it remaining:1:48:55 elapsed:1:11:35 postfix:- (1084:tqdm_logging.py:145)
12:09:34.005 [I] Progress on: 576it/1.50kit rate:7.1s/it remaining:1:48:41 elapsed:1:11:49 postfix:- (1084:tqdm_logging.py:145)
12:09:48.127 [I] Progress on: 578it/1.50kit rate:7.1s/it remaining:1:48:28 elapsed:1:12:03 postfix:- (1084:tqdm_logging.py:145)
12:10:02.244 [I] Progress on: 580it/1.50kit rate:7.1s/it remaining:1:48:14 elapsed:1:12:17 postfix:- (1084:tqdm_logging.py:145)
12:10:16.359 [I] Progress on: 582it/1.50kit rate:7.1s/it remaining:1:48:00 elapsed:1:12:31 postfix:- (1084:tqdm_logging.py:145)
12:10:30.471 [I] Progress on: 584it/1.50kit rate:7.1s/it remaining:1:47:45 elapsed:1:12:45 postfix:- (1084:tqdm_logging.py:145)
12:10:44.587 [I] Progress on: 586it/1.50kit rate:7.1s/it remaining:1:47:31 elapsed:1:12:59 postfix:- (1084:tqdm_logging.py:145)
12:10:58.707 [I] Progress on: 588it/1.50kit rate:7.1s/it remaining:1:47:18 elapsed:1:13:13 postfix:- (1084:tqdm_logging.py:145)
12:11:12.823 [I] Progress on: 590it/1.50kit rate:7.1s/it remaining:1:47:04 elapsed:1:13:28 postfix:- (1084:tqdm_logging.py:145)
12:11:26.944 [I] Progress on: 592it/1.50kit rate:7.1s/it remaining:1:46:51 elapsed:1:13:42 postfix:- (1084:tqdm_logging.py:145)
12:11:41.068 [I] Progress on: 594it/1.50kit rate:7.1s/it remaining:1:46:38 elapsed:1:13:56 postfix:- (1084:tqdm_logging.py:145)
12:11:55.159 [I] Progress on: 596it/1.50kit rate:7.1s/it remaining:1:46:16 elapsed:1:14:10 postfix:- (1084:tqdm_logging.py:145)
12:12:09.277 [I] Progress on: 598it/1.50kit rate:7.1s/it remaining:1:46:04 elapsed:1:14:24 postfix:- (1084:tqdm_logging.py:145)
12:12:23.399 [I] Progress on: 600it/1.50kit rate:7.1s/it remaining:1:45:53 elapsed:1:14:38 postfix:- (1084:tqdm_logging.py:145)


Step 600: grad_norm=0.0451, loss=0.0040, param_norm=1804.2385
12:12:37.180 [I] Progress on: 600it/1.50kit rate:7.1s/it remaining:1:45:53 elapsed:1:14:52 postfix:- (1084:tqdm_logging.py:145)
12:12:51.765 [I] Progress on: 604it/1.50kit rate:6.8s/it remaining:1:41:11 elapsed:1:15:06 postfix:- (1084:tqdm_logging.py:145)
12:13:05.827 [I] Progress on: 606it/1.50kit rate:6.9s/it remaining:1:42:53 elapsed:1:15:21 postfix:- (1084:tqdm_logging.py:145)
12:13:19.927 [I] Progress on: 608it/1.50kit rate:7.0s/it remaining:1:43:45 elapsed:1:15:35 postfix:- (1084:tqdm_logging.py:145)
12:13:34.040 [I] Progress on: 610it/1.50kit rate:7.0s/it remaining:1:44:05 elapsed:1:15:49 postfix:- (1084:tqdm_logging.py:145)
12:13:48.143 [I] Progress on: 612it/1.50kit rate:7.0s/it remaining:1:44:05 elapsed:1:16:03 postfix:- (1084:tqdm_logging.py:145)
12:14:02.261 [I] Progress on: 614it/1.50kit rate:7.0s/it remaining:1:44:03 elapsed:1:16:17 postfix:- (1084:tqdm_logging.py:145)
12:14:16.379 [I] Progress on: 616it/1.50kit rate:7.1s/it remaining:1:43:55 elapsed:1:16:31 postfix:- (1084:tqdm_logging.py:145)
12:14:30.481 [I] Progress on: 618it/1.50kit rate:7.1s/it remaining:1:43:40 elapsed:1:16:45 postfix:- (1084:tqdm_logging.py:145)
12:14:44.616 [I] Progress on: 620it/1.50kit rate:7.1s/it remaining:1:43:32 elapsed:1:16:59 postfix:- (1084:tqdm_logging.py:145)
12:14:58.719 [I] Progress on: 622it/1.50kit rate:7.1s/it remaining:1:43:15 elapsed:1:17:13 postfix:- (1084:tqdm_logging.py:145)
12:15:13.232 [I] Progress on: 624it/1.50kit rate:7.1s/it remaining:1:44:16 elapsed:1:17:28 postfix:- (1084:tqdm_logging.py:145)
12:15:27.331 [I] Progress on: 626it/1.50kit rate:7.1s/it remaining:1:43:22 elapsed:1:17:42 postfix:- (1084:tqdm_logging.py:145)
12:15:41.450 [I] Progress on: 628it/1.50kit rate:7.1s/it remaining:1:42:51 elapsed:1:17:56 postfix:- (1084:tqdm_logging.py:145)
12:15:55.537 [I] Progress on: 630it/1.50kit rate:7.1s/it remaining:1:42:21 elapsed:1:18:10 postfix:- (1084:tqdm_logging.py:145)
12:16:09.638 [I] Progress on: 632it/1.50kit rate:7.1s/it remaining:1:42:01 elapsed:1:18:24 postfix:- (1084:tqdm_logging.py:145)
12:16:23.733 [I] Progress on: 634it/1.50kit rate:7.0s/it remaining:1:41:44 elapsed:1:18:38 postfix:- (1084:tqdm_logging.py:145)
12:16:37.846 [I] Progress on: 636it/1.50kit rate:7.1s/it remaining:1:41:32 elapsed:1:18:53 postfix:- (1084:tqdm_logging.py:145)
12:16:51.950 [I] Progress on: 638it/1.50kit rate:7.1s/it remaining:1:41:19 elapsed:1:19:07 postfix:- (1084:tqdm_logging.py:145)
12:17:06.064 [I] Progress on: 640it/1.50kit rate:7.1s/it remaining:1:41:07 elapsed:1:19:21 postfix:- (1084:tqdm_logging.py:145)
12:17:20.150 [I] Progress on: 642it/1.50kit rate:7.0s/it remaining:1:40:47 elapsed:1:19:35 postfix:- (1084:tqdm_logging.py:145)
12:17:34.259 [I] Progress on: 644it/1.50kit rate:7.1s/it remaining:1:40:37 elapsed:1:19:49 postfix:- (1084:tqdm_logging.py:145)
12:17:48.367 [I] Progress on: 646it/1.50kit rate:7.1s/it remaining:1:40:23 elapsed:1:20:03 postfix:- (1084:tqdm_logging.py:145)
12:18:02.484 [I] Progress on: 648it/1.50kit rate:7.1s/it remaining:1:40:11 elapsed:1:20:17 postfix:- (1084:tqdm_logging.py:145)
12:18:16.602 [I] Progress on: 650it/1.50kit rate:7.1s/it remaining:1:39:59 elapsed:1:20:31 postfix:- (1084:tqdm_logging.py:145)


Step 650: grad_norm=0.0486, loss=0.0039, param_norm=1804.2402
12:18:30.363 [I] Progress on: 650it/1.50kit rate:7.1s/it remaining:1:39:59 elapsed:1:20:45 postfix:- (1084:tqdm_logging.py:145)
12:18:44.901 [I] Progress on: 654it/1.50kit rate:6.8s/it remaining:1:35:26 elapsed:1:21:00 postfix:- (1084:tqdm_logging.py:145)
12:18:59.010 [I] Progress on: 656it/1.50kit rate:6.9s/it remaining:1:37:15 elapsed:1:21:14 postfix:- (1084:tqdm_logging.py:145)
12:19:13.132 [I] Progress on: 658it/1.50kit rate:7.0s/it remaining:1:38:05 elapsed:1:21:28 postfix:- (1084:tqdm_logging.py:145)
12:19:27.245 [I] Progress on: 660it/1.50kit rate:7.0s/it remaining:1:38:20 elapsed:1:21:42 postfix:- (1084:tqdm_logging.py:145)
12:19:41.353 [I] Progress on: 662it/1.50kit rate:7.0s/it remaining:1:38:19 elapsed:1:21:56 postfix:- (1084:tqdm_logging.py:145)
12:19:55.472 [I] Progress on: 664it/1.50kit rate:7.1s/it remaining:1:38:14 elapsed:1:22:10 postfix:- (1084:tqdm_logging.py:145)
12:20:09.590 [I] Progress on: 666it/1.50kit rate:7.1s/it remaining:1:38:03 elapsed:1:22:24 postfix:- (1084:tqdm_logging.py:145)
12:20:23.698 [I] Progress on: 668it/1.50kit rate:7.1s/it remaining:1:37:49 elapsed:1:22:38 postfix:- (1084:tqdm_logging.py:145)
12:20:38.290 [I] Progress on: 670it/1.50kit rate:7.2s/it remaining:1:39:36 elapsed:1:22:53 postfix:- (1084:tqdm_logging.py:145)
12:20:51.917 [I] Progress on: 672it/1.50kit rate:7.0s/it remaining:1:36:57 elapsed:1:23:07 postfix:- (1084:tqdm_logging.py:145)
12:21:06.021 [I] Progress on: 674it/1.50kit rate:7.0s/it remaining:1:36:53 elapsed:1:23:21 postfix:- (1084:tqdm_logging.py:145)
12:21:20.131 [I] Progress on: 676it/1.50kit rate:7.0s/it remaining:1:36:46 elapsed:1:23:35 postfix:- (1084:tqdm_logging.py:145)
12:21:34.245 [I] Progress on: 678it/1.50kit rate:7.1s/it remaining:1:36:37 elapsed:1:23:49 postfix:- (1084:tqdm_logging.py:145)
12:21:48.355 [I] Progress on: 680it/1.50kit rate:7.1s/it remaining:1:36:24 elapsed:1:24:03 postfix:- (1084:tqdm_logging.py:145)
12:22:02.432 [I] Progress on: 682it/1.50kit rate:7.0s/it remaining:1:36:03 elapsed:1:24:17 postfix:- (1084:tqdm_logging.py:145)
12:22:16.545 [I] Progress on: 684it/1.50kit rate:7.1s/it remaining:1:35:53 elapsed:1:24:31 postfix:- (1084:tqdm_logging.py:145)
12:22:30.666 [I] Progress on: 686it/1.50kit rate:7.1s/it remaining:1:35:43 elapsed:1:24:45 postfix:- (1084:tqdm_logging.py:145)
12:22:44.782 [I] Progress on: 688it/1.50kit rate:7.1s/it remaining:1:35:29 elapsed:1:24:59 postfix:- (1084:tqdm_logging.py:145)
12:22:58.896 [I] Progress on: 690it/1.50kit rate:7.1s/it remaining:1:35:16 elapsed:1:25:14 postfix:- (1084:tqdm_logging.py:145)
12:23:13.002 [I] Progress on: 692it/1.50kit rate:7.1s/it remaining:1:35:01 elapsed:1:25:28 postfix:- (1084:tqdm_logging.py:145)
12:23:27.147 [I] Progress on: 694it/1.50kit rate:7.1s/it remaining:1:34:47 elapsed:1:25:42 postfix:- (1084:tqdm_logging.py:145)
12:23:41.285 [I] Progress on: 696it/1.50kit rate:7.1s/it remaining:1:34:40 elapsed:1:25:56 postfix:- (1084:tqdm_logging.py:145)
12:23:55.372 [I] Progress on: 698it/1.50kit rate:7.1s/it remaining:1:34:17 elapsed:1:26:10 postfix:- (1084:tqdm_logging.py:145)
12:24:09.470 [I] Progress on: 700it/1.50kit rate:7.1s/it remaining:1:34:01 elapsed:1:26:24 postfix:- (1084:tqdm_logging.py:145)


Step 700: grad_norm=0.0511, loss=0.0039, param_norm=1804.2418
12:24:23.237 [I] Progress on: 700it/1.50kit rate:7.1s/it remaining:1:34:01 elapsed:1:26:38 postfix:- (1084:tqdm_logging.py:145)
12:24:37.807 [I] Progress on: 704it/1.50kit rate:6.8s/it remaining:1:29:51 elapsed:1:26:52 postfix:- (1084:tqdm_logging.py:145)
12:24:51.952 [I] Progress on: 706it/1.50kit rate:6.9s/it remaining:1:31:40 elapsed:1:27:07 postfix:- (1084:tqdm_logging.py:145)
12:25:06.039 [I] Progress on: 708it/1.50kit rate:7.0s/it remaining:1:32:14 elapsed:1:27:21 postfix:- (1084:tqdm_logging.py:145)
12:25:20.161 [I] Progress on: 710it/1.50kit rate:7.0s/it remaining:1:32:30 elapsed:1:27:35 postfix:- (1084:tqdm_logging.py:145)
12:25:34.263 [I] Progress on: 712it/1.50kit rate:7.0s/it remaining:1:32:26 elapsed:1:27:49 postfix:- (1084:tqdm_logging.py:145)
12:25:48.373 [I] Progress on: 714it/1.50kit rate:7.0s/it remaining:1:32:18 elapsed:1:28:03 postfix:- (1084:tqdm_logging.py:145)
12:26:02.503 [I] Progress on: 716it/1.50kit rate:7.1s/it remaining:1:32:14 elapsed:1:28:17 postfix:- (1084:tqdm_logging.py:145)
12:26:16.598 [I] Progress on: 718it/1.50kit rate:7.1s/it remaining:1:31:55 elapsed:1:28:31 postfix:- (1084:tqdm_logging.py:145)
12:26:30.705 [I] Progress on: 720it/1.50kit rate:7.1s/it remaining:1:31:42 elapsed:1:28:45 postfix:- (1084:tqdm_logging.py:145)
12:26:44.821 [I] Progress on: 722it/1.50kit rate:7.1s/it remaining:1:31:29 elapsed:1:29:00 postfix:- (1084:tqdm_logging.py:145)
12:26:58.945 [I] Progress on: 724it/1.50kit rate:7.1s/it remaining:1:31:16 elapsed:1:29:14 postfix:- (1084:tqdm_logging.py:145)
12:27:13.058 [I] Progress on: 726it/1.50kit rate:7.1s/it remaining:1:31:01 elapsed:1:29:28 postfix:- (1084:tqdm_logging.py:145)
12:27:27.172 [I] Progress on: 728it/1.50kit rate:7.1s/it remaining:1:30:46 elapsed:1:29:42 postfix:- (1084:tqdm_logging.py:145)
12:27:41.268 [I] Progress on: 730it/1.50kit rate:7.1s/it remaining:1:30:30 elapsed:1:29:56 postfix:- (1084:tqdm_logging.py:145)
12:27:55.390 [I] Progress on: 732it/1.50kit rate:7.1s/it remaining:1:30:20 elapsed:1:30:10 postfix:- (1084:tqdm_logging.py:145)
12:28:09.497 [I] Progress on: 734it/1.50kit rate:7.1s/it remaining:1:30:05 elapsed:1:30:24 postfix:- (1084:tqdm_logging.py:145)
12:28:23.597 [I] Progress on: 736it/1.50kit rate:7.1s/it remaining:1:29:49 elapsed:1:30:38 postfix:- (1084:tqdm_logging.py:145)
12:28:37.722 [I] Progress on: 738it/1.50kit rate:7.1s/it remaining:1:29:39 elapsed:1:30:52 postfix:- (1084:tqdm_logging.py:145)
12:28:51.831 [I] Progress on: 740it/1.50kit rate:7.1s/it remaining:1:29:23 elapsed:1:31:07 postfix:- (1084:tqdm_logging.py:145)
12:29:05.936 [I] Progress on: 742it/1.50kit rate:7.1s/it remaining:1:29:08 elapsed:1:31:21 postfix:- (1084:tqdm_logging.py:145)
12:29:20.020 [I] Progress on: 744it/1.50kit rate:7.0s/it remaining:1:28:48 elapsed:1:31:35 postfix:- (1084:tqdm_logging.py:145)
12:29:34.138 [I] Progress on: 746it/1.50kit rate:7.1s/it remaining:1:28:38 elapsed:1:31:49 postfix:- (1084:tqdm_logging.py:145)
12:29:48.279 [I] Progress on: 748it/1.50kit rate:7.1s/it remaining:1:28:31 elapsed:1:32:03 postfix:- (1084:tqdm_logging.py:145)
12:30:02.386 [I] Progress on: 750it/1.50kit rate:7.1s/it remaining:1:28:14 elapsed:1:32:17 postfix:- (1084:tqdm_logging.py:145)


Step 750: grad_norm=0.0501, loss=0.0036, param_norm=1804.2430
12:30:16.178 [I] Progress on: 750it/1.50kit rate:7.1s/it remaining:1:28:14 elapsed:1:32:31 postfix:- (1084:tqdm_logging.py:145)
12:30:30.753 [I] Progress on: 754it/1.50kit rate:6.8s/it remaining:1:24:14 elapsed:1:32:45 postfix:- (1084:tqdm_logging.py:145)
12:30:44.839 [I] Progress on: 756it/1.50kit rate:6.9s/it remaining:1:25:42 elapsed:1:33:00 postfix:- (1084:tqdm_logging.py:145)
12:30:58.959 [I] Progress on: 758it/1.50kit rate:7.0s/it remaining:1:26:24 elapsed:1:33:14 postfix:- (1084:tqdm_logging.py:145)
12:31:13.074 [I] Progress on: 760it/1.50kit rate:7.0s/it remaining:1:26:36 elapsed:1:33:28 postfix:- (1084:tqdm_logging.py:145)
12:31:27.169 [I] Progress on: 762it/1.50kit rate:7.0s/it remaining:1:26:31 elapsed:1:33:42 postfix:- (1084:tqdm_logging.py:145)
12:31:41.277 [I] Progress on: 764it/1.50kit rate:7.0s/it remaining:1:26:24 elapsed:1:33:56 postfix:- (1084:tqdm_logging.py:145)
12:31:55.374 [I] Progress on: 766it/1.50kit rate:7.0s/it remaining:1:26:11 elapsed:1:34:10 postfix:- (1084:tqdm_logging.py:145)
12:32:09.489 [I] Progress on: 768it/1.50kit rate:7.1s/it remaining:1:26:01 elapsed:1:34:24 postfix:- (1084:tqdm_logging.py:145)
12:32:23.612 [I] Progress on: 770it/1.50kit rate:7.1s/it remaining:1:25:51 elapsed:1:34:38 postfix:- (1084:tqdm_logging.py:145)
12:32:37.714 [I] Progress on: 772it/1.50kit rate:7.1s/it remaining:1:25:36 elapsed:1:34:52 postfix:- (1084:tqdm_logging.py:145)
12:32:51.817 [I] Progress on: 774it/1.50kit rate:7.1s/it remaining:1:25:22 elapsed:1:35:07 postfix:- (1084:tqdm_logging.py:145)
12:33:05.927 [I] Progress on: 776it/1.50kit rate:7.1s/it remaining:1:25:08 elapsed:1:35:21 postfix:- (1084:tqdm_logging.py:145)
12:33:20.040 [I] Progress on: 778it/1.50kit rate:7.1s/it remaining:1:24:54 elapsed:1:35:35 postfix:- (1084:tqdm_logging.py:145)
12:33:34.139 [I] Progress on: 780it/1.50kit rate:7.1s/it remaining:1:24:37 elapsed:1:35:49 postfix:- (1084:tqdm_logging.py:145)
12:33:48.268 [I] Progress on: 782it/1.50kit rate:7.1s/it remaining:1:24:27 elapsed:1:36:03 postfix:- (1084:tqdm_logging.py:145)
12:34:02.383 [I] Progress on: 784it/1.50kit rate:7.1s/it remaining:1:24:13 elapsed:1:36:17 postfix:- (1084:tqdm_logging.py:145)
12:34:16.481 [I] Progress on: 786it/1.50kit rate:7.1s/it remaining:1:23:56 elapsed:1:36:31 postfix:- (1084:tqdm_logging.py:145)
12:34:30.587 [I] Progress on: 788it/1.50kit rate:7.1s/it remaining:1:23:42 elapsed:1:36:45 postfix:- (1084:tqdm_logging.py:145)
12:34:44.716 [I] Progress on: 790it/1.50kit rate:7.1s/it remaining:1:23:32 elapsed:1:36:59 postfix:- (1084:tqdm_logging.py:145)
12:34:58.811 [I] Progress on: 792it/1.50kit rate:7.1s/it remaining:1:23:14 elapsed:1:37:14 postfix:- (1084:tqdm_logging.py:145)
12:35:12.942 [I] Progress on: 794it/1.50kit rate:7.1s/it remaining:1:23:04 elapsed:1:37:28 postfix:- (1084:tqdm_logging.py:145)
12:35:27.039 [I] Progress on: 796it/1.50kit rate:7.1s/it remaining:1:22:46 elapsed:1:37:42 postfix:- (1084:tqdm_logging.py:145)
12:35:41.151 [I] Progress on: 798it/1.50kit rate:7.1s/it remaining:1:22:32 elapsed:1:37:56 postfix:- (1084:tqdm_logging.py:145)
12:35:55.282 [I] Progress on: 800it/1.50kit rate:7.1s/it remaining:1:22:22 elapsed:1:38:10 postfix:- (1084:tqdm_logging.py:145)


Step 800: grad_norm=0.0439, loss=0.0034, param_norm=1804.2444
12:36:09.001 [I] Progress on: 800it/1.50kit rate:7.1s/it remaining:1:22:22 elapsed:1:38:24 postfix:- (1084:tqdm_logging.py:145)
12:36:24.084 [I] Progress on: 804it/1.50kit rate:6.9s/it remaining:1:20:24 elapsed:1:38:39 postfix:- (1084:tqdm_logging.py:145)
12:36:37.632 [I] Progress on: 806it/1.50kit rate:6.9s/it remaining:1:19:31 elapsed:1:38:52 postfix:- (1084:tqdm_logging.py:145)
12:36:51.762 [I] Progress on: 808it/1.50kit rate:7.0s/it remaining:1:20:24 elapsed:1:39:06 postfix:- (1084:tqdm_logging.py:145)
12:37:05.870 [I] Progress on: 810it/1.50kit rate:7.0s/it remaining:1:20:39 elapsed:1:39:21 postfix:- (1084:tqdm_logging.py:145)
12:37:19.979 [I] Progress on: 812it/1.50kit rate:7.0s/it remaining:1:20:40 elapsed:1:39:35 postfix:- (1084:tqdm_logging.py:145)
12:37:34.104 [I] Progress on: 814it/1.50kit rate:7.0s/it remaining:1:20:35 elapsed:1:39:49 postfix:- (1084:tqdm_logging.py:145)
12:37:48.219 [I] Progress on: 816it/1.50kit rate:7.1s/it remaining:1:20:24 elapsed:1:40:03 postfix:- (1084:tqdm_logging.py:145)
12:38:02.334 [I] Progress on: 818it/1.50kit rate:7.1s/it remaining:1:20:12 elapsed:1:40:17 postfix:- (1084:tqdm_logging.py:145)
12:38:16.453 [I] Progress on: 820it/1.50kit rate:7.1s/it remaining:1:20:00 elapsed:1:40:31 postfix:- (1084:tqdm_logging.py:145)
12:38:30.544 [I] Progress on: 822it/1.50kit rate:7.1s/it remaining:1:19:41 elapsed:1:40:45 postfix:- (1084:tqdm_logging.py:145)
12:38:44.659 [I] Progress on: 824it/1.50kit rate:7.1s/it remaining:1:19:28 elapsed:1:40:59 postfix:- (1084:tqdm_logging.py:145)
12:38:58.768 [I] Progress on: 826it/1.50kit rate:7.1s/it remaining:1:19:14 elapsed:1:41:13 postfix:- (1084:tqdm_logging.py:145)
12:39:12.875 [I] Progress on: 828it/1.50kit rate:7.1s/it remaining:1:18:59 elapsed:1:41:28 postfix:- (1084:tqdm_logging.py:145)
12:39:27.003 [I] Progress on: 830it/1.50kit rate:7.1s/it remaining:1:18:49 elapsed:1:41:42 postfix:- (1084:tqdm_logging.py:145)
12:39:41.122 [I] Progress on: 832it/1.50kit rate:7.1s/it remaining:1:18:32 elapsed:1:41:56 postfix:- (1084:tqdm_logging.py:145)
12:39:55.237 [I] Progress on: 834it/1.50kit rate:7.1s/it remaining:1:18:21 elapsed:1:42:10 postfix:- (1084:tqdm_logging.py:145)
12:40:09.330 [I] Progress on: 836it/1.50kit rate:7.1s/it remaining:1:18:04 elapsed:1:42:24 postfix:- (1084:tqdm_logging.py:145)
12:40:23.427 [I] Progress on: 838it/1.50kit rate:7.1s/it remaining:1:17:47 elapsed:1:42:38 postfix:- (1084:tqdm_logging.py:145)
12:40:37.528 [I] Progress on: 840it/1.50kit rate:7.1s/it remaining:1:17:33 elapsed:1:42:52 postfix:- (1084:tqdm_logging.py:145)
12:40:51.615 [I] Progress on: 842it/1.50kit rate:7.0s/it remaining:1:17:17 elapsed:1:43:06 postfix:- (1084:tqdm_logging.py:145)
12:41:05.735 [I] Progress on: 844it/1.50kit rate:7.1s/it remaining:1:17:07 elapsed:1:43:20 postfix:- (1084:tqdm_logging.py:145)
12:41:19.839 [I] Progress on: 846it/1.50kit rate:7.1s/it remaining:1:16:53 elapsed:1:43:35 postfix:- (1084:tqdm_logging.py:145)
12:41:33.948 [I] Progress on: 848it/1.50kit rate:7.1s/it remaining:1:16:39 elapsed:1:43:49 postfix:- (1084:tqdm_logging.py:145)
12:41:48.045 [I] Progress on: 850it/1.50kit rate:7.1s/it remaining:1:16:23 elapsed:1:44:03 postfix:- (1084:tqdm_logging.py:145)


Step 850: grad_norm=0.0455, loss=0.0033, param_norm=1804.2455
12:42:01.820 [I] Progress on: 850it/1.50kit rate:7.1s/it remaining:1:16:23 elapsed:1:44:17 postfix:- (1084:tqdm_logging.py:145)
12:42:16.367 [I] Progress on: 854it/1.50kit rate:6.8s/it remaining:1:12:55 elapsed:1:44:31 postfix:- (1084:tqdm_logging.py:145)
12:42:30.470 [I] Progress on: 856it/1.50kit rate:6.9s/it remaining:1:14:13 elapsed:1:44:45 postfix:- (1084:tqdm_logging.py:145)
12:42:44.582 [I] Progress on: 858it/1.50kit rate:7.0s/it remaining:1:14:46 elapsed:1:44:59 postfix:- (1084:tqdm_logging.py:145)
12:42:58.675 [I] Progress on: 860it/1.50kit rate:7.0s/it remaining:1:14:51 elapsed:1:45:13 postfix:- (1084:tqdm_logging.py:145)
12:43:12.818 [I] Progress on: 862it/1.50kit rate:7.0s/it remaining:1:14:55 elapsed:1:45:28 postfix:- (1084:tqdm_logging.py:145)
12:43:26.912 [I] Progress on: 864it/1.50kit rate:7.0s/it remaining:1:14:42 elapsed:1:45:42 postfix:- (1084:tqdm_logging.py:145)
12:43:41.004 [I] Progress on: 866it/1.50kit rate:7.0s/it remaining:1:14:28 elapsed:1:45:56 postfix:- (1084:tqdm_logging.py:145)
12:43:55.104 [I] Progress on: 868it/1.50kit rate:7.0s/it remaining:1:14:15 elapsed:1:46:10 postfix:- (1084:tqdm_logging.py:145)
12:44:09.243 [I] Progress on: 870it/1.50kit rate:7.1s/it remaining:1:14:07 elapsed:1:46:24 postfix:- (1084:tqdm_logging.py:145)
12:44:23.377 [I] Progress on: 872it/1.50kit rate:7.1s/it remaining:1:13:56 elapsed:1:46:38 postfix:- (1084:tqdm_logging.py:145)
12:44:37.456 [I] Progress on: 874it/1.50kit rate:7.1s/it remaining:1:13:35 elapsed:1:46:52 postfix:- (1084:tqdm_logging.py:145)
12:44:51.561 [I] Progress on: 876it/1.50kit rate:7.1s/it remaining:1:13:21 elapsed:1:47:06 postfix:- (1084:tqdm_logging.py:145)
12:45:05.664 [I] Progress on: 878it/1.50kit rate:7.1s/it remaining:1:13:08 elapsed:1:47:20 postfix:- (1084:tqdm_logging.py:145)
12:45:19.771 [I] Progress on: 880it/1.50kit rate:7.1s/it remaining:1:12:53 elapsed:1:47:34 postfix:- (1084:tqdm_logging.py:145)
12:45:33.908 [I] Progress on: 882it/1.50kit rate:7.1s/it remaining:1:12:44 elapsed:1:47:49 postfix:- (1084:tqdm_logging.py:145)
12:45:48.011 [I] Progress on: 884it/1.50kit rate:7.1s/it remaining:1:12:28 elapsed:1:48:03 postfix:- (1084:tqdm_logging.py:145)
12:46:02.075 [I] Progress on: 886it/1.50kit rate:7.0s/it remaining:1:12:05 elapsed:1:48:17 postfix:- (1084:tqdm_logging.py:145)
12:46:16.208 [I] Progress on: 888it/1.50kit rate:7.1s/it remaining:1:11:58 elapsed:1:48:31 postfix:- (1084:tqdm_logging.py:145)
12:46:30.309 [I] Progress on: 890it/1.50kit rate:7.1s/it remaining:1:11:42 elapsed:1:48:45 postfix:- (1084:tqdm_logging.py:145)
12:46:44.449 [I] Progress on: 892it/1.50kit rate:7.1s/it remaining:1:11:33 elapsed:1:48:59 postfix:- (1084:tqdm_logging.py:145)
12:46:58.526 [I] Progress on: 894it/1.50kit rate:7.1s/it remaining:1:11:12 elapsed:1:49:13 postfix:- (1084:tqdm_logging.py:145)
12:47:12.636 [I] Progress on: 896it/1.50kit rate:7.1s/it remaining:1:10:59 elapsed:1:49:27 postfix:- (1084:tqdm_logging.py:145)
12:47:26.747 [I] Progress on: 898it/1.50kit rate:7.1s/it remaining:1:10:45 elapsed:1:49:41 postfix:- (1084:tqdm_logging.py:145)
12:47:40.848 [I] Progress on: 900it/1.50kit rate:7.1s/it remaining:1:10:30 elapsed:1:49:56 postfix:- (1084:tqdm_logging.py:145)


Step 900: grad_norm=0.0463, loss=0.0035, param_norm=1804.2463
12:47:54.648 [I] Progress on: 900it/1.50kit rate:7.1s/it remaining:1:10:30 elapsed:1:50:09 postfix:- (1084:tqdm_logging.py:145)
12:48:09.250 [I] Progress on: 904it/1.50kit rate:6.8s/it remaining:1:07:20 elapsed:1:50:24 postfix:- (1084:tqdm_logging.py:145)
12:48:23.341 [I] Progress on: 906it/1.50kit rate:6.9s/it remaining:1:08:27 elapsed:1:50:38 postfix:- (1084:tqdm_logging.py:145)
12:48:37.474 [I] Progress on: 908it/1.50kit rate:7.0s/it remaining:1:08:57 elapsed:1:50:52 postfix:- (1084:tqdm_logging.py:145)
12:48:51.558 [I] Progress on: 910it/1.50kit rate:7.0s/it remaining:1:08:59 elapsed:1:51:06 postfix:- (1084:tqdm_logging.py:145)
12:49:05.665 [I] Progress on: 912it/1.50kit rate:7.0s/it remaining:1:08:56 elapsed:1:51:20 postfix:- (1084:tqdm_logging.py:145)
12:49:19.784 [I] Progress on: 914it/1.50kit rate:7.0s/it remaining:1:08:49 elapsed:1:51:34 postfix:- (1084:tqdm_logging.py:145)
12:49:33.896 [I] Progress on: 916it/1.50kit rate:7.1s/it remaining:1:08:38 elapsed:1:51:49 postfix:- (1084:tqdm_logging.py:145)
12:49:48.011 [I] Progress on: 918it/1.50kit rate:7.1s/it remaining:1:08:26 elapsed:1:52:03 postfix:- (1084:tqdm_logging.py:145)
12:50:02.103 [I] Progress on: 920it/1.50kit rate:7.1s/it remaining:1:08:10 elapsed:1:52:17 postfix:- (1084:tqdm_logging.py:145)
12:50:16.218 [I] Progress on: 922it/1.50kit rate:7.1s/it remaining:1:07:57 elapsed:1:52:31 postfix:- (1084:tqdm_logging.py:145)
12:50:30.320 [I] Progress on: 924it/1.50kit rate:7.1s/it remaining:1:07:43 elapsed:1:52:45 postfix:- (1084:tqdm_logging.py:145)
12:50:44.419 [I] Progress on: 926it/1.50kit rate:7.1s/it remaining:1:07:27 elapsed:1:52:59 postfix:- (1084:tqdm_logging.py:145)
12:50:58.509 [I] Progress on: 928it/1.50kit rate:7.0s/it remaining:1:07:10 elapsed:1:53:13 postfix:- (1084:tqdm_logging.py:145)
12:51:12.619 [I] Progress on: 930it/1.50kit rate:7.0s/it remaining:1:06:58 elapsed:1:53:27 postfix:- (1084:tqdm_logging.py:145)
12:51:26.708 [I] Progress on: 932it/1.50kit rate:7.0s/it remaining:1:06:42 elapsed:1:53:41 postfix:- (1084:tqdm_logging.py:145)
12:51:40.828 [I] Progress on: 934it/1.50kit rate:7.1s/it remaining:1:06:31 elapsed:1:53:56 postfix:- (1084:tqdm_logging.py:145)
12:51:54.937 [I] Progress on: 936it/1.50kit rate:7.1s/it remaining:1:06:17 elapsed:1:54:10 postfix:- (1084:tqdm_logging.py:145)
12:52:09.658 [I] Progress on: 938it/1.50kit rate:7.2s/it remaining:1:07:47 elapsed:1:54:24 postfix:- (1084:tqdm_logging.py:145)
12:52:23.190 [I] Progress on: 940it/1.50kit rate:7.0s/it remaining:1:05:34 elapsed:1:54:38 postfix:- (1084:tqdm_logging.py:145)
12:52:37.278 [I] Progress on: 942it/1.50kit rate:7.0s/it remaining:1:05:28 elapsed:1:54:52 postfix:- (1084:tqdm_logging.py:145)
12:52:51.396 [I] Progress on: 944it/1.50kit rate:7.1s/it remaining:1:05:22 elapsed:1:55:06 postfix:- (1084:tqdm_logging.py:145)
12:53:05.506 [I] Progress on: 946it/1.50kit rate:7.1s/it remaining:1:05:10 elapsed:1:55:20 postfix:- (1084:tqdm_logging.py:145)
12:53:19.617 [I] Progress on: 948it/1.50kit rate:7.1s/it remaining:1:04:57 elapsed:1:55:34 postfix:- (1084:tqdm_logging.py:145)
12:53:33.736 [I] Progress on: 950it/1.50kit rate:7.1s/it remaining:1:04:44 elapsed:1:55:48 postfix:- (1084:tqdm_logging.py:145)


Step 950: grad_norm=0.0417, loss=0.0033, param_norm=1804.2474
12:53:47.472 [I] Progress on: 950it/1.50kit rate:7.1s/it remaining:1:04:44 elapsed:1:56:02 postfix:- (1084:tqdm_logging.py:145)
12:54:01.971 [I] Progress on: 954it/1.50kit rate:6.8s/it remaining:1:01:33 elapsed:1:56:17 postfix:- (1084:tqdm_logging.py:145)
12:54:16.081 [I] Progress on: 956it/1.50kit rate:6.9s/it remaining:1:02:40 elapsed:1:56:31 postfix:- (1084:tqdm_logging.py:145)
12:54:30.184 [I] Progress on: 958it/1.50kit rate:7.0s/it remaining:1:03:04 elapsed:1:56:45 postfix:- (1084:tqdm_logging.py:145)
12:54:44.317 [I] Progress on: 960it/1.50kit rate:7.0s/it remaining:1:03:13 elapsed:1:56:59 postfix:- (1084:tqdm_logging.py:145)
12:54:58.443 [I] Progress on: 962it/1.50kit rate:7.0s/it remaining:1:03:08 elapsed:1:57:13 postfix:- (1084:tqdm_logging.py:145)
12:55:12.567 [I] Progress on: 964it/1.50kit rate:7.1s/it remaining:1:03:00 elapsed:1:57:27 postfix:- (1084:tqdm_logging.py:145)
12:55:26.670 [I] Progress on: 966it/1.50kit rate:7.1s/it remaining:1:02:46 elapsed:1:57:41 postfix:- (1084:tqdm_logging.py:145)
12:55:40.794 [I] Progress on: 968it/1.50kit rate:7.1s/it remaining:1:02:35 elapsed:1:57:55 postfix:- (1084:tqdm_logging.py:145)
12:55:54.887 [I] Progress on: 970it/1.50kit rate:7.1s/it remaining:1:02:18 elapsed:1:58:10 postfix:- (1084:tqdm_logging.py:145)
12:56:08.995 [I] Progress on: 972it/1.50kit rate:7.1s/it remaining:1:02:04 elapsed:1:58:24 postfix:- (1084:tqdm_logging.py:145)
12:56:23.145 [I] Progress on: 974it/1.50kit rate:7.1s/it remaining:1:01:55 elapsed:1:58:38 postfix:- (1084:tqdm_logging.py:145)
12:56:37.211 [I] Progress on: 976it/1.50kit rate:7.1s/it remaining:1:01:34 elapsed:1:58:52 postfix:- (1084:tqdm_logging.py:145)
12:56:51.306 [I] Progress on: 978it/1.50kit rate:7.0s/it remaining:1:01:19 elapsed:1:59:06 postfix:- (1084:tqdm_logging.py:145)
12:57:05.420 [I] Progress on: 980it/1.50kit rate:7.1s/it remaining:1:01:07 elapsed:1:59:20 postfix:- (1084:tqdm_logging.py:145)
12:57:19.518 [I] Progress on: 982it/1.50kit rate:7.1s/it remaining:1:00:53 elapsed:1:59:34 postfix:- (1084:tqdm_logging.py:145)
12:57:33.629 [I] Progress on: 984it/1.50kit rate:7.1s/it remaining:1:00:39 elapsed:1:59:48 postfix:- (1084:tqdm_logging.py:145)
12:57:47.713 [I] Progress on: 986it/1.50kit rate:7.0s/it remaining:1:00:22 elapsed:2:00:02 postfix:- (1084:tqdm_logging.py:145)
12:58:01.829 [I] Progress on: 988it/1.50kit rate:7.1s/it remaining:1:00:11 elapsed:2:00:17 postfix:- (1084:tqdm_logging.py:145)
12:58:15.976 [I] Progress on: 990it/1.50kit rate:7.1s/it remaining:1:00:02 elapsed:2:00:31 postfix:- (1084:tqdm_logging.py:145)
12:58:30.054 [I] Progress on: 992it/1.50kit rate:7.0s/it remaining:59:41 elapsed:2:00:45 postfix:- (1084:tqdm_logging.py:145)
12:58:44.153 [I] Progress on: 994it/1.50kit rate:7.0s/it remaining:59:27 elapsed:2:00:59 postfix:- (1084:tqdm_logging.py:145)
12:58:58.273 [I] Progress on: 996it/1.50kit rate:7.1s/it remaining:59:16 elapsed:2:01:13 postfix:- (1084:tqdm_logging.py:145)
12:59:12.374 [I] Progress on: 998it/1.50kit rate:7.1s/it remaining:59:01 elapsed:2:01:27 postfix:- (1084:tqdm_logging.py:145)
12:59:26.473 [I] Progress on: 1.00kit/1.50kit rate:7.1s/it remaining:58:46 elapsed:2:01:41 postfix:- (1084:tqdm_logging.py:145)


Step 1000: grad_norm=0.0437, loss=0.0034, param_norm=1804.2480
12:59:40.264 [I] Progress on: 1.00kit/1.50kit rate:7.1s/it remaining:58:46 elapsed:2:01:55 postfix:- (1084:tqdm_logging.py:145)
12:59:40.310 [I] [process=0][thread=MainThread][wait_until_finished] No Save Finalize thread to wait for. Returning. (1084:checkpoint_manager.py:1987)
12:59:40.311 [I] [process=0] Saving checkpoint at step 1000                                       (1084:checkpoint_manager.py:1408)
12:59:40.312 [I] [process=0] Started async saving checkpoint to /workspace/openpi/checkpoints/pi05_mecharm_trim_corr_v5_lora/trim_corr_v5/1000. (1084:async_checkpointer.py:439)
12:59:40.313 [I] Using ThreadSafeKeyValueSignalingClient                                          (1084:signaling_client.py:332)
12:59:40.317 [I] Creating tmp directory /workspace/openpi/checkpoints/pi05_mecharm_trim_corr_v5_lora/trim_corr_v5/1000.orbax-checkpoint-tmp-0 (1084:atomicity.py:144)
12:59:40.336 [I] Wrote Metadata={'item_handlers': None, 'metrics': {}, 'performance_metrics': {}, 'init_timestamp_nsecs': 1793969980334486312, 'commit_timestamp_nsecs': None, 'custom_metadata': {}}, json={"item_handlers": null, "metrics": {}, "performance_metrics": {}, "init_timestamp_nsecs": 1793969980334486312, "commit_timestamp_nsecs": null, "custom_metadata": {}} to /workspace/openpi/checkpoints/pi05_mecharm_trim_corr_v5_lora/trim_corr_v5/1000.orbax-checkpoint-tmp-0/_CHECKPOINT_METADATA (1084:checkpoint.py:186)
12:59:40.339 [I] Creating tmp directory /workspace/openpi/checkpoints/pi05_mecharm_trim_corr_v5_lora/trim_corr_v5/1000.orbax-checkpoint-tmp-0/assets.orbax-checkpoint-tmp-1 (1084:atomicity.py:144)
12:59:40.339 [I] Creating tmp directory /workspace/openpi/checkpoints/pi05_mecharm_trim_corr_v5_lora/trim_corr_v5/1000.orbax-checkpoint-tmp-0/params.orbax-checkpoint-tmp-2 (1084:atomicity.py:144)
12:59:40.340 [I] Creating tmp directory /workspace/openpi/checkpoints/pi05_mecharm_trim_corr_v5_lora/trim_corr_v5/1000.orbax-checkpoint-tmp-0/train_state.orbax-checkpoint-tmp-3 (1084:atomicity.py:144)
12:59:40.360 [I] Transferring arrays to host memory with options: use_replica_parallel=True, enable_pinned_host_transfer=False (1084:replica_slices.py:341)
12:59:43.136 [I] Transferring arrays to host memory with options: use_replica_parallel=True, enable_pinned_host_transfer=False (1084:replica_slices.py:341)
12:59:43.148 [I] Array name: 'params.PaliGemma.llm.layers.mlp.gating_einsum.value', global shape: (18, 2, 2048, 16384), write shape: (18, 2, 2048, 16384), chosen chunk shape: (18, 2, 2048, 8192) (1084:tensorstore_utils.py:408)
12:59:43.189 [I] [process=0][thread=array_type_handler] Wrote 71 array_metadata.ArrayMetadata to /workspace/openpi/checkpoints/pi05_mecharm_trim_corr_v5_lora/trim_corr_v5/1000.orbax-checkpoint-tmp-0/params.orbax-checkpoint-tmp-2/array_metadatas/process_0 (1084:array_metadata_store.py:198)
12:59:45.334 [I] [process=0] /jax/checkpoint/write/blocking_bytes_per_sec: 1.4 GiB/s (total bytes: 7.2 GiB) (time elapsed: 5 seconds) (per-host) (1084:base_pytree_checkpoint_handler.py:114)
12:59:45.337 [I] [process=0] /jax/checkpoint/write/blocking_bytes_per_sec: 711.7 MiB/s (total bytes: 3.5 GiB) (time elapsed: 5 seconds) (per-host) (1084:base_pytree_checkpoint_handler.py:114)
12:59:45.342 [I] [process=0][thread=async_save] Background save thread started.                   (1084:async_checkpointer.py:77)
12:59:45.342 [I] Finished blocking save in 5.03 seconds. Continuing to save asynchronously to /workspace/openpi/checkpoints/pi05_mecharm_trim_corr_v5_lora/trim_corr_v5/1000. (1084:async_checkpointer.py:548)
12:59:45.343 [I] [process=0][thread=MainThread][step=1000] Starting CheckpointManager Save Finalize thread=save_finalize (1084:checkpoint_manager.py:1453)
12:59:45.344 [I] [process=0][thread=save_finalize] Waiting for background save thread=async_save. (1084:async_checkpointer.py:258)
12:59:45.344 [I] {'step': 1000, 'event_type': 'save', 'directory': '/workspace/openpi/checkpoints/pi05_mecharm_trim_corr_v5_lora/trim_corr_v5', 'reached_preemption': False, 'preemption_received_at': None, 'synchronous': False, 'wait_for_prev_start_time': 1793969980.3103168, 'wait_for_prev_duration_secs': 0.00020432472229003906, 'checkpointer_blocking_start_time': 1793969980.311409, 'checkpointer_blocking_duration_secs': 5.0317113399505615, 'get_old_steps_start_time': 1793969985.34319, 'get_old_steps_duration_secs': 1.4543533325195312e-05, 'checkpoint_manager_blocking_start_time': 1793969980.3097968, 'checkpoint_manager_blocking_duration_secs': 5.0346527099609375} (1084:standard_logger.py:34)
12:59:45.381 [I] [process=0][thread=array_type_handler] Wrote 105 array_metadata.ArrayMetadata to /workspace/openpi/checkpoints/pi05_mecharm_trim_corr_v5_lora/trim_corr_v5/1000.orbax-checkpoint-tmp-0/train_state.orbax-checkpoint-tmp-3/array_metadatas/process_0 (1084:array_metadata_store.py:198)
12:59:48.934 [I] [process=0] /jax/checkpoint/write/bytes_per_sec: 856.7 MiB/s (total bytes: 7.2 GiB) (time elapsed: 8 seconds) (per-host) (1084:base_pytree_checkpoint_handler.py:114)
12:59:48.934 [I] [process=0] /jax/checkpoint/write/bytes_per_sec: 414.1 MiB/s (total bytes: 3.5 GiB) (time elapsed: 8 seconds) (per-host) (1084:base_pytree_checkpoint_handler.py:114)
12:59:48.934 [I] [process=0][thread=async_save] 5 Handler Commit operations completed.            (1084:async_checkpointer.py:87)
12:59:48.935 [I] Read Metadata={'item_handlers': None, 'metrics': {}, 'performance_metrics': {}, 'init_timestamp_nsecs': 1793969980334486312, 'commit_timestamp_nsecs': None, 'custom_metadata': {}} from /workspace/openpi/checkpoints/pi05_mecharm_trim_corr_v5_lora/trim_corr_v5/1000.orbax-checkpoint-tmp-0/_CHECKPOINT_METADATA (1084:checkpoint.py:226)
12:59:48.935 [I] Renaming /workspace/openpi/checkpoints/pi05_mecharm_trim_corr_v5_lora/trim_corr_v5/1000.orbax-checkpoint-tmp-0/assets.orbax-checkpoint-tmp-1 to /workspace/openpi/checkpoints/pi05_mecharm_trim_corr_v5_lora/trim_corr_v5/1000.orbax-checkpoint-tmp-0/assets (1084:atomicity.py:289)
12:59:48.936 [I] Updated Metadata={'item_handlers': {'assets': 'openpi.training.checkpoints.CallbackHandler', 'params': 'orbax.checkpoint._src.handlers.pytree_checkpoint_handler.PyTreeCheckpointHandler', 'train_state': 'orbax.checkpoint._src.handlers.pytree_checkpoint_handler.PyTreeCheckpointHandler'}, 'metrics': {}, 'performance_metrics': {}, 'init_timestamp_nsecs': 1793969980334486312, 'commit_timestamp_nsecs': None, 'custom_metadata': {}} to /workspace/openpi/checkpoints/pi05_mecharm_trim_corr_v5_lora/trim_corr_v5/1000.orbax-checkpoint-tmp-0/_CHECKPOINT_METADATA (1084:checkpoint.py:245)
12:59:48.940 [I] [process=0][thread=async_save] Skipped cross-host ArrayMetadata validation because only one process is found: process_index=0. (1084:array_metadata_store.py:362)
12:59:48.946 [I] Renaming /workspace/openpi/checkpoints/pi05_mecharm_trim_corr_v5_lora/trim_corr_v5/1000.orbax-checkpoint-tmp-0/train_state.orbax-checkpoint-tmp-3 to /workspace/openpi/checkpoints/pi05_mecharm_trim_corr_v5_lora/trim_corr_v5/1000.orbax-checkpoint-tmp-0/train_state (1084:atomicity.py:289)
12:59:48.949 [I] [process=0][thread=async_save] Skipped cross-host ArrayMetadata validation because only one process is found: process_index=0. (1084:array_metadata_store.py:362)
12:59:48.956 [I] Renaming /workspace/openpi/checkpoints/pi05_mecharm_trim_corr_v5_lora/trim_corr_v5/1000.orbax-checkpoint-tmp-0/params.orbax-checkpoint-tmp-2 to /workspace/openpi/checkpoints/pi05_mecharm_trim_corr_v5_lora/trim_corr_v5/1000.orbax-checkpoint-tmp-0/params (1084:atomicity.py:289)
12:59:48.957 [I] Renaming /workspace/openpi/checkpoints/pi05_mecharm_trim_corr_v5_lora/trim_corr_v5/1000.orbax-checkpoint-tmp-0 to /workspace/openpi/checkpoints/pi05_mecharm_trim_corr_v5_lora/trim_corr_v5/1000 (1084:atomicity.py:289)
12:59:48.958 [I] [process=0][thread=async_save] Finished saving checkpoint (finalized tmp dir) to `/workspace/openpi/checkpoints/pi05_mecharm_trim_corr_v5_lora/trim_corr_v5/1000`. (1084:atomicity.py:573)
12:59:48.958 [I] Finished asynchronous save (blocking + background) in 8.65 seconds to /workspace/openpi/checkpoints/pi05_mecharm_trim_corr_v5_lora/trim_corr_v5/1000 (1084:async_checkpointer.py:412)
12:59:48.958 [I] [process=0][thread=async_save] Background save thread done.                      (1084:async_checkpointer.py:143)
12:59:48.959 [I] [process=0][thread=save_finalize] Done with waiting for background save thread=async_save. (1084:async_checkpointer.py:266)
12:59:48.959 [I] [process=0][thread=save_finalize] No errors found in background save thread=async_save. (1084:async_checkpointer.py:276)
12:59:48.959 [I] [process=0][thread=save_finalize][step=1000] CheckpointManager Save Finalize is syncing with other hosts... (1084:checkpoint_manager.py:2095)
12:59:48.959 [I] [process=0][thread=save_finalize][step=1000] CheckpointManager Save Finalize is done on all hosts. (1084:checkpoint_manager.py:2110)
12:59:52.808 [I] Progress on: 1.00kit/1.50kit rate:7.4s/it remaining:1:01:08 elapsed:2:02:07 postfix:- (1084:tqdm_logging.py:145)
13:00:06.896 [I] Progress on: 1.00kit/1.50kit rate:7.2s/it remaining:59:30 elapsed:2:02:22 postfix:- (1084:tqdm_logging.py:145)
13:00:21.020 [I] Progress on: 1.01kit/1.50kit rate:7.1s/it remaining:58:38 elapsed:2:02:36 postfix:- (1084:tqdm_logging.py:145)
13:00:35.147 [I] Progress on: 1.01kit/1.50kit rate:7.1s/it remaining:58:06 elapsed:2:02:50 postfix:- (1084:tqdm_logging.py:145)
13:00:49.251 [I] Progress on: 1.01kit/1.50kit rate:7.1s/it remaining:57:40 elapsed:2:03:04 postfix:- (1084:tqdm_logging.py:145)
13:01:03.403 [I] Progress on: 1.01kit/1.50kit rate:7.1s/it remaining:57:26 elapsed:2:03:18 postfix:- (1084:tqdm_logging.py:145)
13:01:17.548 [I] Progress on: 1.01kit/1.50kit rate:7.1s/it remaining:57:11 elapsed:2:03:32 postfix:- (1084:tqdm_logging.py:145)
13:01:31.688 [I] Progress on: 1.02kit/1.50kit rate:7.1s/it remaining:56:55 elapsed:2:03:46 postfix:- (1084:tqdm_logging.py:145)
13:01:45.829 [I] Progress on: 1.02kit/1.50kit rate:7.1s/it remaining:56:41 elapsed:2:04:01 postfix:- (1084:tqdm_logging.py:145)
13:01:59.955 [I] Progress on: 1.02kit/1.50kit rate:7.1s/it remaining:56:24 elapsed:2:04:15 postfix:- (1084:tqdm_logging.py:145)
13:02:14.113 [I] Progress on: 1.02kit/1.50kit rate:7.1s/it remaining:56:13 elapsed:2:04:29 postfix:- (1084:tqdm_logging.py:145)
13:02:28.236 [I] Progress on: 1.02kit/1.50kit rate:7.1s/it remaining:55:57 elapsed:2:04:43 postfix:- (1084:tqdm_logging.py:145)
13:02:42.404 [I] Progress on: 1.03kit/1.50kit rate:7.1s/it remaining:55:47 elapsed:2:04:57 postfix:- (1084:tqdm_logging.py:145)
13:02:56.537 [I] Progress on: 1.03kit/1.50kit rate:7.1s/it remaining:55:31 elapsed:2:05:11 postfix:- (1084:tqdm_logging.py:145)
13:03:10.664 [I] Progress on: 1.03kit/1.50kit rate:7.1s/it remaining:55:14 elapsed:2:05:25 postfix:- (1084:tqdm_logging.py:145)
13:03:24.806 [I] Progress on: 1.03kit/1.50kit rate:7.1s/it remaining:55:01 elapsed:2:05:39 postfix:- (1084:tqdm_logging.py:145)
13:03:38.971 [I] Progress on: 1.03kit/1.50kit rate:7.1s/it remaining:54:50 elapsed:2:05:54 postfix:- (1084:tqdm_logging.py:145)
13:03:53.100 [I] Progress on: 1.04kit/1.50kit rate:7.1s/it remaining:54:33 elapsed:2:06:08 postfix:- (1084:tqdm_logging.py:145)
13:04:07.206 [I] Progress on: 1.04kit/1.50kit rate:7.1s/it remaining:54:14 elapsed:2:06:22 postfix:- (1084:tqdm_logging.py:145)
13:04:21.342 [I] Progress on: 1.04kit/1.50kit rate:7.1s/it remaining:54:02 elapsed:2:06:36 postfix:- (1084:tqdm_logging.py:145)
13:04:35.479 [I] Progress on: 1.04kit/1.50kit rate:7.1s/it remaining:53:49 elapsed:2:06:50 postfix:- (1084:tqdm_logging.py:145)
13:04:49.642 [I] Progress on: 1.04kit/1.50kit rate:7.1s/it remaining:53:39 elapsed:2:07:04 postfix:- (1084:tqdm_logging.py:145)
13:05:03.802 [I] Progress on: 1.05kit/1.50kit rate:7.1s/it remaining:53:25 elapsed:2:07:18 postfix:- (1084:tqdm_logging.py:145)
13:05:17.951 [I] Progress on: 1.05kit/1.50kit rate:7.1s/it remaining:53:11 elapsed:2:07:33 postfix:- (1084:tqdm_logging.py:145)


Step 1050: grad_norm=0.0399, loss=0.0028, param_norm=1804.2487
13:05:38.808 [I] Progress on: 1.05kit/1.50kit rate:7.1s/it remaining:53:02 elapsed:2:07:53 postfix:- (1084:tqdm_logging.py:145)
13:05:53.415 [I] Progress on: 1.05kit/1.50kit rate:6.8s/it remaining:50:29 elapsed:2:08:08 postfix:- (1084:tqdm_logging.py:145)
13:06:07.563 [I] Progress on: 1.06kit/1.50kit rate:6.9s/it remaining:51:19 elapsed:2:08:22 postfix:- (1084:tqdm_logging.py:145)
13:06:21.698 [I] Progress on: 1.06kit/1.50kit rate:7.0s/it remaining:51:35 elapsed:2:08:36 postfix:- (1084:tqdm_logging.py:145)
13:06:35.845 [I] Progress on: 1.06kit/1.50kit rate:7.0s/it remaining:51:37 elapsed:2:08:51 postfix:- (1084:tqdm_logging.py:145)
13:06:50.008 [I] Progress on: 1.06kit/1.50kit rate:7.1s/it remaining:51:33 elapsed:2:09:05 postfix:- (1084:tqdm_logging.py:145)
13:07:04.133 [I] Progress on: 1.06kit/1.50kit rate:7.1s/it remaining:51:19 elapsed:2:09:19 postfix:- (1084:tqdm_logging.py:145)
13:07:18.298 [I] Progress on: 1.07kit/1.50kit rate:7.1s/it remaining:51:09 elapsed:2:09:33 postfix:- (1084:tqdm_logging.py:145)
13:07:32.450 [I] Progress on: 1.07kit/1.50kit rate:7.1s/it remaining:50:56 elapsed:2:09:47 postfix:- (1084:tqdm_logging.py:145)
13:07:46.596 [I] Progress on: 1.07kit/1.50kit rate:7.1s/it remaining:50:41 elapsed:2:10:01 postfix:- (1084:tqdm_logging.py:145)
13:08:01.338 [I] Progress on: 1.07kit/1.50kit rate:7.3s/it remaining:51:45 elapsed:2:10:16 postfix:- (1084:tqdm_logging.py:145)
13:08:14.862 [I] Progress on: 1.07kit/1.50kit rate:7.0s/it remaining:49:57 elapsed:2:10:30 postfix:- (1084:tqdm_logging.py:145)
13:08:28.981 [I] Progress on: 1.08kit/1.50kit rate:7.1s/it remaining:49:50 elapsed:2:10:44 postfix:- (1084:tqdm_logging.py:145)
13:08:43.108 [I] Progress on: 1.08kit/1.50kit rate:7.1s/it remaining:49:40 elapsed:2:10:58 postfix:- (1084:tqdm_logging.py:145)
13:08:57.224 [I] Progress on: 1.08kit/1.50kit rate:7.1s/it remaining:49:26 elapsed:2:11:12 postfix:- (1084:tqdm_logging.py:145)
13:09:11.333 [I] Progress on: 1.08kit/1.50kit rate:7.1s/it remaining:49:10 elapsed:2:11:26 postfix:- (1084:tqdm_logging.py:145)
13:09:25.443 [I] Progress on: 1.08kit/1.50kit rate:7.1s/it remaining:48:55 elapsed:2:11:40 postfix:- (1084:tqdm_logging.py:145)
13:09:39.559 [I] Progress on: 1.09kit/1.50kit rate:7.1s/it remaining:48:41 elapsed:2:11:54 postfix:- (1084:tqdm_logging.py:145)
13:09:53.704 [I] Progress on: 1.09kit/1.50kit rate:7.1s/it remaining:48:30 elapsed:2:12:08 postfix:- (1084:tqdm_logging.py:145)
13:10:07.821 [I] Progress on: 1.09kit/1.50kit rate:7.1s/it remaining:48:15 elapsed:2:12:23 postfix:- (1084:tqdm_logging.py:145)
13:10:21.931 [I] Progress on: 1.09kit/1.50kit rate:7.1s/it remaining:47:59 elapsed:2:12:37 postfix:- (1084:tqdm_logging.py:145)
13:10:36.036 [I] Progress on: 1.09kit/1.50kit rate:7.1s/it remaining:47:44 elapsed:2:12:51 postfix:- (1084:tqdm_logging.py:145)
13:10:50.166 [I] Progress on: 1.10kit/1.50kit rate:7.1s/it remaining:47:32 elapsed:2:13:05 postfix:- (1084:tqdm_logging.py:145)
13:11:04.285 [I] Progress on: 1.10kit/1.50kit rate:7.1s/it remaining:47:18 elapsed:2:13:19 postfix:- (1084:tqdm_logging.py:145)
13:11:18.425 [I] Progress on: 1.10kit/1.50kit rate:7.1s/it remaining:47:05 elapsed:2:13:33 postfix:- (1084:tqdm_logging.py:145)


Step 1100: grad_norm=0.0454, loss=0.0037, param_norm=1804.2490
13:11:32.184 [I] Progress on: 1.10kit/1.50kit rate:7.1s/it remaining:47:05 elapsed:2:13:47 postfix:- (1084:tqdm_logging.py:145)
13:11:46.753 [I] Progress on: 1.10kit/1.50kit rate:6.8s/it remaining:44:43 elapsed:2:14:01 postfix:- (1084:tqdm_logging.py:145)
13:12:00.893 [I] Progress on: 1.11kit/1.50kit rate:6.9s/it remaining:45:28 elapsed:2:14:16 postfix:- (1084:tqdm_logging.py:145)
13:12:15.019 [I] Progress on: 1.11kit/1.50kit rate:7.0s/it remaining:45:42 elapsed:2:14:30 postfix:- (1084:tqdm_logging.py:145)
13:12:29.130 [I] Progress on: 1.11kit/1.50kit rate:7.0s/it remaining:45:40 elapsed:2:14:44 postfix:- (1084:tqdm_logging.py:145)
13:12:43.249 [I] Progress on: 1.11kit/1.50kit rate:7.0s/it remaining:45:32 elapsed:2:14:58 postfix:- (1084:tqdm_logging.py:145)
13:12:57.398 [I] Progress on: 1.11kit/1.50kit rate:7.1s/it remaining:45:25 elapsed:2:15:12 postfix:- (1084:tqdm_logging.py:145)
13:13:11.509 [I] Progress on: 1.12kit/1.50kit rate:7.1s/it remaining:45:10 elapsed:2:15:26 postfix:- (1084:tqdm_logging.py:145)
13:13:25.641 [I] Progress on: 1.12kit/1.50kit rate:7.1s/it remaining:44:58 elapsed:2:15:40 postfix:- (1084:tqdm_logging.py:145)
13:13:39.754 [I] Progress on: 1.12kit/1.50kit rate:7.1s/it remaining:44:42 elapsed:2:15:54 postfix:- (1084:tqdm_logging.py:145)
13:13:53.891 [I] Progress on: 1.12kit/1.50kit rate:7.1s/it remaining:44:30 elapsed:2:16:09 postfix:- (1084:tqdm_logging.py:145)
13:14:08.016 [I] Progress on: 1.12kit/1.50kit rate:7.1s/it remaining:44:16 elapsed:2:16:23 postfix:- (1084:tqdm_logging.py:145)
13:14:22.120 [I] Progress on: 1.13kit/1.50kit rate:7.1s/it remaining:43:59 elapsed:2:16:37 postfix:- (1084:tqdm_logging.py:145)
13:14:36.253 [I] Progress on: 1.13kit/1.50kit rate:7.1s/it remaining:43:46 elapsed:2:16:51 postfix:- (1084:tqdm_logging.py:145)
13:14:50.379 [I] Progress on: 1.13kit/1.50kit rate:7.1s/it remaining:43:33 elapsed:2:17:05 postfix:- (1084:tqdm_logging.py:145)
13:15:04.513 [I] Progress on: 1.13kit/1.50kit rate:7.1s/it remaining:43:19 elapsed:2:17:19 postfix:- (1084:tqdm_logging.py:145)
13:15:18.646 [I] Progress on: 1.13kit/1.50kit rate:7.1s/it remaining:43:05 elapsed:2:17:33 postfix:- (1084:tqdm_logging.py:145)
13:15:32.765 [I] Progress on: 1.14kit/1.50kit rate:7.1s/it remaining:42:51 elapsed:2:17:47 postfix:- (1084:tqdm_logging.py:145)
13:15:46.867 [I] Progress on: 1.14kit/1.50kit rate:7.1s/it remaining:42:34 elapsed:2:18:02 postfix:- (1084:tqdm_logging.py:145)
13:16:00.991 [I] Progress on: 1.14kit/1.50kit rate:7.1s/it remaining:42:21 elapsed:2:18:16 postfix:- (1084:tqdm_logging.py:145)
13:16:15.138 [I] Progress on: 1.14kit/1.50kit rate:7.1s/it remaining:42:10 elapsed:2:18:30 postfix:- (1084:tqdm_logging.py:145)
13:16:29.265 [I] Progress on: 1.14kit/1.50kit rate:7.1s/it remaining:41:55 elapsed:2:18:44 postfix:- (1084:tqdm_logging.py:145)
13:16:43.396 [I] Progress on: 1.15kit/1.50kit rate:7.1s/it remaining:41:41 elapsed:2:18:58 postfix:- (1084:tqdm_logging.py:145)
13:16:57.501 [I] Progress on: 1.15kit/1.50kit rate:7.1s/it remaining:41:25 elapsed:2:19:12 postfix:- (1084:tqdm_logging.py:145)
13:17:11.638 [I] Progress on: 1.15kit/1.50kit rate:7.1s/it remaining:41:12 elapsed:2:19:26 postfix:- (1084:tqdm_logging.py:145)


Step 1150: grad_norm=0.0399, loss=0.0029, param_norm=1804.2496
13:17:25.396 [I] Progress on: 1.15kit/1.50kit rate:7.1s/it remaining:41:12 elapsed:2:19:40 postfix:- (1084:tqdm_logging.py:145)
13:17:39.972 [I] Progress on: 1.15kit/1.50kit rate:6.8s/it remaining:39:06 elapsed:2:19:55 postfix:- (1084:tqdm_logging.py:145)
13:17:54.090 [I] Progress on: 1.16kit/1.50kit rate:6.9s/it remaining:39:40 elapsed:2:20:09 postfix:- (1084:tqdm_logging.py:145)
13:18:08.198 [I] Progress on: 1.16kit/1.50kit rate:7.0s/it remaining:39:49 elapsed:2:20:23 postfix:- (1084:tqdm_logging.py:145)
13:18:22.332 [I] Progress on: 1.16kit/1.50kit rate:7.0s/it remaining:39:49 elapsed:2:20:37 postfix:- (1084:tqdm_logging.py:145)
13:18:36.462 [I] Progress on: 1.16kit/1.50kit rate:7.0s/it remaining:39:41 elapsed:2:20:51 postfix:- (1084:tqdm_logging.py:145)
13:18:50.579 [I] Progress on: 1.16kit/1.50kit rate:7.1s/it remaining:39:29 elapsed:2:21:05 postfix:- (1084:tqdm_logging.py:145)
13:19:04.699 [I] Progress on: 1.17kit/1.50kit rate:7.1s/it remaining:39:16 elapsed:2:21:19 postfix:- (1084:tqdm_logging.py:145)
13:19:18.851 [I] Progress on: 1.17kit/1.50kit rate:7.1s/it remaining:39:06 elapsed:2:21:34 postfix:- (1084:tqdm_logging.py:145)
13:19:32.963 [I] Progress on: 1.17kit/1.50kit rate:7.1s/it remaining:38:50 elapsed:2:21:48 postfix:- (1084:tqdm_logging.py:145)
13:19:47.097 [I] Progress on: 1.17kit/1.50kit rate:7.1s/it remaining:38:37 elapsed:2:22:02 postfix:- (1084:tqdm_logging.py:145)
13:20:01.197 [I] Progress on: 1.17kit/1.50kit rate:7.1s/it remaining:38:20 elapsed:2:22:16 postfix:- (1084:tqdm_logging.py:145)
13:20:15.342 [I] Progress on: 1.18kit/1.50kit rate:7.1s/it remaining:38:08 elapsed:2:22:30 postfix:- (1084:tqdm_logging.py:145)
13:20:29.479 [I] Progress on: 1.18kit/1.50kit rate:7.1s/it remaining:37:55 elapsed:2:22:44 postfix:- (1084:tqdm_logging.py:145)
13:20:43.604 [I] Progress on: 1.18kit/1.50kit rate:7.1s/it remaining:37:41 elapsed:2:22:58 postfix:- (1084:tqdm_logging.py:145)
13:20:57.702 [I] Progress on: 1.18kit/1.50kit rate:7.1s/it remaining:37:24 elapsed:2:23:12 postfix:- (1084:tqdm_logging.py:145)
13:21:11.826 [I] Progress on: 1.18kit/1.50kit rate:7.1s/it remaining:37:10 elapsed:2:23:27 postfix:- (1084:tqdm_logging.py:145)
13:21:25.967 [I] Progress on: 1.19kit/1.50kit rate:7.1s/it remaining:36:57 elapsed:2:23:41 postfix:- (1084:tqdm_logging.py:145)
13:21:40.112 [I] Progress on: 1.19kit/1.50kit rate:7.1s/it remaining:36:45 elapsed:2:23:55 postfix:- (1084:tqdm_logging.py:145)
13:21:54.221 [I] Progress on: 1.19kit/1.50kit rate:7.1s/it remaining:36:29 elapsed:2:24:09 postfix:- (1084:tqdm_logging.py:145)
13:22:08.351 [I] Progress on: 1.19kit/1.50kit rate:7.1s/it remaining:36:15 elapsed:2:24:23 postfix:- (1084:tqdm_logging.py:145)
13:22:22.469 [I] Progress on: 1.19kit/1.50kit rate:7.1s/it remaining:36:00 elapsed:2:24:37 postfix:- (1084:tqdm_logging.py:145)
13:22:36.590 [I] Progress on: 1.20kit/1.50kit rate:7.1s/it remaining:35:46 elapsed:2:24:51 postfix:- (1084:tqdm_logging.py:145)
13:22:50.709 [I] Progress on: 1.20kit/1.50kit rate:7.1s/it remaining:35:32 elapsed:2:25:05 postfix:- (1084:tqdm_logging.py:145)
13:23:04.842 [I] Progress on: 1.20kit/1.50kit rate:7.1s/it remaining:35:18 elapsed:2:25:20 postfix:- (1084:tqdm_logging.py:145)


Step 1200: grad_norm=0.0451, loss=0.0033, param_norm=1804.2500
13:23:18.596 [I] Progress on: 1.20kit/1.50kit rate:7.1s/it remaining:35:18 elapsed:2:25:33 postfix:- (1084:tqdm_logging.py:145)
13:23:33.171 [I] Progress on: 1.20kit/1.50kit rate:6.8s/it remaining:33:26 elapsed:2:25:48 postfix:- (1084:tqdm_logging.py:145)
13:23:47.857 [I] Progress on: 1.21kit/1.50kit rate:7.1s/it remaining:34:44 elapsed:2:26:03 postfix:- (1084:tqdm_logging.py:145)
13:24:01.427 [I] Progress on: 1.21kit/1.50kit rate:7.0s/it remaining:33:53 elapsed:2:26:16 postfix:- (1084:tqdm_logging.py:145)
13:24:15.553 [I] Progress on: 1.21kit/1.50kit rate:7.0s/it remaining:33:55 elapsed:2:26:30 postfix:- (1084:tqdm_logging.py:145)
13:24:29.680 [I] Progress on: 1.21kit/1.50kit rate:7.0s/it remaining:33:48 elapsed:2:26:44 postfix:- (1084:tqdm_logging.py:145)
13:24:43.797 [I] Progress on: 1.21kit/1.50kit rate:7.1s/it remaining:33:36 elapsed:2:26:58 postfix:- (1084:tqdm_logging.py:145)
13:24:57.927 [I] Progress on: 1.22kit/1.50kit rate:7.1s/it remaining:33:24 elapsed:2:27:13 postfix:- (1084:tqdm_logging.py:145)
13:25:12.058 [I] Progress on: 1.22kit/1.50kit rate:7.1s/it remaining:33:11 elapsed:2:27:27 postfix:- (1084:tqdm_logging.py:145)
13:25:26.171 [I] Progress on: 1.22kit/1.50kit rate:7.1s/it remaining:32:56 elapsed:2:27:41 postfix:- (1084:tqdm_logging.py:145)
13:25:40.303 [I] Progress on: 1.22kit/1.50kit rate:7.1s/it remaining:32:43 elapsed:2:27:55 postfix:- (1084:tqdm_logging.py:145)
13:25:54.425 [I] Progress on: 1.22kit/1.50kit rate:7.1s/it remaining:32:29 elapsed:2:28:09 postfix:- (1084:tqdm_logging.py:145)
13:26:08.546 [I] Progress on: 1.23kit/1.50kit rate:7.1s/it remaining:32:14 elapsed:2:28:23 postfix:- (1084:tqdm_logging.py:145)
13:26:22.672 [I] Progress on: 1.23kit/1.50kit rate:7.1s/it remaining:32:00 elapsed:2:28:37 postfix:- (1084:tqdm_logging.py:145)
13:26:36.809 [I] Progress on: 1.23kit/1.50kit rate:7.1s/it remaining:31:47 elapsed:2:28:51 postfix:- (1084:tqdm_logging.py:145)
13:26:50.915 [I] Progress on: 1.23kit/1.50kit rate:7.1s/it remaining:31:32 elapsed:2:29:06 postfix:- (1084:tqdm_logging.py:145)
13:27:05.049 [I] Progress on: 1.23kit/1.50kit rate:7.1s/it remaining:31:18 elapsed:2:29:20 postfix:- (1084:tqdm_logging.py:145)
13:27:19.173 [I] Progress on: 1.24kit/1.50kit rate:7.1s/it remaining:31:04 elapsed:2:29:34 postfix:- (1084:tqdm_logging.py:145)
13:27:33.299 [I] Progress on: 1.24kit/1.50kit rate:7.1s/it remaining:30:49 elapsed:2:29:48 postfix:- (1084:tqdm_logging.py:145)
13:27:47.440 [I] Progress on: 1.24kit/1.50kit rate:7.1s/it remaining:30:37 elapsed:2:30:02 postfix:- (1084:tqdm_logging.py:145)
13:28:01.517 [I] Progress on: 1.24kit/1.50kit rate:7.1s/it remaining:30:20 elapsed:2:30:16 postfix:- (1084:tqdm_logging.py:145)
13:28:15.629 [I] Progress on: 1.24kit/1.50kit rate:7.1s/it remaining:30:06 elapsed:2:30:30 postfix:- (1084:tqdm_logging.py:145)
13:28:29.748 [I] Progress on: 1.25kit/1.50kit rate:7.1s/it remaining:29:52 elapsed:2:30:44 postfix:- (1084:tqdm_logging.py:145)
13:28:43.874 [I] Progress on: 1.25kit/1.50kit rate:7.1s/it remaining:29:39 elapsed:2:30:59 postfix:- (1084:tqdm_logging.py:145)
13:28:57.984 [I] Progress on: 1.25kit/1.50kit rate:7.1s/it remaining:29:24 elapsed:2:31:13 postfix:- (1084:tqdm_logging.py:145)


Step 1250: grad_norm=0.0403, loss=0.0027, param_norm=1804.2506
13:29:11.772 [I] Progress on: 1.25kit/1.50kit rate:7.1s/it remaining:29:24 elapsed:2:31:26 postfix:- (1084:tqdm_logging.py:145)
13:29:26.314 [I] Progress on: 1.25kit/1.50kit rate:6.8s/it remaining:27:45 elapsed:2:31:41 postfix:- (1084:tqdm_logging.py:145)
13:29:40.448 [I] Progress on: 1.26kit/1.50kit rate:6.9s/it remaining:28:09 elapsed:2:31:55 postfix:- (1084:tqdm_logging.py:145)
13:29:54.558 [I] Progress on: 1.26kit/1.50kit rate:7.0s/it remaining:28:11 elapsed:2:32:09 postfix:- (1084:tqdm_logging.py:145)
13:30:08.699 [I] Progress on: 1.26kit/1.50kit rate:7.0s/it remaining:28:07 elapsed:2:32:23 postfix:- (1084:tqdm_logging.py:145)
13:30:22.817 [I] Progress on: 1.26kit/1.50kit rate:7.0s/it remaining:27:56 elapsed:2:32:38 postfix:- (1084:tqdm_logging.py:145)
13:30:36.952 [I] Progress on: 1.26kit/1.50kit rate:7.1s/it remaining:27:45 elapsed:2:32:52 postfix:- (1084:tqdm_logging.py:145)
13:30:51.051 [I] Progress on: 1.27kit/1.50kit rate:7.1s/it remaining:27:30 elapsed:2:33:06 postfix:- (1084:tqdm_logging.py:145)
13:31:05.194 [I] Progress on: 1.27kit/1.50kit rate:7.1s/it remaining:27:18 elapsed:2:33:20 postfix:- (1084:tqdm_logging.py:145)
13:31:19.326 [I] Progress on: 1.27kit/1.50kit rate:7.1s/it remaining:27:04 elapsed:2:33:34 postfix:- (1084:tqdm_logging.py:145)
13:31:33.449 [I] Progress on: 1.27kit/1.50kit rate:7.1s/it remaining:26:50 elapsed:2:33:48 postfix:- (1084:tqdm_logging.py:145)
13:31:47.571 [I] Progress on: 1.27kit/1.50kit rate:7.1s/it remaining:26:36 elapsed:2:34:02 postfix:- (1084:tqdm_logging.py:145)
13:32:01.704 [I] Progress on: 1.28kit/1.50kit rate:7.1s/it remaining:26:22 elapsed:2:34:16 postfix:- (1084:tqdm_logging.py:145)
13:32:15.843 [I] Progress on: 1.28kit/1.50kit rate:7.1s/it remaining:26:09 elapsed:2:34:31 postfix:- (1084:tqdm_logging.py:145)
13:32:29.954 [I] Progress on: 1.28kit/1.50kit rate:7.1s/it remaining:25:53 elapsed:2:34:45 postfix:- (1084:tqdm_logging.py:145)
13:32:44.091 [I] Progress on: 1.28kit/1.50kit rate:7.1s/it remaining:25:40 elapsed:2:34:59 postfix:- (1084:tqdm_logging.py:145)
13:32:58.218 [I] Progress on: 1.28kit/1.50kit rate:7.1s/it remaining:25:25 elapsed:2:35:13 postfix:- (1084:tqdm_logging.py:145)
13:33:12.326 [I] Progress on: 1.29kit/1.50kit rate:7.1s/it remaining:25:10 elapsed:2:35:27 postfix:- (1084:tqdm_logging.py:145)
13:33:26.467 [I] Progress on: 1.29kit/1.50kit rate:7.1s/it remaining:24:57 elapsed:2:35:41 postfix:- (1084:tqdm_logging.py:145)
13:33:40.589 [I] Progress on: 1.29kit/1.50kit rate:7.1s/it remaining:24:43 elapsed:2:35:55 postfix:- (1084:tqdm_logging.py:145)
13:33:54.719 [I] Progress on: 1.29kit/1.50kit rate:7.1s/it remaining:24:29 elapsed:2:36:09 postfix:- (1084:tqdm_logging.py:145)
13:34:08.834 [I] Progress on: 1.29kit/1.50kit rate:7.1s/it remaining:24:14 elapsed:2:36:24 postfix:- (1084:tqdm_logging.py:145)
13:34:22.944 [I] Progress on: 1.30kit/1.50kit rate:7.1s/it remaining:24:00 elapsed:2:36:38 postfix:- (1084:tqdm_logging.py:145)
13:34:37.090 [I] Progress on: 1.30kit/1.50kit rate:7.1s/it remaining:23:47 elapsed:2:36:52 postfix:- (1084:tqdm_logging.py:145)
13:34:51.223 [I] Progress on: 1.30kit/1.50kit rate:7.1s/it remaining:23:32 elapsed:2:37:06 postfix:- (1084:tqdm_logging.py:145)


Step 1300: grad_norm=0.0399, loss=0.0025, param_norm=1804.2509
13:35:04.983 [I] Progress on: 1.30kit/1.50kit rate:7.1s/it remaining:23:32 elapsed:2:37:20 postfix:- (1084:tqdm_logging.py:145)
13:35:19.535 [I] Progress on: 1.30kit/1.50kit rate:6.8s/it remaining:22:08 elapsed:2:37:34 postfix:- (1084:tqdm_logging.py:145)
13:35:33.669 [I] Progress on: 1.31kit/1.50kit rate:6.9s/it remaining:22:23 elapsed:2:37:48 postfix:- (1084:tqdm_logging.py:145)
13:35:47.776 [I] Progress on: 1.31kit/1.50kit rate:7.0s/it remaining:22:22 elapsed:2:38:02 postfix:- (1084:tqdm_logging.py:145)
13:36:01.890 [I] Progress on: 1.31kit/1.50kit rate:7.0s/it remaining:22:14 elapsed:2:38:17 postfix:- (1084:tqdm_logging.py:145)
13:36:16.044 [I] Progress on: 1.31kit/1.50kit rate:7.0s/it remaining:22:05 elapsed:2:38:31 postfix:- (1084:tqdm_logging.py:145)
13:36:30.166 [I] Progress on: 1.31kit/1.50kit rate:7.1s/it remaining:21:52 elapsed:2:38:45 postfix:- (1084:tqdm_logging.py:145)
13:36:44.311 [I] Progress on: 1.32kit/1.50kit rate:7.1s/it remaining:21:40 elapsed:2:38:59 postfix:- (1084:tqdm_logging.py:145)
13:36:58.389 [I] Progress on: 1.32kit/1.50kit rate:7.1s/it remaining:21:23 elapsed:2:39:13 postfix:- (1084:tqdm_logging.py:145)
13:37:12.542 [I] Progress on: 1.32kit/1.50kit rate:7.1s/it remaining:21:11 elapsed:2:39:27 postfix:- (1084:tqdm_logging.py:145)
13:37:26.660 [I] Progress on: 1.32kit/1.50kit rate:7.1s/it remaining:20:57 elapsed:2:39:41 postfix:- (1084:tqdm_logging.py:145)
13:37:40.774 [I] Progress on: 1.32kit/1.50kit rate:7.1s/it remaining:20:42 elapsed:2:39:55 postfix:- (1084:tqdm_logging.py:145)
13:37:54.894 [I] Progress on: 1.33kit/1.50kit rate:7.1s/it remaining:20:28 elapsed:2:40:10 postfix:- (1084:tqdm_logging.py:145)
13:38:09.012 [I] Progress on: 1.33kit/1.50kit rate:7.1s/it remaining:20:14 elapsed:2:40:24 postfix:- (1084:tqdm_logging.py:145)
13:38:23.152 [I] Progress on: 1.33kit/1.50kit rate:7.1s/it remaining:20:00 elapsed:2:40:38 postfix:- (1084:tqdm_logging.py:145)
13:38:37.289 [I] Progress on: 1.33kit/1.50kit rate:7.1s/it remaining:19:47 elapsed:2:40:52 postfix:- (1084:tqdm_logging.py:145)
13:38:51.421 [I] Progress on: 1.33kit/1.50kit rate:7.1s/it remaining:19:33 elapsed:2:41:06 postfix:- (1084:tqdm_logging.py:145)
13:39:05.537 [I] Progress on: 1.34kit/1.50kit rate:7.1s/it remaining:19:18 elapsed:2:41:20 postfix:- (1084:tqdm_logging.py:145)
13:39:19.641 [I] Progress on: 1.34kit/1.50kit rate:7.1s/it remaining:19:03 elapsed:2:41:34 postfix:- (1084:tqdm_logging.py:145)
13:39:34.393 [I] Progress on: 1.34kit/1.50kit rate:7.2s/it remaining:19:19 elapsed:2:41:49 postfix:- (1084:tqdm_logging.py:145)
13:39:47.918 [I] Progress on: 1.34kit/1.50kit rate:7.0s/it remaining:18:31 elapsed:2:42:03 postfix:- (1084:tqdm_logging.py:145)
13:40:02.047 [I] Progress on: 1.34kit/1.50kit rate:7.1s/it remaining:18:20 elapsed:2:42:17 postfix:- (1084:tqdm_logging.py:145)
13:40:16.160 [I] Progress on: 1.35kit/1.50kit rate:7.1s/it remaining:18:07 elapsed:2:42:31 postfix:- (1084:tqdm_logging.py:145)
13:40:30.305 [I] Progress on: 1.35kit/1.50kit rate:7.1s/it remaining:17:54 elapsed:2:42:45 postfix:- (1084:tqdm_logging.py:145)
13:40:44.457 [I] Progress on: 1.35kit/1.50kit rate:7.1s/it remaining:17:41 elapsed:2:42:59 postfix:- (1084:tqdm_logging.py:145)


Step 1350: grad_norm=0.0447, loss=0.0031, param_norm=1804.2512
13:40:58.198 [I] Progress on: 1.35kit/1.50kit rate:7.1s/it remaining:17:41 elapsed:2:43:13 postfix:- (1084:tqdm_logging.py:145)
13:41:12.699 [I] Progress on: 1.35kit/1.50kit rate:6.8s/it remaining:16:27 elapsed:2:43:27 postfix:- (1084:tqdm_logging.py:145)
13:41:26.842 [I] Progress on: 1.36kit/1.50kit rate:6.9s/it remaining:16:36 elapsed:2:43:42 postfix:- (1084:tqdm_logging.py:145)
13:41:40.970 [I] Progress on: 1.36kit/1.50kit rate:7.0s/it remaining:16:33 elapsed:2:43:56 postfix:- (1084:tqdm_logging.py:145)
13:41:55.090 [I] Progress on: 1.36kit/1.50kit rate:7.0s/it remaining:16:23 elapsed:2:44:10 postfix:- (1084:tqdm_logging.py:145)
13:42:09.202 [I] Progress on: 1.36kit/1.50kit rate:7.0s/it remaining:16:11 elapsed:2:44:24 postfix:- (1084:tqdm_logging.py:145)
13:42:23.349 [I] Progress on: 1.36kit/1.50kit rate:7.1s/it remaining:15:59 elapsed:2:44:38 postfix:- (1084:tqdm_logging.py:145)
13:42:37.436 [I] Progress on: 1.37kit/1.50kit rate:7.1s/it remaining:15:45 elapsed:2:44:52 postfix:- (1084:tqdm_logging.py:145)
13:42:51.544 [I] Progress on: 1.37kit/1.50kit rate:7.1s/it remaining:15:31 elapsed:2:45:06 postfix:- (1084:tqdm_logging.py:145)
13:43:05.659 [I] Progress on: 1.37kit/1.50kit rate:7.1s/it remaining:15:17 elapsed:2:45:20 postfix:- (1084:tqdm_logging.py:145)
13:43:19.772 [I] Progress on: 1.37kit/1.50kit rate:7.1s/it remaining:15:03 elapsed:2:45:34 postfix:- (1084:tqdm_logging.py:145)
13:43:33.885 [I] Progress on: 1.37kit/1.50kit rate:7.1s/it remaining:14:49 elapsed:2:45:49 postfix:- (1084:tqdm_logging.py:145)
13:43:48.032 [I] Progress on: 1.38kit/1.50kit rate:7.1s/it remaining:14:36 elapsed:2:46:03 postfix:- (1084:tqdm_logging.py:145)
13:44:02.158 [I] Progress on: 1.38kit/1.50kit rate:7.1s/it remaining:14:21 elapsed:2:46:17 postfix:- (1084:tqdm_logging.py:145)
13:44:16.278 [I] Progress on: 1.38kit/1.50kit rate:7.1s/it remaining:14:07 elapsed:2:46:31 postfix:- (1084:tqdm_logging.py:145)
13:44:30.375 [I] Progress on: 1.38kit/1.50kit rate:7.1s/it remaining:13:52 elapsed:2:46:45 postfix:- (1084:tqdm_logging.py:145)
13:44:44.526 [I] Progress on: 1.38kit/1.50kit rate:7.1s/it remaining:13:39 elapsed:2:46:59 postfix:- (1084:tqdm_logging.py:145)
13:44:58.648 [I] Progress on: 1.39kit/1.50kit rate:7.1s/it remaining:13:25 elapsed:2:47:13 postfix:- (1084:tqdm_logging.py:145)
13:45:13.085 [I] Progress on: 1.39kit/1.50kit rate:7.1s/it remaining:13:18 elapsed:2:47:28 postfix:- (1084:tqdm_logging.py:145)
13:45:27.208 [I] Progress on: 1.39kit/1.50kit rate:7.1s/it remaining:13:00 elapsed:2:47:42 postfix:- (1084:tqdm_logging.py:145)
13:45:41.352 [I] Progress on: 1.39kit/1.50kit rate:7.1s/it remaining:12:44 elapsed:2:47:56 postfix:- (1084:tqdm_logging.py:145)
13:45:55.481 [I] Progress on: 1.39kit/1.50kit rate:7.1s/it remaining:12:29 elapsed:2:48:10 postfix:- (1084:tqdm_logging.py:145)
13:46:09.605 [I] Progress on: 1.40kit/1.50kit rate:7.1s/it remaining:12:14 elapsed:2:48:24 postfix:- (1084:tqdm_logging.py:145)
13:46:23.754 [I] Progress on: 1.40kit/1.50kit rate:7.1s/it remaining:12:01 elapsed:2:48:38 postfix:- (1084:tqdm_logging.py:145)
13:46:37.848 [I] Progress on: 1.40kit/1.50kit rate:7.1s/it remaining:11:45 elapsed:2:48:53 postfix:- (1084:tqdm_logging.py:145)


Step 1400: grad_norm=0.0406, loss=0.0028, param_norm=1804.2515
13:46:51.661 [I] Progress on: 1.40kit/1.50kit rate:7.1s/it remaining:11:45 elapsed:2:49:06 postfix:- (1084:tqdm_logging.py:145)
13:47:06.253 [I] Progress on: 1.40kit/1.50kit rate:6.8s/it remaining:10:51 elapsed:2:49:21 postfix:- (1084:tqdm_logging.py:145)
13:47:20.357 [I] Progress on: 1.41kit/1.50kit rate:6.9s/it remaining:10:50 elapsed:2:49:35 postfix:- (1084:tqdm_logging.py:145)
13:47:34.456 [I] Progress on: 1.41kit/1.50kit rate:7.0s/it remaining:10:42 elapsed:2:49:49 postfix:- (1084:tqdm_logging.py:145)
13:47:48.571 [I] Progress on: 1.41kit/1.50kit rate:7.0s/it remaining:10:31 elapsed:2:50:03 postfix:- (1084:tqdm_logging.py:145)
13:48:02.686 [I] Progress on: 1.41kit/1.50kit rate:7.0s/it remaining:10:19 elapsed:2:50:17 postfix:- (1084:tqdm_logging.py:145)
13:48:16.805 [I] Progress on: 1.41kit/1.50kit rate:7.0s/it remaining:10:06 elapsed:2:50:31 postfix:- (1084:tqdm_logging.py:145)
13:48:30.931 [I] Progress on: 1.42kit/1.50kit rate:7.1s/it remaining:09:52 elapsed:2:50:46 postfix:- (1084:tqdm_logging.py:145)
13:48:45.050 [I] Progress on: 1.42kit/1.50kit rate:7.1s/it remaining:09:38 elapsed:2:51:00 postfix:- (1084:tqdm_logging.py:145)
13:48:59.166 [I] Progress on: 1.42kit/1.50kit rate:7.1s/it remaining:09:24 elapsed:2:51:14 postfix:- (1084:tqdm_logging.py:145)
13:49:13.274 [I] Progress on: 1.42kit/1.50kit rate:7.1s/it remaining:09:10 elapsed:2:51:28 postfix:- (1084:tqdm_logging.py:145)
13:49:27.401 [I] Progress on: 1.42kit/1.50kit rate:7.1s/it remaining:08:56 elapsed:2:51:42 postfix:- (1084:tqdm_logging.py:145)
13:49:41.518 [I] Progress on: 1.43kit/1.50kit rate:7.1s/it remaining:08:42 elapsed:2:51:56 postfix:- (1084:tqdm_logging.py:145)
13:49:55.657 [I] Progress on: 1.43kit/1.50kit rate:7.1s/it remaining:08:28 elapsed:2:52:10 postfix:- (1084:tqdm_logging.py:145)
13:50:09.776 [I] Progress on: 1.43kit/1.50kit rate:7.1s/it remaining:08:14 elapsed:2:52:24 postfix:- (1084:tqdm_logging.py:145)
13:50:23.893 [I] Progress on: 1.43kit/1.50kit rate:7.1s/it remaining:08:00 elapsed:2:52:39 postfix:- (1084:tqdm_logging.py:145)
13:50:38.023 [I] Progress on: 1.43kit/1.50kit rate:7.1s/it remaining:07:46 elapsed:2:52:53 postfix:- (1084:tqdm_logging.py:145)
13:50:52.152 [I] Progress on: 1.44kit/1.50kit rate:7.1s/it remaining:07:31 elapsed:2:53:07 postfix:- (1084:tqdm_logging.py:145)
13:51:06.293 [I] Progress on: 1.44kit/1.50kit rate:7.1s/it remaining:07:18 elapsed:2:53:21 postfix:- (1084:tqdm_logging.py:145)
13:51:20.434 [I] Progress on: 1.44kit/1.50kit rate:7.1s/it remaining:07:04 elapsed:2:53:35 postfix:- (1084:tqdm_logging.py:145)
13:51:34.549 [I] Progress on: 1.44kit/1.50kit rate:7.1s/it remaining:06:49 elapsed:2:53:49 postfix:- (1084:tqdm_logging.py:145)
13:51:48.666 [I] Progress on: 1.44kit/1.50kit rate:7.1s/it remaining:06:35 elapsed:2:54:03 postfix:- (1084:tqdm_logging.py:145)
13:52:02.793 [I] Progress on: 1.45kit/1.50kit rate:7.1s/it remaining:06:21 elapsed:2:54:17 postfix:- (1084:tqdm_logging.py:145)
13:52:16.944 [I] Progress on: 1.45kit/1.50kit rate:7.1s/it remaining:06:07 elapsed:2:54:32 postfix:- (1084:tqdm_logging.py:145)
13:52:31.059 [I] Progress on: 1.45kit/1.50kit rate:7.1s/it remaining:05:53 elapsed:2:54:46 postfix:- (1084:tqdm_logging.py:145)


Step 1450: grad_norm=0.0395, loss=0.0030, param_norm=1804.2515
13:52:44.807 [I] Progress on: 1.45kit/1.50kit rate:7.1s/it remaining:05:53 elapsed:2:54:59 postfix:- (1084:tqdm_logging.py:145)
13:52:59.318 [I] Progress on: 1.45kit/1.50kit rate:6.8s/it remaining:05:11 elapsed:2:55:14 postfix:- (1084:tqdm_logging.py:145)
13:53:13.449 [I] Progress on: 1.46kit/1.50kit rate:6.9s/it remaining:05:04 elapsed:2:55:28 postfix:- (1084:tqdm_logging.py:145)
13:53:27.571 [I] Progress on: 1.46kit/1.50kit rate:7.0s/it remaining:04:53 elapsed:2:55:42 postfix:- (1084:tqdm_logging.py:145)
13:53:41.716 [I] Progress on: 1.46kit/1.50kit rate:7.0s/it remaining:04:41 elapsed:2:55:56 postfix:- (1084:tqdm_logging.py:145)
13:53:55.843 [I] Progress on: 1.46kit/1.50kit rate:7.0s/it remaining:04:27 elapsed:2:56:11 postfix:- (1084:tqdm_logging.py:145)
13:54:09.968 [I] Progress on: 1.46kit/1.50kit rate:7.1s/it remaining:04:14 elapsed:2:56:25 postfix:- (1084:tqdm_logging.py:145)
13:54:24.092 [I] Progress on: 1.47kit/1.50kit rate:7.1s/it remaining:04:00 elapsed:2:56:39 postfix:- (1084:tqdm_logging.py:145)
13:54:38.200 [I] Progress on: 1.47kit/1.50kit rate:7.1s/it remaining:03:45 elapsed:2:56:53 postfix:- (1084:tqdm_logging.py:145)
13:54:52.345 [I] Progress on: 1.47kit/1.50kit rate:7.1s/it remaining:03:31 elapsed:2:57:07 postfix:- (1084:tqdm_logging.py:145)
13:55:06.455 [I] Progress on: 1.47kit/1.50kit rate:7.1s/it remaining:03:17 elapsed:2:57:21 postfix:- (1084:tqdm_logging.py:145)
13:55:21.102 [I] Progress on: 1.47kit/1.50kit rate:7.2s/it remaining:03:07 elapsed:2:57:36 postfix:- (1084:tqdm_logging.py:145)
13:55:34.704 [I] Progress on: 1.48kit/1.50kit rate:7.0s/it remaining:02:48 elapsed:2:57:49 postfix:- (1084:tqdm_logging.py:145)
13:55:48.843 [I] Progress on: 1.48kit/1.50kit rate:7.0s/it remaining:02:35 elapsed:2:58:04 postfix:- (1084:tqdm_logging.py:145)
13:56:02.974 [I] Progress on: 1.48kit/1.50kit rate:7.1s/it remaining:02:21 elapsed:2:58:18 postfix:- (1084:tqdm_logging.py:145)
13:56:17.098 [I] Progress on: 1.48kit/1.50kit rate:7.1s/it remaining:02:07 elapsed:2:58:32 postfix:- (1084:tqdm_logging.py:145)
13:56:31.235 [I] Progress on: 1.48kit/1.50kit rate:7.1s/it remaining:01:53 elapsed:2:58:46 postfix:- (1084:tqdm_logging.py:145)
13:56:45.346 [I] Progress on: 1.49kit/1.50kit rate:7.1s/it remaining:01:38 elapsed:2:59:00 postfix:- (1084:tqdm_logging.py:145)
13:56:59.466 [I] Progress on: 1.49kit/1.50kit rate:7.1s/it remaining:01:24 elapsed:2:59:14 postfix:- (1084:tqdm_logging.py:145)
13:57:13.596 [I] Progress on: 1.49kit/1.50kit rate:7.1s/it remaining:01:10 elapsed:2:59:28 postfix:- (1084:tqdm_logging.py:145)
13:57:27.712 [I] Progress on: 1.49kit/1.50kit rate:7.1s/it remaining:00:56 elapsed:2:59:42 postfix:- (1084:tqdm_logging.py:145)
13:57:41.855 [I] Progress on: 1.49kit/1.50kit rate:7.1s/it remaining:00:42 elapsed:2:59:57 postfix:- (1084:tqdm_logging.py:145)
13:57:55.993 [I] Progress on: 1.50kit/1.50kit rate:7.1s/it remaining:00:28 elapsed:3:00:11 postfix:- (1084:tqdm_logging.py:145)
13:58:10.116 [I] Progress on: 1.50kit/1.50kit rate:7.1s/it remaining:00:14 elapsed:3:00:25 postfix:- (1084:tqdm_logging.py:145)
13:58:24.240 [I] [process=0][thread=MainThread][wait_until_finished] No Save Finalize thread to wait for. Returning. (1084:checkpoint_manager.py:1987)
13:58:24.241 [I] [process=0] Saving checkpoint at step 1499                                       (1084:checkpoint_manager.py:1408)
13:58:24.242 [I] [process=0] Started async saving checkpoint to /workspace/openpi/checkpoints/pi05_mecharm_trim_corr_v5_lora/trim_corr_v5/1499. (1084:async_checkpointer.py:439)
13:58:24.254 [I] Creating tmp directory /workspace/openpi/checkpoints/pi05_mecharm_trim_corr_v5_lora/trim_corr_v5/1499.orbax-checkpoint-tmp-4 (1084:atomicity.py:144)
13:58:24.260 [I] Creating tmp directory /workspace/openpi/checkpoints/pi05_mecharm_trim_corr_v5_lora/trim_corr_v5/1499.orbax-checkpoint-tmp-4/params.orbax-checkpoint-tmp-6 (1084:atomicity.py:144)
13:58:24.260 [I] Creating tmp directory /workspace/openpi/checkpoints/pi05_mecharm_trim_corr_v5_lora/trim_corr_v5/1499.orbax-checkpoint-tmp-4/assets.orbax-checkpoint-tmp-5 (1084:atomicity.py:144)
13:58:24.261 [I] Creating tmp directory /workspace/openpi/checkpoints/pi05_mecharm_trim_corr_v5_lora/trim_corr_v5/1499.orbax-checkpoint-tmp-4/train_state.orbax-checkpoint-tmp-7 (1084:atomicity.py:144)
13:58:24.274 [I] Transferring arrays to host memory with options: use_replica_parallel=True, enable_pinned_host_transfer=False (1084:replica_slices.py:341)
13:58:33.591 [I] Transferring arrays to host memory with options: use_replica_parallel=True, enable_pinned_host_transfer=False (1084:replica_slices.py:341)
13:58:33.605 [I] Array name: 'params.PaliGemma.llm.layers.mlp.gating_einsum.value', global shape: (18, 2, 2048, 16384), write shape: (18, 2, 2048, 16384), chosen chunk shape: (18, 2, 2048, 8192) (1084:tensorstore_utils.py:408)
13:58:33.648 [I] [process=0][thread=array_type_handler] Wrote 71 array_metadata.ArrayMetadata to /workspace/openpi/checkpoints/pi05_mecharm_trim_corr_v5_lora/trim_corr_v5/1499.orbax-checkpoint-tmp-4/params.orbax-checkpoint-tmp-6/array_metadatas/process_0 (1084:array_metadata_store.py:198)
13:58:36.057 [I] [process=0] /jax/checkpoint/write/blocking_bytes_per_sec: 625.0 MiB/s (total bytes: 7.2 GiB) (time elapsed: 11 seconds) (per-host) (1084:base_pytree_checkpoint_handler.py:114)
13:58:36.064 [I] [process=0] /jax/checkpoint/write/blocking_bytes_per_sec: 301.6 MiB/s (total bytes: 3.5 GiB) (time elapsed: 11 seconds) (per-host) (1084:base_pytree_checkpoint_handler.py:114)
13:58:36.074 [I] [process=0][thread=async_save] Background save thread started.                   (1084:async_checkpointer.py:77)
13:58:36.076 [I] Finished blocking save in 11.83 seconds. Continuing to save asynchronously to /workspace/openpi/checkpoints/pi05_mecharm_trim_corr_v5_lora/trim_corr_v5/1499. (1084:async_checkpointer.py:548)
13:58:36.078 [I] Preserving Checkpoint[step=1000 | time=2026-11-06 12:59:45.343153+00:00]: (Reason: on keep_period=1000). (1084:checkpoint_manager.py:1949)
13:58:36.078 [I] [process=0][thread=MainThread][step=1499] Starting CheckpointManager Save Finalize thread=save_finalize (1084:checkpoint_manager.py:1453)
13:58:36.080 [I] [process=0][thread=save_finalize] Waiting for background save thread=async_save. (1084:async_checkpointer.py:258)
13:58:36.081 [I] {'step': 1499, 'event_type': 'save', 'directory': '/workspace/openpi/checkpoints/pi05_mecharm_trim_corr_v5_lora/trim_corr_v5', 'reached_preemption': False, 'preemption_received_at': None, 'synchronous': False, 'wait_for_prev_start_time': 1793973504.2408435, 'wait_for_prev_duration_secs': 0.00017714500427246094, 'checkpointer_blocking_start_time': 1793973504.2415597, 'checkpointer_blocking_duration_secs': 11.836369276046753, 'get_old_steps_start_time': 1793973516.0779653, 'get_old_steps_duration_secs': 0.0005509853363037109, 'checkpoint_manager_blocking_start_time': 1793973504.2404659, 'checkpoint_manager_blocking_duration_secs': 11.841144323348999} (1084:standard_logger.py:34)
13:58:36.082 [I] Progress on: 1.50kit/1.50kit rate:10.6s/it remaining:00:00 elapsed:3:00:51 postfix:- (1084:tqdm_logging.py:145)
13:58:36.082 [I] Progress on: 1.50kit/1.50kit rate:10.6s/it remaining:00:00 elapsed:3:00:51 postfix:- (1084:tqdm_logging.py:145)
13:58:36.082 [I] Waiting for checkpoint manager to finish                                         (1084:train.py:275)
13:58:36.083 [I] [process=0][thread=MainThread][step=1499][wait_until_finished] Waiting for Save Finalize thread (save_finalize) to complete. (1084:checkpoint_manager.py:1998)
13:58:36.128 [I] [process=0][thread=array_type_handler] Wrote 105 array_metadata.ArrayMetadata to /workspace/openpi/checkpoints/pi05_mecharm_trim_corr_v5_lora/trim_corr_v5/1499.orbax-checkpoint-tmp-4/train_state.orbax-checkpoint-tmp-7/array_metadatas/process_0 (1084:array_metadata_store.py:198)
13:58:39.664 [I] [process=0] /jax/checkpoint/write/bytes_per_sec: 478.8 MiB/s (total bytes: 7.2 GiB) (time elapsed: 15 seconds) (per-host) (1084:base_pytree_checkpoint_handler.py:114)
13:58:39.664 [I] [process=0] /jax/checkpoint/write/bytes_per_sec: 231.2 MiB/s (total bytes: 3.5 GiB) (time elapsed: 15 seconds) (per-host) (1084:base_pytree_checkpoint_handler.py:114)
13:58:39.664 [I] [process=0][thread=async_save] 5 Handler Commit operations completed.            (1084:async_checkpointer.py:87)
13:58:39.665 [I] Renaming /workspace/openpi/checkpoints/pi05_mecharm_trim_corr_v5_lora/trim_corr_v5/1499.orbax-checkpoint-tmp-4/assets.orbax-checkpoint-tmp-5 to /workspace/openpi/checkpoints/pi05_mecharm_trim_corr_v5_lora/trim_corr_v5/1499.orbax-checkpoint-tmp-4/assets (1084:atomicity.py:289)
13:58:39.667 [I] [process=0][thread=async_save] Skipped cross-host ArrayMetadata validation because only one process is found: process_index=0. (1084:array_metadata_store.py:362)
13:58:39.671 [I] Renaming /workspace/openpi/checkpoints/pi05_mecharm_trim_corr_v5_lora/trim_corr_v5/1499.orbax-checkpoint-tmp-4/train_state.orbax-checkpoint-tmp-7 to /workspace/openpi/checkpoints/pi05_mecharm_trim_corr_v5_lora/trim_corr_v5/1499.orbax-checkpoint-tmp-4/train_state (1084:atomicity.py:289)
13:58:39.672 [I] [process=0][thread=async_save] Skipped cross-host ArrayMetadata validation because only one process is found: process_index=0. (1084:array_metadata_store.py:362)
13:58:39.680 [I] Renaming /workspace/openpi/checkpoints/pi05_mecharm_trim_corr_v5_lora/trim_corr_v5/1499.orbax-checkpoint-tmp-4/params.orbax-checkpoint-tmp-6 to /workspace/openpi/checkpoints/pi05_mecharm_trim_corr_v5_lora/trim_corr_v5/1499.orbax-checkpoint-tmp-4/params (1084:atomicity.py:289)
13:58:39.681 [I] Renaming /workspace/openpi/checkpoints/pi05_mecharm_trim_corr_v5_lora/trim_corr_v5/1499.orbax-checkpoint-tmp-4 to /workspace/openpi/checkpoints/pi05_mecharm_trim_corr_v5_lora/trim_corr_v5/1499 (1084:atomicity.py:289)
13:58:39.681 [I] [process=0][thread=async_save] Finished saving checkpoint (finalized tmp dir) to `/workspace/openpi/checkpoints/pi05_mecharm_trim_corr_v5_lora/trim_corr_v5/1499`. (1084:atomicity.py:573)
13:58:39.681 [I] Finished asynchronous save (blocking + background) in 15.44 seconds to /workspace/openpi/checkpoints/pi05_mecharm_trim_corr_v5_lora/trim_corr_v5/1499 (1084:async_checkpointer.py:412)
13:58:39.681 [I] [process=0][thread=async_save] Background save thread done.                      (1084:async_checkpointer.py:143)
13:58:39.682 [I] [process=0][thread=save_finalize] Done with waiting for background save thread=async_save. (1084:async_checkpointer.py:266)
13:58:39.682 [I] [process=0][thread=save_finalize] No errors found in background save thread=async_save. (1084:async_checkpointer.py:276)
13:58:39.682 [I] [process=0][thread=save_finalize][step=1499] CheckpointManager Save Finalize is syncing with other hosts... (1084:checkpoint_manager.py:2095)
13:58:39.682 [I] [process=0][thread=save_finalize][step=1499] CheckpointManager Save Finalize is done on all hosts. (1084:checkpoint_manager.py:2110)
13:58:39.682 [I] [process=0][thread=MainThread][step=1499][wait_until_finished] Done waiting for Save Finalize thread (save_finalize) running at step=1499. (1084:checkpoint_manager.py:2010)
