13:20:22.369 [I] Running on: 2b07075eed8d                                                         (1143:train.py:196)
INFO:2026-10-31 13:20:22,794:jax._src.xla_bridge:925: Unable to initialize backend 'rocm': module 'jaxlib.xla_extension' has no attribute 'GpuAllocatorConfig'
13:20:22.794 [I] Unable to initialize backend 'rocm': module 'jaxlib.xla_extension' has no attribute 'GpuAllocatorConfig' (1143:xla_bridge.py:925)
INFO:2026-10-31 13:20:22,799:jax._src.xla_bridge:925: Unable to initialize backend 'tpu': INTERNAL: Failed to open libtpu.so: libtpu.so: cannot open shared object file: No such file or directory
13:20:22.799 [I] Unable to initialize backend 'tpu': INTERNAL: Failed to open libtpu.so: libtpu.so: cannot open shared object file: No such file or directory (1143:xla_bridge.py:925)
13:20:23.780 [I] Created BasePyTreeCheckpointHandler: use_ocdbt=True, use_zarr3=False, pytree_metadata_options=PyTreeMetadataOptions(support_rich_types=False), array_metadata_store=<orbax.checkpoint._src.metadata.array_metadata_store.Store object at 0x7f3614dcbf10> (1143:base_pytree_checkpoint_handler.py:334)
13:20:23.780 [I] Created BasePyTreeCheckpointHandler: use_ocdbt=True, use_zarr3=False, pytree_metadata_options=PyTreeMetadataOptions(support_rich_types=False), array_metadata_store=<orbax.checkpoint._src.metadata.array_metadata_store.Store object at 0x7f3614dcbf10> (1143:base_pytree_checkpoint_handler.py:334)
13:20:23.781 [I] [thread=MainThread] Failed to get flag value for EXPERIMENTAL_ORBAX_USE_DISTRIBUTED_PROCESS_ID. (1143:multihost.py:390)
13:20:23.781 [I] [process=0][thread=MainThread] CheckpointManager init: checkpointers=None, item_names=None, item_handlers={'assets': <openpi.training.checkpoints.CallbackHandler object at 0x7f3476393cd0>, 'train_state': <orbax.checkpoint._src.handlers.pytree_checkpoint_handler.PyTreeCheckpointHandler object at 0x7f347615c190>, 'params': <orbax.checkpoint._src.handlers.pytree_checkpoint_handler.PyTreeCheckpointHandler object at 0x7f34774d1f50>}, handler_registry=None (1143:checkpoint_manager.py:620)
13:20:23.781 [I] Deferred registration for item: "assets". Adding handler `<openpi.training.checkpoints.CallbackHandler object at 0x7f3476393cd0>` for item "assets" and save args `<class 'openpi.training.checkpoints.CallbackSave'>` and restore args `<class 'openpi.training.checkpoints.CallbackRestore'>` to `_handler_registry`. (1143:composite_checkpoint_handler.py:234)
13:20:23.781 [I] Deferred registration for item: "train_state". Adding handler `<orbax.checkpoint._src.handlers.pytree_checkpoint_handler.PyTreeCheckpointHandler object at 0x7f347615c190>` for item "train_state" and save args `<class 'orbax.checkpoint._src.handlers.pytree_checkpoint_handler.PyTreeSaveArgs'>` and restore args `<class 'orbax.checkpoint._src.handlers.pytree_checkpoint_handler.PyTreeRestoreArgs'>` to `_handler_registry`. (1143:composite_checkpoint_handler.py:234)
13:20:23.781 [I] Deferred registration for item: "params". Adding handler `<orbax.checkpoint._src.handlers.pytree_checkpoint_handler.PyTreeCheckpointHandler object at 0x7f34774d1f50>` for item "params" and save args `<class 'orbax.checkpoint._src.handlers.pytree_checkpoint_handler.PyTreeSaveArgs'>` and restore args `<class 'orbax.checkpoint._src.handlers.pytree_checkpoint_handler.PyTreeRestoreArgs'>` to `_handler_registry`. (1143:composite_checkpoint_handler.py:234)
13:20:23.781 [I] Deferred registration for item: "metrics". Adding handler `<orbax.checkpoint._src.handlers.json_checkpoint_handler.JsonCheckpointHandler object at 0x7f34766021d0>` for item "metrics" and save args `<class 'orbax.checkpoint._src.handlers.json_checkpoint_handler.JsonSaveArgs'>` and restore args `<class 'orbax.checkpoint._src.handlers.json_checkpoint_handler.JsonRestoreArgs'>` to `_handler_registry`. (1143:composite_checkpoint_handler.py:234)
13:20:23.781 [I] Initialized registry DefaultCheckpointHandlerRegistry({('assets', <class 'openpi.training.checkpoints.CallbackSave'>): <openpi.training.checkpoints.CallbackHandler object at 0x7f3476393cd0>, ('assets', <class 'openpi.training.checkpoints.CallbackRestore'>): <openpi.training.checkpoints.CallbackHandler object at 0x7f3476393cd0>, ('train_state', <class 'orbax.checkpoint._src.handlers.pytree_checkpoint_handler.PyTreeSaveArgs'>): <orbax.checkpoint._src.handlers.pytree_checkpoint_handler.PyTreeCheckpointHandler object at 0x7f347615c190>, ('train_state', <class 'orbax.checkpoint._src.handlers.pytree_checkpoint_handler.PyTreeRestoreArgs'>): <orbax.checkpoint._src.handlers.pytree_checkpoint_handler.PyTreeCheckpointHandler object at 0x7f347615c190>, ('params', <class 'orbax.checkpoint._src.handlers.pytree_checkpoint_handler.PyTreeSaveArgs'>): <orbax.checkpoint._src.handlers.pytree_checkpoint_handler.PyTreeCheckpointHandler object at 0x7f34774d1f50>, ('params', <class 'orbax.checkpoint._src.handlers.pytree_checkpoint_handler.PyTreeRestoreArgs'>): <orbax.checkpoint._src.handlers.pytree_checkpoint_handler.PyTreeCheckpointHandler object at 0x7f34774d1f50>, ('metrics', <class 'orbax.checkpoint._src.handlers.json_checkpoint_handler.JsonSaveArgs'>): <orbax.checkpoint._src.handlers.json_checkpoint_handler.JsonCheckpointHandler object at 0x7f34766021d0>, ('metrics', <class 'orbax.checkpoint._src.handlers.json_checkpoint_handler.JsonRestoreArgs'>): <orbax.checkpoint._src.handlers.json_checkpoint_handler.JsonCheckpointHandler object at 0x7f34766021d0>}). (1143:composite_checkpoint_handler.py:502)
13:20:23.782 [I] orbax-checkpoint version: 0.11.13                                                (1143:abstract_checkpointer.py:35)
13:20:23.782 [I] [process=0][thread=MainThread] Using barrier_sync_fn: <function get_barrier_sync_fn.<locals>.<lambda> at 0x7f3475dda7a0> timeout: 7200 secs and primary_host=0 for async checkpoint writes (1143:async_checkpointer.py:170)
13:20:23.782 [I] Found 0 checkpoint steps in /workspace/openpi/checkpoints/pi05_mecharm_trim_corr_lora/trim_corr_v3 (1143:checkpoint_manager.py:1701)
13:20:23.782 [I] [process=0][thread=MainThread] CheckpointManager created,  primary_host=0, CheckpointManagerOptions=CheckpointManagerOptions(save_interval_steps=1, max_to_keep=1, keep_time_interval=None, keep_period=1000, should_keep_fn=None, best_fn=None, best_mode='max', keep_checkpoints_without_metrics=True, step_prefix=None, step_format_fixed_length=None, step_name_format=None, create=False, cleanup_tmp_directories=False, save_on_steps=frozenset(), single_host_load_and_broadcast=False, todelete_subdir=None, enable_background_delete=False, read_only=False, enable_async_checkpointing=True, async_options=AsyncOptions(timeout_secs=7200, barrier_sync_fn=None, post_finalization_callback=None, create_directories_asynchronously=True), multiprocessing_options=MultiprocessingOptions(primary_host=0, active_processes=None, barrier_sync_key_prefix=None), should_save_fn=None, file_options=FileOptions(path_permission_mode=None), save_root_metadata=True, temporary_path_class=None, save_decision_policy=None, prevent_write_metrics=False), root_directory=/workspace/openpi/checkpoints/pi05_mecharm_trim_corr_lora/trim_corr_v3: <orbax.checkpoint.checkpoint_manager.CheckpointManager object at 0x7f3476210d90> (1143:checkpoint_manager.py:801)
13:20:23.792 [I] Downloading gs://big_vision/paligemma_tokenizer.model to /root/.cache/openpi/big_vision/paligemma_tokenizer.model (1143:download.py:86)
13:20:24.179 [I] Progress on: -/4264023 rate:- remaining:? elapsed:00:00 postfix:-                (1143:tqdm_logging.py:145)
13:20:25.181 [I] Progress on: 4.07MiB/4.07MiB rate:4,260,153.2iB/s remaining:00:00 elapsed:00:01 postfix:- (1143:tqdm_logging.py:145)
13:20:25.322 [I] Loaded norm stats from /workspace/openpi/assets/pi05_mecharm_trim_corr_lora/local/mecharm_redblock_box_teach_v1_trim2_corr_v3 (1143:config.py:197)
13:20:25.323 [I] data_config: DataConfig(repo_id='local/mecharm_redblock_box_teach_v1_trim2_corr_v3', asset_id='local/mecharm_redblock_box_teach_v1_trim2_corr_v3', norm_stats={'state': NormStats(mean=array([  0.11632019,  47.20463562,   4.25571823,   3.47346711,
       -49.80432129, -32.3266716 ,   0.42522314]), std=array([33.52104568, 21.01031494, 13.69488239,  5.18986511, 21.06582069,
        3.66200471,  0.49437681]), q01=array([-61.62614893,   1.48315994, -34.35241028,  -7.64541887,
       -86.05951349, -40.59999847,   0.        ]), q99=array([ 44.61974199,  87.87268339,  30.92747083,  16.58067341,
        -0.260924  , -26.45196724,   0.9998    ])), 'actions': NormStats(mean=array([ 0.192315  ,  0.027426  , -0.01522407,  0.04785711, -0.08074823,
       -0.05333935,  0.42522326]), std=array([9.28794479, 9.00317574, 4.14188147, 1.60169351, 9.50789356,
       0.9818573 , 0.49437684]), q01=array([-21.17317298, -23.18362675, -14.27432952,  -5.18250646,
       -26.67564539,  -3.16441521,   0.        ]), q99=array([27.84317563, 22.359031  , 11.17634333,  4.02140303, 25.71650509,
        3.24633011,  0.9998    ]))}, repack_transforms=Group(inputs=[RepackTransform(structure={'observation/image': 'image', 'observation/state': 'state', 'actions': 'actions', 'prompt': 'prompt'})], outputs=()), data_transforms=Group(inputs=(MecharmInputs(model_type=<ModelType.PI05: 'pi05'>), DeltaActions(mask=(True, True, True, True, True, True, False))), outputs=(AbsoluteActions(mask=(True, True, True, True, True, True, False)), MecharmOutputs())), model_transforms=Group(inputs=[InjectDefaultPrompt(prompt=None), ResizeImages(height=224, width=224), TokenizePrompt(tokenizer=<openpi.models.tokenizer.PaligemmaTokenizer object at 0x7f3476393950>, discrete_state_input=True), PadStatesAndActions(model_action_dim=32)], outputs=()), use_quantile_norm=True, action_sequence_keys=('actions',), prompt_from_task=True, rlds_data_dir=None, action_space=None, datasets=()) (1143:data_loader.py:243)

Downloading data:   0%|          | 0/24 [00:00<?, ?files/s]
Downloading data: 100%|██████████| 24/24 [00:00<00:00, 111600.11files/s]

Generating train split: 0 examples [00:00, ? examples/s]
Generating train split: 496 examples [00:00, 2369.91 examples/s]
Generating train split: 826 examples [00:00, 2239.10 examples/s]
Generating train split: 1261 examples [00:00, 2455.92 examples/s]
Generating train split: 1841 examples [00:00, 3127.10 examples/s]
Generating train split: 2420 examples [00:00, 2972.05 examples/s]
Generating train split: 3051 examples [00:00, 3545.91 examples/s]
Generating train split: 3443 examples [00:01, 3203.04 examples/s]
13:20:28.035 [I] local_batch_size: 32                                                             (1143:data_loader.py:324)
13:20:42.264 [I] Initialized data loader:
[0].images['base_0_rgb']: (32, 224, 224, 3)@float32
[0].images['left_wrist_0_rgb']: (32, 224, 224, 3)@float32
[0].images['right_wrist_0_rgb']: (32, 224, 224, 3)@float32
[0].image_masks['base_0_rgb']: (32,)@bool
[0].image_masks['left_wrist_0_rgb']: (32,)@bool
[0].image_masks['right_wrist_0_rgb']: (32,)@bool
[0].state: (32, 32)@float32
[0].tokenized_prompt: (32, 200)@int32
[0].tokenized_prompt_mask: (32, 200)@bool
[1]: (32, 10, 32)@float32 (1143:train.py:227)
13:20:43.483 [I] Created BasePyTreeCheckpointHandler: use_ocdbt=True, use_zarr3=False, pytree_metadata_options=PyTreeMetadataOptions(support_rich_types=False), array_metadata_store=<orbax.checkpoint._src.metadata.array_metadata_store.Store object at 0x7f3614dcbf10> (1143:base_pytree_checkpoint_handler.py:334)
13:20:43.513 [I] Restoring checkpoint from /workspace/ckpt/3999/params.                           (1143:checkpointer.py:298)
13:20:48.394 [I] [process=0] /jax/checkpoint/read/bytes_per_sec: 1.5 GiB/s (total bytes: 7.2 GiB) (time elapsed: 4 seconds) (per-host) (1143:base_pytree_checkpoint_handler.py:114)
13:20:48.395 [I] Finished restoring checkpoint in 4.88 seconds from /workspace/ckpt/3999/params.  (1143:checkpointer.py:309)
13:20:51.375 [I] Initialized train state:
['PaliGemma']['img']['Transformer']['encoder_norm']['bias'].value: (1152,)@float32
['PaliGemma']['img']['Transformer']['encoder_norm']['scale'].value: (1152,)@float32
['PaliGemma']['img']['Transformer']['encoderblock']['LayerNorm_0']['bias'].value: (27, 1152)@float32
['PaliGemma']['img']['Transformer']['encoderblock']['LayerNorm_0']['scale'].value: (27, 1152)@float32
['PaliGemma']['img']['Transformer']['encoderblock']['LayerNorm_1']['bias'].value: (27, 1152)@float32
['PaliGemma']['img']['Transformer']['encoderblock']['LayerNorm_1']['scale'].value: (27, 1152)@float32
['PaliGemma']['img']['Transformer']['encoderblock']['MlpBlock_0']['Dense_0']['bias'].value: (27, 4304)@float32
['PaliGemma']['img']['Transformer']['encoderblock']['MlpBlock_0']['Dense_0']['kernel'].value: (27, 1152, 4304)@float32
['PaliGemma']['img']['Transformer']['encoderblock']['MlpBlock_0']['Dense_1']['bias'].value: (27, 1152)@float32
['PaliGemma']['img']['Transformer']['encoderblock']['MlpBlock_0']['Dense_1']['kernel'].value: (27, 4304, 1152)@float32
['PaliGemma']['img']['Transformer']['encoderblock']['MultiHeadDotProductAttention_0']['key']['bias'].value: (27, 16, 72)@float32
['PaliGemma']['img']['Transformer']['encoderblock']['MultiHeadDotProductAttention_0']['key']['kernel'].value: (27, 1152, 16, 72)@float32
['PaliGemma']['img']['Transformer']['encoderblock']['MultiHeadDotProductAttention_0']['out']['bias'].value: (27, 1152)@float32
['PaliGemma']['img']['Transformer']['encoderblock']['MultiHeadDotProductAttention_0']['out']['kernel'].value: (27, 16, 72, 1152)@float32
['PaliGemma']['img']['Transformer']['encoderblock']['MultiHeadDotProductAttention_0']['query']['bias'].value: (27, 16, 72)@float32
['PaliGemma']['img']['Transformer']['encoderblock']['MultiHeadDotProductAttention_0']['query']['kernel'].value: (27, 1152, 16, 72)@float32
['PaliGemma']['img']['Transformer']['encoderblock']['MultiHeadDotProductAttention_0']['value']['bias'].value: (27, 16, 72)@float32
['PaliGemma']['img']['Transformer']['encoderblock']['MultiHeadDotProductAttention_0']['value']['kernel'].value: (27, 1152, 16, 72)@float32
['PaliGemma']['img']['embedding']['bias'].value: (1152,)@float32
['PaliGemma']['img']['embedding']['kernel'].value: (14, 14, 3, 1152)@float32
['PaliGemma']['img']['head']['bias'].value: (2048,)@float32
['PaliGemma']['img']['head']['kernel'].value: (1152, 2048)@float32
['PaliGemma']['img']['pos_embedding'].value: (1, 256, 1152)@float32
['PaliGemma']['llm']['embedder']['input_embedding'].value: (257152, 2048)@bfloat16
['PaliGemma']['llm']['final_norm']['scale'].value: (2048,)@bfloat16
['PaliGemma']['llm']['final_norm_1']['Dense_0']['bias'].value: (3072,)@bfloat16
['PaliGemma']['llm']['final_norm_1']['Dense_0']['kernel'].value: (1024, 3072)@bfloat16
['PaliGemma']['llm']['layers']['attn']['attn_vec_einsum']['lora_a'].value: (18, 8, 256, 16)@float32
['PaliGemma']['llm']['layers']['attn']['attn_vec_einsum']['lora_b'].value: (18, 8, 16, 2048)@float32
['PaliGemma']['llm']['layers']['attn']['attn_vec_einsum']['w'].value: (18, 8, 256, 2048)@bfloat16
['PaliGemma']['llm']['layers']['attn']['attn_vec_einsum_1']['lora_a'].value: (18, 8, 256, 32)@float32
['PaliGemma']['llm']['layers']['attn']['attn_vec_einsum_1']['lora_b'].value: (18, 8, 32, 1024)@float32
['PaliGemma']['llm']['layers']['attn']['attn_vec_einsum_1']['w'].value: (18, 8, 256, 1024)@bfloat16
['PaliGemma']['llm']['layers']['attn']['kv_einsum']['lora_a'].value: (18, 2, 1, 2048, 16)@float32
['PaliGemma']['llm']['layers']['attn']['kv_einsum']['lora_b'].value: (18, 2, 1, 16, 256)@float32
['PaliGemma']['llm']['layers']['attn']['kv_einsum']['w'].value: (18, 2, 1, 2048, 256)@bfloat16
['PaliGemma']['llm']['layers']['attn']['kv_einsum_1']['lora_a'].value: (18, 2, 1, 1024, 32)@float32
['PaliGemma']['llm']['layers']['attn']['kv_einsum_1']['lora_b'].value: (18, 2, 1, 32, 256)@float32
['PaliGemma']['llm']['layers']['attn']['kv_einsum_1']['w'].value: (18, 2, 1, 1024, 256)@bfloat16
['PaliGemma']['llm']['layers']['attn']['q_einsum']['lora_a'].value: (18, 8, 2048, 16)@float32
['PaliGemma']['llm']['layers']['attn']['q_einsum']['lora_b'].value: (18, 8, 16, 256)@float32
['PaliGemma']['llm']['layers']['attn']['q_einsum']['w'].value: (18, 8, 2048, 256)@bfloat16
['PaliGemma']['llm']['layers']['attn']['q_einsum_1']['lora_a'].value: (18, 8, 1024, 32)@float32
['PaliGemma']['llm']['layers']['attn']['q_einsum_1']['lora_b'].value: (18, 8, 32, 256)@float32
['PaliGemma']['llm']['layers']['attn']['q_einsum_1']['w'].value: (18, 8, 1024, 256)@bfloat16
['PaliGemma']['llm']['layers']['mlp']['gating_einsum'].value: (18, 2, 2048, 16384)@bfloat16
['PaliGemma']['llm']['layers']['mlp']['gating_einsum_lora_a'].value: (18, 2, 2048, 16)@float32
['PaliGemma']['llm']['layers']['mlp']['gating_einsum_lora_b'].value: (18, 2, 16, 16384)@float32
['PaliGemma']['llm']['layers']['mlp']['linear'].value: (18, 16384, 2048)@bfloat16
['PaliGemma']['llm']['layers']['mlp']['linear_lora_a'].value: (18, 16384, 16)@float32
['PaliGemma']['llm']['layers']['mlp']['linear_lora_b'].value: (18, 16, 2048)@float32
['PaliGemma']['llm']['layers']['mlp_1']['gating_einsum'].value: (18, 2, 1024, 4096)@bfloat16
['PaliGemma']['llm']['layers']['mlp_1']['gating_einsum_lora_a'].value: (18, 2, 1024, 32)@float32
['PaliGemma']['llm']['layers']['mlp_1']['gating_einsum_lora_b'].value: (18, 2, 32, 4096)@float32
['PaliGemma']['llm']['layers']['mlp_1']['linear'].value: (18, 4096, 1024)@bfloat16
['PaliGemma']['llm']['layers']['mlp_1']['linear_lora_a'].value: (18, 4096, 32)@float32
['PaliGemma']['llm']['layers']['mlp_1']['linear_lora_b'].value: (18, 32, 1024)@float32
['PaliGemma']['llm']['layers']['pre_attention_norm']['scale'].value: (18, 2048)@bfloat16
['PaliGemma']['llm']['layers']['pre_attention_norm_1']['Dense_0']['bias'].value: (18, 3072)@bfloat16
['PaliGemma']['llm']['layers']['pre_attention_norm_1']['Dense_0']['kernel'].value: (18, 1024, 3072)@bfloat16
['PaliGemma']['llm']['layers']['pre_ffw_norm']['scale'].value: (18, 2048)@bfloat16
['PaliGemma']['llm']['layers']['pre_ffw_norm_1']['Dense_0']['bias'].value: (18, 3072)@bfloat16
['PaliGemma']['llm']['layers']['pre_ffw_norm_1']['Dense_0']['kernel'].value: (18, 1024, 3072)@bfloat16
['action_in_proj']['bias'].value: (1024,)@float32
['action_in_proj']['kernel'].value: (32, 1024)@float32
['action_out_proj']['bias'].value: (32,)@float32
['action_out_proj']['kernel'].value: (1024, 32)@float32
['time_mlp_in']['bias'].value: (1024,)@float32
['time_mlp_in']['kernel'].value: (1024, 1024)@float32
['time_mlp_out']['bias'].value: (1024,)@float32
['time_mlp_out']['kernel'].value: (1024, 1024)@float32 (1143:train.py:238)
13:20:51.384 [I] Progress on: -/1500 rate:- remaining:? elapsed:00:00 postfix:-                   (1143:tqdm_logging.py:145)


13:25:37.985 [I] Progress on: -/1500 rate:- remaining:? elapsed:04:46 postfix:-                   (1143:tqdm_logging.py:145)
13:25:52.304 [I] Progress on: 4.00it/1.50kit rate:43.6s/it remaining:18:05:50 elapsed:05:00 postfix:- (1143:tqdm_logging.py:145)
13:26:06.233 [I] Progress on: 6.00it/1.50kit rate:22.4s/it remaining:9:17:50 elapsed:05:14 postfix:- (1143:tqdm_logging.py:145)
13:26:20.192 [I] Progress on: 8.00it/1.50kit rate:14.1s/it remaining:5:49:27 elapsed:05:28 postfix:- (1143:tqdm_logging.py:145)
13:26:34.185 [I] Progress on: 10.0it/1.50kit rate:10.4s/it remaining:4:17:02 elapsed:05:42 postfix:- (1143:tqdm_logging.py:145)
13:26:48.182 [I] Progress on: 12.0it/1.50kit rate:8.6s/it remaining:3:33:40 elapsed:05:56 postfix:- (1143:tqdm_logging.py:145)
13:27:02.241 [I] Progress on: 14.0it/1.50kit rate:7.8s/it remaining:3:13:14 elapsed:06:10 postfix:- (1143:tqdm_logging.py:145)
13:27:16.246 [I] Progress on: 16.0it/1.50kit rate:7.4s/it remaining:3:02:53 elapsed:06:24 postfix:- (1143:tqdm_logging.py:145)
13:27:30.291 [I] Progress on: 18.0it/1.50kit rate:7.2s/it remaining:2:57:56 elapsed:06:38 postfix:- (1143:tqdm_logging.py:145)
13:27:44.365 [I] Progress on: 20.0it/1.50kit rate:7.1s/it remaining:2:55:32 elapsed:06:52 postfix:- (1143:tqdm_logging.py:145)
13:27:58.402 [I] Progress on: 22.0it/1.50kit rate:7.1s/it remaining:2:54:06 elapsed:07:07 postfix:- (1143:tqdm_logging.py:145)
13:28:12.467 [I] Progress on: 24.0it/1.50kit rate:7.1s/it remaining:2:53:27 elapsed:07:21 postfix:- (1143:tqdm_logging.py:145)
13:28:26.559 [I] Progress on: 26.0it/1.50kit rate:7.0s/it remaining:2:53:10 elapsed:07:35 postfix:- (1143:tqdm_logging.py:145)
13:28:40.623 [I] Progress on: 28.0it/1.50kit rate:7.0s/it remaining:2:52:45 elapsed:07:49 postfix:- (1143:tqdm_logging.py:145)
13:28:54.692 [I] Progress on: 30.0it/1.50kit rate:7.0s/it remaining:2:52:26 elapsed:08:03 postfix:- (1143:tqdm_logging.py:145)
13:29:08.773 [I] Progress on: 32.0it/1.50kit rate:7.0s/it remaining:2:52:14 elapsed:08:17 postfix:- (1143:tqdm_logging.py:145)
13:29:22.856 [I] Progress on: 34.0it/1.50kit rate:7.0s/it remaining:2:52:02 elapsed:08:31 postfix:- (1143:tqdm_logging.py:145)
13:29:36.941 [I] Progress on: 36.0it/1.50kit rate:7.0s/it remaining:2:51:49 elapsed:08:45 postfix:- (1143:tqdm_logging.py:145)
13:29:51.047 [I] Progress on: 38.0it/1.50kit rate:7.0s/it remaining:2:51:44 elapsed:08:59 postfix:- (1143:tqdm_logging.py:145)
13:30:05.144 [I] Progress on: 40.0it/1.50kit rate:7.0s/it remaining:2:51:30 elapsed:09:13 postfix:- (1143:tqdm_logging.py:145)
13:30:19.245 [I] Progress on: 42.0it/1.50kit rate:7.0s/it remaining:2:51:18 elapsed:09:27 postfix:- (1143:tqdm_logging.py:145)
13:30:33.346 [I] Progress on: 44.0it/1.50kit rate:7.1s/it remaining:2:51:05 elapsed:09:41 postfix:- (1143:tqdm_logging.py:145)
13:30:47.441 [I] Progress on: 46.0it/1.50kit rate:7.0s/it remaining:2:50:49 elapsed:09:56 postfix:- (1143:tqdm_logging.py:145)
13:31:01.531 [I] Progress on: 48.0it/1.50kit rate:7.0s/it remaining:2:50:34 elapsed:10:10 postfix:- (1143:tqdm_logging.py:145)
13:31:15.628 [I] Progress on: 50.0it/1.50kit rate:7.0s/it remaining:2:50:20 elapsed:10:24 postfix:- (1143:tqdm_logging.py:145)


13:31:29.482 [I] Progress on: 50.0it/1.50kit rate:7.0s/it remaining:2:50:20 elapsed:10:38 postfix:- (1143:tqdm_logging.py:145)
13:31:43.999 [I] Progress on: 54.0it/1.50kit rate:6.8s/it remaining:2:43:06 elapsed:10:52 postfix:- (1143:tqdm_logging.py:145)
13:31:58.071 [I] Progress on: 56.0it/1.50kit rate:6.9s/it remaining:2:46:10 elapsed:11:06 postfix:- (1143:tqdm_logging.py:145)
13:32:12.160 [I] Progress on: 58.0it/1.50kit rate:7.0s/it remaining:2:47:39 elapsed:11:20 postfix:- (1143:tqdm_logging.py:145)
13:32:26.235 [I] Progress on: 60.0it/1.50kit rate:7.0s/it remaining:2:48:11 elapsed:11:34 postfix:- (1143:tqdm_logging.py:145)
13:32:40.300 [I] Progress on: 62.0it/1.50kit rate:7.0s/it remaining:2:48:15 elapsed:11:48 postfix:- (1143:tqdm_logging.py:145)
13:32:54.382 [I] Progress on: 64.0it/1.50kit rate:7.0s/it remaining:2:48:16 elapsed:12:02 postfix:- (1143:tqdm_logging.py:145)
13:33:08.455 [I] Progress on: 66.0it/1.50kit rate:7.0s/it remaining:2:48:06 elapsed:12:17 postfix:- (1143:tqdm_logging.py:145)
13:33:22.526 [I] Progress on: 68.0it/1.50kit rate:7.0s/it remaining:2:47:53 elapsed:12:31 postfix:- (1143:tqdm_logging.py:145)
13:33:36.608 [I] Progress on: 70.0it/1.50kit rate:7.0s/it remaining:2:47:44 elapsed:12:45 postfix:- (1143:tqdm_logging.py:145)
13:33:50.683 [I] Progress on: 72.0it/1.50kit rate:7.0s/it remaining:2:47:30 elapsed:12:59 postfix:- (1143:tqdm_logging.py:145)
13:34:04.778 [I] Progress on: 74.0it/1.50kit rate:7.0s/it remaining:2:47:23 elapsed:13:13 postfix:- (1143:tqdm_logging.py:145)
13:34:18.862 [I] Progress on: 76.0it/1.50kit rate:7.0s/it remaining:2:47:08 elapsed:13:27 postfix:- (1143:tqdm_logging.py:145)
13:34:32.940 [I] Progress on: 78.0it/1.50kit rate:7.0s/it remaining:2:46:50 elapsed:13:41 postfix:- (1143:tqdm_logging.py:145)
13:34:47.022 [I] Progress on: 80.0it/1.50kit rate:7.0s/it remaining:2:46:37 elapsed:13:55 postfix:- (1143:tqdm_logging.py:145)
13:35:01.118 [I] Progress on: 82.0it/1.50kit rate:7.0s/it remaining:2:46:27 elapsed:14:09 postfix:- (1143:tqdm_logging.py:145)
13:35:15.197 [I] Progress on: 84.0it/1.50kit rate:7.0s/it remaining:2:46:10 elapsed:14:23 postfix:- (1143:tqdm_logging.py:145)
13:35:29.275 [I] Progress on: 86.0it/1.50kit rate:7.0s/it remaining:2:45:55 elapsed:14:37 postfix:- (1143:tqdm_logging.py:145)
13:35:43.348 [I] Progress on: 88.0it/1.50kit rate:7.0s/it remaining:2:45:38 elapsed:14:51 postfix:- (1143:tqdm_logging.py:145)
13:35:57.423 [I] Progress on: 90.0it/1.50kit rate:7.0s/it remaining:2:45:22 elapsed:15:06 postfix:- (1143:tqdm_logging.py:145)
13:36:11.516 [I] Progress on: 92.0it/1.50kit rate:7.0s/it remaining:2:45:16 elapsed:15:20 postfix:- (1143:tqdm_logging.py:145)
13:36:25.617 [I] Progress on: 94.0it/1.50kit rate:7.0s/it remaining:2:45:08 elapsed:15:34 postfix:- (1143:tqdm_logging.py:145)
13:36:39.728 [I] Progress on: 96.0it/1.50kit rate:7.1s/it remaining:2:44:59 elapsed:15:48 postfix:- (1143:tqdm_logging.py:145)
13:36:53.826 [I] Progress on: 98.0it/1.50kit rate:7.1s/it remaining:2:44:45 elapsed:16:02 postfix:- (1143:tqdm_logging.py:145)
13:37:07.918 [I] Progress on: 100it/1.50kit rate:7.0s/it remaining:2:44:26 elapsed:16:16 postfix:- (1143:tqdm_logging.py:145)


13:37:21.654 [I] Progress on: 100it/1.50kit rate:7.0s/it remaining:2:44:26 elapsed:16:30 postfix:- (1143:tqdm_logging.py:145)
13:37:36.211 [I] Progress on: 104it/1.50kit rate:6.8s/it remaining:2:37:22 elapsed:16:44 postfix:- (1143:tqdm_logging.py:145)
13:37:50.308 [I] Progress on: 106it/1.50kit rate:6.9s/it remaining:2:40:32 elapsed:16:58 postfix:- (1143:tqdm_logging.py:145)
13:38:04.384 [I] Progress on: 108it/1.50kit rate:6.9s/it remaining:2:40:42 elapsed:17:12 postfix:- (1143:tqdm_logging.py:145)
13:38:18.476 [I] Progress on: 110it/1.50kit rate:7.0s/it remaining:2:41:52 elapsed:17:27 postfix:- (1143:tqdm_logging.py:145)
13:38:32.565 [I] Progress on: 112it/1.50kit rate:7.0s/it remaining:2:42:18 elapsed:17:41 postfix:- (1143:tqdm_logging.py:145)
13:38:46.640 [I] Progress on: 114it/1.50kit rate:7.0s/it remaining:2:42:18 elapsed:17:55 postfix:- (1143:tqdm_logging.py:145)
13:39:00.721 [I] Progress on: 116it/1.50kit rate:7.0s/it remaining:2:42:14 elapsed:18:09 postfix:- (1143:tqdm_logging.py:145)
13:39:14.815 [I] Progress on: 118it/1.50kit rate:7.0s/it remaining:2:42:09 elapsed:18:23 postfix:- (1143:tqdm_logging.py:145)
13:39:28.894 [I] Progress on: 120it/1.50kit rate:7.0s/it remaining:2:41:55 elapsed:18:37 postfix:- (1143:tqdm_logging.py:145)
13:39:43.009 [I] Progress on: 122it/1.50kit rate:7.1s/it remaining:2:41:55 elapsed:18:51 postfix:- (1143:tqdm_logging.py:145)
13:39:57.089 [I] Progress on: 124it/1.50kit rate:7.0s/it remaining:2:41:36 elapsed:19:05 postfix:- (1143:tqdm_logging.py:145)
13:40:11.142 [I] Progress on: 126it/1.50kit rate:7.0s/it remaining:2:41:09 elapsed:19:19 postfix:- (1143:tqdm_logging.py:145)
13:40:25.214 [I] Progress on: 128it/1.50kit rate:7.0s/it remaining:2:40:54 elapsed:19:33 postfix:- (1143:tqdm_logging.py:145)
13:40:39.296 [I] Progress on: 130it/1.50kit rate:7.0s/it remaining:2:40:44 elapsed:19:47 postfix:- (1143:tqdm_logging.py:145)
13:40:53.368 [I] Progress on: 132it/1.50kit rate:7.0s/it remaining:2:40:28 elapsed:20:01 postfix:- (1143:tqdm_logging.py:145)
13:41:07.450 [I] Progress on: 134it/1.50kit rate:7.0s/it remaining:2:40:16 elapsed:20:16 postfix:- (1143:tqdm_logging.py:145)
13:41:21.519 [I] Progress on: 136it/1.50kit rate:7.0s/it remaining:2:39:59 elapsed:20:30 postfix:- (1143:tqdm_logging.py:145)
13:41:35.602 [I] Progress on: 138it/1.50kit rate:7.0s/it remaining:2:39:48 elapsed:20:44 postfix:- (1143:tqdm_logging.py:145)
13:41:49.667 [I] Progress on: 140it/1.50kit rate:7.0s/it remaining:2:39:29 elapsed:20:58 postfix:- (1143:tqdm_logging.py:145)
13:42:03.747 [I] Progress on: 142it/1.50kit rate:7.0s/it remaining:2:39:18 elapsed:21:12 postfix:- (1143:tqdm_logging.py:145)
13:42:17.832 [I] Progress on: 144it/1.50kit rate:7.0s/it remaining:2:39:07 elapsed:21:26 postfix:- (1143:tqdm_logging.py:145)
13:42:31.918 [I] Progress on: 146it/1.50kit rate:7.0s/it remaining:2:38:54 elapsed:21:40 postfix:- (1143:tqdm_logging.py:145)
13:42:45.990 [I] Progress on: 148it/1.50kit rate:7.0s/it remaining:2:38:36 elapsed:21:54 postfix:- (1143:tqdm_logging.py:145)
13:43:00.064 [I] Progress on: 150it/1.50kit rate:7.0s/it remaining:2:38:21 elapsed:22:08 postfix:- (1143:tqdm_logging.py:145)


13:43:13.782 [I] Progress on: 150it/1.50kit rate:7.0s/it remaining:2:38:21 elapsed:22:22 postfix:- (1143:tqdm_logging.py:145)
13:43:28.302 [I] Progress on: 154it/1.50kit rate:6.8s/it remaining:2:31:31 elapsed:22:36 postfix:- (1143:tqdm_logging.py:145)
13:43:42.371 [I] Progress on: 156it/1.50kit rate:6.9s/it remaining:2:34:30 elapsed:22:50 postfix:- (1143:tqdm_logging.py:145)
13:43:56.448 [I] Progress on: 158it/1.50kit rate:7.0s/it remaining:2:35:52 elapsed:23:05 postfix:- (1143:tqdm_logging.py:145)
13:44:10.544 [I] Progress on: 160it/1.50kit rate:7.0s/it remaining:2:36:32 elapsed:23:19 postfix:- (1143:tqdm_logging.py:145)
13:44:24.621 [I] Progress on: 162it/1.50kit rate:7.0s/it remaining:2:36:36 elapsed:23:33 postfix:- (1143:tqdm_logging.py:145)
13:44:38.690 [I] Progress on: 164it/1.50kit rate:7.0s/it remaining:2:36:30 elapsed:23:47 postfix:- (1143:tqdm_logging.py:145)
13:44:52.766 [I] Progress on: 166it/1.50kit rate:7.0s/it remaining:2:36:22 elapsed:24:01 postfix:- (1143:tqdm_logging.py:145)
13:45:06.849 [I] Progress on: 168it/1.50kit rate:7.0s/it remaining:2:36:13 elapsed:24:15 postfix:- (1143:tqdm_logging.py:145)
13:45:20.936 [I] Progress on: 170it/1.50kit rate:7.0s/it remaining:2:36:03 elapsed:24:29 postfix:- (1143:tqdm_logging.py:145)
13:45:35.012 [I] Progress on: 172it/1.50kit rate:7.0s/it remaining:2:35:47 elapsed:24:43 postfix:- (1143:tqdm_logging.py:145)
13:45:49.084 [I] Progress on: 174it/1.50kit rate:7.0s/it remaining:2:35:31 elapsed:24:57 postfix:- (1143:tqdm_logging.py:145)
13:46:03.166 [I] Progress on: 176it/1.50kit rate:7.0s/it remaining:2:35:18 elapsed:25:11 postfix:- (1143:tqdm_logging.py:145)
13:46:17.260 [I] Progress on: 178it/1.50kit rate:7.0s/it remaining:2:35:06 elapsed:25:25 postfix:- (1143:tqdm_logging.py:145)
13:46:31.320 [I] Progress on: 180it/1.50kit rate:7.0s/it remaining:2:34:46 elapsed:25:39 postfix:- (1143:tqdm_logging.py:145)
13:46:45.397 [I] Progress on: 182it/1.50kit rate:7.0s/it remaining:2:34:34 elapsed:25:54 postfix:- (1143:tqdm_logging.py:145)
13:46:59.473 [I] Progress on: 184it/1.50kit rate:7.0s/it remaining:2:34:21 elapsed:26:08 postfix:- (1143:tqdm_logging.py:145)
13:47:13.553 [I] Progress on: 186it/1.50kit rate:7.0s/it remaining:2:34:09 elapsed:26:22 postfix:- (1143:tqdm_logging.py:145)
13:47:27.628 [I] Progress on: 188it/1.50kit rate:7.0s/it remaining:2:33:54 elapsed:26:36 postfix:- (1143:tqdm_logging.py:145)
13:47:41.715 [I] Progress on: 190it/1.50kit rate:7.0s/it remaining:2:33:42 elapsed:26:50 postfix:- (1143:tqdm_logging.py:145)
13:47:55.791 [I] Progress on: 192it/1.50kit rate:7.0s/it remaining:2:33:27 elapsed:27:04 postfix:- (1143:tqdm_logging.py:145)
13:48:09.866 [I] Progress on: 194it/1.50kit rate:7.0s/it remaining:2:33:10 elapsed:27:18 postfix:- (1143:tqdm_logging.py:145)
13:48:23.942 [I] Progress on: 196it/1.50kit rate:7.0s/it remaining:2:32:57 elapsed:27:32 postfix:- (1143:tqdm_logging.py:145)
13:48:38.015 [I] Progress on: 198it/1.50kit rate:7.0s/it remaining:2:32:42 elapsed:27:46 postfix:- (1143:tqdm_logging.py:145)
13:48:52.091 [I] Progress on: 200it/1.50kit rate:7.0s/it remaining:2:32:28 elapsed:28:00 postfix:- (1143:tqdm_logging.py:145)


13:49:05.867 [I] Progress on: 200it/1.50kit rate:7.0s/it remaining:2:32:28 elapsed:28:14 postfix:- (1143:tqdm_logging.py:145)
13:49:20.426 [I] Progress on: 204it/1.50kit rate:6.8s/it remaining:2:26:04 elapsed:28:29 postfix:- (1143:tqdm_logging.py:145)
13:49:34.494 [I] Progress on: 206it/1.50kit rate:6.9s/it remaining:2:28:50 elapsed:28:43 postfix:- (1143:tqdm_logging.py:145)
13:49:48.573 [I] Progress on: 208it/1.50kit rate:7.0s/it remaining:2:30:06 elapsed:28:57 postfix:- (1143:tqdm_logging.py:145)
13:50:02.662 [I] Progress on: 210it/1.50kit rate:7.0s/it remaining:2:30:40 elapsed:29:11 postfix:- (1143:tqdm_logging.py:145)
13:50:16.739 [I] Progress on: 212it/1.50kit rate:7.0s/it remaining:2:30:47 elapsed:29:25 postfix:- (1143:tqdm_logging.py:145)
13:50:31.371 [I] Progress on: 214it/1.50kit rate:7.2s/it remaining:2:34:15 elapsed:29:39 postfix:- (1143:tqdm_logging.py:145)
13:50:44.904 [I] Progress on: 216it/1.50kit rate:7.0s/it remaining:2:29:50 elapsed:29:53 postfix:- (1143:tqdm_logging.py:145)
13:50:58.975 [I] Progress on: 218it/1.50kit rate:7.0s/it remaining:2:29:57 elapsed:30:07 postfix:- (1143:tqdm_logging.py:145)
13:51:13.053 [I] Progress on: 220it/1.50kit rate:7.0s/it remaining:2:29:57 elapsed:30:21 postfix:- (1143:tqdm_logging.py:145)
13:51:27.131 [I] Progress on: 222it/1.50kit rate:7.0s/it remaining:2:29:49 elapsed:30:35 postfix:- (1143:tqdm_logging.py:145)
13:51:41.226 [I] Progress on: 224it/1.50kit rate:7.0s/it remaining:2:29:44 elapsed:30:49 postfix:- (1143:tqdm_logging.py:145)
13:51:55.315 [I] Progress on: 226it/1.50kit rate:7.0s/it remaining:2:29:33 elapsed:31:03 postfix:- (1143:tqdm_logging.py:145)
13:52:09.403 [I] Progress on: 228it/1.50kit rate:7.0s/it remaining:2:29:19 elapsed:31:18 postfix:- (1143:tqdm_logging.py:145)
13:52:23.492 [I] Progress on: 230it/1.50kit rate:7.0s/it remaining:2:29:06 elapsed:31:32 postfix:- (1143:tqdm_logging.py:145)
13:52:37.567 [I] Progress on: 232it/1.50kit rate:7.0s/it remaining:2:28:47 elapsed:31:46 postfix:- (1143:tqdm_logging.py:145)
13:52:51.645 [I] Progress on: 234it/1.50kit rate:7.0s/it remaining:2:28:32 elapsed:32:00 postfix:- (1143:tqdm_logging.py:145)
13:53:05.737 [I] Progress on: 236it/1.50kit rate:7.0s/it remaining:2:28:23 elapsed:32:14 postfix:- (1143:tqdm_logging.py:145)
13:53:19.819 [I] Progress on: 238it/1.50kit rate:7.0s/it remaining:2:28:07 elapsed:32:28 postfix:- (1143:tqdm_logging.py:145)
13:53:33.912 [I] Progress on: 240it/1.50kit rate:7.0s/it remaining:2:27:55 elapsed:32:42 postfix:- (1143:tqdm_logging.py:145)
13:53:47.993 [I] Progress on: 242it/1.50kit rate:7.0s/it remaining:2:27:39 elapsed:32:56 postfix:- (1143:tqdm_logging.py:145)
13:54:02.098 [I] Progress on: 244it/1.50kit rate:7.0s/it remaining:2:27:30 elapsed:33:10 postfix:- (1143:tqdm_logging.py:145)
13:54:16.177 [I] Progress on: 246it/1.50kit rate:7.0s/it remaining:2:27:11 elapsed:33:24 postfix:- (1143:tqdm_logging.py:145)
13:54:30.269 [I] Progress on: 248it/1.50kit rate:7.0s/it remaining:2:27:00 elapsed:33:38 postfix:- (1143:tqdm_logging.py:145)
13:54:44.353 [I] Progress on: 250it/1.50kit rate:7.0s/it remaining:2:26:44 elapsed:33:52 postfix:- (1143:tqdm_logging.py:145)


13:54:58.118 [I] Progress on: 250it/1.50kit rate:7.0s/it remaining:2:26:44 elapsed:34:06 postfix:- (1143:tqdm_logging.py:145)
13:55:12.665 [I] Progress on: 254it/1.50kit rate:6.8s/it remaining:2:20:25 elapsed:34:21 postfix:- (1143:tqdm_logging.py:145)
13:55:26.759 [I] Progress on: 256it/1.50kit rate:6.9s/it remaining:2:23:12 elapsed:34:35 postfix:- (1143:tqdm_logging.py:145)
13:55:40.856 [I] Progress on: 258it/1.50kit rate:7.0s/it remaining:2:24:27 elapsed:34:49 postfix:- (1143:tqdm_logging.py:145)
13:55:54.952 [I] Progress on: 260it/1.50kit rate:7.0s/it remaining:2:24:57 elapsed:35:03 postfix:- (1143:tqdm_logging.py:145)
13:56:09.047 [I] Progress on: 262it/1.50kit rate:7.0s/it remaining:2:25:03 elapsed:35:17 postfix:- (1143:tqdm_logging.py:145)
13:56:23.137 [I] Progress on: 264it/1.50kit rate:7.0s/it remaining:2:24:57 elapsed:35:31 postfix:- (1143:tqdm_logging.py:145)
13:56:37.231 [I] Progress on: 266it/1.50kit rate:7.0s/it remaining:2:24:50 elapsed:35:45 postfix:- (1143:tqdm_logging.py:145)
13:56:51.324 [I] Progress on: 268it/1.50kit rate:7.0s/it remaining:2:24:39 elapsed:35:59 postfix:- (1143:tqdm_logging.py:145)
13:57:05.416 [I] Progress on: 270it/1.50kit rate:7.0s/it remaining:2:24:25 elapsed:36:14 postfix:- (1143:tqdm_logging.py:145)
13:57:19.516 [I] Progress on: 272it/1.50kit rate:7.0s/it remaining:2:24:15 elapsed:36:28 postfix:- (1143:tqdm_logging.py:145)
13:57:33.609 [I] Progress on: 274it/1.50kit rate:7.0s/it remaining:2:24:00 elapsed:36:42 postfix:- (1143:tqdm_logging.py:145)
13:57:47.709 [I] Progress on: 276it/1.50kit rate:7.0s/it remaining:2:23:48 elapsed:36:56 postfix:- (1143:tqdm_logging.py:145)
13:58:01.803 [I] Progress on: 278it/1.50kit rate:7.0s/it remaining:2:23:31 elapsed:37:10 postfix:- (1143:tqdm_logging.py:145)
13:58:15.901 [I] Progress on: 280it/1.50kit rate:7.0s/it remaining:2:23:17 elapsed:37:24 postfix:- (1143:tqdm_logging.py:145)
13:58:30.003 [I] Progress on: 282it/1.50kit rate:7.0s/it remaining:2:23:05 elapsed:37:38 postfix:- (1143:tqdm_logging.py:145)
13:58:44.102 [I] Progress on: 284it/1.50kit rate:7.1s/it remaining:2:22:52 elapsed:37:52 postfix:- (1143:tqdm_logging.py:145)
13:58:58.175 [I] Progress on: 286it/1.50kit rate:7.0s/it remaining:2:22:31 elapsed:38:06 postfix:- (1143:tqdm_logging.py:145)
13:59:12.271 [I] Progress on: 288it/1.50kit rate:7.0s/it remaining:2:22:19 elapsed:38:20 postfix:- (1143:tqdm_logging.py:145)
13:59:26.359 [I] Progress on: 290it/1.50kit rate:7.0s/it remaining:2:22:05 elapsed:38:34 postfix:- (1143:tqdm_logging.py:145)
13:59:40.463 [I] Progress on: 292it/1.50kit rate:7.0s/it remaining:2:21:55 elapsed:38:49 postfix:- (1143:tqdm_logging.py:145)
13:59:54.558 [I] Progress on: 294it/1.50kit rate:7.0s/it remaining:2:21:40 elapsed:39:03 postfix:- (1143:tqdm_logging.py:145)
14:00:08.669 [I] Progress on: 296it/1.50kit rate:7.1s/it remaining:2:21:31 elapsed:39:17 postfix:- (1143:tqdm_logging.py:145)
14:00:22.785 [I] Progress on: 298it/1.50kit rate:7.1s/it remaining:2:21:20 elapsed:39:31 postfix:- (1143:tqdm_logging.py:145)
14:00:36.877 [I] Progress on: 300it/1.50kit rate:7.1s/it remaining:2:21:00 elapsed:39:45 postfix:- (1143:tqdm_logging.py:145)


14:00:50.609 [I] Progress on: 300it/1.50kit rate:7.1s/it remaining:2:21:00 elapsed:39:59 postfix:- (1143:tqdm_logging.py:145)
14:01:05.149 [I] Progress on: 304it/1.50kit rate:6.8s/it remaining:2:14:48 elapsed:40:13 postfix:- (1143:tqdm_logging.py:145)
14:01:19.241 [I] Progress on: 306it/1.50kit rate:6.9s/it remaining:2:17:27 elapsed:40:27 postfix:- (1143:tqdm_logging.py:145)
14:01:33.356 [I] Progress on: 308it/1.50kit rate:7.0s/it remaining:2:18:45 elapsed:40:41 postfix:- (1143:tqdm_logging.py:145)
14:01:47.443 [I] Progress on: 310it/1.50kit rate:7.0s/it remaining:2:19:08 elapsed:40:56 postfix:- (1143:tqdm_logging.py:145)
14:02:01.540 [I] Progress on: 312it/1.50kit rate:7.0s/it remaining:2:19:15 elapsed:41:10 postfix:- (1143:tqdm_logging.py:145)
14:02:15.628 [I] Progress on: 314it/1.50kit rate:7.0s/it remaining:2:19:07 elapsed:41:24 postfix:- (1143:tqdm_logging.py:145)
14:02:29.727 [I] Progress on: 316it/1.50kit rate:7.0s/it remaining:2:19:00 elapsed:41:38 postfix:- (1143:tqdm_logging.py:145)
14:02:43.833 [I] Progress on: 318it/1.50kit rate:7.0s/it remaining:2:18:51 elapsed:41:52 postfix:- (1143:tqdm_logging.py:145)
14:02:57.928 [I] Progress on: 320it/1.50kit rate:7.0s/it remaining:2:18:37 elapsed:42:06 postfix:- (1143:tqdm_logging.py:145)
14:03:12.003 [I] Progress on: 322it/1.50kit rate:7.0s/it remaining:2:17:16 elapsed:42:20 postfix:- (1143:tqdm_logging.py:145)
14:03:26.104 [I] Progress on: 324it/1.50kit rate:7.0s/it remaining:2:17:33 elapsed:42:34 postfix:- (1143:tqdm_logging.py:145)
14:03:40.211 [I] Progress on: 326it/1.50kit rate:7.0s/it remaining:2:17:35 elapsed:42:48 postfix:- (1143:tqdm_logging.py:145)
14:03:54.330 [I] Progress on: 328it/1.50kit rate:7.0s/it remaining:2:17:33 elapsed:43:02 postfix:- (1143:tqdm_logging.py:145)
14:04:08.425 [I] Progress on: 330it/1.50kit rate:7.0s/it remaining:2:17:16 elapsed:43:17 postfix:- (1143:tqdm_logging.py:145)
14:04:22.538 [I] Progress on: 332it/1.50kit rate:7.0s/it remaining:2:17:08 elapsed:43:31 postfix:- (1143:tqdm_logging.py:145)
14:04:36.631 [I] Progress on: 334it/1.50kit rate:7.0s/it remaining:2:16:50 elapsed:43:45 postfix:- (1143:tqdm_logging.py:145)
14:04:50.728 [I] Progress on: 336it/1.50kit rate:7.0s/it remaining:2:16:37 elapsed:43:59 postfix:- (1143:tqdm_logging.py:145)
14:05:04.819 [I] Progress on: 338it/1.50kit rate:7.0s/it remaining:2:16:20 elapsed:44:13 postfix:- (1143:tqdm_logging.py:145)
14:05:18.920 [I] Progress on: 340it/1.50kit rate:7.0s/it remaining:2:16:07 elapsed:44:27 postfix:- (1143:tqdm_logging.py:145)
14:05:33.017 [I] Progress on: 342it/1.50kit rate:7.0s/it remaining:2:15:52 elapsed:44:41 postfix:- (1143:tqdm_logging.py:145)
14:05:47.126 [I] Progress on: 344it/1.50kit rate:7.0s/it remaining:2:15:42 elapsed:44:55 postfix:- (1143:tqdm_logging.py:145)
14:06:01.223 [I] Progress on: 346it/1.50kit rate:7.0s/it remaining:2:15:26 elapsed:45:09 postfix:- (1143:tqdm_logging.py:145)
14:06:15.334 [I] Progress on: 348it/1.50kit rate:7.0s/it remaining:2:15:17 elapsed:45:23 postfix:- (1143:tqdm_logging.py:145)
14:06:29.422 [I] Progress on: 350it/1.50kit rate:7.0s/it remaining:2:14:57 elapsed:45:38 postfix:- (1143:tqdm_logging.py:145)


14:06:43.224 [I] Progress on: 350it/1.50kit rate:7.0s/it remaining:2:14:57 elapsed:45:51 postfix:- (1143:tqdm_logging.py:145)
14:06:57.748 [I] Progress on: 354it/1.50kit rate:6.8s/it remaining:2:09:04 elapsed:46:06 postfix:- (1143:tqdm_logging.py:145)
14:07:11.845 [I] Progress on: 356it/1.50kit rate:6.9s/it remaining:2:11:36 elapsed:46:20 postfix:- (1143:tqdm_logging.py:145)
14:07:25.930 [I] Progress on: 358it/1.50kit rate:7.0s/it remaining:2:12:39 elapsed:46:34 postfix:- (1143:tqdm_logging.py:145)
14:07:40.012 [I] Progress on: 360it/1.50kit rate:7.0s/it remaining:2:13:03 elapsed:46:48 postfix:- (1143:tqdm_logging.py:145)
14:07:54.081 [I] Progress on: 362it/1.50kit rate:7.0s/it remaining:2:13:02 elapsed:47:02 postfix:- (1143:tqdm_logging.py:145)
14:08:08.166 [I] Progress on: 364it/1.50kit rate:7.0s/it remaining:2:13:00 elapsed:47:16 postfix:- (1143:tqdm_logging.py:145)
14:08:22.258 [I] Progress on: 366it/1.50kit rate:7.0s/it remaining:2:12:53 elapsed:47:30 postfix:- (1143:tqdm_logging.py:145)
14:08:36.339 [I] Progress on: 368it/1.50kit rate:7.0s/it remaining:2:12:40 elapsed:47:44 postfix:- (1143:tqdm_logging.py:145)
14:08:50.431 [I] Progress on: 370it/1.50kit rate:7.0s/it remaining:2:12:30 elapsed:47:59 postfix:- (1143:tqdm_logging.py:145)
14:09:04.524 [I] Progress on: 372it/1.50kit rate:7.0s/it remaining:2:12:18 elapsed:48:13 postfix:- (1143:tqdm_logging.py:145)
14:09:18.596 [I] Progress on: 374it/1.50kit rate:7.0s/it remaining:2:11:56 elapsed:48:27 postfix:- (1143:tqdm_logging.py:145)
14:09:32.691 [I] Progress on: 376it/1.50kit rate:7.0s/it remaining:2:11:49 elapsed:48:41 postfix:- (1143:tqdm_logging.py:145)
14:09:46.780 [I] Progress on: 378it/1.50kit rate:7.0s/it remaining:2:11:34 elapsed:48:55 postfix:- (1143:tqdm_logging.py:145)
14:10:00.880 [I] Progress on: 380it/1.50kit rate:7.0s/it remaining:2:11:24 elapsed:49:09 postfix:- (1143:tqdm_logging.py:145)
14:10:14.966 [I] Progress on: 382it/1.50kit rate:7.0s/it remaining:2:11:07 elapsed:49:23 postfix:- (1143:tqdm_logging.py:145)
14:10:29.062 [I] Progress on: 384it/1.50kit rate:7.0s/it remaining:2:10:55 elapsed:49:37 postfix:- (1143:tqdm_logging.py:145)
14:10:43.155 [I] Progress on: 386it/1.50kit rate:7.0s/it remaining:2:10:41 elapsed:49:51 postfix:- (1143:tqdm_logging.py:145)
14:10:57.239 [I] Progress on: 388it/1.50kit rate:7.0s/it remaining:2:10:25 elapsed:50:05 postfix:- (1143:tqdm_logging.py:145)
14:11:11.328 [I] Progress on: 390it/1.50kit rate:7.0s/it remaining:2:10:09 elapsed:50:19 postfix:- (1143:tqdm_logging.py:145)
14:11:25.408 [I] Progress on: 392it/1.50kit rate:7.0s/it remaining:2:09:54 elapsed:50:34 postfix:- (1143:tqdm_logging.py:145)
14:11:39.501 [I] Progress on: 394it/1.50kit rate:7.0s/it remaining:2:09:43 elapsed:50:48 postfix:- (1143:tqdm_logging.py:145)
14:11:53.584 [I] Progress on: 396it/1.50kit rate:7.0s/it remaining:2:09:28 elapsed:51:02 postfix:- (1143:tqdm_logging.py:145)
14:12:07.672 [I] Progress on: 398it/1.50kit rate:7.0s/it remaining:2:09:13 elapsed:51:16 postfix:- (1143:tqdm_logging.py:145)
14:12:21.749 [I] Progress on: 400it/1.50kit rate:7.0s/it remaining:2:08:57 elapsed:51:30 postfix:- (1143:tqdm_logging.py:145)


14:12:35.500 [I] Progress on: 400it/1.50kit rate:7.0s/it remaining:2:08:57 elapsed:51:44 postfix:- (1143:tqdm_logging.py:145)
14:12:50.008 [I] Progress on: 404it/1.50kit rate:6.8s/it remaining:2:03:20 elapsed:51:58 postfix:- (1143:tqdm_logging.py:145)
14:13:04.133 [I] Progress on: 406it/1.50kit rate:6.9s/it remaining:2:05:59 elapsed:52:12 postfix:- (1143:tqdm_logging.py:145)
14:13:18.193 [I] Progress on: 408it/1.50kit rate:7.0s/it remaining:2:06:53 elapsed:52:26 postfix:- (1143:tqdm_logging.py:145)
14:13:32.299 [I] Progress on: 410it/1.50kit rate:7.0s/it remaining:2:07:24 elapsed:52:40 postfix:- (1143:tqdm_logging.py:145)
14:13:46.405 [I] Progress on: 412it/1.50kit rate:7.0s/it remaining:2:07:32 elapsed:52:55 postfix:- (1143:tqdm_logging.py:145)
14:14:00.498 [I] Progress on: 414it/1.50kit rate:7.0s/it remaining:2:07:26 elapsed:53:09 postfix:- (1143:tqdm_logging.py:145)
14:14:14.602 [I] Progress on: 416it/1.50kit rate:7.0s/it remaining:2:07:18 elapsed:53:23 postfix:- (1143:tqdm_logging.py:145)
14:14:28.697 [I] Progress on: 418it/1.50kit rate:7.0s/it remaining:2:07:05 elapsed:53:37 postfix:- (1143:tqdm_logging.py:145)
14:14:42.793 [I] Progress on: 420it/1.50kit rate:7.0s/it remaining:2:06:51 elapsed:53:51 postfix:- (1143:tqdm_logging.py:145)
14:14:56.899 [I] Progress on: 422it/1.50kit rate:7.0s/it remaining:2:06:39 elapsed:54:05 postfix:- (1143:tqdm_logging.py:145)
14:15:11.001 [I] Progress on: 424it/1.50kit rate:7.1s/it remaining:2:06:26 elapsed:54:19 postfix:- (1143:tqdm_logging.py:145)
14:15:25.093 [I] Progress on: 426it/1.50kit rate:7.0s/it remaining:2:06:10 elapsed:54:33 postfix:- (1143:tqdm_logging.py:145)
14:15:39.685 [I] Progress on: 428it/1.50kit rate:7.2s/it remaining:2:08:36 elapsed:54:48 postfix:- (1143:tqdm_logging.py:145)
14:15:53.306 [I] Progress on: 430it/1.50kit rate:7.0s/it remaining:2:05:17 elapsed:55:01 postfix:- (1143:tqdm_logging.py:145)
14:16:07.375 [I] Progress on: 432it/1.50kit rate:7.0s/it remaining:2:05:12 elapsed:55:15 postfix:- (1143:tqdm_logging.py:145)
14:16:21.479 [I] Progress on: 434it/1.50kit rate:7.0s/it remaining:2:05:08 elapsed:55:30 postfix:- (1143:tqdm_logging.py:145)
14:16:35.575 [I] Progress on: 436it/1.50kit rate:7.0s/it remaining:2:04:57 elapsed:55:44 postfix:- (1143:tqdm_logging.py:145)
14:16:49.671 [I] Progress on: 438it/1.50kit rate:7.0s/it remaining:2:04:43 elapsed:55:58 postfix:- (1143:tqdm_logging.py:145)
14:17:03.770 [I] Progress on: 440it/1.50kit rate:7.0s/it remaining:2:04:30 elapsed:56:12 postfix:- (1143:tqdm_logging.py:145)
14:17:17.893 [I] Progress on: 442it/1.50kit rate:7.1s/it remaining:2:04:24 elapsed:56:26 postfix:- (1143:tqdm_logging.py:145)
14:17:31.962 [I] Progress on: 444it/1.50kit rate:7.0s/it remaining:2:03:58 elapsed:56:40 postfix:- (1143:tqdm_logging.py:145)
14:17:46.045 [I] Progress on: 446it/1.50kit rate:7.0s/it remaining:2:03:44 elapsed:56:54 postfix:- (1143:tqdm_logging.py:145)
14:18:00.124 [I] Progress on: 448it/1.50kit rate:7.0s/it remaining:2:03:27 elapsed:57:08 postfix:- (1143:tqdm_logging.py:145)
14:18:14.204 [I] Progress on: 450it/1.50kit rate:7.0s/it remaining:2:03:13 elapsed:57:22 postfix:- (1143:tqdm_logging.py:145)


14:18:27.915 [I] Progress on: 450it/1.50kit rate:7.0s/it remaining:2:03:13 elapsed:57:36 postfix:- (1143:tqdm_logging.py:145)
14:18:42.417 [I] Progress on: 454it/1.50kit rate:6.8s/it remaining:1:57:43 elapsed:57:51 postfix:- (1143:tqdm_logging.py:145)
14:18:56.487 [I] Progress on: 456it/1.50kit rate:6.9s/it remaining:2:00:00 elapsed:58:05 postfix:- (1143:tqdm_logging.py:145)
14:19:10.557 [I] Progress on: 458it/1.50kit rate:7.0s/it remaining:2:01:00 elapsed:58:19 postfix:- (1143:tqdm_logging.py:145)
14:19:24.637 [I] Progress on: 460it/1.50kit rate:7.0s/it remaining:2:01:25 elapsed:58:33 postfix:- (1143:tqdm_logging.py:145)
14:19:38.719 [I] Progress on: 462it/1.50kit rate:7.0s/it remaining:2:01:31 elapsed:58:47 postfix:- (1143:tqdm_logging.py:145)
14:19:52.792 [I] Progress on: 464it/1.50kit rate:7.0s/it remaining:2:01:24 elapsed:59:01 postfix:- (1143:tqdm_logging.py:145)
14:20:06.864 [I] Progress on: 466it/1.50kit rate:7.0s/it remaining:2:01:13 elapsed:59:15 postfix:- (1143:tqdm_logging.py:145)
14:20:20.937 [I] Progress on: 468it/1.50kit rate:7.0s/it remaining:2:01:00 elapsed:59:29 postfix:- (1143:tqdm_logging.py:145)
14:20:35.012 [I] Progress on: 470it/1.50kit rate:7.0s/it remaining:2:00:47 elapsed:59:43 postfix:- (1143:tqdm_logging.py:145)
14:20:49.088 [I] Progress on: 472it/1.50kit rate:7.0s/it remaining:2:00:34 elapsed:59:57 postfix:- (1143:tqdm_logging.py:145)
14:21:03.166 [I] Progress on: 474it/1.50kit rate:7.0s/it remaining:2:00:21 elapsed:1:00:11 postfix:- (1143:tqdm_logging.py:145)
14:21:17.263 [I] Progress on: 476it/1.50kit rate:7.0s/it remaining:2:00:12 elapsed:1:00:25 postfix:- (1143:tqdm_logging.py:145)
14:21:31.336 [I] Progress on: 478it/1.50kit rate:7.0s/it remaining:1:59:54 elapsed:1:00:39 postfix:- (1143:tqdm_logging.py:145)
14:21:45.414 [I] Progress on: 480it/1.50kit rate:7.0s/it remaining:1:59:40 elapsed:1:00:54 postfix:- (1143:tqdm_logging.py:145)
14:21:59.494 [I] Progress on: 482it/1.50kit rate:7.0s/it remaining:1:59:26 elapsed:1:01:08 postfix:- (1143:tqdm_logging.py:145)
14:22:13.612 [I] Progress on: 484it/1.50kit rate:7.0s/it remaining:1:59:22 elapsed:1:01:22 postfix:- (1143:tqdm_logging.py:145)
14:22:27.705 [I] Progress on: 486it/1.50kit rate:7.0s/it remaining:1:59:05 elapsed:1:01:36 postfix:- (1143:tqdm_logging.py:145)
14:22:41.813 [I] Progress on: 488it/1.50kit rate:7.1s/it remaining:1:58:55 elapsed:1:01:50 postfix:- (1143:tqdm_logging.py:145)
14:22:55.923 [I] Progress on: 490it/1.50kit rate:7.1s/it remaining:1:58:44 elapsed:1:02:04 postfix:- (1143:tqdm_logging.py:145)
14:23:10.024 [I] Progress on: 492it/1.50kit rate:7.1s/it remaining:1:58:28 elapsed:1:02:18 postfix:- (1143:tqdm_logging.py:145)
14:23:24.138 [I] Progress on: 494it/1.50kit rate:7.1s/it remaining:1:58:17 elapsed:1:02:32 postfix:- (1143:tqdm_logging.py:145)
14:23:38.241 [I] Progress on: 496it/1.50kit rate:7.1s/it remaining:1:58:01 elapsed:1:02:46 postfix:- (1143:tqdm_logging.py:145)
14:23:52.338 [I] Progress on: 498it/1.50kit rate:7.1s/it remaining:1:57:44 elapsed:1:03:00 postfix:- (1143:tqdm_logging.py:145)
14:24:06.440 [I] Progress on: 500it/1.50kit rate:7.1s/it remaining:1:57:30 elapsed:1:03:15 postfix:- (1143:tqdm_logging.py:145)


14:24:20.185 [I] Progress on: 500it/1.50kit rate:7.1s/it remaining:1:57:30 elapsed:1:03:28 postfix:- (1143:tqdm_logging.py:145)
14:24:34.711 [I] Progress on: 504it/1.50kit rate:6.8s/it remaining:1:52:16 elapsed:1:03:43 postfix:- (1143:tqdm_logging.py:145)
14:24:48.802 [I] Progress on: 506it/1.50kit rate:6.9s/it remaining:1:54:25 elapsed:1:03:57 postfix:- (1143:tqdm_logging.py:145)
14:25:02.895 [I] Progress on: 508it/1.50kit rate:7.0s/it remaining:1:55:22 elapsed:1:04:11 postfix:- (1143:tqdm_logging.py:145)
14:25:17.020 [I] Progress on: 510it/1.50kit rate:7.0s/it remaining:1:55:50 elapsed:1:04:25 postfix:- (1143:tqdm_logging.py:145)
14:25:31.105 [I] Progress on: 512it/1.50kit rate:7.0s/it remaining:1:55:49 elapsed:1:04:39 postfix:- (1143:tqdm_logging.py:145)
14:25:45.196 [I] Progress on: 514it/1.50kit rate:7.0s/it remaining:1:55:41 elapsed:1:04:53 postfix:- (1143:tqdm_logging.py:145)
14:25:59.293 [I] Progress on: 516it/1.50kit rate:7.0s/it remaining:1:55:31 elapsed:1:05:07 postfix:- (1143:tqdm_logging.py:145)
14:26:13.372 [I] Progress on: 518it/1.50kit rate:7.0s/it remaining:1:55:15 elapsed:1:05:21 postfix:- (1143:tqdm_logging.py:145)
14:26:27.449 [I] Progress on: 520it/1.50kit rate:7.0s/it remaining:1:54:59 elapsed:1:05:36 postfix:- (1143:tqdm_logging.py:145)
14:26:41.519 [I] Progress on: 522it/1.50kit rate:7.0s/it remaining:1:54:43 elapsed:1:05:50 postfix:- (1143:tqdm_logging.py:145)
14:26:55.592 [I] Progress on: 524it/1.50kit rate:7.0s/it remaining:1:54:28 elapsed:1:06:04 postfix:- (1143:tqdm_logging.py:145)
14:27:09.672 [I] Progress on: 526it/1.50kit rate:7.0s/it remaining:1:54:15 elapsed:1:06:18 postfix:- (1143:tqdm_logging.py:145)
14:27:23.748 [I] Progress on: 528it/1.50kit rate:7.0s/it remaining:1:54:00 elapsed:1:06:32 postfix:- (1143:tqdm_logging.py:145)
14:27:37.838 [I] Progress on: 530it/1.50kit rate:7.0s/it remaining:1:53:50 elapsed:1:06:46 postfix:- (1143:tqdm_logging.py:145)
14:27:51.918 [I] Progress on: 532it/1.50kit rate:7.0s/it remaining:1:53:35 elapsed:1:07:00 postfix:- (1143:tqdm_logging.py:145)
14:28:05.998 [I] Progress on: 534it/1.50kit rate:7.0s/it remaining:1:53:21 elapsed:1:07:14 postfix:- (1143:tqdm_logging.py:145)
14:28:20.093 [I] Progress on: 536it/1.50kit rate:7.0s/it remaining:1:52:27 elapsed:1:07:28 postfix:- (1143:tqdm_logging.py:145)
14:28:34.192 [I] Progress on: 538it/1.50kit rate:7.0s/it remaining:1:52:38 elapsed:1:07:42 postfix:- (1143:tqdm_logging.py:145)
14:28:48.291 [I] Progress on: 540it/1.50kit rate:7.0s/it remaining:1:52:36 elapsed:1:07:56 postfix:- (1143:tqdm_logging.py:145)
14:29:02.383 [I] Progress on: 542it/1.50kit rate:7.0s/it remaining:1:52:25 elapsed:1:08:10 postfix:- (1143:tqdm_logging.py:145)
14:29:16.473 [I] Progress on: 544it/1.50kit rate:7.0s/it remaining:1:52:13 elapsed:1:08:25 postfix:- (1143:tqdm_logging.py:145)
14:29:30.569 [I] Progress on: 546it/1.50kit rate:7.0s/it remaining:1:52:01 elapsed:1:08:39 postfix:- (1143:tqdm_logging.py:145)
14:29:44.653 [I] Progress on: 548it/1.50kit rate:7.0s/it remaining:1:51:46 elapsed:1:08:53 postfix:- (1143:tqdm_logging.py:145)
14:29:58.751 [I] Progress on: 550it/1.50kit rate:7.0s/it remaining:1:51:34 elapsed:1:09:07 postfix:- (1143:tqdm_logging.py:145)


14:30:12.491 [I] Progress on: 550it/1.50kit rate:7.0s/it remaining:1:51:34 elapsed:1:09:21 postfix:- (1143:tqdm_logging.py:145)
14:30:26.993 [I] Progress on: 554it/1.50kit rate:6.8s/it remaining:1:46:31 elapsed:1:09:35 postfix:- (1143:tqdm_logging.py:145)
14:30:41.088 [I] Progress on: 556it/1.50kit rate:6.9s/it remaining:1:48:37 elapsed:1:09:49 postfix:- (1143:tqdm_logging.py:145)
14:30:55.189 [I] Progress on: 558it/1.50kit rate:7.0s/it remaining:1:49:34 elapsed:1:10:03 postfix:- (1143:tqdm_logging.py:145)
14:31:09.264 [I] Progress on: 560it/1.50kit rate:7.0s/it remaining:1:49:49 elapsed:1:10:17 postfix:- (1143:tqdm_logging.py:145)
14:31:23.371 [I] Progress on: 562it/1.50kit rate:7.0s/it remaining:1:49:56 elapsed:1:10:31 postfix:- (1143:tqdm_logging.py:145)
14:31:37.465 [I] Progress on: 564it/1.50kit rate:7.0s/it remaining:1:49:49 elapsed:1:10:46 postfix:- (1143:tqdm_logging.py:145)
14:31:51.550 [I] Progress on: 566it/1.50kit rate:7.0s/it remaining:1:49:37 elapsed:1:11:00 postfix:- (1143:tqdm_logging.py:145)
14:32:05.623 [I] Progress on: 568it/1.50kit rate:7.0s/it remaining:1:49:20 elapsed:1:11:14 postfix:- (1143:tqdm_logging.py:145)
14:32:19.713 [I] Progress on: 570it/1.50kit rate:7.0s/it remaining:1:49:09 elapsed:1:11:28 postfix:- (1143:tqdm_logging.py:145)
14:32:33.809 [I] Progress on: 572it/1.50kit rate:7.0s/it remaining:1:48:58 elapsed:1:11:42 postfix:- (1143:tqdm_logging.py:145)
14:32:47.890 [I] Progress on: 574it/1.50kit rate:7.0s/it remaining:1:48:41 elapsed:1:11:56 postfix:- (1143:tqdm_logging.py:145)
14:33:01.980 [I] Progress on: 576it/1.50kit rate:7.0s/it remaining:1:48:28 elapsed:1:12:10 postfix:- (1143:tqdm_logging.py:145)
14:33:16.075 [I] Progress on: 578it/1.50kit rate:7.0s/it remaining:1:48:16 elapsed:1:12:24 postfix:- (1143:tqdm_logging.py:145)
14:33:30.179 [I] Progress on: 580it/1.50kit rate:7.0s/it remaining:1:48:05 elapsed:1:12:38 postfix:- (1143:tqdm_logging.py:145)
14:33:44.297 [I] Progress on: 582it/1.50kit rate:7.1s/it remaining:1:47:55 elapsed:1:12:52 postfix:- (1143:tqdm_logging.py:145)
14:33:58.404 [I] Progress on: 584it/1.50kit rate:7.1s/it remaining:1:47:41 elapsed:1:13:07 postfix:- (1143:tqdm_logging.py:145)
14:34:12.503 [I] Progress on: 586it/1.50kit rate:7.1s/it remaining:1:47:26 elapsed:1:13:21 postfix:- (1143:tqdm_logging.py:145)
14:34:26.617 [I] Progress on: 588it/1.50kit rate:7.1s/it remaining:1:47:14 elapsed:1:13:35 postfix:- (1143:tqdm_logging.py:145)
14:34:40.732 [I] Progress on: 590it/1.50kit rate:7.1s/it remaining:1:47:01 elapsed:1:13:49 postfix:- (1143:tqdm_logging.py:145)
14:34:54.820 [I] Progress on: 592it/1.50kit rate:7.1s/it remaining:1:46:42 elapsed:1:14:03 postfix:- (1143:tqdm_logging.py:145)
14:35:08.926 [I] Progress on: 594it/1.50kit rate:7.1s/it remaining:1:46:29 elapsed:1:14:17 postfix:- (1143:tqdm_logging.py:145)
14:35:23.023 [I] Progress on: 596it/1.50kit rate:7.1s/it remaining:1:46:13 elapsed:1:14:31 postfix:- (1143:tqdm_logging.py:145)
14:35:37.115 [I] Progress on: 598it/1.50kit rate:7.0s/it remaining:1:45:56 elapsed:1:14:45 postfix:- (1143:tqdm_logging.py:145)
14:35:51.206 [I] Progress on: 600it/1.50kit rate:7.0s/it remaining:1:45:42 elapsed:1:14:59 postfix:- (1143:tqdm_logging.py:145)


14:36:04.920 [I] Progress on: 600it/1.50kit rate:7.0s/it remaining:1:45:42 elapsed:1:15:13 postfix:- (1143:tqdm_logging.py:145)
14:36:19.437 [I] Progress on: 604it/1.50kit rate:6.8s/it remaining:1:40:53 elapsed:1:15:28 postfix:- (1143:tqdm_logging.py:145)
14:36:33.508 [I] Progress on: 606it/1.50kit rate:6.9s/it remaining:1:42:47 elapsed:1:15:42 postfix:- (1143:tqdm_logging.py:145)
14:36:47.589 [I] Progress on: 608it/1.50kit rate:7.0s/it remaining:1:43:37 elapsed:1:15:56 postfix:- (1143:tqdm_logging.py:145)
14:37:01.652 [I] Progress on: 610it/1.50kit rate:7.0s/it remaining:1:43:51 elapsed:1:16:10 postfix:- (1143:tqdm_logging.py:145)
14:37:15.726 [I] Progress on: 612it/1.50kit rate:7.0s/it remaining:1:43:53 elapsed:1:16:24 postfix:- (1143:tqdm_logging.py:145)
14:37:29.805 [I] Progress on: 614it/1.50kit rate:7.0s/it remaining:1:43:48 elapsed:1:16:38 postfix:- (1143:tqdm_logging.py:145)
14:37:43.866 [I] Progress on: 616it/1.50kit rate:7.0s/it remaining:1:43:35 elapsed:1:16:52 postfix:- (1143:tqdm_logging.py:145)
14:37:57.943 [I] Progress on: 618it/1.50kit rate:7.0s/it remaining:1:43:24 elapsed:1:17:06 postfix:- (1143:tqdm_logging.py:145)
14:38:12.020 [I] Progress on: 620it/1.50kit rate:7.0s/it remaining:1:43:12 elapsed:1:17:20 postfix:- (1143:tqdm_logging.py:145)
14:38:26.101 [I] Progress on: 622it/1.50kit rate:7.0s/it remaining:1:43:00 elapsed:1:17:34 postfix:- (1143:tqdm_logging.py:145)
14:38:40.186 [I] Progress on: 624it/1.50kit rate:7.0s/it remaining:1:42:48 elapsed:1:17:48 postfix:- (1143:tqdm_logging.py:145)
14:38:54.274 [I] Progress on: 626it/1.50kit rate:7.0s/it remaining:1:42:37 elapsed:1:18:02 postfix:- (1143:tqdm_logging.py:145)
14:39:08.337 [I] Progress on: 628it/1.50kit rate:7.0s/it remaining:1:42:19 elapsed:1:18:16 postfix:- (1143:tqdm_logging.py:145)
14:39:22.426 [I] Progress on: 630it/1.50kit rate:7.0s/it remaining:1:42:06 elapsed:1:18:31 postfix:- (1143:tqdm_logging.py:145)
14:39:36.537 [I] Progress on: 632it/1.50kit rate:7.0s/it remaining:1:41:58 elapsed:1:18:45 postfix:- (1143:tqdm_logging.py:145)
14:39:50.633 [I] Progress on: 634it/1.50kit rate:7.0s/it remaining:1:41:44 elapsed:1:18:59 postfix:- (1143:tqdm_logging.py:145)
14:40:04.733 [I] Progress on: 636it/1.50kit rate:7.0s/it remaining:1:41:30 elapsed:1:19:13 postfix:- (1143:tqdm_logging.py:145)
14:40:18.848 [I] Progress on: 638it/1.50kit rate:7.1s/it remaining:1:41:19 elapsed:1:19:27 postfix:- (1143:tqdm_logging.py:145)
14:40:32.956 [I] Progress on: 640it/1.50kit rate:7.1s/it remaining:1:41:06 elapsed:1:19:41 postfix:- (1143:tqdm_logging.py:145)
14:40:47.618 [I] Progress on: 642it/1.50kit rate:7.2s/it remaining:1:43:15 elapsed:1:19:56 postfix:- (1143:tqdm_logging.py:145)
14:41:01.152 [I] Progress on: 644it/1.50kit rate:7.0s/it remaining:1:40:06 elapsed:1:20:09 postfix:- (1143:tqdm_logging.py:145)
14:41:15.261 [I] Progress on: 646it/1.50kit rate:7.0s/it remaining:1:40:08 elapsed:1:20:23 postfix:- (1143:tqdm_logging.py:145)
14:41:29.372 [I] Progress on: 648it/1.50kit rate:7.0s/it remaining:1:40:03 elapsed:1:20:37 postfix:- (1143:tqdm_logging.py:145)
14:41:43.486 [I] Progress on: 650it/1.50kit rate:7.1s/it remaining:1:39:54 elapsed:1:20:52 postfix:- (1143:tqdm_logging.py:145)


14:41:57.283 [I] Progress on: 650it/1.50kit rate:7.1s/it remaining:1:39:54 elapsed:1:21:05 postfix:- (1143:tqdm_logging.py:145)
14:42:11.837 [I] Progress on: 654it/1.50kit rate:6.8s/it remaining:1:35:25 elapsed:1:21:20 postfix:- (1143:tqdm_logging.py:145)
14:42:25.945 [I] Progress on: 656it/1.50kit rate:6.9s/it remaining:1:37:15 elapsed:1:21:34 postfix:- (1143:tqdm_logging.py:145)
14:42:40.055 [I] Progress on: 658it/1.50kit rate:7.0s/it remaining:1:38:02 elapsed:1:21:48 postfix:- (1143:tqdm_logging.py:145)
14:42:54.155 [I] Progress on: 660it/1.50kit rate:7.0s/it remaining:1:38:15 elapsed:1:22:02 postfix:- (1143:tqdm_logging.py:145)
14:43:08.267 [I] Progress on: 662it/1.50kit rate:7.0s/it remaining:1:38:16 elapsed:1:22:16 postfix:- (1143:tqdm_logging.py:145)
14:43:22.368 [I] Progress on: 664it/1.50kit rate:7.0s/it remaining:1:38:08 elapsed:1:22:30 postfix:- (1143:tqdm_logging.py:145)
14:43:36.474 [I] Progress on: 666it/1.50kit rate:7.0s/it remaining:1:37:57 elapsed:1:22:45 postfix:- (1143:tqdm_logging.py:145)
14:43:50.582 [I] Progress on: 668it/1.50kit rate:7.1s/it remaining:1:37:46 elapsed:1:22:59 postfix:- (1143:tqdm_logging.py:145)
14:44:04.703 [I] Progress on: 670it/1.50kit rate:7.1s/it remaining:1:37:36 elapsed:1:23:13 postfix:- (1143:tqdm_logging.py:145)
14:44:18.806 [I] Progress on: 672it/1.50kit rate:7.1s/it remaining:1:37:20 elapsed:1:23:27 postfix:- (1143:tqdm_logging.py:145)
14:44:32.907 [I] Progress on: 674it/1.50kit rate:7.1s/it remaining:1:37:05 elapsed:1:23:41 postfix:- (1143:tqdm_logging.py:145)
14:44:47.022 [I] Progress on: 676it/1.50kit rate:7.1s/it remaining:1:36:53 elapsed:1:23:55 postfix:- (1143:tqdm_logging.py:145)
14:45:01.126 [I] Progress on: 678it/1.50kit rate:7.1s/it remaining:1:36:37 elapsed:1:24:09 postfix:- (1143:tqdm_logging.py:145)
14:45:15.251 [I] Progress on: 680it/1.50kit rate:7.1s/it remaining:1:36:27 elapsed:1:24:23 postfix:- (1143:tqdm_logging.py:145)
14:45:29.368 [I] Progress on: 682it/1.50kit rate:7.1s/it remaining:1:36:12 elapsed:1:24:37 postfix:- (1143:tqdm_logging.py:145)
14:45:43.468 [I] Progress on: 684it/1.50kit rate:7.1s/it remaining:1:35:55 elapsed:1:24:52 postfix:- (1143:tqdm_logging.py:145)
14:45:57.583 [I] Progress on: 686it/1.50kit rate:7.1s/it remaining:1:35:43 elapsed:1:25:06 postfix:- (1143:tqdm_logging.py:145)
14:46:11.691 [I] Progress on: 688it/1.50kit rate:7.1s/it remaining:1:35:28 elapsed:1:25:20 postfix:- (1143:tqdm_logging.py:145)
14:46:25.824 [I] Progress on: 690it/1.50kit rate:7.1s/it remaining:1:35:19 elapsed:1:25:34 postfix:- (1143:tqdm_logging.py:145)
14:46:39.931 [I] Progress on: 692it/1.50kit rate:7.1s/it remaining:1:35:01 elapsed:1:25:48 postfix:- (1143:tqdm_logging.py:145)
14:46:54.043 [I] Progress on: 694it/1.50kit rate:7.1s/it remaining:1:34:47 elapsed:1:26:02 postfix:- (1143:tqdm_logging.py:145)
14:47:08.162 [I] Progress on: 696it/1.50kit rate:7.1s/it remaining:1:34:34 elapsed:1:26:16 postfix:- (1143:tqdm_logging.py:145)
14:47:22.270 [I] Progress on: 698it/1.50kit rate:7.1s/it remaining:1:34:18 elapsed:1:26:30 postfix:- (1143:tqdm_logging.py:145)
14:47:36.370 [I] Progress on: 700it/1.50kit rate:7.1s/it remaining:1:34:01 elapsed:1:26:44 postfix:- (1143:tqdm_logging.py:145)


14:47:50.116 [I] Progress on: 700it/1.50kit rate:7.1s/it remaining:1:34:01 elapsed:1:26:58 postfix:- (1143:tqdm_logging.py:145)
14:48:04.613 [I] Progress on: 704it/1.50kit rate:6.8s/it remaining:1:29:38 elapsed:1:27:13 postfix:- (1143:tqdm_logging.py:145)
14:48:18.710 [I] Progress on: 706it/1.50kit rate:6.9s/it remaining:1:31:22 elapsed:1:27:27 postfix:- (1143:tqdm_logging.py:145)
14:48:32.788 [I] Progress on: 708it/1.50kit rate:7.0s/it remaining:1:32:03 elapsed:1:27:41 postfix:- (1143:tqdm_logging.py:145)
14:48:46.858 [I] Progress on: 710it/1.50kit rate:7.0s/it remaining:1:32:13 elapsed:1:27:55 postfix:- (1143:tqdm_logging.py:145)
14:49:00.937 [I] Progress on: 712it/1.50kit rate:7.0s/it remaining:1:32:13 elapsed:1:28:09 postfix:- (1143:tqdm_logging.py:145)
14:49:15.010 [I] Progress on: 714it/1.50kit rate:7.0s/it remaining:1:32:05 elapsed:1:28:23 postfix:- (1143:tqdm_logging.py:145)
14:49:29.086 [I] Progress on: 716it/1.50kit rate:7.0s/it remaining:1:31:54 elapsed:1:28:37 postfix:- (1143:tqdm_logging.py:145)
14:49:43.159 [I] Progress on: 718it/1.50kit rate:7.0s/it remaining:1:31:41 elapsed:1:28:51 postfix:- (1143:tqdm_logging.py:145)
14:49:57.230 [I] Progress on: 720it/1.50kit rate:7.0s/it remaining:1:31:27 elapsed:1:29:05 postfix:- (1143:tqdm_logging.py:145)
14:50:11.307 [I] Progress on: 722it/1.50kit rate:7.0s/it remaining:1:31:14 elapsed:1:29:19 postfix:- (1143:tqdm_logging.py:145)
14:50:25.376 [I] Progress on: 724it/1.50kit rate:7.0s/it remaining:1:30:59 elapsed:1:29:33 postfix:- (1143:tqdm_logging.py:145)
14:50:39.465 [I] Progress on: 726it/1.50kit rate:7.0s/it remaining:1:30:49 elapsed:1:29:48 postfix:- (1143:tqdm_logging.py:145)
14:50:53.531 [I] Progress on: 728it/1.50kit rate:7.0s/it remaining:1:30:31 elapsed:1:30:02 postfix:- (1143:tqdm_logging.py:145)
14:51:07.606 [I] Progress on: 730it/1.50kit rate:7.0s/it remaining:1:30:17 elapsed:1:30:16 postfix:- (1143:tqdm_logging.py:145)
14:51:21.684 [I] Progress on: 732it/1.50kit rate:7.0s/it remaining:1:30:04 elapsed:1:30:30 postfix:- (1143:tqdm_logging.py:145)
14:51:35.759 [I] Progress on: 734it/1.50kit rate:7.0s/it remaining:1:29:50 elapsed:1:30:44 postfix:- (1143:tqdm_logging.py:145)
14:51:49.835 [I] Progress on: 736it/1.50kit rate:7.0s/it remaining:1:29:37 elapsed:1:30:58 postfix:- (1143:tqdm_logging.py:145)
14:52:03.903 [I] Progress on: 738it/1.50kit rate:7.0s/it remaining:1:29:21 elapsed:1:31:12 postfix:- (1143:tqdm_logging.py:145)
14:52:17.985 [I] Progress on: 740it/1.50kit rate:7.0s/it remaining:1:29:09 elapsed:1:31:26 postfix:- (1143:tqdm_logging.py:145)
14:52:32.058 [I] Progress on: 742it/1.50kit rate:7.0s/it remaining:1:28:54 elapsed:1:31:40 postfix:- (1143:tqdm_logging.py:145)
14:52:46.135 [I] Progress on: 744it/1.50kit rate:7.0s/it remaining:1:28:40 elapsed:1:31:54 postfix:- (1143:tqdm_logging.py:145)
14:53:00.212 [I] Progress on: 746it/1.50kit rate:7.0s/it remaining:1:28:26 elapsed:1:32:08 postfix:- (1143:tqdm_logging.py:145)
14:53:14.302 [I] Progress on: 748it/1.50kit rate:7.0s/it remaining:1:28:14 elapsed:1:32:22 postfix:- (1143:tqdm_logging.py:145)
14:53:28.381 [I] Progress on: 750it/1.50kit rate:7.0s/it remaining:1:27:24 elapsed:1:32:36 postfix:- (1143:tqdm_logging.py:145)


14:53:42.095 [I] Progress on: 750it/1.50kit rate:7.0s/it remaining:1:27:24 elapsed:1:32:50 postfix:- (1143:tqdm_logging.py:145)
14:53:56.615 [I] Progress on: 754it/1.50kit rate:6.7s/it remaining:1:23:49 elapsed:1:33:05 postfix:- (1143:tqdm_logging.py:145)
14:54:10.689 [I] Progress on: 756it/1.50kit rate:6.9s/it remaining:1:25:27 elapsed:1:33:19 postfix:- (1143:tqdm_logging.py:145)
14:54:24.762 [I] Progress on: 758it/1.50kit rate:7.0s/it remaining:1:26:08 elapsed:1:33:33 postfix:- (1143:tqdm_logging.py:145)
14:54:38.842 [I] Progress on: 760it/1.50kit rate:7.0s/it remaining:1:26:22 elapsed:1:33:47 postfix:- (1143:tqdm_logging.py:145)
14:54:52.914 [I] Progress on: 762it/1.50kit rate:7.0s/it remaining:1:26:20 elapsed:1:34:01 postfix:- (1143:tqdm_logging.py:145)
14:55:06.990 [I] Progress on: 764it/1.50kit rate:7.0s/it remaining:1:26:13 elapsed:1:34:15 postfix:- (1143:tqdm_logging.py:145)
14:55:21.067 [I] Progress on: 766it/1.50kit rate:7.0s/it remaining:1:26:02 elapsed:1:34:29 postfix:- (1143:tqdm_logging.py:145)
14:55:35.134 [I] Progress on: 768it/1.50kit rate:7.0s/it remaining:1:25:48 elapsed:1:34:43 postfix:- (1143:tqdm_logging.py:145)
14:55:49.197 [I] Progress on: 770it/1.50kit rate:7.0s/it remaining:1:25:34 elapsed:1:34:57 postfix:- (1143:tqdm_logging.py:145)
14:56:03.274 [I] Progress on: 772it/1.50kit rate:7.0s/it remaining:1:25:22 elapsed:1:35:11 postfix:- (1143:tqdm_logging.py:145)
14:56:17.352 [I] Progress on: 774it/1.50kit rate:7.0s/it remaining:1:25:09 elapsed:1:35:25 postfix:- (1143:tqdm_logging.py:145)
14:56:31.422 [I] Progress on: 776it/1.50kit rate:7.0s/it remaining:1:24:53 elapsed:1:35:40 postfix:- (1143:tqdm_logging.py:145)
14:56:45.496 [I] Progress on: 778it/1.50kit rate:7.0s/it remaining:1:24:40 elapsed:1:35:54 postfix:- (1143:tqdm_logging.py:145)
14:56:59.574 [I] Progress on: 780it/1.50kit rate:7.0s/it remaining:1:24:26 elapsed:1:36:08 postfix:- (1143:tqdm_logging.py:145)
14:57:13.660 [I] Progress on: 782it/1.50kit rate:7.0s/it remaining:1:24:15 elapsed:1:36:22 postfix:- (1143:tqdm_logging.py:145)
14:57:27.732 [I] Progress on: 784it/1.50kit rate:7.0s/it remaining:1:23:59 elapsed:1:36:36 postfix:- (1143:tqdm_logging.py:145)
14:57:41.803 [I] Progress on: 786it/1.50kit rate:7.0s/it remaining:1:23:44 elapsed:1:36:50 postfix:- (1143:tqdm_logging.py:145)
14:57:55.877 [I] Progress on: 788it/1.50kit rate:7.0s/it remaining:1:23:30 elapsed:1:37:04 postfix:- (1143:tqdm_logging.py:145)
14:58:09.958 [I] Progress on: 790it/1.50kit rate:7.0s/it remaining:1:23:17 elapsed:1:37:18 postfix:- (1143:tqdm_logging.py:145)
14:58:24.057 [I] Progress on: 792it/1.50kit rate:7.0s/it remaining:1:23:07 elapsed:1:37:32 postfix:- (1143:tqdm_logging.py:145)
14:58:38.131 [I] Progress on: 794it/1.50kit rate:7.0s/it remaining:1:22:50 elapsed:1:37:46 postfix:- (1143:tqdm_logging.py:145)
14:58:52.214 [I] Progress on: 796it/1.50kit rate:7.0s/it remaining:1:22:36 elapsed:1:38:00 postfix:- (1143:tqdm_logging.py:145)
14:59:06.302 [I] Progress on: 798it/1.50kit rate:7.0s/it remaining:1:22:24 elapsed:1:38:14 postfix:- (1143:tqdm_logging.py:145)
14:59:20.383 [I] Progress on: 800it/1.50kit rate:7.0s/it remaining:1:22:09 elapsed:1:38:28 postfix:- (1143:tqdm_logging.py:145)


14:59:34.114 [I] Progress on: 800it/1.50kit rate:7.0s/it remaining:1:22:09 elapsed:1:38:42 postfix:- (1143:tqdm_logging.py:145)
14:59:48.618 [I] Progress on: 804it/1.50kit rate:6.8s/it remaining:1:18:21 elapsed:1:38:57 postfix:- (1143:tqdm_logging.py:145)
15:00:02.709 [I] Progress on: 806it/1.50kit rate:6.9s/it remaining:1:19:49 elapsed:1:39:11 postfix:- (1143:tqdm_logging.py:145)
15:00:16.830 [I] Progress on: 808it/1.50kit rate:7.0s/it remaining:1:20:31 elapsed:1:39:25 postfix:- (1143:tqdm_logging.py:145)
15:00:30.916 [I] Progress on: 810it/1.50kit rate:7.0s/it remaining:1:20:38 elapsed:1:39:39 postfix:- (1143:tqdm_logging.py:145)
15:00:45.006 [I] Progress on: 812it/1.50kit rate:7.0s/it remaining:1:20:36 elapsed:1:39:53 postfix:- (1143:tqdm_logging.py:145)
15:00:59.087 [I] Progress on: 814it/1.50kit rate:7.0s/it remaining:1:20:25 elapsed:1:40:07 postfix:- (1143:tqdm_logging.py:145)
15:01:13.164 [I] Progress on: 816it/1.50kit rate:7.0s/it remaining:1:20:13 elapsed:1:40:21 postfix:- (1143:tqdm_logging.py:145)
15:01:27.235 [I] Progress on: 818it/1.50kit rate:7.0s/it remaining:1:19:58 elapsed:1:40:35 postfix:- (1143:tqdm_logging.py:145)
15:01:41.309 [I] Progress on: 820it/1.50kit rate:7.0s/it remaining:1:19:44 elapsed:1:40:49 postfix:- (1143:tqdm_logging.py:145)
15:01:55.384 [I] Progress on: 822it/1.50kit rate:7.0s/it remaining:1:19:30 elapsed:1:41:04 postfix:- (1143:tqdm_logging.py:145)
15:02:09.463 [I] Progress on: 824it/1.50kit rate:7.0s/it remaining:1:19:17 elapsed:1:41:18 postfix:- (1143:tqdm_logging.py:145)
15:02:23.539 [I] Progress on: 826it/1.50kit rate:7.0s/it remaining:1:19:03 elapsed:1:41:32 postfix:- (1143:tqdm_logging.py:145)
15:02:37.608 [I] Progress on: 828it/1.50kit rate:7.0s/it remaining:1:18:48 elapsed:1:41:46 postfix:- (1143:tqdm_logging.py:145)
15:02:51.692 [I] Progress on: 830it/1.50kit rate:7.0s/it remaining:1:18:36 elapsed:1:42:00 postfix:- (1143:tqdm_logging.py:145)
15:03:05.776 [I] Progress on: 832it/1.50kit rate:7.0s/it remaining:1:18:23 elapsed:1:42:14 postfix:- (1143:tqdm_logging.py:145)
15:03:19.836 [I] Progress on: 834it/1.50kit rate:7.0s/it remaining:1:18:05 elapsed:1:42:28 postfix:- (1143:tqdm_logging.py:145)
15:03:33.915 [I] Progress on: 836it/1.50kit rate:7.0s/it remaining:1:17:53 elapsed:1:42:42 postfix:- (1143:tqdm_logging.py:145)
15:03:47.984 [I] Progress on: 838it/1.50kit rate:7.0s/it remaining:1:17:37 elapsed:1:42:56 postfix:- (1143:tqdm_logging.py:145)
15:04:02.062 [I] Progress on: 840it/1.50kit rate:7.0s/it remaining:1:17:24 elapsed:1:43:10 postfix:- (1143:tqdm_logging.py:145)
15:04:16.140 [I] Progress on: 842it/1.50kit rate:7.0s/it remaining:1:17:10 elapsed:1:43:24 postfix:- (1143:tqdm_logging.py:145)
15:04:30.215 [I] Progress on: 844it/1.50kit rate:7.0s/it remaining:1:16:56 elapsed:1:43:38 postfix:- (1143:tqdm_logging.py:145)
15:04:44.290 [I] Progress on: 846it/1.50kit rate:7.0s/it remaining:1:16:42 elapsed:1:43:52 postfix:- (1143:tqdm_logging.py:145)
15:04:58.370 [I] Progress on: 848it/1.50kit rate:7.0s/it remaining:1:16:29 elapsed:1:44:06 postfix:- (1143:tqdm_logging.py:145)
15:05:12.444 [I] Progress on: 850it/1.50kit rate:7.0s/it remaining:1:16:14 elapsed:1:44:21 postfix:- (1143:tqdm_logging.py:145)


15:05:26.155 [I] Progress on: 850it/1.50kit rate:7.0s/it remaining:1:16:14 elapsed:1:44:34 postfix:- (1143:tqdm_logging.py:145)
15:05:40.636 [I] Progress on: 854it/1.50kit rate:6.7s/it remaining:1:12:38 elapsed:1:44:49 postfix:- (1143:tqdm_logging.py:145)
15:05:55.274 [I] Progress on: 856it/1.50kit rate:7.1s/it remaining:1:15:49 elapsed:1:45:03 postfix:- (1143:tqdm_logging.py:145)
15:06:08.794 [I] Progress on: 858it/1.50kit rate:6.9s/it remaining:1:14:13 elapsed:1:45:17 postfix:- (1143:tqdm_logging.py:145)
15:06:22.878 [I] Progress on: 860it/1.50kit rate:7.0s/it remaining:1:14:36 elapsed:1:45:31 postfix:- (1143:tqdm_logging.py:145)
15:06:36.942 [I] Progress on: 862it/1.50kit rate:7.0s/it remaining:1:14:37 elapsed:1:45:45 postfix:- (1143:tqdm_logging.py:145)
15:06:51.033 [I] Progress on: 864it/1.50kit rate:7.0s/it remaining:1:14:34 elapsed:1:45:59 postfix:- (1143:tqdm_logging.py:145)
15:07:05.104 [I] Progress on: 866it/1.50kit rate:7.0s/it remaining:1:14:23 elapsed:1:46:13 postfix:- (1143:tqdm_logging.py:145)
15:07:19.185 [I] Progress on: 868it/1.50kit rate:7.0s/it remaining:1:14:11 elapsed:1:46:27 postfix:- (1143:tqdm_logging.py:145)
15:07:33.247 [I] Progress on: 870it/1.50kit rate:7.0s/it remaining:1:13:56 elapsed:1:46:41 postfix:- (1143:tqdm_logging.py:145)
15:07:47.334 [I] Progress on: 872it/1.50kit rate:7.0s/it remaining:1:13:45 elapsed:1:46:55 postfix:- (1143:tqdm_logging.py:145)
15:08:01.400 [I] Progress on: 874it/1.50kit rate:7.0s/it remaining:1:13:28 elapsed:1:47:10 postfix:- (1143:tqdm_logging.py:145)
15:08:15.487 [I] Progress on: 876it/1.50kit rate:7.0s/it remaining:1:13:17 elapsed:1:47:24 postfix:- (1143:tqdm_logging.py:145)
15:08:29.546 [I] Progress on: 878it/1.50kit rate:7.0s/it remaining:1:13:00 elapsed:1:47:38 postfix:- (1143:tqdm_logging.py:145)
15:08:43.638 [I] Progress on: 880it/1.50kit rate:7.0s/it remaining:1:12:49 elapsed:1:47:52 postfix:- (1143:tqdm_logging.py:145)
15:08:57.703 [I] Progress on: 882it/1.50kit rate:7.0s/it remaining:1:12:33 elapsed:1:48:06 postfix:- (1143:tqdm_logging.py:145)
15:09:11.780 [I] Progress on: 884it/1.50kit rate:7.0s/it remaining:1:12:19 elapsed:1:48:20 postfix:- (1143:tqdm_logging.py:145)
15:09:25.855 [I] Progress on: 886it/1.50kit rate:7.0s/it remaining:1:12:05 elapsed:1:48:34 postfix:- (1143:tqdm_logging.py:145)
15:09:39.922 [I] Progress on: 888it/1.50kit rate:7.0s/it remaining:1:11:49 elapsed:1:48:48 postfix:- (1143:tqdm_logging.py:145)
15:09:53.993 [I] Progress on: 890it/1.50kit rate:7.0s/it remaining:1:11:33 elapsed:1:49:02 postfix:- (1143:tqdm_logging.py:145)
15:10:08.062 [I] Progress on: 892it/1.50kit rate:7.0s/it remaining:1:11:17 elapsed:1:49:16 postfix:- (1143:tqdm_logging.py:145)
15:10:22.144 [I] Progress on: 894it/1.50kit rate:7.0s/it remaining:1:11:05 elapsed:1:49:30 postfix:- (1143:tqdm_logging.py:145)
15:10:36.219 [I] Progress on: 896it/1.50kit rate:7.0s/it remaining:1:10:50 elapsed:1:49:44 postfix:- (1143:tqdm_logging.py:145)
15:10:50.298 [I] Progress on: 898it/1.50kit rate:7.0s/it remaining:1:10:37 elapsed:1:49:58 postfix:- (1143:tqdm_logging.py:145)
15:11:04.373 [I] Progress on: 900it/1.50kit rate:7.0s/it remaining:1:10:22 elapsed:1:50:12 postfix:- (1143:tqdm_logging.py:145)


15:11:18.087 [I] Progress on: 900it/1.50kit rate:7.0s/it remaining:1:10:22 elapsed:1:50:26 postfix:- (1143:tqdm_logging.py:145)
15:11:32.606 [I] Progress on: 904it/1.50kit rate:6.8s/it remaining:1:07:04 elapsed:1:50:41 postfix:- (1143:tqdm_logging.py:145)
15:11:46.682 [I] Progress on: 906it/1.50kit rate:6.9s/it remaining:1:08:17 elapsed:1:50:55 postfix:- (1143:tqdm_logging.py:145)
15:12:00.761 [I] Progress on: 908it/1.50kit rate:7.0s/it remaining:1:08:46 elapsed:1:51:09 postfix:- (1143:tqdm_logging.py:145)
15:12:14.854 [I] Progress on: 910it/1.50kit rate:7.0s/it remaining:1:08:55 elapsed:1:51:23 postfix:- (1143:tqdm_logging.py:145)
15:12:28.935 [I] Progress on: 912it/1.50kit rate:7.0s/it remaining:1:08:50 elapsed:1:51:37 postfix:- (1143:tqdm_logging.py:145)
15:12:43.032 [I] Progress on: 914it/1.50kit rate:7.0s/it remaining:1:08:43 elapsed:1:51:51 postfix:- (1143:tqdm_logging.py:145)
15:12:57.120 [I] Progress on: 916it/1.50kit rate:7.0s/it remaining:1:08:32 elapsed:1:52:05 postfix:- (1143:tqdm_logging.py:145)
15:13:11.201 [I] Progress on: 918it/1.50kit rate:7.0s/it remaining:1:08:17 elapsed:1:52:19 postfix:- (1143:tqdm_logging.py:145)
15:13:25.298 [I] Progress on: 920it/1.50kit rate:7.0s/it remaining:1:08:05 elapsed:1:52:33 postfix:- (1143:tqdm_logging.py:145)
15:13:39.375 [I] Progress on: 922it/1.50kit rate:7.0s/it remaining:1:07:50 elapsed:1:52:47 postfix:- (1143:tqdm_logging.py:145)
15:13:53.445 [I] Progress on: 924it/1.50kit rate:7.0s/it remaining:1:07:34 elapsed:1:53:02 postfix:- (1143:tqdm_logging.py:145)
15:14:07.527 [I] Progress on: 926it/1.50kit rate:7.0s/it remaining:1:07:20 elapsed:1:53:16 postfix:- (1143:tqdm_logging.py:145)
15:14:21.595 [I] Progress on: 928it/1.50kit rate:7.0s/it remaining:1:07:04 elapsed:1:53:30 postfix:- (1143:tqdm_logging.py:145)
15:14:35.672 [I] Progress on: 930it/1.50kit rate:7.0s/it remaining:1:06:51 elapsed:1:53:44 postfix:- (1143:tqdm_logging.py:145)
15:14:49.748 [I] Progress on: 932it/1.50kit rate:7.0s/it remaining:1:06:37 elapsed:1:53:58 postfix:- (1143:tqdm_logging.py:145)
15:15:03.839 [I] Progress on: 934it/1.50kit rate:7.0s/it remaining:1:06:25 elapsed:1:54:12 postfix:- (1143:tqdm_logging.py:145)
15:15:17.917 [I] Progress on: 936it/1.50kit rate:7.0s/it remaining:1:06:09 elapsed:1:54:26 postfix:- (1143:tqdm_logging.py:145)
15:15:31.991 [I] Progress on: 938it/1.50kit rate:7.0s/it remaining:1:05:54 elapsed:1:54:40 postfix:- (1143:tqdm_logging.py:145)
15:15:46.074 [I] Progress on: 940it/1.50kit rate:7.0s/it remaining:1:05:41 elapsed:1:54:54 postfix:- (1143:tqdm_logging.py:145)
15:16:00.138 [I] Progress on: 942it/1.50kit rate:7.0s/it remaining:1:05:24 elapsed:1:55:08 postfix:- (1143:tqdm_logging.py:145)
15:16:14.417 [I] Progress on: 944it/1.50kit rate:7.1s/it remaining:1:05:45 elapsed:1:55:23 postfix:- (1143:tqdm_logging.py:145)
15:16:28.441 [I] Progress on: 946it/1.50kit rate:7.1s/it remaining:1:05:06 elapsed:1:55:37 postfix:- (1143:tqdm_logging.py:145)
15:16:42.483 [I] Progress on: 948it/1.50kit rate:7.0s/it remaining:1:04:43 elapsed:1:55:51 postfix:- (1143:tqdm_logging.py:145)
15:16:56.554 [I] Progress on: 950it/1.50kit rate:7.0s/it remaining:1:04:29 elapsed:1:56:05 postfix:- (1143:tqdm_logging.py:145)


15:17:10.254 [I] Progress on: 950it/1.50kit rate:7.0s/it remaining:1:04:29 elapsed:1:56:18 postfix:- (1143:tqdm_logging.py:145)
15:17:24.715 [I] Progress on: 954it/1.50kit rate:6.7s/it remaining:1:01:18 elapsed:1:56:33 postfix:- (1143:tqdm_logging.py:145)
15:17:38.755 [I] Progress on: 956it/1.50kit rate:6.9s/it remaining:1:02:23 elapsed:1:56:47 postfix:- (1143:tqdm_logging.py:145)
15:17:52.792 [I] Progress on: 958it/1.50kit rate:7.0s/it remaining:1:02:47 elapsed:1:57:01 postfix:- (1143:tqdm_logging.py:145)
15:18:06.827 [I] Progress on: 960it/1.50kit rate:7.0s/it remaining:1:02:51 elapsed:1:57:15 postfix:- (1143:tqdm_logging.py:145)
15:18:20.856 [I] Progress on: 962it/1.50kit rate:7.0s/it remaining:1:02:45 elapsed:1:57:29 postfix:- (1143:tqdm_logging.py:145)
15:18:34.871 [I] Progress on: 964it/1.50kit rate:7.0s/it remaining:1:02:08 elapsed:1:57:43 postfix:- (1143:tqdm_logging.py:145)
15:18:48.910 [I] Progress on: 966it/1.50kit rate:7.0s/it remaining:1:02:09 elapsed:1:57:57 postfix:- (1143:tqdm_logging.py:145)
15:19:02.971 [I] Progress on: 968it/1.50kit rate:7.0s/it remaining:1:02:06 elapsed:1:58:11 postfix:- (1143:tqdm_logging.py:145)
15:19:16.996 [I] Progress on: 970it/1.50kit rate:7.0s/it remaining:1:01:52 elapsed:1:58:25 postfix:- (1143:tqdm_logging.py:145)
15:19:31.067 [I] Progress on: 972it/1.50kit rate:7.0s/it remaining:1:01:46 elapsed:1:58:39 postfix:- (1143:tqdm_logging.py:145)
15:19:45.121 [I] Progress on: 974it/1.50kit rate:7.0s/it remaining:1:01:34 elapsed:1:58:53 postfix:- (1143:tqdm_logging.py:145)
15:19:59.170 [I] Progress on: 976it/1.50kit rate:7.0s/it remaining:1:01:20 elapsed:1:59:07 postfix:- (1143:tqdm_logging.py:145)
15:20:13.209 [I] Progress on: 978it/1.50kit rate:7.0s/it remaining:1:01:05 elapsed:1:59:21 postfix:- (1143:tqdm_logging.py:145)
15:20:27.263 [I] Progress on: 980it/1.50kit rate:7.0s/it remaining:1:00:53 elapsed:1:59:35 postfix:- (1143:tqdm_logging.py:145)
15:20:41.302 [I] Progress on: 982it/1.50kit rate:7.0s/it remaining:1:00:37 elapsed:1:59:49 postfix:- (1143:tqdm_logging.py:145)
15:20:55.334 [I] Progress on: 984it/1.50kit rate:7.0s/it remaining:1:00:22 elapsed:2:00:03 postfix:- (1143:tqdm_logging.py:145)
15:21:09.387 [I] Progress on: 986it/1.50kit rate:7.0s/it remaining:1:00:10 elapsed:2:00:18 postfix:- (1143:tqdm_logging.py:145)
15:21:23.425 [I] Progress on: 988it/1.50kit rate:7.0s/it remaining:59:55 elapsed:2:00:32 postfix:- (1143:tqdm_logging.py:145)
15:21:37.480 [I] Progress on: 990it/1.50kit rate:7.0s/it remaining:59:42 elapsed:2:00:46 postfix:- (1143:tqdm_logging.py:145)
15:21:51.546 [I] Progress on: 992it/1.50kit rate:7.0s/it remaining:59:31 elapsed:2:01:00 postfix:- (1143:tqdm_logging.py:145)
15:22:05.590 [I] Progress on: 994it/1.50kit rate:7.0s/it remaining:59:14 elapsed:2:01:14 postfix:- (1143:tqdm_logging.py:145)
15:22:19.660 [I] Progress on: 996it/1.50kit rate:7.0s/it remaining:59:03 elapsed:2:01:28 postfix:- (1143:tqdm_logging.py:145)
15:22:33.710 [I] Progress on: 998it/1.50kit rate:7.0s/it remaining:58:44 elapsed:2:01:42 postfix:- (1143:tqdm_logging.py:145)
15:22:47.764 [I] Progress on: 1.00kit/1.50kit rate:7.0s/it remaining:58:31 elapsed:2:01:56 postfix:- (1143:tqdm_logging.py:145)


15:23:01.460 [I] Progress on: 1.00kit/1.50kit rate:7.0s/it remaining:58:31 elapsed:2:02:10 postfix:- (1143:tqdm_logging.py:145)
15:23:01.533 [I] [process=0][thread=MainThread][wait_until_finished] No Save Finalize thread to wait for. Returning. (1143:checkpoint_manager.py:1987)
15:23:01.534 [I] [process=0] Saving checkpoint at step 1000                                       (1143:checkpoint_manager.py:1408)
15:23:01.535 [I] [process=0] Started async saving checkpoint to /workspace/openpi/checkpoints/pi05_mecharm_trim_corr_lora/trim_corr_v3/1000. (1143:async_checkpointer.py:439)
15:23:01.537 [I] Using ThreadSafeKeyValueSignalingClient                                          (1143:signaling_client.py:332)
15:23:01.540 [I] Creating tmp directory /workspace/openpi/checkpoints/pi05_mecharm_trim_corr_lora/trim_corr_v3/1000.orbax-checkpoint-tmp-0 (1143:atomicity.py:144)
15:23:01.564 [I] Wrote Metadata={'item_handlers': None, 'metrics': {}, 'performance_metrics': {}, 'init_timestamp_nsecs': 1793460181562130319, 'commit_timestamp_nsecs': None, 'custom_metadata': {}}, json={"item_handlers": null, "metrics": {}, "performance_metrics": {}, "init_timestamp_nsecs": 1793460181562130319, "commit_timestamp_nsecs": null, "custom_metadata": {}} to /workspace/openpi/checkpoints/pi05_mecharm_trim_corr_lora/trim_corr_v3/1000.orbax-checkpoint-tmp-0/_CHECKPOINT_METADATA (1143:checkpoint.py:186)
15:23:01.567 [I] Creating tmp directory /workspace/openpi/checkpoints/pi05_mecharm_trim_corr_lora/trim_corr_v3/1000.orbax-checkpoint-tmp-0/assets.orbax-checkpoint-tmp-1 (1143:atomicity.py:144)
15:23:01.568 [I] Creating tmp directory /workspace/openpi/checkpoints/pi05_mecharm_trim_corr_lora/trim_corr_v3/1000.orbax-checkpoint-tmp-0/params.orbax-checkpoint-tmp-2 (1143:atomicity.py:144)
15:23:01.568 [I] Creating tmp directory /workspace/openpi/checkpoints/pi05_mecharm_trim_corr_lora/trim_corr_v3/1000.orbax-checkpoint-tmp-0/train_state.orbax-checkpoint-tmp-3 (1143:atomicity.py:144)
15:23:01.589 [I] Transferring arrays to host memory with options: use_replica_parallel=True, enable_pinned_host_transfer=False (1143:replica_slices.py:341)
15:23:23.491 [I] Transferring arrays to host memory with options: use_replica_parallel=True, enable_pinned_host_transfer=False (1143:replica_slices.py:341)
15:23:23.500 [I] Array name: 'params.PaliGemma.llm.layers.mlp.gating_einsum.value', global shape: (18, 2, 2048, 16384), write shape: (18, 2, 2048, 16384), chosen chunk shape: (18, 2, 2048, 8192) (1143:tensorstore_utils.py:408)
15:23:23.558 [I] [process=0][thread=array_type_handler] Wrote 71 array_metadata.ArrayMetadata to /workspace/openpi/checkpoints/pi05_mecharm_trim_corr_lora/trim_corr_v3/1000.orbax-checkpoint-tmp-0/params.orbax-checkpoint-tmp-2/array_metadatas/process_0 (1143:array_metadata_store.py:198)
15:23:26.678 [I] [process=0] /jax/checkpoint/write/blocking_bytes_per_sec: 293.7 MiB/s (total bytes: 7.2 GiB) (time elapsed: 25 seconds) (per-host) (1143:base_pytree_checkpoint_handler.py:114)
15:23:26.685 [I] [process=0] /jax/checkpoint/write/blocking_bytes_per_sec: 141.8 MiB/s (total bytes: 3.5 GiB) (time elapsed: 25 seconds) (per-host) (1143:base_pytree_checkpoint_handler.py:114)
15:23:26.701 [I] [process=0][thread=async_save] Background save thread started.                   (1143:async_checkpointer.py:77)
15:23:26.702 [I] Finished blocking save in 25.17 seconds. Continuing to save asynchronously to /workspace/openpi/checkpoints/pi05_mecharm_trim_corr_lora/trim_corr_v3/1000. (1143:async_checkpointer.py:548)
15:23:26.704 [I] [process=0][thread=MainThread][step=1000] Starting CheckpointManager Save Finalize thread=save_finalize (1143:checkpoint_manager.py:1453)
15:23:26.708 [I] [process=0][thread=save_finalize] Waiting for background save thread=async_save. (1143:async_checkpointer.py:258)
15:23:26.710 [I] {'step': 1000, 'event_type': 'save', 'directory': '/workspace/openpi/checkpoints/pi05_mecharm_trim_corr_lora/trim_corr_v3', 'reached_preemption': False, 'preemption_received_at': None, 'synchronous': False, 'wait_for_prev_start_time': 1793460181.5333178, 'wait_for_prev_duration_secs': 0.0002067089080810547, 'checkpointer_blocking_start_time': 1793460181.5345185, 'checkpointer_blocking_duration_secs': 25.169535636901855, 'get_old_steps_start_time': 1793460206.704104, 'get_old_steps_duration_secs': 1.3113021850585938e-05, 'checkpoint_manager_blocking_start_time': 1793460181.53271, 'checkpoint_manager_blocking_duration_secs': 25.17711877822876} (1143:standard_logger.py:34)
15:23:26.714 [I] Progress on: 1.00kit/1.50kit rate:16.6s/it remaining:2:18:03 elapsed:2:02:35 postfix:- (1143:tqdm_logging.py:145)
15:23:26.769 [I] [process=0][thread=array_type_handler] Wrote 105 array_metadata.ArrayMetadata to /workspace/openpi/checkpoints/pi05_mecharm_trim_corr_lora/trim_corr_v3/1000.orbax-checkpoint-tmp-0/train_state.orbax-checkpoint-tmp-3/array_metadatas/process_0 (1143:array_metadata_store.py:198)
15:23:28.625 [I] [process=0] /jax/checkpoint/write/bytes_per_sec: 272.6 MiB/s (total bytes: 7.2 GiB) (time elapsed: 27 seconds) (per-host) (1143:base_pytree_checkpoint_handler.py:114)
15:23:29.590 [I] [process=0] /jax/checkpoint/write/bytes_per_sec: 127.1 MiB/s (total bytes: 3.5 GiB) (time elapsed: 28 seconds) (per-host) (1143:base_pytree_checkpoint_handler.py:114)
15:23:29.591 [I] [process=0][thread=async_save] 5 Handler Commit operations completed.            (1143:async_checkpointer.py:87)
15:23:29.592 [I] Read Metadata={'item_handlers': None, 'metrics': {}, 'performance_metrics': {}, 'init_timestamp_nsecs': 1793460181562130319, 'commit_timestamp_nsecs': None, 'custom_metadata': {}} from /workspace/openpi/checkpoints/pi05_mecharm_trim_corr_lora/trim_corr_v3/1000.orbax-checkpoint-tmp-0/_CHECKPOINT_METADATA (1143:checkpoint.py:226)
15:23:29.593 [I] Renaming /workspace/openpi/checkpoints/pi05_mecharm_trim_corr_lora/trim_corr_v3/1000.orbax-checkpoint-tmp-0/assets.orbax-checkpoint-tmp-1 to /workspace/openpi/checkpoints/pi05_mecharm_trim_corr_lora/trim_corr_v3/1000.orbax-checkpoint-tmp-0/assets (1143:atomicity.py:289)
15:23:29.596 [I] Updated Metadata={'item_handlers': {'assets': 'openpi.training.checkpoints.CallbackHandler', 'params': 'orbax.checkpoint._src.handlers.pytree_checkpoint_handler.PyTreeCheckpointHandler', 'train_state': 'orbax.checkpoint._src.handlers.pytree_checkpoint_handler.PyTreeCheckpointHandler'}, 'metrics': {}, 'performance_metrics': {}, 'init_timestamp_nsecs': 1793460181562130319, 'commit_timestamp_nsecs': None, 'custom_metadata': {}} to /workspace/openpi/checkpoints/pi05_mecharm_trim_corr_lora/trim_corr_v3/1000.orbax-checkpoint-tmp-0/_CHECKPOINT_METADATA (1143:checkpoint.py:245)
15:23:29.599 [I] [process=0][thread=async_save] Skipped cross-host ArrayMetadata validation because only one process is found: process_index=0. (1143:array_metadata_store.py:362)
15:23:29.606 [I] Renaming /workspace/openpi/checkpoints/pi05_mecharm_trim_corr_lora/trim_corr_v3/1000.orbax-checkpoint-tmp-0/train_state.orbax-checkpoint-tmp-3 to /workspace/openpi/checkpoints/pi05_mecharm_trim_corr_lora/trim_corr_v3/1000.orbax-checkpoint-tmp-0/train_state (1143:atomicity.py:289)
15:23:29.610 [I] [process=0][thread=async_save] Skipped cross-host ArrayMetadata validation because only one process is found: process_index=0. (1143:array_metadata_store.py:362)
15:23:29.615 [I] Renaming /workspace/openpi/checkpoints/pi05_mecharm_trim_corr_lora/trim_corr_v3/1000.orbax-checkpoint-tmp-0/params.orbax-checkpoint-tmp-2 to /workspace/openpi/checkpoints/pi05_mecharm_trim_corr_lora/trim_corr_v3/1000.orbax-checkpoint-tmp-0/params (1143:atomicity.py:289)
15:23:29.616 [I] Renaming /workspace/openpi/checkpoints/pi05_mecharm_trim_corr_lora/trim_corr_v3/1000.orbax-checkpoint-tmp-0 to /workspace/openpi/checkpoints/pi05_mecharm_trim_corr_lora/trim_corr_v3/1000 (1143:atomicity.py:289)
15:23:29.617 [I] [process=0][thread=async_save] Finished saving checkpoint (finalized tmp dir) to `/workspace/openpi/checkpoints/pi05_mecharm_trim_corr_lora/trim_corr_v3/1000`. (1143:atomicity.py:573)
15:23:29.618 [I] Finished asynchronous save (blocking + background) in 28.08 seconds to /workspace/openpi/checkpoints/pi05_mecharm_trim_corr_lora/trim_corr_v3/1000 (1143:async_checkpointer.py:412)
15:23:29.618 [I] [process=0][thread=async_save] Background save thread done.                      (1143:async_checkpointer.py:143)
15:23:29.618 [I] [process=0][thread=save_finalize] Done with waiting for background save thread=async_save. (1143:async_checkpointer.py:266)
15:23:29.618 [I] [process=0][thread=save_finalize] No errors found in background save thread=async_save. (1143:async_checkpointer.py:276)
15:23:29.618 [I] [process=0][thread=save_finalize][step=1000] CheckpointManager Save Finalize is syncing with other hosts... (1143:checkpoint_manager.py:2095)
15:23:29.618 [I] [process=0][thread=save_finalize][step=1000] CheckpointManager Save Finalize is done on all hosts. (1143:checkpoint_manager.py:2110)
15:23:41.077 [I] Progress on: 1.00kit/1.50kit rate:9.3s/it remaining:1:16:52 elapsed:2:02:49 postfix:- (1143:tqdm_logging.py:145)
15:23:55.057 [I] Progress on: 1.01kit/1.50kit rate:8.1s/it remaining:1:06:52 elapsed:2:03:03 postfix:- (1143:tqdm_logging.py:145)
15:24:09.077 [I] Progress on: 1.01kit/1.50kit rate:7.6s/it remaining:1:01:57 elapsed:2:03:17 postfix:- (1143:tqdm_logging.py:145)
15:24:23.102 [I] Progress on: 1.01kit/1.50kit rate:7.3s/it remaining:59:26 elapsed:2:03:31 postfix:- (1143:tqdm_logging.py:145)
15:24:37.150 [I] Progress on: 1.01kit/1.50kit rate:7.1s/it remaining:58:08 elapsed:2:03:45 postfix:- (1143:tqdm_logging.py:145)
15:24:51.210 [I] Progress on: 1.01kit/1.50kit rate:7.1s/it remaining:57:25 elapsed:2:03:59 postfix:- (1143:tqdm_logging.py:145)
15:25:05.321 [I] Progress on: 1.02kit/1.50kit rate:7.1s/it remaining:57:00 elapsed:2:04:13 postfix:- (1143:tqdm_logging.py:145)
15:25:19.371 [I] Progress on: 1.02kit/1.50kit rate:7.0s/it remaining:56:36 elapsed:2:04:27 postfix:- (1143:tqdm_logging.py:145)
15:25:33.461 [I] Progress on: 1.02kit/1.50kit rate:7.0s/it remaining:56:21 elapsed:2:04:42 postfix:- (1143:tqdm_logging.py:145)
15:25:47.558 [I] Progress on: 1.02kit/1.50kit rate:7.0s/it remaining:56:08 elapsed:2:04:56 postfix:- (1143:tqdm_logging.py:145)
15:26:01.660 [I] Progress on: 1.02kit/1.50kit rate:7.0s/it remaining:55:54 elapsed:2:05:10 postfix:- (1143:tqdm_logging.py:145)
15:26:15.753 [I] Progress on: 1.03kit/1.50kit rate:7.0s/it remaining:55:40 elapsed:2:05:24 postfix:- (1143:tqdm_logging.py:145)
15:26:29.832 [I] Progress on: 1.03kit/1.50kit rate:7.0s/it remaining:55:23 elapsed:2:05:38 postfix:- (1143:tqdm_logging.py:145)
15:26:43.913 [I] Progress on: 1.03kit/1.50kit rate:7.0s/it remaining:55:09 elapsed:2:05:52 postfix:- (1143:tqdm_logging.py:145)
15:26:57.979 [I] Progress on: 1.03kit/1.50kit rate:7.0s/it remaining:54:53 elapsed:2:06:06 postfix:- (1143:tqdm_logging.py:145)
15:27:12.053 [I] Progress on: 1.03kit/1.50kit rate:7.0s/it remaining:54:38 elapsed:2:06:20 postfix:- (1143:tqdm_logging.py:145)
15:27:26.134 [I] Progress on: 1.04kit/1.50kit rate:7.0s/it remaining:54:25 elapsed:2:06:34 postfix:- (1143:tqdm_logging.py:145)
15:27:40.209 [I] Progress on: 1.04kit/1.50kit rate:7.0s/it remaining:54:10 elapsed:2:06:48 postfix:- (1143:tqdm_logging.py:145)
15:27:54.286 [I] Progress on: 1.04kit/1.50kit rate:7.0s/it remaining:53:56 elapsed:2:07:02 postfix:- (1143:tqdm_logging.py:145)
15:28:08.360 [I] Progress on: 1.04kit/1.50kit rate:7.0s/it remaining:53:42 elapsed:2:07:16 postfix:- (1143:tqdm_logging.py:145)
15:28:22.428 [I] Progress on: 1.04kit/1.50kit rate:7.0s/it remaining:53:27 elapsed:2:07:31 postfix:- (1143:tqdm_logging.py:145)
15:28:36.510 [I] Progress on: 1.05kit/1.50kit rate:7.0s/it remaining:53:15 elapsed:2:07:45 postfix:- (1143:tqdm_logging.py:145)
15:28:50.577 [I] Progress on: 1.05kit/1.50kit rate:7.0s/it remaining:52:59 elapsed:2:07:59 postfix:- (1143:tqdm_logging.py:145)
15:29:04.651 [I] Progress on: 1.05kit/1.50kit rate:7.0s/it remaining:52:46 elapsed:2:08:13 postfix:- (1143:tqdm_logging.py:145)


15:29:18.379 [I] Progress on: 1.05kit/1.50kit rate:7.0s/it remaining:52:46 elapsed:2:08:26 postfix:- (1143:tqdm_logging.py:145)
15:29:32.872 [I] Progress on: 1.05kit/1.50kit rate:6.7s/it remaining:50:08 elapsed:2:08:41 postfix:- (1143:tqdm_logging.py:145)
15:29:46.935 [I] Progress on: 1.06kit/1.50kit rate:6.9s/it remaining:50:59 elapsed:2:08:55 postfix:- (1143:tqdm_logging.py:145)
15:30:00.987 [I] Progress on: 1.06kit/1.50kit rate:7.0s/it remaining:51:16 elapsed:2:09:09 postfix:- (1143:tqdm_logging.py:145)
15:30:15.037 [I] Progress on: 1.06kit/1.50kit rate:7.0s/it remaining:51:16 elapsed:2:09:23 postfix:- (1143:tqdm_logging.py:145)
15:30:29.080 [I] Progress on: 1.06kit/1.50kit rate:7.0s/it remaining:51:09 elapsed:2:09:37 postfix:- (1143:tqdm_logging.py:145)
15:30:43.134 [I] Progress on: 1.06kit/1.50kit rate:7.0s/it remaining:50:59 elapsed:2:09:51 postfix:- (1143:tqdm_logging.py:145)
15:30:57.181 [I] Progress on: 1.07kit/1.50kit rate:7.0s/it remaining:50:47 elapsed:2:10:05 postfix:- (1143:tqdm_logging.py:145)
15:31:11.218 [I] Progress on: 1.07kit/1.50kit rate:7.0s/it remaining:50:32 elapsed:2:10:19 postfix:- (1143:tqdm_logging.py:145)
15:31:25.779 [I] Progress on: 1.07kit/1.50kit rate:7.2s/it remaining:51:25 elapsed:2:10:34 postfix:- (1143:tqdm_logging.py:145)
15:31:39.320 [I] Progress on: 1.07kit/1.50kit rate:7.0s/it remaining:49:53 elapsed:2:10:47 postfix:- (1143:tqdm_logging.py:145)
15:31:53.391 [I] Progress on: 1.07kit/1.50kit rate:7.0s/it remaining:49:50 elapsed:2:11:02 postfix:- (1143:tqdm_logging.py:145)
15:32:07.417 [I] Progress on: 1.08kit/1.50kit rate:7.0s/it remaining:49:36 elapsed:2:11:16 postfix:- (1143:tqdm_logging.py:145)
15:32:21.462 [I] Progress on: 1.08kit/1.50kit rate:7.0s/it remaining:49:24 elapsed:2:11:30 postfix:- (1143:tqdm_logging.py:145)
15:32:35.491 [I] Progress on: 1.08kit/1.50kit rate:7.0s/it remaining:49:09 elapsed:2:11:44 postfix:- (1143:tqdm_logging.py:145)
15:32:49.546 [I] Progress on: 1.08kit/1.50kit rate:7.0s/it remaining:48:57 elapsed:2:11:58 postfix:- (1143:tqdm_logging.py:145)
15:33:03.570 [I] Progress on: 1.08kit/1.50kit rate:7.0s/it remaining:48:41 elapsed:2:12:12 postfix:- (1143:tqdm_logging.py:145)
15:33:17.624 [I] Progress on: 1.09kit/1.50kit rate:7.0s/it remaining:48:29 elapsed:2:12:26 postfix:- (1143:tqdm_logging.py:145)
15:33:31.650 [I] Progress on: 1.09kit/1.50kit rate:7.0s/it remaining:48:14 elapsed:2:12:40 postfix:- (1143:tqdm_logging.py:145)
15:33:45.708 [I] Progress on: 1.09kit/1.50kit rate:7.0s/it remaining:48:02 elapsed:2:12:54 postfix:- (1143:tqdm_logging.py:145)
15:33:59.737 [I] Progress on: 1.09kit/1.50kit rate:7.0s/it remaining:47:46 elapsed:2:13:08 postfix:- (1143:tqdm_logging.py:145)
15:34:13.785 [I] Progress on: 1.09kit/1.50kit rate:7.0s/it remaining:47:33 elapsed:2:13:22 postfix:- (1143:tqdm_logging.py:145)
15:34:27.808 [I] Progress on: 1.10kit/1.50kit rate:7.0s/it remaining:47:17 elapsed:2:13:36 postfix:- (1143:tqdm_logging.py:145)
15:34:41.851 [I] Progress on: 1.10kit/1.50kit rate:7.0s/it remaining:47:04 elapsed:2:13:50 postfix:- (1143:tqdm_logging.py:145)
15:34:55.878 [I] Progress on: 1.10kit/1.50kit rate:7.0s/it remaining:46:49 elapsed:2:14:04 postfix:- (1143:tqdm_logging.py:145)


15:35:09.553 [I] Progress on: 1.10kit/1.50kit rate:7.0s/it remaining:46:49 elapsed:2:14:18 postfix:- (1143:tqdm_logging.py:145)
15:35:24.014 [I] Progress on: 1.10kit/1.50kit rate:6.7s/it remaining:44:27 elapsed:2:14:32 postfix:- (1143:tqdm_logging.py:145)
15:35:38.055 [I] Progress on: 1.11kit/1.50kit rate:6.9s/it remaining:45:10 elapsed:2:14:46 postfix:- (1143:tqdm_logging.py:145)
15:35:52.096 [I] Progress on: 1.11kit/1.50kit rate:7.0s/it remaining:45:25 elapsed:2:15:00 postfix:- (1143:tqdm_logging.py:145)
15:36:06.129 [I] Progress on: 1.11kit/1.50kit rate:7.0s/it remaining:45:24 elapsed:2:15:14 postfix:- (1143:tqdm_logging.py:145)
15:36:20.175 [I] Progress on: 1.11kit/1.50kit rate:7.0s/it remaining:45:17 elapsed:2:15:28 postfix:- (1143:tqdm_logging.py:145)
15:36:34.218 [I] Progress on: 1.11kit/1.50kit rate:7.0s/it remaining:45:07 elapsed:2:15:42 postfix:- (1143:tqdm_logging.py:145)
15:36:48.252 [I] Progress on: 1.12kit/1.50kit rate:7.0s/it remaining:44:53 elapsed:2:15:56 postfix:- (1143:tqdm_logging.py:145)
15:37:02.301 [I] Progress on: 1.12kit/1.50kit rate:7.0s/it remaining:44:41 elapsed:2:16:10 postfix:- (1143:tqdm_logging.py:145)
15:37:16.348 [I] Progress on: 1.12kit/1.50kit rate:7.0s/it remaining:44:27 elapsed:2:16:24 postfix:- (1143:tqdm_logging.py:145)
15:37:30.438 [I] Progress on: 1.12kit/1.50kit rate:7.0s/it remaining:44:18 elapsed:2:16:39 postfix:- (1143:tqdm_logging.py:145)
15:37:44.518 [I] Progress on: 1.12kit/1.50kit rate:7.0s/it remaining:44:05 elapsed:2:16:53 postfix:- (1143:tqdm_logging.py:145)
15:37:58.630 [I] Progress on: 1.13kit/1.50kit rate:7.0s/it remaining:43:55 elapsed:2:17:07 postfix:- (1143:tqdm_logging.py:145)
15:38:12.725 [I] Progress on: 1.13kit/1.50kit rate:7.0s/it remaining:43:41 elapsed:2:17:21 postfix:- (1143:tqdm_logging.py:145)
15:38:26.815 [I] Progress on: 1.13kit/1.50kit rate:7.0s/it remaining:43:27 elapsed:2:17:35 postfix:- (1143:tqdm_logging.py:145)
15:38:40.915 [I] Progress on: 1.13kit/1.50kit rate:7.0s/it remaining:43:13 elapsed:2:17:49 postfix:- (1143:tqdm_logging.py:145)
15:38:55.037 [I] Progress on: 1.13kit/1.50kit rate:7.1s/it remaining:43:01 elapsed:2:18:03 postfix:- (1143:tqdm_logging.py:145)
15:39:09.133 [I] Progress on: 1.14kit/1.50kit rate:7.1s/it remaining:42:46 elapsed:2:18:17 postfix:- (1143:tqdm_logging.py:145)
15:39:23.225 [I] Progress on: 1.14kit/1.50kit rate:7.0s/it remaining:42:31 elapsed:2:18:31 postfix:- (1143:tqdm_logging.py:145)
15:39:37.296 [I] Progress on: 1.14kit/1.50kit rate:7.0s/it remaining:42:14 elapsed:2:18:45 postfix:- (1143:tqdm_logging.py:145)
15:39:51.373 [I] Progress on: 1.14kit/1.50kit rate:7.0s/it remaining:42:00 elapsed:2:18:59 postfix:- (1143:tqdm_logging.py:145)
15:40:05.449 [I] Progress on: 1.14kit/1.50kit rate:7.0s/it remaining:41:45 elapsed:2:19:14 postfix:- (1143:tqdm_logging.py:145)
15:40:19.524 [I] Progress on: 1.15kit/1.50kit rate:7.0s/it remaining:41:31 elapsed:2:19:28 postfix:- (1143:tqdm_logging.py:145)
15:40:33.597 [I] Progress on: 1.15kit/1.50kit rate:7.0s/it remaining:41:17 elapsed:2:19:42 postfix:- (1143:tqdm_logging.py:145)
15:40:47.672 [I] Progress on: 1.15kit/1.50kit rate:7.0s/it remaining:41:03 elapsed:2:19:56 postfix:- (1143:tqdm_logging.py:145)


15:41:01.399 [I] Progress on: 1.15kit/1.50kit rate:7.0s/it remaining:41:03 elapsed:2:20:10 postfix:- (1143:tqdm_logging.py:145)
15:41:15.922 [I] Progress on: 1.15kit/1.50kit rate:6.8s/it remaining:38:57 elapsed:2:20:24 postfix:- (1143:tqdm_logging.py:145)
15:41:29.995 [I] Progress on: 1.16kit/1.50kit rate:6.9s/it remaining:39:33 elapsed:2:20:38 postfix:- (1143:tqdm_logging.py:145)
15:41:44.076 [I] Progress on: 1.16kit/1.50kit rate:7.0s/it remaining:39:44 elapsed:2:20:52 postfix:- (1143:tqdm_logging.py:145)
15:41:58.146 [I] Progress on: 1.16kit/1.50kit rate:7.0s/it remaining:39:41 elapsed:2:21:06 postfix:- (1143:tqdm_logging.py:145)
15:42:12.224 [I] Progress on: 1.16kit/1.50kit rate:7.0s/it remaining:39:33 elapsed:2:21:20 postfix:- (1143:tqdm_logging.py:145)
15:42:26.298 [I] Progress on: 1.16kit/1.50kit rate:7.0s/it remaining:39:22 elapsed:2:21:34 postfix:- (1143:tqdm_logging.py:145)
15:42:40.374 [I] Progress on: 1.17kit/1.50kit rate:7.0s/it remaining:39:09 elapsed:2:21:48 postfix:- (1143:tqdm_logging.py:145)
15:42:54.446 [I] Progress on: 1.17kit/1.50kit rate:7.0s/it remaining:38:55 elapsed:2:22:03 postfix:- (1143:tqdm_logging.py:145)
15:43:08.524 [I] Progress on: 1.17kit/1.50kit rate:7.0s/it remaining:38:42 elapsed:2:22:17 postfix:- (1143:tqdm_logging.py:145)
15:43:22.599 [I] Progress on: 1.17kit/1.50kit rate:7.0s/it remaining:38:28 elapsed:2:22:31 postfix:- (1143:tqdm_logging.py:145)
15:43:36.673 [I] Progress on: 1.17kit/1.50kit rate:7.0s/it remaining:38:14 elapsed:2:22:45 postfix:- (1143:tqdm_logging.py:145)
15:43:50.753 [I] Progress on: 1.18kit/1.50kit rate:7.0s/it remaining:38:00 elapsed:2:22:59 postfix:- (1143:tqdm_logging.py:145)
15:44:04.800 [I] Progress on: 1.18kit/1.50kit rate:7.0s/it remaining:37:28 elapsed:2:23:13 postfix:- (1143:tqdm_logging.py:145)
15:44:18.876 [I] Progress on: 1.18kit/1.50kit rate:7.0s/it remaining:37:22 elapsed:2:23:27 postfix:- (1143:tqdm_logging.py:145)
15:44:32.947 [I] Progress on: 1.18kit/1.50kit rate:7.0s/it remaining:37:11 elapsed:2:23:41 postfix:- (1143:tqdm_logging.py:145)
15:44:47.031 [I] Progress on: 1.18kit/1.50kit rate:7.0s/it remaining:37:00 elapsed:2:23:55 postfix:- (1143:tqdm_logging.py:145)
15:45:01.101 [I] Progress on: 1.19kit/1.50kit rate:7.0s/it remaining:36:46 elapsed:2:24:09 postfix:- (1143:tqdm_logging.py:145)
15:45:15.184 [I] Progress on: 1.19kit/1.50kit rate:7.0s/it remaining:36:33 elapsed:2:24:23 postfix:- (1143:tqdm_logging.py:145)
15:45:29.256 [I] Progress on: 1.19kit/1.50kit rate:7.0s/it remaining:36:19 elapsed:2:24:37 postfix:- (1143:tqdm_logging.py:145)
15:45:43.333 [I] Progress on: 1.19kit/1.50kit rate:7.0s/it remaining:36:05 elapsed:2:24:51 postfix:- (1143:tqdm_logging.py:145)
15:45:57.403 [I] Progress on: 1.19kit/1.50kit rate:7.0s/it remaining:35:50 elapsed:2:25:06 postfix:- (1143:tqdm_logging.py:145)
15:46:11.483 [I] Progress on: 1.20kit/1.50kit rate:7.0s/it remaining:35:37 elapsed:2:25:20 postfix:- (1143:tqdm_logging.py:145)
15:46:25.555 [I] Progress on: 1.20kit/1.50kit rate:7.0s/it remaining:35:23 elapsed:2:25:34 postfix:- (1143:tqdm_logging.py:145)
15:46:39.628 [I] Progress on: 1.20kit/1.50kit rate:7.0s/it remaining:35:09 elapsed:2:25:48 postfix:- (1143:tqdm_logging.py:145)


15:46:53.373 [I] Progress on: 1.20kit/1.50kit rate:7.0s/it remaining:35:09 elapsed:2:26:01 postfix:- (1143:tqdm_logging.py:145)
15:47:07.847 [I] Progress on: 1.20kit/1.50kit rate:6.7s/it remaining:33:16 elapsed:2:26:16 postfix:- (1143:tqdm_logging.py:145)
15:47:21.914 [I] Progress on: 1.21kit/1.50kit rate:6.9s/it remaining:33:45 elapsed:2:26:30 postfix:- (1143:tqdm_logging.py:145)
15:47:35.993 [I] Progress on: 1.21kit/1.50kit rate:7.0s/it remaining:33:52 elapsed:2:26:44 postfix:- (1143:tqdm_logging.py:145)
15:47:50.069 [I] Progress on: 1.21kit/1.50kit rate:7.0s/it remaining:33:49 elapsed:2:26:58 postfix:- (1143:tqdm_logging.py:145)
15:48:04.150 [I] Progress on: 1.21kit/1.50kit rate:7.0s/it remaining:33:40 elapsed:2:27:12 postfix:- (1143:tqdm_logging.py:145)
15:48:18.219 [I] Progress on: 1.21kit/1.50kit rate:7.0s/it remaining:33:28 elapsed:2:27:26 postfix:- (1143:tqdm_logging.py:145)
15:48:32.294 [I] Progress on: 1.22kit/1.50kit rate:7.0s/it remaining:33:15 elapsed:2:27:40 postfix:- (1143:tqdm_logging.py:145)
15:48:46.368 [I] Progress on: 1.22kit/1.50kit rate:7.0s/it remaining:33:01 elapsed:2:27:54 postfix:- (1143:tqdm_logging.py:145)
15:49:00.451 [I] Progress on: 1.22kit/1.50kit rate:7.0s/it remaining:32:48 elapsed:2:28:09 postfix:- (1143:tqdm_logging.py:145)
15:49:14.516 [I] Progress on: 1.22kit/1.50kit rate:7.0s/it remaining:32:33 elapsed:2:28:23 postfix:- (1143:tqdm_logging.py:145)
15:49:28.599 [I] Progress on: 1.22kit/1.50kit rate:7.0s/it remaining:32:20 elapsed:2:28:37 postfix:- (1143:tqdm_logging.py:145)
15:49:42.670 [I] Progress on: 1.23kit/1.50kit rate:7.0s/it remaining:32:06 elapsed:2:28:51 postfix:- (1143:tqdm_logging.py:145)
15:49:56.762 [I] Progress on: 1.23kit/1.50kit rate:7.0s/it remaining:31:53 elapsed:2:29:05 postfix:- (1143:tqdm_logging.py:145)
15:50:10.824 [I] Progress on: 1.23kit/1.50kit rate:7.0s/it remaining:31:37 elapsed:2:29:19 postfix:- (1143:tqdm_logging.py:145)
15:50:24.910 [I] Progress on: 1.23kit/1.50kit rate:7.0s/it remaining:31:24 elapsed:2:29:33 postfix:- (1143:tqdm_logging.py:145)
15:50:38.986 [I] Progress on: 1.23kit/1.50kit rate:7.0s/it remaining:31:10 elapsed:2:29:47 postfix:- (1143:tqdm_logging.py:145)
15:50:53.070 [I] Progress on: 1.24kit/1.50kit rate:7.0s/it remaining:30:56 elapsed:2:30:01 postfix:- (1143:tqdm_logging.py:145)
15:51:07.156 [I] Progress on: 1.24kit/1.50kit rate:7.0s/it remaining:30:42 elapsed:2:30:15 postfix:- (1143:tqdm_logging.py:145)
15:51:21.258 [I] Progress on: 1.24kit/1.50kit rate:7.0s/it remaining:30:29 elapsed:2:30:29 postfix:- (1143:tqdm_logging.py:145)
15:51:35.350 [I] Progress on: 1.24kit/1.50kit rate:7.0s/it remaining:30:15 elapsed:2:30:43 postfix:- (1143:tqdm_logging.py:145)
15:51:49.459 [I] Progress on: 1.24kit/1.50kit rate:7.0s/it remaining:30:03 elapsed:2:30:58 postfix:- (1143:tqdm_logging.py:145)
15:52:03.545 [I] Progress on: 1.25kit/1.50kit rate:7.0s/it remaining:29:47 elapsed:2:31:12 postfix:- (1143:tqdm_logging.py:145)
15:52:17.646 [I] Progress on: 1.25kit/1.50kit rate:7.0s/it remaining:29:34 elapsed:2:31:26 postfix:- (1143:tqdm_logging.py:145)
15:52:31.720 [I] Progress on: 1.25kit/1.50kit rate:7.0s/it remaining:29:18 elapsed:2:31:40 postfix:- (1143:tqdm_logging.py:145)


15:52:45.458 [I] Progress on: 1.25kit/1.50kit rate:7.0s/it remaining:29:18 elapsed:2:31:54 postfix:- (1143:tqdm_logging.py:145)
15:52:59.980 [I] Progress on: 1.25kit/1.50kit rate:6.7s/it remaining:27:40 elapsed:2:32:08 postfix:- (1143:tqdm_logging.py:145)
15:53:14.064 [I] Progress on: 1.26kit/1.50kit rate:6.9s/it remaining:28:02 elapsed:2:32:22 postfix:- (1143:tqdm_logging.py:145)
15:53:28.135 [I] Progress on: 1.26kit/1.50kit rate:7.0s/it remaining:28:04 elapsed:2:32:36 postfix:- (1143:tqdm_logging.py:145)
15:53:42.227 [I] Progress on: 1.26kit/1.50kit rate:7.0s/it remaining:28:00 elapsed:2:32:50 postfix:- (1143:tqdm_logging.py:145)
15:53:56.300 [I] Progress on: 1.26kit/1.50kit rate:7.0s/it remaining:27:50 elapsed:2:33:04 postfix:- (1143:tqdm_logging.py:145)
15:54:10.378 [I] Progress on: 1.26kit/1.50kit rate:7.0s/it remaining:27:38 elapsed:2:33:18 postfix:- (1143:tqdm_logging.py:145)
15:54:24.458 [I] Progress on: 1.27kit/1.50kit rate:7.0s/it remaining:27:26 elapsed:2:33:33 postfix:- (1143:tqdm_logging.py:145)
15:54:38.536 [I] Progress on: 1.27kit/1.50kit rate:7.0s/it remaining:27:12 elapsed:2:33:47 postfix:- (1143:tqdm_logging.py:145)
15:54:52.608 [I] Progress on: 1.27kit/1.50kit rate:7.0s/it remaining:26:58 elapsed:2:34:01 postfix:- (1143:tqdm_logging.py:145)
15:55:06.678 [I] Progress on: 1.27kit/1.50kit rate:7.0s/it remaining:26:44 elapsed:2:34:15 postfix:- (1143:tqdm_logging.py:145)
15:55:20.752 [I] Progress on: 1.27kit/1.50kit rate:7.0s/it remaining:26:30 elapsed:2:34:29 postfix:- (1143:tqdm_logging.py:145)
15:55:34.829 [I] Progress on: 1.28kit/1.50kit rate:7.0s/it remaining:26:16 elapsed:2:34:43 postfix:- (1143:tqdm_logging.py:145)
15:55:48.906 [I] Progress on: 1.28kit/1.50kit rate:7.0s/it remaining:26:02 elapsed:2:34:57 postfix:- (1143:tqdm_logging.py:145)
15:56:02.980 [I] Progress on: 1.28kit/1.50kit rate:7.0s/it remaining:25:48 elapsed:2:35:11 postfix:- (1143:tqdm_logging.py:145)
15:56:17.057 [I] Progress on: 1.28kit/1.50kit rate:7.0s/it remaining:25:34 elapsed:2:35:25 postfix:- (1143:tqdm_logging.py:145)
15:56:31.611 [I] Progress on: 1.28kit/1.50kit rate:7.2s/it remaining:25:51 elapsed:2:35:40 postfix:- (1143:tqdm_logging.py:145)
15:56:45.212 [I] Progress on: 1.29kit/1.50kit rate:7.0s/it remaining:25:00 elapsed:2:35:53 postfix:- (1143:tqdm_logging.py:145)
15:56:59.278 [I] Progress on: 1.29kit/1.50kit rate:7.0s/it remaining:24:48 elapsed:2:36:07 postfix:- (1143:tqdm_logging.py:145)
15:57:13.364 [I] Progress on: 1.29kit/1.50kit rate:7.0s/it remaining:24:36 elapsed:2:36:21 postfix:- (1143:tqdm_logging.py:145)
15:57:27.432 [I] Progress on: 1.29kit/1.50kit rate:7.0s/it remaining:24:22 elapsed:2:36:36 postfix:- (1143:tqdm_logging.py:145)
15:57:41.514 [I] Progress on: 1.29kit/1.50kit rate:7.0s/it remaining:24:09 elapsed:2:36:50 postfix:- (1143:tqdm_logging.py:145)
15:57:55.586 [I] Progress on: 1.30kit/1.50kit rate:7.0s/it remaining:23:55 elapsed:2:37:04 postfix:- (1143:tqdm_logging.py:145)
15:58:09.658 [I] Progress on: 1.30kit/1.50kit rate:7.0s/it remaining:23:41 elapsed:2:37:18 postfix:- (1143:tqdm_logging.py:145)
15:58:23.727 [I] Progress on: 1.30kit/1.50kit rate:7.0s/it remaining:23:27 elapsed:2:37:32 postfix:- (1143:tqdm_logging.py:145)


15:58:37.453 [I] Progress on: 1.30kit/1.50kit rate:7.0s/it remaining:23:27 elapsed:2:37:46 postfix:- (1143:tqdm_logging.py:145)
15:58:51.913 [I] Progress on: 1.30kit/1.50kit rate:6.7s/it remaining:22:01 elapsed:2:38:00 postfix:- (1143:tqdm_logging.py:145)
15:59:05.948 [I] Progress on: 1.31kit/1.50kit rate:6.9s/it remaining:22:14 elapsed:2:38:14 postfix:- (1143:tqdm_logging.py:145)
15:59:19.987 [I] Progress on: 1.31kit/1.50kit rate:7.0s/it remaining:22:14 elapsed:2:38:28 postfix:- (1143:tqdm_logging.py:145)
15:59:34.020 [I] Progress on: 1.31kit/1.50kit rate:7.0s/it remaining:22:07 elapsed:2:38:42 postfix:- (1143:tqdm_logging.py:145)
15:59:48.065 [I] Progress on: 1.31kit/1.50kit rate:7.0s/it remaining:21:56 elapsed:2:38:56 postfix:- (1143:tqdm_logging.py:145)
16:00:02.133 [I] Progress on: 1.31kit/1.50kit rate:7.0s/it remaining:21:45 elapsed:2:39:10 postfix:- (1143:tqdm_logging.py:145)
16:00:16.191 [I] Progress on: 1.32kit/1.50kit rate:7.0s/it remaining:21:32 elapsed:2:39:24 postfix:- (1143:tqdm_logging.py:145)
16:00:30.227 [I] Progress on: 1.32kit/1.50kit rate:7.0s/it remaining:21:17 elapsed:2:39:38 postfix:- (1143:tqdm_logging.py:145)
16:00:44.257 [I] Progress on: 1.32kit/1.50kit rate:7.0s/it remaining:21:03 elapsed:2:39:52 postfix:- (1143:tqdm_logging.py:145)
16:00:58.293 [I] Progress on: 1.32kit/1.50kit rate:7.0s/it remaining:20:49 elapsed:2:40:06 postfix:- (1143:tqdm_logging.py:145)
16:01:12.338 [I] Progress on: 1.32kit/1.50kit rate:7.0s/it remaining:20:35 elapsed:2:40:20 postfix:- (1143:tqdm_logging.py:145)
16:01:26.391 [I] Progress on: 1.33kit/1.50kit rate:7.0s/it remaining:20:21 elapsed:2:40:35 postfix:- (1143:tqdm_logging.py:145)
16:01:40.445 [I] Progress on: 1.33kit/1.50kit rate:7.0s/it remaining:20:08 elapsed:2:40:49 postfix:- (1143:tqdm_logging.py:145)
16:01:54.503 [I] Progress on: 1.33kit/1.50kit rate:7.0s/it remaining:19:54 elapsed:2:41:03 postfix:- (1143:tqdm_logging.py:145)
16:02:08.532 [I] Progress on: 1.33kit/1.50kit rate:7.0s/it remaining:19:39 elapsed:2:41:17 postfix:- (1143:tqdm_logging.py:145)
16:02:22.566 [I] Progress on: 1.33kit/1.50kit rate:7.0s/it remaining:19:25 elapsed:2:41:31 postfix:- (1143:tqdm_logging.py:145)
16:02:36.614 [I] Progress on: 1.34kit/1.50kit rate:7.0s/it remaining:19:11 elapsed:2:41:45 postfix:- (1143:tqdm_logging.py:145)
16:02:50.661 [I] Progress on: 1.34kit/1.50kit rate:7.0s/it remaining:18:57 elapsed:2:41:59 postfix:- (1143:tqdm_logging.py:145)
16:03:04.719 [I] Progress on: 1.34kit/1.50kit rate:7.0s/it remaining:18:44 elapsed:2:42:13 postfix:- (1143:tqdm_logging.py:145)
16:03:18.805 [I] Progress on: 1.34kit/1.50kit rate:7.0s/it remaining:18:31 elapsed:2:42:27 postfix:- (1143:tqdm_logging.py:145)
16:03:32.883 [I] Progress on: 1.34kit/1.50kit rate:7.0s/it remaining:18:17 elapsed:2:42:41 postfix:- (1143:tqdm_logging.py:145)
16:03:46.979 [I] Progress on: 1.35kit/1.50kit rate:7.0s/it remaining:18:04 elapsed:2:42:55 postfix:- (1143:tqdm_logging.py:145)
16:04:01.077 [I] Progress on: 1.35kit/1.50kit rate:7.0s/it remaining:17:50 elapsed:2:43:09 postfix:- (1143:tqdm_logging.py:145)
16:04:15.174 [I] Progress on: 1.35kit/1.50kit rate:7.0s/it remaining:17:37 elapsed:2:43:23 postfix:- (1143:tqdm_logging.py:145)


16:04:28.909 [I] Progress on: 1.35kit/1.50kit rate:7.0s/it remaining:17:37 elapsed:2:43:37 postfix:- (1143:tqdm_logging.py:145)
16:04:43.422 [I] Progress on: 1.35kit/1.50kit rate:6.8s/it remaining:16:26 elapsed:2:43:52 postfix:- (1143:tqdm_logging.py:145)
16:04:57.519 [I] Progress on: 1.36kit/1.50kit rate:6.9s/it remaining:16:34 elapsed:2:44:06 postfix:- (1143:tqdm_logging.py:145)
16:05:11.613 [I] Progress on: 1.36kit/1.50kit rate:7.0s/it remaining:16:31 elapsed:2:44:20 postfix:- (1143:tqdm_logging.py:145)
16:05:25.707 [I] Progress on: 1.36kit/1.50kit rate:7.0s/it remaining:16:21 elapsed:2:44:34 postfix:- (1143:tqdm_logging.py:145)
16:05:39.792 [I] Progress on: 1.36kit/1.50kit rate:7.0s/it remaining:16:09 elapsed:2:44:48 postfix:- (1143:tqdm_logging.py:145)
16:05:53.878 [I] Progress on: 1.36kit/1.50kit rate:7.0s/it remaining:15:56 elapsed:2:45:02 postfix:- (1143:tqdm_logging.py:145)
16:06:07.956 [I] Progress on: 1.37kit/1.50kit rate:7.0s/it remaining:15:42 elapsed:2:45:16 postfix:- (1143:tqdm_logging.py:145)
16:06:22.033 [I] Progress on: 1.37kit/1.50kit rate:7.0s/it remaining:15:28 elapsed:2:45:30 postfix:- (1143:tqdm_logging.py:145)
16:06:36.102 [I] Progress on: 1.37kit/1.50kit rate:7.0s/it remaining:15:14 elapsed:2:45:44 postfix:- (1143:tqdm_logging.py:145)
16:06:50.174 [I] Progress on: 1.37kit/1.50kit rate:7.0s/it remaining:15:00 elapsed:2:45:58 postfix:- (1143:tqdm_logging.py:145)
16:07:04.255 [I] Progress on: 1.37kit/1.50kit rate:7.0s/it remaining:14:46 elapsed:2:46:12 postfix:- (1143:tqdm_logging.py:145)
16:07:18.328 [I] Progress on: 1.38kit/1.50kit rate:7.0s/it remaining:14:32 elapsed:2:46:26 postfix:- (1143:tqdm_logging.py:145)
16:07:32.409 [I] Progress on: 1.38kit/1.50kit rate:7.0s/it remaining:14:18 elapsed:2:46:41 postfix:- (1143:tqdm_logging.py:145)
16:07:46.476 [I] Progress on: 1.38kit/1.50kit rate:7.0s/it remaining:14:04 elapsed:2:46:55 postfix:- (1143:tqdm_logging.py:145)
16:08:00.552 [I] Progress on: 1.38kit/1.50kit rate:7.0s/it remaining:13:50 elapsed:2:47:09 postfix:- (1143:tqdm_logging.py:145)
16:08:14.626 [I] Progress on: 1.38kit/1.50kit rate:7.0s/it remaining:13:36 elapsed:2:47:23 postfix:- (1143:tqdm_logging.py:145)
16:08:28.701 [I] Progress on: 1.39kit/1.50kit rate:7.0s/it remaining:13:22 elapsed:2:47:37 postfix:- (1143:tqdm_logging.py:145)
16:08:42.782 [I] Progress on: 1.39kit/1.50kit rate:7.0s/it remaining:13:08 elapsed:2:47:51 postfix:- (1143:tqdm_logging.py:145)
16:08:56.867 [I] Progress on: 1.39kit/1.50kit rate:7.0s/it remaining:12:54 elapsed:2:48:05 postfix:- (1143:tqdm_logging.py:145)
16:09:10.942 [I] Progress on: 1.39kit/1.50kit rate:7.0s/it remaining:12:35 elapsed:2:48:19 postfix:- (1143:tqdm_logging.py:145)
16:09:25.016 [I] Progress on: 1.39kit/1.50kit rate:7.0s/it remaining:12:23 elapsed:2:48:33 postfix:- (1143:tqdm_logging.py:145)
16:09:39.082 [I] Progress on: 1.40kit/1.50kit rate:7.0s/it remaining:12:10 elapsed:2:48:47 postfix:- (1143:tqdm_logging.py:145)
16:09:53.160 [I] Progress on: 1.40kit/1.50kit rate:7.0s/it remaining:11:57 elapsed:2:49:01 postfix:- (1143:tqdm_logging.py:145)
16:10:07.234 [I] Progress on: 1.40kit/1.50kit rate:7.0s/it remaining:11:43 elapsed:2:49:15 postfix:- (1143:tqdm_logging.py:145)


16:10:20.956 [I] Progress on: 1.40kit/1.50kit rate:7.0s/it remaining:11:43 elapsed:2:49:29 postfix:- (1143:tqdm_logging.py:145)
16:10:35.465 [I] Progress on: 1.40kit/1.50kit rate:6.7s/it remaining:10:47 elapsed:2:49:44 postfix:- (1143:tqdm_logging.py:145)
16:10:49.509 [I] Progress on: 1.41kit/1.50kit rate:6.9s/it remaining:10:47 elapsed:2:49:58 postfix:- (1143:tqdm_logging.py:145)
16:11:03.550 [I] Progress on: 1.41kit/1.50kit rate:7.0s/it remaining:10:39 elapsed:2:50:12 postfix:- (1143:tqdm_logging.py:145)
16:11:17.583 [I] Progress on: 1.41kit/1.50kit rate:7.0s/it remaining:10:28 elapsed:2:50:26 postfix:- (1143:tqdm_logging.py:145)
16:11:31.632 [I] Progress on: 1.41kit/1.50kit rate:7.0s/it remaining:10:16 elapsed:2:50:40 postfix:- (1143:tqdm_logging.py:145)
16:11:45.679 [I] Progress on: 1.41kit/1.50kit rate:7.0s/it remaining:10:03 elapsed:2:50:54 postfix:- (1143:tqdm_logging.py:145)
16:11:59.727 [I] Progress on: 1.42kit/1.50kit rate:7.0s/it remaining:09:49 elapsed:2:51:08 postfix:- (1143:tqdm_logging.py:145)
16:12:13.762 [I] Progress on: 1.42kit/1.50kit rate:7.0s/it remaining:09:35 elapsed:2:51:22 postfix:- (1143:tqdm_logging.py:145)
16:12:27.799 [I] Progress on: 1.42kit/1.50kit rate:7.0s/it remaining:09:21 elapsed:2:51:36 postfix:- (1143:tqdm_logging.py:145)
16:12:41.837 [I] Progress on: 1.42kit/1.50kit rate:7.0s/it remaining:09:07 elapsed:2:51:50 postfix:- (1143:tqdm_logging.py:145)
16:12:55.886 [I] Progress on: 1.42kit/1.50kit rate:7.0s/it remaining:08:53 elapsed:2:52:04 postfix:- (1143:tqdm_logging.py:145)
16:13:09.936 [I] Progress on: 1.43kit/1.50kit rate:7.0s/it remaining:08:39 elapsed:2:52:18 postfix:- (1143:tqdm_logging.py:145)
16:13:23.990 [I] Progress on: 1.43kit/1.50kit rate:7.0s/it remaining:08:25 elapsed:2:52:32 postfix:- (1143:tqdm_logging.py:145)
16:13:38.040 [I] Progress on: 1.43kit/1.50kit rate:7.0s/it remaining:08:11 elapsed:2:52:46 postfix:- (1143:tqdm_logging.py:145)
16:13:52.083 [I] Progress on: 1.43kit/1.50kit rate:7.0s/it remaining:07:57 elapsed:2:53:00 postfix:- (1143:tqdm_logging.py:145)
16:14:06.134 [I] Progress on: 1.43kit/1.50kit rate:7.0s/it remaining:07:43 elapsed:2:53:14 postfix:- (1143:tqdm_logging.py:145)
16:14:20.174 [I] Progress on: 1.44kit/1.50kit rate:7.0s/it remaining:07:29 elapsed:2:53:28 postfix:- (1143:tqdm_logging.py:145)
16:14:34.224 [I] Progress on: 1.44kit/1.50kit rate:7.0s/it remaining:07:15 elapsed:2:53:42 postfix:- (1143:tqdm_logging.py:145)
16:14:48.256 [I] Progress on: 1.44kit/1.50kit rate:7.0s/it remaining:07:01 elapsed:2:53:56 postfix:- (1143:tqdm_logging.py:145)
16:15:02.296 [I] Progress on: 1.44kit/1.50kit rate:7.0s/it remaining:06:47 elapsed:2:54:10 postfix:- (1143:tqdm_logging.py:145)
16:15:16.349 [I] Progress on: 1.44kit/1.50kit rate:7.0s/it remaining:06:33 elapsed:2:54:24 postfix:- (1143:tqdm_logging.py:145)
16:15:30.386 [I] Progress on: 1.45kit/1.50kit rate:7.0s/it remaining:06:19 elapsed:2:54:39 postfix:- (1143:tqdm_logging.py:145)
16:15:44.423 [I] Progress on: 1.45kit/1.50kit rate:7.0s/it remaining:06:04 elapsed:2:54:53 postfix:- (1143:tqdm_logging.py:145)
16:15:58.463 [I] Progress on: 1.45kit/1.50kit rate:7.0s/it remaining:05:50 elapsed:2:55:07 postfix:- (1143:tqdm_logging.py:145)


16:16:12.172 [I] Progress on: 1.45kit/1.50kit rate:7.0s/it remaining:05:50 elapsed:2:55:20 postfix:- (1143:tqdm_logging.py:145)
16:16:26.684 [I] Progress on: 1.45kit/1.50kit rate:6.7s/it remaining:05:10 elapsed:2:55:35 postfix:- (1143:tqdm_logging.py:145)
16:16:40.758 [I] Progress on: 1.46kit/1.50kit rate:6.9s/it remaining:05:03 elapsed:2:55:49 postfix:- (1143:tqdm_logging.py:145)
16:16:54.857 [I] Progress on: 1.46kit/1.50kit rate:7.0s/it remaining:04:52 elapsed:2:56:03 postfix:- (1143:tqdm_logging.py:145)
16:17:08.936 [I] Progress on: 1.46kit/1.50kit rate:7.0s/it remaining:04:40 elapsed:2:56:17 postfix:- (1143:tqdm_logging.py:145)
16:17:23.026 [I] Progress on: 1.46kit/1.50kit rate:7.0s/it remaining:04:26 elapsed:2:56:31 postfix:- (1143:tqdm_logging.py:145)
16:17:37.109 [I] Progress on: 1.46kit/1.50kit rate:7.0s/it remaining:04:13 elapsed:2:56:45 postfix:- (1143:tqdm_logging.py:145)
16:17:51.196 [I] Progress on: 1.47kit/1.50kit rate:7.0s/it remaining:03:59 elapsed:2:56:59 postfix:- (1143:tqdm_logging.py:145)
16:18:05.280 [I] Progress on: 1.47kit/1.50kit rate:7.0s/it remaining:03:45 elapsed:2:57:13 postfix:- (1143:tqdm_logging.py:145)
16:18:19.351 [I] Progress on: 1.47kit/1.50kit rate:7.0s/it remaining:03:31 elapsed:2:57:27 postfix:- (1143:tqdm_logging.py:145)
16:18:33.440 [I] Progress on: 1.47kit/1.50kit rate:7.0s/it remaining:03:17 elapsed:2:57:42 postfix:- (1143:tqdm_logging.py:145)
16:18:47.518 [I] Progress on: 1.47kit/1.50kit rate:7.0s/it remaining:03:03 elapsed:2:57:56 postfix:- (1143:tqdm_logging.py:145)
16:19:01.589 [I] Progress on: 1.48kit/1.50kit rate:7.0s/it remaining:02:48 elapsed:2:58:10 postfix:- (1143:tqdm_logging.py:145)
16:19:15.663 [I] Progress on: 1.48kit/1.50kit rate:7.0s/it remaining:02:34 elapsed:2:58:24 postfix:- (1143:tqdm_logging.py:145)
16:19:29.735 [I] Progress on: 1.48kit/1.50kit rate:7.0s/it remaining:02:20 elapsed:2:58:38 postfix:- (1143:tqdm_logging.py:145)
16:19:43.857 [I] Progress on: 1.48kit/1.50kit rate:7.0s/it remaining:02:06 elapsed:2:58:52 postfix:- (1143:tqdm_logging.py:145)
16:19:57.897 [I] Progress on: 1.48kit/1.50kit rate:7.0s/it remaining:01:52 elapsed:2:59:06 postfix:- (1143:tqdm_logging.py:145)
16:20:11.973 [I] Progress on: 1.49kit/1.50kit rate:7.0s/it remaining:01:38 elapsed:2:59:20 postfix:- (1143:tqdm_logging.py:145)
16:20:26.050 [I] Progress on: 1.49kit/1.50kit rate:7.0s/it remaining:01:24 elapsed:2:59:34 postfix:- (1143:tqdm_logging.py:145)
16:20:40.123 [I] Progress on: 1.49kit/1.50kit rate:7.0s/it remaining:01:10 elapsed:2:59:48 postfix:- (1143:tqdm_logging.py:145)
16:20:54.190 [I] Progress on: 1.49kit/1.50kit rate:7.0s/it remaining:00:56 elapsed:3:00:02 postfix:- (1143:tqdm_logging.py:145)
16:21:08.263 [I] Progress on: 1.49kit/1.50kit rate:7.0s/it remaining:00:42 elapsed:3:00:16 postfix:- (1143:tqdm_logging.py:145)
16:21:22.352 [I] Progress on: 1.50kit/1.50kit rate:7.0s/it remaining:00:28 elapsed:3:00:30 postfix:- (1143:tqdm_logging.py:145)
16:21:36.975 [I] Progress on: 1.50kit/1.50kit rate:7.2s/it remaining:00:14 elapsed:3:00:45 postfix:- (1143:tqdm_logging.py:145)
16:21:50.491 [I] [process=0][thread=MainThread][wait_until_finished] No Save Finalize thread to wait for. Returning. (1143:checkpoint_manager.py:1987)
16:21:50.492 [I] [process=0] Saving checkpoint at step 1499                                       (1143:checkpoint_manager.py:1408)
16:21:50.493 [I] [process=0] Started async saving checkpoint to /workspace/openpi/checkpoints/pi05_mecharm_trim_corr_lora/trim_corr_v3/1499. (1143:async_checkpointer.py:439)
16:21:50.506 [I] Creating tmp directory /workspace/openpi/checkpoints/pi05_mecharm_trim_corr_lora/trim_corr_v3/1499.orbax-checkpoint-tmp-4 (1143:atomicity.py:144)
16:21:50.512 [I] Creating tmp directory /workspace/openpi/checkpoints/pi05_mecharm_trim_corr_lora/trim_corr_v3/1499.orbax-checkpoint-tmp-4/assets.orbax-checkpoint-tmp-5 (1143:atomicity.py:144)
16:21:50.513 [I] Creating tmp directory /workspace/openpi/checkpoints/pi05_mecharm_trim_corr_lora/trim_corr_v3/1499.orbax-checkpoint-tmp-4/params.orbax-checkpoint-tmp-6 (1143:atomicity.py:144)
16:21:50.513 [I] Creating tmp directory /workspace/openpi/checkpoints/pi05_mecharm_trim_corr_lora/trim_corr_v3/1499.orbax-checkpoint-tmp-4/train_state.orbax-checkpoint-tmp-7 (1143:atomicity.py:144)
16:21:50.527 [I] Transferring arrays to host memory with options: use_replica_parallel=True, enable_pinned_host_transfer=False (1143:replica_slices.py:341)
16:22:01.053 [I] Transferring arrays to host memory with options: use_replica_parallel=True, enable_pinned_host_transfer=False (1143:replica_slices.py:341)
16:22:01.056 [I] Array name: 'params.PaliGemma.llm.layers.mlp.gating_einsum.value', global shape: (18, 2, 2048, 16384), write shape: (18, 2, 2048, 16384), chosen chunk shape: (18, 2, 2048, 8192) (1143:tensorstore_utils.py:408)
16:22:01.122 [I] [process=0][thread=array_type_handler] Wrote 71 array_metadata.ArrayMetadata to /workspace/openpi/checkpoints/pi05_mecharm_trim_corr_lora/trim_corr_v3/1499.orbax-checkpoint-tmp-4/params.orbax-checkpoint-tmp-6/array_metadatas/process_0 (1143:array_metadata_store.py:198)
16:22:03.057 [I] [process=0] /jax/checkpoint/write/blocking_bytes_per_sec: 587.8 MiB/s (total bytes: 7.2 GiB) (time elapsed: 12 seconds) (per-host) (1143:base_pytree_checkpoint_handler.py:114)
16:22:03.060 [I] [process=0] /jax/checkpoint/write/blocking_bytes_per_sec: 283.7 MiB/s (total bytes: 3.5 GiB) (time elapsed: 12 seconds) (per-host) (1143:base_pytree_checkpoint_handler.py:114)
16:22:03.065 [I] [process=0][thread=async_save] Background save thread started.                   (1143:async_checkpointer.py:77)
16:22:03.066 [I] Finished blocking save in 12.57 seconds. Continuing to save asynchronously to /workspace/openpi/checkpoints/pi05_mecharm_trim_corr_lora/trim_corr_v3/1499. (1143:async_checkpointer.py:548)
16:22:03.066 [I] Preserving Checkpoint[step=1000 | time=2026-10-31 15:23:26.704081+00:00]: (Reason: on keep_period=1000). (1143:checkpoint_manager.py:1949)
16:22:03.066 [I] [process=0][thread=MainThread][step=1499] Starting CheckpointManager Save Finalize thread=save_finalize (1143:checkpoint_manager.py:1453)
16:22:03.067 [I] [process=0][thread=save_finalize] Waiting for background save thread=async_save. (1143:async_checkpointer.py:258)
16:22:03.067 [I] {'step': 1499, 'event_type': 'save', 'directory': '/workspace/openpi/checkpoints/pi05_mecharm_trim_corr_lora/trim_corr_v3', 'reached_preemption': False, 'preemption_received_at': None, 'synchronous': False, 'wait_for_prev_start_time': 1793463710.4910753, 'wait_for_prev_duration_secs': 0.00032019615173339844, 'checkpointer_blocking_start_time': 1793463710.492204, 'checkpointer_blocking_duration_secs': 12.574098110198975, 'get_old_steps_start_time': 1793463723.066328, 'get_old_steps_duration_secs': 0.0001480579376220703, 'checkpoint_manager_blocking_start_time': 1793463710.4905412, 'checkpoint_manager_blocking_duration_secs': 12.577093124389648} (1143:standard_logger.py:34)
16:22:03.068 [I] Progress on: 1.50kit/1.50kit rate:10.8s/it remaining:00:00 elapsed:3:01:11 postfix:- (1143:tqdm_logging.py:145)
16:22:03.068 [I] Progress on: 1.50kit/1.50kit rate:10.8s/it remaining:00:00 elapsed:3:01:11 postfix:- (1143:tqdm_logging.py:145)
16:22:03.068 [I] Waiting for checkpoint manager to finish                                         (1143:train.py:275)
16:22:03.069 [I] [process=0][thread=MainThread][step=1499][wait_until_finished] Waiting for Save Finalize thread (save_finalize) to complete. (1143:checkpoint_manager.py:1998)
16:22:03.122 [I] [process=0][thread=array_type_handler] Wrote 105 array_metadata.ArrayMetadata to /workspace/openpi/checkpoints/pi05_mecharm_trim_corr_lora/trim_corr_v3/1499.orbax-checkpoint-tmp-4/train_state.orbax-checkpoint-tmp-7/array_metadatas/process_0 (1143:array_metadata_store.py:198)
16:22:06.607 [I] [process=0] /jax/checkpoint/write/bytes_per_sec: 458.2 MiB/s (total bytes: 7.2 GiB) (time elapsed: 16 seconds) (per-host) (1143:base_pytree_checkpoint_handler.py:114)
16:22:06.607 [I] [process=0] /jax/checkpoint/write/bytes_per_sec: 221.3 MiB/s (total bytes: 3.5 GiB) (time elapsed: 16 seconds) (per-host) (1143:base_pytree_checkpoint_handler.py:114)
16:22:06.607 [I] [process=0][thread=async_save] 5 Handler Commit operations completed.            (1143:async_checkpointer.py:87)
16:22:06.608 [I] Renaming /workspace/openpi/checkpoints/pi05_mecharm_trim_corr_lora/trim_corr_v3/1499.orbax-checkpoint-tmp-4/assets.orbax-checkpoint-tmp-5 to /workspace/openpi/checkpoints/pi05_mecharm_trim_corr_lora/trim_corr_v3/1499.orbax-checkpoint-tmp-4/assets (1143:atomicity.py:289)
16:22:06.611 [I] [process=0][thread=async_save] Skipped cross-host ArrayMetadata validation because only one process is found: process_index=0. (1143:array_metadata_store.py:362)
16:22:06.615 [I] Renaming /workspace/openpi/checkpoints/pi05_mecharm_trim_corr_lora/trim_corr_v3/1499.orbax-checkpoint-tmp-4/train_state.orbax-checkpoint-tmp-7 to /workspace/openpi/checkpoints/pi05_mecharm_trim_corr_lora/trim_corr_v3/1499.orbax-checkpoint-tmp-4/train_state (1143:atomicity.py:289)
16:22:06.618 [I] [process=0][thread=async_save] Skipped cross-host ArrayMetadata validation because only one process is found: process_index=0. (1143:array_metadata_store.py:362)
16:22:06.622 [I] Renaming /workspace/openpi/checkpoints/pi05_mecharm_trim_corr_lora/trim_corr_v3/1499.orbax-checkpoint-tmp-4/params.orbax-checkpoint-tmp-6 to /workspace/openpi/checkpoints/pi05_mecharm_trim_corr_lora/trim_corr_v3/1499.orbax-checkpoint-tmp-4/params (1143:atomicity.py:289)
16:22:06.622 [I] Renaming /workspace/openpi/checkpoints/pi05_mecharm_trim_corr_lora/trim_corr_v3/1499.orbax-checkpoint-tmp-4 to /workspace/openpi/checkpoints/pi05_mecharm_trim_corr_lora/trim_corr_v3/1499 (1143:atomicity.py:289)
16:22:06.623 [I] [process=0][thread=async_save] Finished saving checkpoint (finalized tmp dir) to `/workspace/openpi/checkpoints/pi05_mecharm_trim_corr_lora/trim_corr_v3/1499`. (1143:atomicity.py:573)
16:22:06.623 [I] Finished asynchronous save (blocking + background) in 16.13 seconds to /workspace/openpi/checkpoints/pi05_mecharm_trim_corr_lora/trim_corr_v3/1499 (1143:async_checkpointer.py:412)
16:22:06.623 [I] [process=0][thread=async_save] Background save thread done.                      (1143:async_checkpointer.py:143)
16:22:06.623 [I] [process=0][thread=save_finalize] Done with waiting for background save thread=async_save. (1143:async_checkpointer.py:266)
16:22:06.624 [I] [process=0][thread=save_finalize] No errors found in background save thread=async_save. (1143:async_checkpointer.py:276)
16:22:06.624 [I] [process=0][thread=save_finalize][step=1499] CheckpointManager Save Finalize is syncing with other hosts... (1143:checkpoint_manager.py:2095)
16:22:06.624 [I] [process=0][thread=save_finalize][step=1499] CheckpointManager Save Finalize is done on all hosts. (1143:checkpoint_manager.py:2110)
16:22:06.624 [I] [process=0][thread=MainThread][step=1499][wait_until_finished] Done waiting for Save Finalize thread (save_finalize) running at step=1499. (1143:checkpoint_manager.py:2010)
Step 0: grad_norm=0.0573, loss=0.0075, param_norm=1804.1436
Step 50: grad_norm=0.0902, loss=0.0125, param_norm=1804.1436
Step 100: grad_norm=0.0542, loss=0.0077, param_norm=1804.1445
Step 150: grad_norm=0.0483, loss=0.0060, param_norm=1804.1471
Step 200: grad_norm=0.0446, loss=0.0051, param_norm=1804.1493
Step 250: grad_norm=0.0447, loss=0.0048, param_norm=1804.1520
Step 300: grad_norm=0.0463, loss=0.0051, param_norm=1804.1549
Step 350: grad_norm=0.0469, loss=0.0048, param_norm=1804.1571
Step 400: grad_norm=0.0412, loss=0.0043, param_norm=1804.1592
Step 450: grad_norm=0.0520, loss=0.0049, param_norm=1804.1616
Step 500: grad_norm=0.0435, loss=0.0042, param_norm=1804.1635
Step 550: grad_norm=0.0468, loss=0.0042, param_norm=1804.1653
Step 600: grad_norm=0.0446, loss=0.0040, param_norm=1804.1671
Step 650: grad_norm=0.0470, loss=0.0038, param_norm=1804.1692
Step 700: grad_norm=0.0413, loss=0.0036, param_norm=1804.1705
Step 750: grad_norm=0.0462, loss=0.0039, param_norm=1804.1721
Step 800: grad_norm=0.0452, loss=0.0037, param_norm=1804.1731
Step 850: grad_norm=0.0433, loss=0.0033, param_norm=1804.1741
Step 900: grad_norm=0.0415, loss=0.0034, param_norm=1804.1752
Step 950: grad_norm=0.0409, loss=0.0031, param_norm=1804.1760
Step 1000: grad_norm=0.0461, loss=0.0034, param_norm=1804.1766
Step 1050: grad_norm=0.0448, loss=0.0032, param_norm=1804.1771
Step 1100: grad_norm=0.0433, loss=0.0033, param_norm=1804.1777
Step 1150: grad_norm=0.0374, loss=0.0029, param_norm=1804.1783
Step 1200: grad_norm=0.0452, loss=0.0033, param_norm=1804.1783
Step 1250: grad_norm=0.0383, loss=0.0027, param_norm=1804.1791
Step 1300: grad_norm=0.0362, loss=0.0025, param_norm=1804.1797
Step 1350: grad_norm=0.0426, loss=0.0030, param_norm=1804.1799
Step 1400: grad_norm=0.0368, loss=0.0028, param_norm=1804.1799
Step 1450: grad_norm=0.0369, loss=0.0028, param_norm=1804.1803