[LOG   ] ターミナルの出力を残します → /Volumes/KIOXIA/r2r2r-data/lab/Log22/train/lab_train_20261106_195143.txt
[LAB   ] 2026-11-06 19:51:43  学習  GPU=NVIDIA A40 (借りられなければ NVIDIA RTX A6000 NVIDIA L40S)
         --model corr3 (PixelstoMotors/pi05-mecharm-trim-corr-lora-v2 → PixelstoMotors/pi05-mecharm-trim-corr-lora-v3)  データ mecharm_redblock_box_teach_v1_trim2_corr_v5
         pi05_mecharm_trim_corr_v5_lora --exp-name=trim_corr_v5  steps=1500  init=/workspace/ckpt/corr_v2_1499/params  上限 5 時間  記録 /Volumes/KIOXIA/r2r2r-data/lab/Log22/train

=== 0. 送るものと Mac の道具を確かめる  (19:51:43)
[DATA  ] tar 116993536 バイト  (/Volumes/KIOXIA/r2r2r-data/datasets/mecharm_redblock_box_teach_v1_trim2_corr_v5/mecharm_redblock_box_teach_v1_trim2_corr_v5_raw_episodes.tar)
[PROV  ] 始める時点のコード (284dfd7) と入力 3 つの指紋を測った
[MAC   ] 終わるまでスリープしないようにした (caffeinate pid 7188). ふたは閉じない
[WARN  ] 電源につながっていない. バッテリーだとスリープを止めきれないので, 電源につないでください
    (1 秒)

=== 1. Pod を作る  (19:51:44)
[POD   ] NVIDIA A40 を借ります (空きがなければ 30 秒おきに 5 回まで)
[POD   ] 作った: id uirn5j6iy4xbyx  GPU=NVIDIA A40  1時間 0.59 ドル  (記録 /Volumes/KIOXIA/r2r2r-data/lab/Log22/train/lab_pod.json)
[POD   ] 上限: 11-07 00:51 (5 時間) を過ぎたら, ログを持ち帰って消します
    (2 秒)

=== 2. Pod が立ち上がるまで待つ  (19:51:46)
[POD   ] SSH の IP とポートを API から受け取った (<POD_IP>:<SSH_PORT>)
[POD   ] 入れた: a4e3c12d84ee
NVIDIA A40, 46068 MiB
[WATCH ] Pod 側の見張りを置いた: 11-07 01:51 を過ぎたら Pod が自分を消す (Pod 用の鍵. Mac の鍵は送っていない)
    (37 秒)

=== 3a. スクリプトを送る  (19:52:23)
    (52 秒)

=== 3b. huggingface_hub を入れ, 鍵を渡す  (19:53:15)
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager, possibly rendering your system unusable. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv. Use the --root-user-action option if you know what you are doing and want to suppress this warning.
Hint: The `hf-cli` skill is not installed. Run `hf skills add -g` to teach your AI agents how to use the `hf` CLI.
✓ Logged in
  user: PixelstoMotors
    (10 秒)

=== 3c. 続きから学習するモデルを取る (PixelstoMotors/pi05-mecharm-trim-corr-lora-v2 → /workspace/ckpt/corr_v2_1499)  (19:53:25)

Fetching 28 files:   0%|          | 0/28 [00:00<?, ?it/s]
Fetching 28 files:   4%|▎         | 1/28 [00:00<00:03,  8.75it/s]
Fetching 28 files:  32%|███▏      | 9/28 [00:00<00:00, 37.51it/s]
Fetching 28 files:  46%|████▋     | 13/28 [00:04<00:05,  2.51it/s]
Fetching 28 files:  54%|█████▎    | 15/28 [00:04<00:04,  2.82it/s]
Fetching 28 files:  61%|██████    | 17/28 [00:06<00:04,  2.22it/s]
Fetching 28 files:  64%|██████▍   | 18/28 [00:06<00:04,  2.20it/s]
Fetching 28 files:  68%|██████▊   | 19/28 [00:06<00:03,  2.28it/s]
Fetching 28 files:  71%|███████▏  | 20/28 [00:07<00:03,  2.64it/s]
Fetching 28 files:  75%|███████▌  | 21/28 [00:07<00:02,  2.88it/s]
Fetching 28 files:  89%|████████▉ | 25/28 [00:08<00:00,  3.81it/s]
Fetching 28 files:  93%|█████████▎| 26/28 [00:08<00:00,  3.00it/s]
Fetching 28 files:  96%|█████████▋| 27/28 [00:10<00:00,  1.83it/s]✓ Downloaded
  path: /workspace/ckpt/corr_v2_1499

Fetching 28 files: 100%|██████████| 28/28 [00:14<00:00,  1.31s/it]
Fetching 28 files: 100%|██████████| 28/28 [00:14<00:00,  1.96it/s]
6.0G	/workspace/ckpt/corr_v2_1499
    (17 秒)

=== 3d. openpi を取る (215abfb)  (19:53:42)
215abfb docs(droid): fix config search instruction (#1023)
    (16 秒)

=== 3e. uv sync  (19:53:58)
downloading uv 0.13.7 x86_64-unknown-linux-gnu
WARN: The following commands are shadowed by other commands in your PATH: uv uvx
Using CPython 3.11.13 interpreter at: /usr/bin/python3.11
Creating virtual environment at: .venv
Resolved 279 packages in 4ms
   Building openpi-client @ file:///workspace/openpi/packages/openpi-client
   Building openpi @ file:///workspace/openpi
   Updating https://github.com/huggingface/lerobot (0cf864870cf29f4738d3ade893e6fd13fbd7cdb5)
Downloading opencv-python (60.1MiB)
Downloading nvidia-nvjitlink-cu12 (18.8MiB)
Downloading cmake (26.6MiB)
Downloading jax-cuda12-plugin (15.9MiB)
Downloading nvidia-cublas-cu12 (374.9MiB)
Downloading sympy (6.0MiB)
Downloading llvmlite (40.4MiB)
Downloading nvidia-cusolver-cu12 (150.9MiB)
Downloading nvidia-cuda-nvrtc-cu12 (22.6MiB)
Downloading pyarrow (40.3MiB)
Downloading nvidia-curand-cu12 (53.7MiB)
Downloading rerun-sdk (68.1MiB)
Downloading nvidia-cufft-cu12 (190.9MiB)
Downloading nvidia-cusparse-cu12 (206.5MiB)
Downloading nvidia-cudnn-cu12 (544.5MiB)
Downloading torch (783.1MiB)
Downloading hf-transfer (3.4MiB)
Downloading opencv-python-headless (47.7MiB)
Downloading scipy (35.9MiB)
Downloading lxml (4.7MiB)
Downloading polars (34.6MiB)
Downloading wandb (20.4MiB)
Downloading imageio-ffmpeg (28.1MiB)
Downloading pandas (12.5MiB)
Downloading nvidia-cuda-nvcc-cu12 (38.7MiB)
Downloading h5py (4.3MiB)
Downloading triton (148.5MiB)
Downloading jax-cuda12-pjrt (99.8MiB)
Downloading numcodecs (8.4MiB)
Downloading jaxlib (100.2MiB)
Downloading nvidia-cusparselt-cu12 (149.5MiB)
Downloading labmaze (4.7MiB)
Downloading diffusers (3.4MiB)
Downloading nvidia-nccl-cu12 (192.0MiB)
Downloading mujoco (4.4MiB)
Downloading pyopengl (3.0MiB)
Downloading tensorstore (17.5MiB)
Downloading nvidia-cuda-cupti-cu12 (8.5MiB)
Downloading ruff (11.0MiB)
Downloading av (34.7MiB)
Downloading dm-control (37.5MiB)
Downloading virtualenv (5.8MiB)
Downloading torchvision (7.1MiB)
Downloading hf-xet (4.9MiB)
Downloading numba (3.7MiB)
Downloading transformers (10.3MiB)
Downloading pillow (4.4MiB)
Downloading numpy (17.4MiB)
Downloading matplotlib (8.2MiB)
Downloading fonttools (4.7MiB)
 Downloaded pyopengl
 Downloaded diffusers
Downloading debugpy (3.0MiB)
Downloading tokenizers (2.9MiB)
 Downloaded hf-transfer
Downloading jax (2.3MiB)
 Downloaded numba
Downloading widgetsnbextension (2.1MiB)
 Downloaded h5py
Downloading ml-dtypes (2.1MiB)
 Downloaded mujoco
 Downloaded pillow
 Downloaded labmaze
Downloading networkx (1.9MiB)
Downloading pydantic-core (1.9MiB)
Downloading aiohttp (1.6MiB)
 Downloaded lxml
 Downloaded fonttools
Downloading kiwisolver (1.4MiB)
Downloading jedi (1.5MiB)
 Downloaded hf-xet
Downloading torchcodec (1.3MiB)
 Downloaded virtualenv
Downloading sentencepiece (1.2MiB)
 Downloaded widgetsnbextension
 Downloaded jax
 Downloaded sympy
Downloading pygments (1.2MiB)
Downloading setuptools (1.1MiB)
Downloading nvidia-cufile-cu12 (1.1MiB)
 Downloaded kiwisolver
 Downloaded tokenizers
 Downloaded debugpy
 Downloaded aiohttp
 Downloaded jedi
Downloading pymunk (1.0MiB)
 Downloaded ml-dtypes
 Downloaded pydantic-core
 Downloaded torchcodec
 Downloaded networkx
 Downloaded torchvision
 Downloaded sentencepiece
 Downloaded nvidia-cufile-cu12
 Downloaded pygments
 Downloaded setuptools
 Downloaded pymunk
 Downloaded matplotlib
 Downloaded numcodecs
 Downloaded nvidia-cuda-cupti-cu12
   Building evdev==1.9.2
   Building antlr4-python3-runtime==4.9.3
 Downloaded transformers
 Downloaded ruff
 Downloaded pandas
 Downloaded jax-cuda12-plugin
 Downloaded numpy
 Downloaded tensorstore
      Built openpi @ file:///workspace/openpi
      Built openpi-client @ file:///workspace/openpi/packages/openpi-client
 Downloaded nvidia-nvjitlink-cu12
 Downloaded wandb
 Downloaded nvidia-cuda-nvrtc-cu12
      Built antlr4-python3-runtime==4.9.3
 Downloaded imageio-ffmpeg
 Downloaded cmake
 Downloaded polars
 Downloaded av
 Downloaded scipy
 Downloaded dm-control
 Downloaded nvidia-cuda-nvcc-cu12
 Downloaded pyarrow
 Downloaded llvmlite
 Downloaded opencv-python-headless
 Downloaded nvidia-curand-cu12
      Built evdev==1.9.2
 Downloaded opencv-python
    Updated https://github.com/huggingface/lerobot (0cf864870cf29f4738d3ade893e6fd13fbd7cdb5)
   Building lerobot @ git+https://github.com/huggingface/lerobot@0cf864870cf29f4738d3ade893e6fd13fbd7cdb5
 Downloaded rerun-sdk
      Built lerobot @ git+https://github.com/huggingface/lerobot@0cf864870cf29f4738d3ade893e6fd13fbd7cdb5
 Downloaded jaxlib
 Downloaded jax-cuda12-pjrt
 Downloaded triton
 Downloaded nvidia-cusolver-cu12
 Downloaded nvidia-cusparselt-cu12
 Downloaded nvidia-cufft-cu12
 Downloaded nvidia-nccl-cu12
 Downloaded nvidia-cusparse-cu12
 Downloaded nvidia-cublas-cu12
 Downloaded nvidia-cudnn-cu12
 Downloaded torch
Prepared 240 packages in 14.67s
Installed 240 packages in 1.86s
 + absl-py==2.3.0
 + aiohappyeyeballs==2.6.1
 + aiohttp==3.12.4
 + aiosignal==1.3.2
 + annotated-types==0.7.0
 + antlr4-python3-runtime==4.9.3
 + asttokens==3.0.0
 + attrs==25.3.0
 + augmax==0.4.1
 + av==17.0.0
 + beartype==0.19.0
 + beautifulsoup4==4.13.4
 + blinker==1.9.0
 + cachetools==5.5.2
 + certifi==2025.4.26
 + cffi==1.17.1
 + cfgv==3.4.0
 + charset-normalizer==3.4.2
 + chex==0.1.90
 + click==8.2.1
 + cloudpickle==3.1.1
 + cmake==4.0.2
 + comm==0.2.2
 + contourpy==1.3.2
 + crc32c==2.7.1
 + cycler==0.12.1
 + datasets==3.6.0
 + debugpy==1.8.14
 + decorator==5.2.1
 + deepdiff==8.5.0
 + diffusers==0.33.1
 + dill==0.3.8
 + distlib==0.3.9
 + dm-control==1.0.14
 + dm-env==1.6
 + dm-tree==0.1.9
 + docker-pycreds==0.4.0
 + docstring-parser==0.16
 + donfig==0.8.1.post1
 + draccus==0.10.0
 + einops==0.8.1
 + equinox==0.12.2
 + etils==1.12.2
 + evdev==1.9.2
 + executing==2.2.0
 + farama-notifications==0.0.4
 + filelock==3.18.0
 + flask==3.1.1
 + flatbuffers==25.2.10
 + flax==0.10.2
 + fonttools==4.58.1
 + frozenlist==1.6.0
 + fsspec==2025.3.0
 + gcsfs==2025.3.0
 + gdown==5.2.0
 + gitdb==4.0.12
 + gitpython==3.1.44
 + glfw==2.9.0
 + google-api-core==2.24.2
 + google-auth==2.40.2
 + google-auth-oauthlib==1.2.2
 + google-cloud-core==2.4.3
 + google-cloud-storage==3.1.0
 + google-crc32c==1.7.1
 + google-resumable-media==2.7.2
 + googleapis-common-protos==1.70.0
 + gym-aloha==0.1.1
 + gymnasium==0.29.1
 + h5py==3.13.0
 + hf-transfer==0.1.9
 + hf-xet==1.1.2
 + huggingface-hub==0.32.3
 + humanize==4.12.3
 + identify==2.6.12
 + idna==3.10
 + imageio==2.37.0
 + imageio-ffmpeg==0.6.0
 + importlib-metadata==8.7.0
 + importlib-resources==6.5.2
 + iniconfig==2.1.0
 + inquirerpy==0.3.4
 + ipykernel==6.29.5
 + ipython==9.2.0
 + ipython-pygments-lexers==1.1.1
 + ipywidgets==8.1.7
 + itsdangerous==2.2.0
 + jax==0.5.3
 + jax-cuda12-pjrt==0.5.3
 + jax-cuda12-plugin==0.5.3
 + jaxlib==0.5.3
 + jaxtyping==0.2.36
 + jedi==0.19.2
 + jinja2==3.1.6
 + jsonlines==4.0.0
 + jupyter-client==8.6.3
 + jupyter-core==5.8.1
 + jupyterlab-widgets==3.0.15
 + kiwisolver==1.4.8
 + labmaze==1.0.6
 + lerobot==0.1.0 (from git+https://github.com/huggingface/lerobot@0cf864870cf29f4738d3ade893e6fd13fbd7cdb5)
 + llvmlite==0.44.0
 + lxml==5.4.0
 + markdown-it-py==3.0.0
 + markupsafe==3.0.2
 + matplotlib==3.10.3
 + matplotlib-inline==0.1.7
 + mdurl==0.1.2
 + mergedeep==1.3.4
 + ml-collections==1.0.0
 + ml-dtypes==0.4.1
 + mpmath==1.3.0
 + msgpack==1.1.0
 + mujoco==2.3.7
 + multidict==6.4.4
 + multiprocess==0.70.16
 + mypy-extensions==1.1.0
 + nest-asyncio==1.6.0
 + networkx==3.5
 + nodeenv==1.9.1
 + numba==0.61.2
 + numcodecs==0.16.1
 + numpy==1.26.4
 + numpydantic==1.6.9
 + nvidia-cublas-cu12==12.6.4.1
 + nvidia-cuda-cupti-cu12==12.6.80
 + nvidia-cuda-nvcc-cu12==12.9.41
 + nvidia-cuda-nvrtc-cu12==12.6.77
 + nvidia-cuda-runtime-cu12==12.6.77
 + nvidia-cudnn-cu12==9.5.1.17
 + nvidia-cufft-cu12==11.3.0.4
 + nvidia-cufile-cu12==1.11.1.6
 + nvidia-curand-cu12==10.3.7.77
 + nvidia-cusolver-cu12==11.7.1.2
 + nvidia-cusparse-cu12==12.5.4.2
 + nvidia-cusparselt-cu12==0.6.3
 + nvidia-ml-py==12.575.51
 + nvidia-nccl-cu12==2.26.2
 + nvidia-nvjitlink-cu12==12.6.85
 + nvidia-nvtx-cu12==12.6.77
 + oauthlib==3.2.2
 + omegaconf==2.3.0
 + opencv-python==4.11.0.86
 + opencv-python-headless==4.11.0.86
 + openpi==0.1.0 (from file:///workspace/openpi)
 + openpi-client==0.1.0 (from file:///workspace/openpi/packages/openpi-client)
 + opt-einsum==3.4.0
 + optax==0.2.4
 + orbax-checkpoint==0.11.13
 + orderly-set==5.4.1
 + packaging==25.0
 + pandas==2.2.3
 + parso==0.8.4
 + pexpect==4.9.0
 + pfzy==0.3.4
 + pillow==11.2.1
 + platformdirs==4.3.8
 + pluggy==1.6.0
 + polars==1.30.0
 + pre-commit==4.2.0
 + prompt-toolkit==3.0.51
 + propcache==0.3.1
 + proto-plus==1.26.1
 + protobuf==4.25.8
 + psutil==7.0.0
 + ptyprocess==0.7.0
 + pure-eval==0.2.3
 + pyarrow==20.0.0
 + pyasn1==0.6.1
 + pyasn1-modules==0.4.2
 + pycparser==2.22
 + pydantic==2.11.5
 + pydantic-core==2.33.2
 + pygments==2.19.1
 + pymunk==7.0.0
 + pynput==1.8.1
 + pynvml==12.0.0
 + pyopengl==3.1.9
 + pyparsing==3.2.3
 + pysocks==1.7.1
 + pytest==8.3.5
 + python-dateutil==2.9.0.post0
 + python-xlib==0.33
 + pytz==2025.2
 + pyyaml==6.0.2
 + pyyaml-include==1.4.1
 + pyzmq==26.4.0
 + regex==2024.11.6
 + requests==2.32.3
 + requests-oauthlib==2.0.0
 + rerun-sdk==0.23.1
 + rich==14.0.0
 + rsa==4.9.1
 + ruff==0.11.12
 + safetensors==0.5.3
 + scipy==1.15.3
 + sentencepiece==0.2.0
 + sentry-sdk==2.29.1
 + setproctitle==1.3.6
 + setuptools==80.9.0
 + shtab==1.7.2
 + simplejson==3.20.1
 + six==1.17.0
 + smmap==5.0.2
 + soupsieve==2.7
 + stack-data==0.6.3
 + sympy==1.14.0
 + tensorstore==0.1.74
 + termcolor==3.1.0
 + tokenizers==0.21.1
 + toml==0.10.2
 + toolz==1.0.0
 + torch==2.7.1
 + torchcodec==0.4.0
 + torchvision==0.22.1
 + tornado==6.5.1
 + tqdm==4.67.1
 + tqdm-loggable==0.2
 + traitlets==5.14.3
 + transformers==4.53.2
 + treescope==0.1.9
 + triton==3.3.1
 + typeguard==4.4.2
 + typing-extensions==4.13.2
 + typing-inspect==0.9.0
 + typing-inspection==0.4.1
 + tyro==0.9.22
 + tzdata==2025.2
 + urllib3==2.4.0
 + virtualenv==20.31.2
 + wadler-lindig==0.1.6
 + wandb==0.19.11
 + wcwidth==0.2.13
 + websockets==15.0.1
 + werkzeug==3.1.3
 + widgetsnbextension==4.0.14
 + wrapt==1.14.1
 + xxhash==3.5.0
 + yarl==1.20.0
 + zarr==3.0.8
 + zipp==3.22.0
    (20 秒)

=== 3f. addon・設定・assets  (19:54:18)
installed: policies/mecharm_policy.py + config.py (pi05_mecharm_lora)
installed: pi05_mecharm_trim_lora (repo_id=local/mecharm_redblock_box_teach_v1_trim2, 4,000 steps)
[PROV  ] openpi 215abfb217dbac7d5f1273282331b9b1866c0479 (2 ファイル変更・addon と設定を入れたあと)
    (7 秒)

=== 4a. tar を送る  (19:54:25)
-rwx------ 1 root root 116993536 Nov  6 10:56 /workspace/data/mecharm_redblock_box_teach_v1_trim2_corr_v5_raw_episodes.tar
    (103 秒)

=== 4b. pod_prep_trim_corr.sh 1500 /workspace/ckpt/corr_v2_1499/params v5  (19:56:08)
== 1. tar を広げる → /workspace/data/mecharm_redblock_box_teach_v1_trim2_corr_v5
   エピソード 30 本 (Mac で作ったときは 30 本)
== 2. 学習設定 pi05_mecharm_trim_corr_v5_lora を足す (steps=1500 init=/workspace/ckpt/corr_v2_1499/params)
installed: pi05_mecharm_trim_corr_v5_lora (repo_id=local/mecharm_redblock_box_teach_v1_trim2_corr_v5, 1500 steps, init=/workspace/ckpt/corr_v2_1499/params)
== 3. LeRobot 形式に変換 (local/mecharm_redblock_box_teach_v1_trim2_corr_v5)

Map:   0%|          | 0/118 [00:00<?, ? examples/s]
Map:  84%|████████▍ | 99/118 [00:00<00:00, 972.36 examples/s]
Map: 100%|██████████| 118/118 [00:00<00:00, 846.40 examples/s]

Creating parquet from Arrow format:   0%|          | 0/2 [00:00<?, ?ba/s]
Creating parquet from Arrow format: 100%|██████████| 2/2 [00:00<00:00, 90.30ba/s]
  pi05_20261031_175527: 118 steps

Map:   0%|          | 0/152 [00:00<?, ? examples/s]
Map:  85%|████████▍ | 129/152 [00:00<00:00, 1271.54 examples/s]
Map: 100%|██████████| 152/152 [00:00<00:00, 1084.97 examples/s]

Creating parquet from Arrow format:   0%|          | 0/2 [00:00<?, ?ba/s]
Creating parquet from Arrow format: 100%|██████████| 2/2 [00:00<00:00, 89.94ba/s]
  pi05_20261031_180435: 152 steps

Map:   0%|          | 0/126 [00:00<?, ? examples/s]
Map: 100%|██████████| 126/126 [00:00<00:00, 1098.62 examples/s]
Map: 100%|██████████| 126/126 [00:00<00:00, 1070.64 examples/s]

Creating parquet from Arrow format:   0%|          | 0/2 [00:00<?, ?ba/s]
Creating parquet from Arrow format: 100%|██████████| 2/2 [00:00<00:00, 83.71ba/s]
  pi05_20261031_181946: 126 steps

Map:   0%|          | 0/133 [00:00<?, ? examples/s]
Map: 100%|██████████| 133/133 [00:00<00:00, 1219.33 examples/s]
Map: 100%|██████████| 133/133 [00:00<00:00, 1191.78 examples/s]

Creating parquet from Arrow format:   0%|          | 0/2 [00:00<?, ?ba/s]
Creating parquet from Arrow format: 100%|██████████| 2/2 [00:00<00:00, 114.57ba/s]
  pi05_20261031_184016: 133 steps

Map:   0%|          | 0/115 [00:00<?, ? examples/s]
Map: 100%|██████████| 115/115 [00:00<00:00, 1259.43 examples/s]

Creating parquet from Arrow format:   0%|          | 0/2 [00:00<?, ?ba/s]
Creating parquet from Arrow format: 100%|██████████| 2/2 [00:00<00:00, 88.77ba/s]
  pi05_20261101_190347: 115 steps

Map:   0%|          | 0/145 [00:00<?, ? examples/s]
Map:  90%|████████▉ | 130/145 [00:00<00:00, 1280.71 examples/s]
Map: 100%|██████████| 145/145 [00:00<00:00, 1096.39 examples/s]

Creating parquet from Arrow format:   0%|          | 0/2 [00:00<?, ?ba/s]
Creating parquet from Arrow format: 100%|██████████| 2/2 [00:00<00:00, 77.33ba/s]
  pi05_20261101_190854: 145 steps

Map:   0%|          | 0/104 [00:00<?, ? examples/s]
Map: 100%|██████████| 104/104 [00:00<00:00, 1184.46 examples/s]

Creating parquet from Arrow format:   0%|          | 0/2 [00:00<?, ?ba/s]
Creating parquet from Arrow format: 100%|██████████| 2/2 [00:00<00:00, 100.10ba/s]
  pi05_20261101_192101: 104 steps

Map:   0%|          | 0/188 [00:00<?, ? examples/s]
Map:  49%|████▉     | 92/188 [00:00<00:00, 898.09 examples/s]
Map: 100%|██████████| 188/188 [00:00<00:00, 933.81 examples/s]
Map: 100%|██████████| 188/188 [00:00<00:00, 915.68 examples/s]

Creating parquet from Arrow format:   0%|          | 0/2 [00:00<?, ?ba/s]
Creating parquet from Arrow format: 100%|██████████| 2/2 [00:00<00:00, 76.15ba/s]
  pi05_20261106_183926: 188 steps

Map:   0%|          | 0/190 [00:00<?, ? examples/s]
Map:  65%|██████▌   | 124/190 [00:00<00:00, 1223.93 examples/s]
Map: 100%|██████████| 190/190 [00:00<00:00, 1116.45 examples/s]

Creating parquet from Arrow format:   0%|          | 0/2 [00:00<?, ?ba/s]
Creating parquet from Arrow format: 100%|██████████| 2/2 [00:00<00:00, 82.95ba/s]
  pi05_20261106_184452: 190 steps

Map:   0%|          | 0/178 [00:00<?, ? examples/s]
Map:  77%|███████▋  | 137/178 [00:00<00:00, 1355.49 examples/s]
Map: 100%|██████████| 178/178 [00:00<00:00, 1201.71 examples/s]

Creating parquet from Arrow format:   0%|          | 0/2 [00:00<?, ?ba/s]
Creating parquet from Arrow format: 100%|██████████| 2/2 [00:00<00:00, 105.06ba/s]
  pi05_20261106_184650: 178 steps

Map:   0%|          | 0/135 [00:00<?, ? examples/s]
Map: 100%|██████████| 135/135 [00:00<00:00, 1279.14 examples/s]
Map: 100%|██████████| 135/135 [00:00<00:00, 1249.93 examples/s]

Creating parquet from Arrow format:   0%|          | 0/2 [00:00<?, ?ba/s]
Creating parquet from Arrow format: 100%|██████████| 2/2 [00:00<00:00, 92.10ba/s]
  pi05_20261106_184906: 135 steps

Map:   0%|          | 0/144 [00:00<?, ? examples/s]
Map:  98%|█████████▊| 141/144 [00:00<00:00, 1397.81 examples/s]
Map: 100%|██████████| 144/144 [00:00<00:00, 1260.94 examples/s]

Creating parquet from Arrow format:   0%|          | 0/2 [00:00<?, ?ba/s]
Creating parquet from Arrow format: 100%|██████████| 2/2 [00:00<00:00, 88.36ba/s]
  run_0001_20261019_183454: 144 steps

Map:   0%|          | 0/147 [00:00<?, ? examples/s]
Map:  67%|██████▋   | 98/147 [00:00<00:00, 968.23 examples/s]
Map: 100%|██████████| 147/147 [00:00<00:00, 863.83 examples/s]

Creating parquet from Arrow format:   0%|          | 0/2 [00:00<?, ?ba/s]
Creating parquet from Arrow format: 100%|██████████| 2/2 [00:00<00:00, 55.79ba/s]
  run_0001_20261019_191257: 147 steps

Map:   0%|          | 0/144 [00:00<?, ? examples/s]
Map:  95%|█████████▌| 137/144 [00:00<00:00, 1355.84 examples/s]
Map: 100%|██████████| 144/144 [00:00<00:00, 1175.39 examples/s]

Creating parquet from Arrow format:   0%|          | 0/2 [00:00<?, ?ba/s]
Creating parquet from Arrow format: 100%|██████████| 2/2 [00:00<00:00, 79.65ba/s]
  run_0001_20261019_191347: 144 steps

Map:   0%|          | 0/163 [00:00<?, ? examples/s]
Map:  72%|███████▏  | 118/163 [00:00<00:00, 1163.69 examples/s]
Map: 100%|██████████| 163/163 [00:00<00:00, 956.02 examples/s] 

Creating parquet from Arrow format:   0%|          | 0/2 [00:00<?, ?ba/s]
Creating parquet from Arrow format: 100%|██████████| 2/2 [00:00<00:00, 61.18ba/s]
  run_0001_20261019_191441: 163 steps

Map:   0%|          | 0/139 [00:00<?, ? examples/s]
Map:  73%|███████▎  | 102/139 [00:00<00:00, 1005.16 examples/s]
Map: 100%|██████████| 139/139 [00:00<00:00, 864.78 examples/s] 

Creating parquet from Arrow format:   0%|          | 0/2 [00:00<?, ?ba/s]
Creating parquet from Arrow format: 100%|██████████| 2/2 [00:00<00:00, 73.55ba/s]
  run_0001_20261019_191541: 139 steps

Map:   0%|          | 0/138 [00:00<?, ? examples/s]
Map:  76%|███████▌  | 105/138 [00:00<00:00, 1039.93 examples/s]
Map: 100%|██████████| 138/138 [00:00<00:00, 972.58 examples/s] 

Creating parquet from Arrow format:   0%|          | 0/2 [00:00<?, ?ba/s]
Creating parquet from Arrow format: 100%|██████████| 2/2 [00:00<00:00, 84.37ba/s]
  run_0001_20261019_191635: 138 steps

Map:   0%|          | 0/140 [00:00<?, ? examples/s]
Map:  97%|█████████▋| 136/140 [00:00<00:00, 1347.88 examples/s]
Map: 100%|██████████| 140/140 [00:00<00:00, 1168.10 examples/s]

Creating parquet from Arrow format:   0%|          | 0/2 [00:00<?, ?ba/s]
Creating parquet from Arrow format: 100%|██████████| 2/2 [00:00<00:00, 96.59ba/s]
  run_0001_20261019_192343: 140 steps

Map:   0%|          | 0/138 [00:00<?, ? examples/s]
Map:  95%|█████████▍| 131/138 [00:00<00:00, 1302.34 examples/s]
Map: 100%|██████████| 138/138 [00:00<00:00, 1118.87 examples/s]

Creating parquet from Arrow format:   0%|          | 0/2 [00:00<?, ?ba/s]
Creating parquet from Arrow format: 100%|██████████| 2/2 [00:00<00:00, 114.55ba/s]
  run_0001_20261019_192438: 138 steps

Map:   0%|          | 0/140 [00:00<?, ? examples/s]
Map:  99%|█████████▊| 138/140 [00:00<00:00, 1361.52 examples/s]
Map: 100%|██████████| 140/140 [00:00<00:00, 1194.44 examples/s]

Creating parquet from Arrow format:   0%|          | 0/2 [00:00<?, ?ba/s]
Creating parquet from Arrow format: 100%|██████████| 2/2 [00:00<00:00, 76.66ba/s]
  run_0001_20261019_192539: 140 steps

Map:   0%|          | 0/137 [00:00<?, ? examples/s]
Map:  85%|████████▌ | 117/137 [00:00<00:00, 1149.13 examples/s]
Map: 100%|██████████| 137/137 [00:00<00:00, 1042.42 examples/s]

Creating parquet from Arrow format:   0%|          | 0/2 [00:00<?, ?ba/s]
Creating parquet from Arrow format: 100%|██████████| 2/2 [00:00<00:00, 78.92ba/s]
  run_0001_20261019_192633: 137 steps

Map:   0%|          | 0/164 [00:00<?, ? examples/s]
Map:  58%|█████▊    | 95/164 [00:00<00:00, 926.76 examples/s]
Map: 100%|██████████| 164/164 [00:00<00:00, 803.84 examples/s]

Creating parquet from Arrow format:   0%|          | 0/2 [00:00<?, ?ba/s]
Creating parquet from Arrow format: 100%|██████████| 2/2 [00:00<00:00, 50.97ba/s]
  run_0001_20261019_193404: 164 steps

Map:   0%|          | 0/159 [00:00<?, ? examples/s]
Map:  67%|██████▋   | 107/159 [00:00<00:00, 1055.78 examples/s]
Map: 100%|██████████| 159/159 [00:00<00:00, 945.11 examples/s] 

Creating parquet from Arrow format:   0%|          | 0/2 [00:00<?, ?ba/s]
Creating parquet from Arrow format: 100%|██████████| 2/2 [00:00<00:00, 57.83ba/s]
  run_0001_20261019_193459: 159 steps

Map:   0%|          | 0/156 [00:00<?, ? examples/s]
Map:  76%|███████▌  | 118/156 [00:00<00:00, 1163.21 examples/s]
Map: 100%|██████████| 156/156 [00:00<00:00, 1098.09 examples/s]

Creating parquet from Arrow format:   0%|          | 0/2 [00:00<?, ?ba/s]
Creating parquet from Arrow format: 100%|██████████| 2/2 [00:00<00:00, 117.17ba/s]
  run_0001_20261019_193617: 156 steps

Map:   0%|          | 0/165 [00:00<?, ? examples/s]
Map:  79%|███████▉  | 130/165 [00:00<00:00, 1279.45 examples/s]
Map: 100%|██████████| 165/165 [00:00<00:00, 1118.68 examples/s]

Creating parquet from Arrow format:   0%|          | 0/2 [00:00<?, ?ba/s]
Creating parquet from Arrow format: 100%|██████████| 2/2 [00:00<00:00, 91.08ba/s]
  run_0001_20261019_193825: 165 steps

Map:   0%|          | 0/151 [00:00<?, ? examples/s]
Map:  89%|████████▉ | 135/151 [00:00<00:00, 1339.38 examples/s]
Map: 100%|██████████| 151/151 [00:00<00:00, 1141.87 examples/s]

Creating parquet from Arrow format:   0%|          | 0/2 [00:00<?, ?ba/s]
Creating parquet from Arrow format: 100%|██████████| 2/2 [00:00<00:00, 57.94ba/s]
  run_0001_20261019_193932: 151 steps

Map:   0%|          | 0/123 [00:00<?, ? examples/s]
Map: 100%|██████████| 123/123 [00:00<00:00, 1060.43 examples/s]
Map: 100%|██████████| 123/123 [00:00<00:00, 1034.37 examples/s]

Creating parquet from Arrow format:   0%|          | 0/2 [00:00<?, ?ba/s]
Creating parquet from Arrow format: 100%|██████████| 2/2 [00:00<00:00, 77.36ba/s]
  run_0001_20261019_194818: 123 steps

Map:   0%|          | 0/126 [00:00<?, ? examples/s]
Map:  92%|█████████▏| 116/126 [00:00<00:00, 1150.81 examples/s]
Map: 100%|██████████| 126/126 [00:00<00:00, 1008.68 examples/s]

Creating parquet from Arrow format:   0%|          | 0/2 [00:00<?, ?ba/s]
Creating parquet from Arrow format: 100%|██████████| 2/2 [00:00<00:00, 74.45ba/s]
  run_0001_20261019_194913: 126 steps

Map:   0%|          | 0/123 [00:00<?, ? examples/s]
Map:  97%|█████████▋| 119/123 [00:00<00:00, 1174.84 examples/s]
Map: 100%|██████████| 123/123 [00:00<00:00, 999.16 examples/s] 

Creating parquet from Arrow format:   0%|          | 0/2 [00:00<?, ?ba/s]
Creating parquet from Arrow format: 100%|██████████| 2/2 [00:00<00:00, 83.82ba/s]
  run_0001_20261019_195023: 123 steps

Map:   0%|          | 0/120 [00:00<?, ? examples/s]
Map: 100%|██████████| 120/120 [00:00<00:00, 1110.76 examples/s]
Map: 100%|██████████| 120/120 [00:00<00:00, 1085.65 examples/s]

Creating parquet from Arrow format:   0%|          | 0/2 [00:00<?, ?ba/s]
Creating parquet from Arrow format: 100%|██████████| 2/2 [00:00<00:00, 94.67ba/s]
  run_0001_20261019_195120: 120 steps
done: 30 episodes / 4301 frames → /root/.cache/huggingface/lerobot/local/mecharm_redblock_box_teach_v1_trim2_corr_v5
== 4. 正規化の値は trim2 の値をそのまま置く (計算し直さない)
   /workspace/ckpt/corr_v2_1499/assets/local/mecharm_redblock_box_teach_v1_trim2_corr_v4b/norm_stats.json → /workspace/openpi/assets/pi05_mecharm_trim_corr_v5_lora/local/mecharm_redblock_box_teach_v1_trim2_corr_v5/norm_stats.json (sha256 80346837eb12…)
== 5. 準備できました. 学習はこのコマンドで始める (このスクリプトでは流さない):
   cd /workspace/openpi && XLA_PYTHON_CLIENT_MEM_FRACTION=0.9 uv run scripts/train.py pi05_mecharm_trim_corr_v5_lora --exp-name=trim_corr_v5 --overwrite 2>&1 | tee /workspace/logs/train_trim_corr_v5.log
    (48 秒)

=== 5. 学習を始める  (19:56:56)
[TRAIN ] 学習を始めました (Pod の中で setsid nohup. Mac とのつながりが切れても続く)
         ログ /workspace/logs/train_trim_corr_v5.log  GPU /workspace/logs/gpu_train.csv (30 秒おき)
    (6 秒)

=== 6. 学習を見守る (5 分おき)  (19:57:02)
[TRAIN ] 19:57  step -/1500  loss -  経過 0:00  上限まで 4:54

========================================================================
 学習は Pod の中で回っています. ここまで来れば寝てかまいません.
 このターミナルは閉じない・Mac のふたは閉じない (電源につないだまま).
 終わると通知が出ます. まとめは /Volumes/KIOXIA/r2r2r-data/lab/Log22/train/result_*.txt
========================================================================

[TRAIN ] 20:02  step 0/1500  loss 0.0129 (Step 0 の行)  経過 0:05  上限まで 4:49
[TRAIN ] 20:07  step 44/1500  loss 0.0129 (Step 0 の行)  経過 0:10  上限まで 4:44
[TRAIN ] 20:12  step 88/1500  loss 0.0066 (Step 50 の行)  経過 0:15  上限まで 4:39
[TRAIN ] 20:17  step 130/1500  loss 0.0048 (Step 100 の行)  経過 0:20  上限まで 4:34
[TRAIN ] 20:22  step 174/1500  loss 0.0051 (Step 150 の行)  経過 0:25  上限まで 4:29
[TRAIN ] 20:27  step 216/1500  loss 0.0052 (Step 200 の行)  経過 0:30  上限まで 4:24
[TRAIN ] 20:32  step 260/1500  loss 0.0045 (Step 250 の行)  経過 0:35  上限まで 4:19
[TRAIN ] 20:37  step 300/1500  loss 0.0044 (Step 300 の行)  経過 0:40  上限まで 4:14
[TRAIN ] 20:42  step 346/1500  loss 0.0044 (Step 300 の行)  経過 0:45  上限まで 4:09
[TRAIN ] 20:47  step 388/1500  loss 0.0042 (Step 350 の行)  経過 0:50  上限まで 4:04
[TRAIN ] 20:52  step 430/1500  loss 0.0039 (Step 400 の行)  経過 0:55  上限まで 3:59
[TRAIN ] 20:57  step 474/1500  loss 0.0043 (Step 450 の行)  経過 1:00  上限まで 3:54
[TRAIN ] 21:02  step 516/1500  loss 0.0041 (Step 500 の行)  経過 1:05  上限まで 3:49
[TRAIN ] 21:07  step 560/1500  loss 0.0043 (Step 550 の行)  経過 1:10  上限まで 3:44
[TRAIN ] 21:12  step 600/1500  loss 0.0040 (Step 600 の行)  経過 1:15  上限まで 3:38
[TRAIN ] 21:17  step 646/1500  loss 0.0040 (Step 600 の行)  経過 1:20  上限まで 3:33
[TRAIN ] 21:22  step 688/1500  loss 0.0039 (Step 650 の行)  経過 1:25  上限まで 3:28
[TRAIN ] 21:27  step 732/1500  loss 0.0039 (Step 700 の行)  経過 1:30  上限まで 3:23
[TRAIN ] 21:33  step 774/1500  loss 0.0036 (Step 750 の行)  経過 1:35  上限まで 3:18
[TRAIN ] 21:38  step 818/1500  loss 0.0034 (Step 800 の行)  経過 1:41  上限まで 3:13
[TRAIN ] 21:43  step 860/1500  loss 0.0033 (Step 850 の行)  経過 1:46  上限まで 3:08
[TRAIN ] 21:48  step 904/1500  loss 0.0035 (Step 900 の行)  経過 1:51  上限まで 3:03
[TRAIN ] 21:53  step 948/1500  loss 0.0035 (Step 900 の行)  経過 1:56  上限まで 2:58
[TRAIN ] 21:58  step 990/1500  loss 0.0033 (Step 950 の行)  経過 2:01  上限まで 2:53
[TRAIN ] 22:03  step 1030/1500  loss 0.0034 (Step 1000 の行)  経過 2:06  上限まで 2:48
[TRAIN ] 22:08  step 1080/1500  loss 0.0028 (Step 1050 の行)  経過 2:11  上限まで 2:43
[TRAIN ] 22:13  step 1120/1500  loss 0.0037 (Step 1100 の行)  経過 2:16  上限まで 2:38
[TRAIN ] 22:18  step 1160/1500  loss 0.0029 (Step 1150 の行)  経過 2:21  上限まで 2:33
[TRAIN ] 22:23  step 1200/1500  loss 0.0033 (Step 1200 の行)  経過 2:26  上限まで 2:28
[TRAIN ] 22:28  step 1250/1500  loss 0.0033 (Step 1200 の行)  経過 2:31  上限まで 2:23
[TRAIN ] 22:33  step 1290/1500  loss 0.0027 (Step 1250 の行)  経過 2:36  上限まで 2:18
[TRAIN ] 22:38  step 1330/1500  loss 0.0025 (Step 1300 の行)  経過 2:41  上限まで 2:12
[TRAIN ] 22:43  step 1380/1500  loss 0.0031 (Step 1350 の行)  経過 2:46  上限まで 2:07
[TRAIN ] 22:48  step 1420/1500  loss 0.0028 (Step 1400 の行)  経過 2:51  上限まで 2:02
[TRAIN ] 22:53  step 1460/1500  loss 0.0030 (Step 1450 の行)  経過 2:56  上限まで 1:57
[TRAIN ] 22:58  step 1500/1500  loss 0.0030 (Step 1450 の行)  経過 3:01  上限まで 1:52
    (10917 秒)

[TRAIN ] 学習が終わりました (最後のチェックポイント 1499, loss 0.0030)
[WATCH ] Pod 側の見張りを 01:58 まで延ばした (上げる・持ち帰る間に消えないように)

=== 7a. Pod 側の一覧 (1499)  (22:59:02)
[LIST  ] 35 ファイル 6335848626 バイト (train_state なし) → /Volumes/KIOXIA/r2r2r-data/lab/Log22/train/pod_files_1499.tsv
    (2 秒)

=== 7b. Hugging Face へ上げる (PixelstoMotors/pi05-mecharm-trim-corr-lora-v3・非公開)  (22:59:04)
[HF    ] 上げるもの: 35 ファイル 6,335,848,626 バイト (train_state/ は上げない)
HFUP ok PixelstoMotors/pi05-mecharm-trim-corr-lora-v3 に 35 ファイル 6,335,848,626 バイトを上げ, 名前・バイト数が全部合うのを確かめた
    (14 秒)

=== 7d. ログを持ち帰る  (22:59:18)
[LOG   ] train_trim_corr_v5.log を持ち帰った → /Volumes/KIOXIA/r2r2r-data/lab/Log22/train/train_trim_corr_v5.log (981 行)
[LOG   ] gpu_train.csv を持ち帰った → /Volumes/KIOXIA/r2r2r-data/lab/Log22/train/gpu_train.csv (367 行)
[LOG   ] train_run.log を持ち帰った → /Volumes/KIOXIA/r2r2r-data/lab/Log22/train/train_run.log (2 行)
[LOG   ] prep_trim_corr.log を持ち帰った → /Volumes/KIOXIA/r2r2r-data/lab/Log22/train/prep_trim_corr.log (100 行)
[LOG   ] watchdog.log は Pod にない (持ち帰れなかった)
[LOG   ] Pod 側のチェックポイントの一覧 → /Volumes/KIOXIA/r2r2r-data/lab/Log22/train/pod_checkpoints.tsv (122 ファイル)
    (28 秒)

=== Pod を消す (id uirn5j6iy4xbyx)  (22:59:46)
[STOP  ] 削除 (Terminate) を送った (22:59:48)
[STOP  ] Pod (id uirn5j6iy4xbyx) が消えたことを API で確かめた
[COST  ] 借りていた時間 188 分 2 秒 (3.13 時間) × 1時間 0.59 ドル = 約 1.85 ドル (約 292 円. 1ドル=158円)  GPU=NVIDIA A40

=== 7e. KIOXIA へ Hugging Face から取る  (22:59:48)
[KIOXIA] Hugging Face (PixelstoMotors/pi05-mecharm-trim-corr-lora-v3) から取ります → /Volumes/KIOXIA/r2r2r-data/checkpoints/pi05_mecharm_trim_corr_lora_v3/.incoming_1499_20261106_225949_7138
Still waiting to acquire lock on /Volumes/KIOXIA/r2r2r-data/checkpoints/pi05_mecharm_trim_corr_lora_v3/.incoming_1499_20261106_225949_7138/.cache/huggingface/.gitignore.lock (elapsed: 0.1 seconds)
Still waiting to acquire lock on /Volumes/KIOXIA/r2r2r-data/checkpoints/pi05_mecharm_trim_corr_lora_v3/.incoming_1499_20261106_225949_7138/.cache/huggingface/.gitignore.lock (elapsed: 0.1 seconds)
path=/Volumes/KIOXIA/r2r2r-data/checkpoints/pi05_mecharm_trim_corr_lora_v3/.incoming_1499_20261106_225949_7138
[KIOXIA] /Volumes/KIOXIA/r2r2r-data/checkpoints/pi05_mecharm_trim_corr_lora_v3/1499 に置いた (35 ファイル 6335848626 バイト. Pod 側と名前・バイト数が全部合った)
    (254 秒)

=== 由来 (train_provenance.json) を書く
[PROV  ] /Volumes/KIOXIA/r2r2r-data/lab/Log22/train/train_provenance.json
[PROV  ] /Volumes/KIOXIA/r2r2r-data/checkpoints/pi05_mecharm_trim_corr_lora_v3/1499/train_provenance.json
[PROV  ] hf: url=https://huggingface.co/PixelstoMotors/pi05-mecharm-trim-corr-lora-v3/commit/b8a5ccbc68c8ac6dadf30a4d074ee66bfadee9c9
[PROV  ] 由来: /Volumes/KIOXIA/r2r2r-data/lab/Log22/train/train_provenance.json ・ /Volumes/KIOXIA/r2r2r-data/checkpoints/pi05_mecharm_trim_corr_lora_v3/1499/train_provenance.json ・ Hugging Face (PixelstoMotors/pi05-mecharm-trim-corr-lora-v3) にも置いた

=== かかった時間
  0. 送るものと Mac の道具を確かめる      1 秒
  1. Pod を作る                          2 秒
  2. Pod が立ち上がるまで待つ     37 秒
  3a. スクリプトを送る             52 秒
  3b. huggingface_hub を入れ, 鍵を渡す     10 秒
  3c. 続きから学習するモデルを取る (PixelstoMotors/pi05-mecharm-trim-corr-lora-v2 → /workspace/ckpt/corr_v2_1499)     17 秒
  3d. openpi を取る (215abfb)           16 秒
  3e. uv sync                              20 秒
  3f. addon・設定・assets               7 秒
  4a. tar を送る                       103 秒
  4b. pod_prep_trim_corr.sh 1500 /workspace/ckpt/corr_v2_1499/params v5     48 秒
  5. 学習を始める                     6 秒
  6. 学習を見守る (5 分おき)   10917 秒
  7a. Pod 側の一覧 (1499)               2 秒
  7b. Hugging Face へ上げる (PixelstoMotors/pi05-mecharm-trim-corr-lora-v3・非公開)     14 秒
  7d. ログを持ち帰る                28 秒
  7e. KIOXIA へ Hugging Face から取る    254 秒

[RESULT] 成功: 学習が終わりました (step 1499, loss 0.0030). Hugging Face と KIOXIA に置き, Pod を消しました. 188 分 2 秒 (3.13 時間) × 1時間 0.59 ドル = 約 1.85 ドル (約 292 円. 1ドル=158円)
[RESULT] まとめ → /Volumes/KIOXIA/r2r2r-data/lab/Log22/train/result_20261106_195143.txt