[LOG ] ターミナルの出力を残します → /Volumes/KIOXIA/r2r2r-data/lab/Log27/train/lab_train_20261109_153622.txt [LAB ] 2026-11-09 15:36:22 学習 GPU=NVIDIA A40 (借りられなければ NVIDIA RTX A6000 NVIDIA L40S) --model deep1 (π0.5 base → PixelstoMotors/pi05-mecharm-deep-even10-lora-v1) データ mecharm_redblock_box_deep_even10_v1 pi05_mecharm_deep_even10_lora --exp-name=deep_even10_v1 steps=3000 init=base 上限 7 時間 記録 /Volumes/KIOXIA/r2r2r-data/lab/Log27/train === 0. 送るものと Mac の道具を確かめる (15:36:22) [DATA ] tar の sha256 c7097302c784… (由来に残す) [DATA ] tar 193353728 バイト (/Volumes/KIOXIA/r2r2r-data/datasets/mecharm_redblock_box_deep_even10_v1/mecharm_redblock_box_deep_even10_v1_raw_episodes.tar) [PROV ] 始める時点のコード (8db7028) と入力 3 つの指紋を測った [MAC ] 終わるまでスリープしないようにした (caffeinate pid 8863). ふたは閉じない (1 秒) === 1. Pod を作る (15:36:23) [POD ] NVIDIA A40 を借ります (空きがなければ 30 秒おきに 5 回まで) [POD ] 作った: id arn304g7xu5wfk GPU=NVIDIA A40 1時間 0.59 ドル (記録 /Volumes/KIOXIA/r2r2r-data/lab/Log27/train/lab_pod.json) [POD ] 上限: 11-09 22:36 (7 時間) を過ぎたら, ログを持ち帰って消します (3 秒) === 2. Pod が立ち上がるまで待つ (15:36:26) [POD ] SSH の IP とポートを API から受け取った (:) [POD ] 入れた: 59541e1c2271 NVIDIA A40, 46068 MiB [WATCH ] Pod 側の見張りを置いた: 11-09 23:36 を過ぎたら Pod が自分を消す (Pod 用の鍵. Mac の鍵は送っていない) (57 秒) === 3a. スクリプトを送る (15:37:23) (75 秒) === 3b. huggingface_hub を入れ, 鍵を渡す (15:38:38) WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager, possibly rendering your system unusable. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv. Use the --root-user-action option if you know what you are doing and want to suppress this warning. Hint: The `hf-cli` skill is not installed. Run `hf skills add -g` to teach your AI agents how to use the `hf` CLI. ✓ Logged in user: PixelstoMotors (10 秒) === 3c. 続きの元は取らない (π0.5 base から. 重みは学習の最初に openpi が取る) (15:38:48) (0 秒) === 3d. openpi を取る (215abfb) (15:38:48) 215abfb docs(droid): fix config search instruction (#1023) (20 秒) === 3e. uv sync (15:39:08) downloading uv 0.13.7 x86_64-unknown-linux-gnu WARN: The following commands are shadowed by other commands in your PATH: uv uvx Using CPython 3.11.13 interpreter at: /usr/bin/python3.11 Creating virtual environment at: .venv Resolved 279 packages in 4ms Building openpi-client @ file:///workspace/openpi/packages/openpi-client Updating https://github.com/huggingface/lerobot (0cf864870cf29f4738d3ade893e6fd13fbd7cdb5) Building openpi @ file:///workspace/openpi Downloading hf-transfer (3.4MiB) Downloading torchvision (7.1MiB) Downloading imageio-ffmpeg (28.1MiB) Downloading opencv-python (60.1MiB) Downloading virtualenv (5.8MiB) Downloading pyarrow (40.3MiB) Downloading matplotlib (8.2MiB) Downloading h5py (4.3MiB) Downloading diffusers (3.4MiB) Downloading nvidia-nvjitlink-cu12 (18.8MiB) Downloading lxml (4.7MiB) Downloading opencv-python-headless (47.7MiB) Downloading polars (34.6MiB) Downloading cmake (26.6MiB) Downloading hf-xet (4.9MiB) Downloading pillow (4.4MiB) Downloading nvidia-cuda-nvrtc-cu12 (22.6MiB) Downloading dm-control (37.5MiB) Downloading ruff (11.0MiB) Downloading nvidia-cublas-cu12 (374.9MiB) Downloading nvidia-curand-cu12 (53.7MiB) Downloading mujoco (4.4MiB) Downloading nvidia-cuda-cupti-cu12 (8.5MiB) Downloading nvidia-nccl-cu12 (192.0MiB) Downloading nvidia-cusparselt-cu12 (149.5MiB) Downloading nvidia-cusparse-cu12 (206.5MiB) Downloading nvidia-cusolver-cu12 (150.9MiB) Downloading pyopengl (3.0MiB) Downloading wandb (20.4MiB) Downloading fonttools (4.7MiB) Downloading numpy (17.4MiB) Downloading labmaze (4.7MiB) Downloading numcodecs (8.4MiB) Downloading nvidia-cufft-cu12 (190.9MiB) Downloading rerun-sdk (68.1MiB) Downloading numba (3.7MiB) Downloading transformers (10.3MiB) Downloading jaxlib (100.2MiB) Downloading pandas (12.5MiB) Downloading nvidia-cuda-nvcc-cu12 (38.7MiB) Downloading jax-cuda12-plugin (15.9MiB) Downloading jax-cuda12-pjrt (99.8MiB) Downloading av (34.7MiB) Downloading triton (148.5MiB) Downloading llvmlite (40.4MiB) Downloading torch (783.1MiB) Downloading scipy (35.9MiB) Downloading tensorstore (17.5MiB) Downloading nvidia-cudnn-cu12 (544.5MiB) Downloading sympy (6.0MiB) Downloaded pyopengl Downloading debugpy (3.0MiB) Downloaded diffusers Downloaded hf-transfer Downloading tokenizers (2.9MiB) Downloading jax (2.3MiB) Downloaded numba Downloading widgetsnbextension (2.1MiB) Downloaded h5py Downloading ml-dtypes (2.1MiB) Downloaded pillow Downloaded mujoco Downloading networkx (1.9MiB) Downloading pydantic-core (1.9MiB) Downloaded fonttools Downloaded labmaze Downloaded lxml Downloading aiohttp (1.6MiB) Downloading jedi (1.5MiB) Downloading kiwisolver (1.4MiB) Downloaded hf-xet Downloading torchcodec (1.3MiB) Downloaded virtualenv Downloading sentencepiece (1.2MiB) Downloaded jax Downloading pygments (1.2MiB) Downloaded widgetsnbextension Downloading setuptools (1.1MiB) Downloaded sympy Downloaded debugpy Downloading pymunk (1.0MiB) Downloading nvidia-cufile-cu12 (1.1MiB) Downloaded kiwisolver Downloaded tokenizers Downloaded torchcodec Downloaded ml-dtypes Downloaded aiohttp Downloaded networkx Downloaded pydantic-core Downloaded jedi Downloaded torchvision Downloaded sentencepiece Downloaded pygments Downloaded setuptools Downloaded pymunk Downloaded nvidia-cufile-cu12 Building evdev==1.9.2 Downloaded nvidia-cuda-cupti-cu12 Downloaded matplotlib Downloaded numcodecs Building antlr4-python3-runtime==4.9.3 Downloaded transformers Downloaded ruff Downloaded pandas Built openpi-client @ file:///workspace/openpi/packages/openpi-client Built openpi @ file:///workspace/openpi Downloaded jax-cuda12-plugin Built antlr4-python3-runtime==4.9.3 Downloaded numpy Downloaded tensorstore Downloaded nvidia-nvjitlink-cu12 Downloaded wandb Downloaded nvidia-cuda-nvrtc-cu12 Downloaded imageio-ffmpeg Downloaded cmake Built evdev==1.9.2 Downloaded polars Downloaded av Downloaded scipy Downloaded nvidia-cuda-nvcc-cu12 Downloaded dm-control Downloaded pyarrow Downloaded llvmlite Updated https://github.com/huggingface/lerobot (0cf864870cf29f4738d3ade893e6fd13fbd7cdb5) Building lerobot @ git+https://github.com/huggingface/lerobot@0cf864870cf29f4738d3ade893e6fd13fbd7cdb5 Downloaded opencv-python-headless Downloaded nvidia-curand-cu12 Downloaded opencv-python Built lerobot @ git+https://github.com/huggingface/lerobot@0cf864870cf29f4738d3ade893e6fd13fbd7cdb5 Downloaded rerun-sdk Downloaded jax-cuda12-pjrt Downloaded jaxlib Downloaded nvidia-cusparselt-cu12 Downloaded triton Downloaded nvidia-cusolver-cu12 Downloaded nvidia-nccl-cu12 Downloaded nvidia-cufft-cu12 Downloaded nvidia-cusparse-cu12 Downloaded nvidia-cublas-cu12 Downloaded nvidia-cudnn-cu12 Downloaded torch Prepared 240 packages in 15.21s Installed 240 packages in 1.87s + absl-py==2.3.0 + aiohappyeyeballs==2.6.1 + aiohttp==3.12.4 + aiosignal==1.3.2 + annotated-types==0.7.0 + antlr4-python3-runtime==4.9.3 + asttokens==3.0.0 + attrs==25.3.0 + augmax==0.4.1 + av==17.0.0 + beartype==0.19.0 + beautifulsoup4==4.13.4 + blinker==1.9.0 + cachetools==5.5.2 + certifi==2025.4.26 + cffi==1.17.1 + cfgv==3.4.0 + charset-normalizer==3.4.2 + chex==0.1.90 + click==8.2.1 + cloudpickle==3.1.1 + cmake==4.0.2 + comm==0.2.2 + contourpy==1.3.2 + crc32c==2.7.1 + cycler==0.12.1 + datasets==3.6.0 + debugpy==1.8.14 + decorator==5.2.1 + deepdiff==8.5.0 + diffusers==0.33.1 + dill==0.3.8 + distlib==0.3.9 + dm-control==1.0.14 + dm-env==1.6 + dm-tree==0.1.9 + docker-pycreds==0.4.0 + docstring-parser==0.16 + donfig==0.8.1.post1 + draccus==0.10.0 + einops==0.8.1 + equinox==0.12.2 + etils==1.12.2 + evdev==1.9.2 + executing==2.2.0 + farama-notifications==0.0.4 + filelock==3.18.0 + flask==3.1.1 + flatbuffers==25.2.10 + flax==0.10.2 + fonttools==4.58.1 + frozenlist==1.6.0 + fsspec==2025.3.0 + gcsfs==2025.3.0 + gdown==5.2.0 + gitdb==4.0.12 + gitpython==3.1.44 + glfw==2.9.0 + google-api-core==2.24.2 + google-auth==2.40.2 + google-auth-oauthlib==1.2.2 + google-cloud-core==2.4.3 + google-cloud-storage==3.1.0 + google-crc32c==1.7.1 + google-resumable-media==2.7.2 + googleapis-common-protos==1.70.0 + gym-aloha==0.1.1 + gymnasium==0.29.1 + h5py==3.13.0 + hf-transfer==0.1.9 + hf-xet==1.1.2 + huggingface-hub==0.32.3 + humanize==4.12.3 + identify==2.6.12 + idna==3.10 + imageio==2.37.0 + imageio-ffmpeg==0.6.0 + importlib-metadata==8.7.0 + importlib-resources==6.5.2 + iniconfig==2.1.0 + inquirerpy==0.3.4 + ipykernel==6.29.5 + ipython==9.2.0 + ipython-pygments-lexers==1.1.1 + ipywidgets==8.1.7 + itsdangerous==2.2.0 + jax==0.5.3 + jax-cuda12-pjrt==0.5.3 + jax-cuda12-plugin==0.5.3 + jaxlib==0.5.3 + jaxtyping==0.2.36 + jedi==0.19.2 + jinja2==3.1.6 + jsonlines==4.0.0 + jupyter-client==8.6.3 + jupyter-core==5.8.1 + jupyterlab-widgets==3.0.15 + kiwisolver==1.4.8 + labmaze==1.0.6 + lerobot==0.1.0 (from git+https://github.com/huggingface/lerobot@0cf864870cf29f4738d3ade893e6fd13fbd7cdb5) + llvmlite==0.44.0 + lxml==5.4.0 + markdown-it-py==3.0.0 + markupsafe==3.0.2 + matplotlib==3.10.3 + matplotlib-inline==0.1.7 + mdurl==0.1.2 + mergedeep==1.3.4 + ml-collections==1.0.0 + ml-dtypes==0.4.1 + mpmath==1.3.0 + msgpack==1.1.0 + mujoco==2.3.7 + multidict==6.4.4 + multiprocess==0.70.16 + mypy-extensions==1.1.0 + nest-asyncio==1.6.0 + networkx==3.5 + nodeenv==1.9.1 + numba==0.61.2 + numcodecs==0.16.1 + numpy==1.26.4 + numpydantic==1.6.9 + nvidia-cublas-cu12==12.6.4.1 + nvidia-cuda-cupti-cu12==12.6.80 + nvidia-cuda-nvcc-cu12==12.9.41 + nvidia-cuda-nvrtc-cu12==12.6.77 + nvidia-cuda-runtime-cu12==12.6.77 + nvidia-cudnn-cu12==9.5.1.17 + nvidia-cufft-cu12==11.3.0.4 + nvidia-cufile-cu12==1.11.1.6 + nvidia-curand-cu12==10.3.7.77 + nvidia-cusolver-cu12==11.7.1.2 + nvidia-cusparse-cu12==12.5.4.2 + nvidia-cusparselt-cu12==0.6.3 + nvidia-ml-py==12.575.51 + nvidia-nccl-cu12==2.26.2 + nvidia-nvjitlink-cu12==12.6.85 + nvidia-nvtx-cu12==12.6.77 + oauthlib==3.2.2 + omegaconf==2.3.0 + opencv-python==4.11.0.86 + opencv-python-headless==4.11.0.86 + openpi==0.1.0 (from file:///workspace/openpi) + openpi-client==0.1.0 (from file:///workspace/openpi/packages/openpi-client) + opt-einsum==3.4.0 + optax==0.2.4 + orbax-checkpoint==0.11.13 + orderly-set==5.4.1 + packaging==25.0 + pandas==2.2.3 + parso==0.8.4 + pexpect==4.9.0 + pfzy==0.3.4 + pillow==11.2.1 + platformdirs==4.3.8 + pluggy==1.6.0 + polars==1.30.0 + pre-commit==4.2.0 + prompt-toolkit==3.0.51 + propcache==0.3.1 + proto-plus==1.26.1 + protobuf==4.25.8 + psutil==7.0.0 + ptyprocess==0.7.0 + pure-eval==0.2.3 + pyarrow==20.0.0 + pyasn1==0.6.1 + pyasn1-modules==0.4.2 + pycparser==2.22 + pydantic==2.11.5 + pydantic-core==2.33.2 + pygments==2.19.1 + pymunk==7.0.0 + pynput==1.8.1 + pynvml==12.0.0 + pyopengl==3.1.9 + pyparsing==3.2.3 + pysocks==1.7.1 + pytest==8.3.5 + python-dateutil==2.9.0.post0 + python-xlib==0.33 + pytz==2025.2 + pyyaml==6.0.2 + pyyaml-include==1.4.1 + pyzmq==26.4.0 + regex==2024.11.6 + requests==2.32.3 + requests-oauthlib==2.0.0 + rerun-sdk==0.23.1 + rich==14.0.0 + rsa==4.9.1 + ruff==0.11.12 + safetensors==0.5.3 + scipy==1.15.3 + sentencepiece==0.2.0 + sentry-sdk==2.29.1 + setproctitle==1.3.6 + setuptools==80.9.0 + shtab==1.7.2 + simplejson==3.20.1 + six==1.17.0 + smmap==5.0.2 + soupsieve==2.7 + stack-data==0.6.3 + sympy==1.14.0 + tensorstore==0.1.74 + termcolor==3.1.0 + tokenizers==0.21.1 + toml==0.10.2 + toolz==1.0.0 + torch==2.7.1 + torchcodec==0.4.0 + torchvision==0.22.1 + tornado==6.5.1 + tqdm==4.67.1 + tqdm-loggable==0.2 + traitlets==5.14.3 + transformers==4.53.2 + treescope==0.1.9 + triton==3.3.1 + typeguard==4.4.2 + typing-extensions==4.13.2 + typing-inspect==0.9.0 + typing-inspection==0.4.1 + tyro==0.9.22 + tzdata==2025.2 + urllib3==2.4.0 + virtualenv==20.31.2 + wadler-lindig==0.1.6 + wandb==0.19.11 + wcwidth==0.2.13 + websockets==15.0.1 + werkzeug==3.1.3 + widgetsnbextension==4.0.14 + wrapt==1.14.1 + xxhash==3.5.0 + yarl==1.20.0 + zarr==3.0.8 + zipp==3.22.0 (21 秒) === 3f. addon・設定・assets (15:39:29) installed: policies/mecharm_policy.py + config.py (pi05_mecharm_lora) installed: pi05_mecharm_trim_lora (repo_id=local/mecharm_redblock_box_teach_v1_trim2, 4,000 steps) [PROV ] openpi 215abfb217dbac7d5f1273282331b9b1866c0479 (2 ファイル変更・addon と設定を入れたあと) (7 秒) === 4a. tar を送る (15:39:36) -rwx------ 1 root root 193353728 Nov 9 06:41 /workspace/data/mecharm_redblock_box_deep_even10_v1_raw_episodes.tar (133 秒) === 4b. pod_prep_deep.sh 3000 (変換・正規化の値を測り直す) (15:41:49) == 1. tar を広げる → /workspace/data/mecharm_redblock_box_deep_even10_v1 エピソード 50 本 (Mac で作ったときは 50 本) == 2. 学習設定 pi05_mecharm_deep_even10_lora を足す (steps=3000. π0.5 base から) installed: pi05_mecharm_deep_even10_lora (repo_id=local/mecharm_redblock_box_deep_even10_v1, 3000 steps, init=π0.5 base) == 3. LeRobot 形式に変換 (local/mecharm_redblock_box_deep_even10_v1) Map: 0%| | 0/118 [00:00&1 | tee /workspace/logs/train_deep_even10_v1.log [NORM ] 正規化の値 (sha256 813fcb8cccd1…) を持ち帰った → /Volumes/KIOXIA/r2r2r-data/lab/Log27/train/norm_stats.json (158 秒) === 5. 学習を始める (15:44:27) [TRAIN ] 学習を始めました (Pod の中で setsid nohup. Mac とのつながりが切れても続く) ログ /workspace/logs/train_deep_even10_v1.log GPU /workspace/logs/gpu_train.csv (30 秒おき) (6 秒) === 6. 学習を見守る (5 分おき) (15:44:33) [TRAIN ] 15:44 step -/3000 loss - 経過 0:00 上限まで 6:51 ======================================================================== 学習は Pod の中で回っています. ここまで来れば寝てかまいません. このターミナルは閉じない・Mac のふたは閉じない (電源につないだまま). 終わると通知が出ます. まとめは /Volumes/KIOXIA/r2r2r-data/lab/Log27/train/result_*.txt ======================================================================== [TRAIN ] 15:49 step -/3000 loss - 経過 0:05 上限まで 6:46 [TRAIN ] 15:54 step 34/3000 loss 0.1015 (Step 0 の行) 経過 0:10 上限まで 6:41 [TRAIN ] 15:59 step 78/3000 loss 0.0611 (Step 50 の行) 経過 0:15 上限まで 6:36 [TRAIN ] 16:04 step 120/3000 loss 0.0422 (Step 100 の行) 経過 0:20 上限まで 6:31 [TRAIN ] 16:09 step 162/3000 loss 0.0296 (Step 150 の行) 経過 0:25 上限まで 6:26 [TRAIN ] 16:14 step 206/3000 loss 0.0213 (Step 200 の行) 経過 0:30 上限まで 6:21 [TRAIN ] 16:19 step 248/3000 loss 0.0213 (Step 200 の行) 経過 0:35 上限まで 6:16 [TRAIN ] 16:24 step 292/3000 loss 0.0186 (Step 250 の行) 経過 0:40 上限まで 6:11 [TRAIN ] 16:30 step 334/3000 loss 0.0169 (Step 300 の行) 経過 0:45 上限まで 6:06 [TRAIN ] 16:35 step 376/3000 loss 0.0143 (Step 350 の行) 経過 0:50 上限まで 6:01 [TRAIN ] 16:40 step 420/3000 loss 0.0124 (Step 400 の行) 経過 0:55 上限まで 5:56 [TRAIN ] 16:45 step 462/3000 loss 0.0126 (Step 450 の行) 経過 1:00 上限まで 5:51 [TRAIN ] 16:50 step 506/3000 loss 0.0113 (Step 500 の行) 経過 1:05 上限まで 5:46 [TRAIN ] 16:55 step 548/3000 loss 0.0113 (Step 500 の行) 経過 1:10 上限まで 5:41 [TRAIN ] 17:00 step 590/3000 loss 0.0116 (Step 550 の行) 経過 1:15 上限まで 5:36 [TRAIN ] 17:05 step 634/3000 loss 0.0111 (Step 600 の行) 経過 1:20 上限まで 5:31 [TRAIN ] 17:10 step 676/3000 loss 0.0100 (Step 650 の行) 経過 1:25 上限まで 5:26 [TRAIN ] 17:15 step 720/3000 loss 0.0086 (Step 700 の行) 経過 1:30 上限まで 5:21 [TRAIN ] 17:20 step 762/3000 loss 0.0089 (Step 750 の行) 経過 1:35 上限まで 5:15 [TRAIN ] 17:25 step 806/3000 loss 0.0083 (Step 800 の行) 経過 1:40 上限まで 5:10 [TRAIN ] 17:30 step 848/3000 loss 0.0083 (Step 800 の行) 経過 1:46 上限まで 5:05 [TRAIN ] 17:35 step 890/3000 loss 0.0083 (Step 850 の行) 経過 1:51 上限まで 5:00 [TRAIN ] 17:40 step 934/3000 loss 0.0080 (Step 900 の行) 経過 1:56 上限まで 4:55 [TRAIN ] 17:45 step 976/3000 loss 0.0075 (Step 950 の行) 経過 2:01 上限まで 4:50 [TRAIN ] 17:50 step 1020/3000 loss 0.0072 (Step 1000 の行) 経過 2:06 上限まで 4:45 [TRAIN ] 17:55 step 1060/3000 loss 0.0069 (Step 1050 の行) 経過 2:11 上限まで 4:40 [TRAIN ] 18:00 step 1100/3000 loss 0.0075 (Step 1100 の行) 経過 2:16 上限まで 4:35 [TRAIN ] 18:05 step 1150/3000 loss 0.0075 (Step 1100 の行) 経過 2:21 上限まで 4:30 [TRAIN ] 18:10 step 1190/3000 loss 0.0066 (Step 1150 の行) 経過 2:26 上限まで 4:25 [TRAIN ] 18:16 step 1230/3000 loss 0.0066 (Step 1200 の行) 経過 2:31 上限まで 4:20 [TRAIN ] 18:21 step 1280/3000 loss 0.0062 (Step 1250 の行) 経過 2:36 上限まで 4:15 [TRAIN ] 18:26 step 1320/3000 loss 0.0064 (Step 1300 の行) 経過 2:41 上限まで 4:10 [TRAIN ] 18:31 step 1360/3000 loss 0.0063 (Step 1350 の行) 経過 2:46 上限まで 4:05 [TRAIN ] 18:36 step 1400/3000 loss 0.0061 (Step 1400 の行) 経過 2:51 上限まで 4:00 [TRAIN ] 18:41 step 1450/3000 loss 0.0061 (Step 1400 の行) 経過 2:56 上限まで 3:55 [TRAIN ] 18:46 step 1490/3000 loss 0.0058 (Step 1450 の行) 経過 3:01 上限まで 3:50 [TRAIN ] 18:51 step 1530/3000 loss 0.0058 (Step 1500 の行) 経過 3:06 上限まで 3:45 [TRAIN ] 18:56 step 1580/3000 loss 0.0059 (Step 1550 の行) 経過 3:11 上限まで 3:40 [TRAIN ] 19:01 step 1620/3000 loss 0.0058 (Step 1600 の行) 経過 3:16 上限まで 3:35 [TRAIN ] 19:06 step 1660/3000 loss 0.0058 (Step 1650 の行) 経過 3:21 上限まで 3:29 [TRAIN ] 19:11 step 1700/3000 loss 0.0056 (Step 1700 の行) 経過 3:26 上限まで 3:24 [TRAIN ] 19:16 step 1750/3000 loss 0.0056 (Step 1700 の行) 経過 3:32 上限まで 3:19 [TRAIN ] 19:21 step 1790/3000 loss 0.0054 (Step 1750 の行) 経過 3:37 上限まで 3:14 [TRAIN ] 19:26 step 1830/3000 loss 0.0056 (Step 1800 の行) 経過 3:42 上限まで 3:09 [TRAIN ] 19:31 step 1880/3000 loss 0.0055 (Step 1850 の行) 経過 3:47 上限まで 3:04 [TRAIN ] 19:36 step 1920/3000 loss 0.0053 (Step 1900 の行) 経過 3:52 上限まで 2:59 [TRAIN ] 19:41 step 1960/3000 loss 0.0053 (Step 1950 の行) 経過 3:57 上限まで 2:54 [TRAIN ] 19:46 step 2000/3000 loss 0.0048 (Step 2000 の行) 経過 4:02 上限まで 2:49 [TRAIN ] 19:51 step 2050/3000 loss 0.0048 (Step 2000 の行) 経過 4:07 上限まで 2:44 [TRAIN ] 19:56 step 2090/3000 loss 0.0055 (Step 2050 の行) 経過 4:12 上限まで 2:39 [TRAIN ] 20:02 step 2130/3000 loss 0.0047 (Step 2100 の行) 経過 4:17 上限まで 2:34 [TRAIN ] 20:07 step 2180/3000 loss 0.0046 (Step 2150 の行) 経過 4:22 上限まで 2:29 [TRAIN ] 20:12 step 2220/3000 loss 0.0047 (Step 2200 の行) 経過 4:27 上限まで 2:24 [TRAIN ] 20:17 step 2260/3000 loss 0.0046 (Step 2250 の行) 経過 4:32 上限まで 2:19 [TRAIN ] 20:22 step 2300/3000 loss 0.0047 (Step 2300 の行) 経過 4:37 上限まで 2:14 [TRAIN ] 20:27 step 2350/3000 loss 0.0047 (Step 2300 の行) 経過 4:42 上限まで 2:09 [TRAIN ] 20:32 step 2390/3000 loss 0.0041 (Step 2350 の行) 経過 4:47 上限まで 2:04 [TRAIN ] 20:37 step 2430/3000 loss 0.0044 (Step 2400 の行) 経過 4:52 上限まで 1:59 [TRAIN ] 20:42 step 2480/3000 loss 0.0043 (Step 2450 の行) 経過 4:57 上限まで 1:54 [TRAIN ] 20:47 step 2520/3000 loss 0.0043 (Step 2500 の行) 経過 5:02 上限まで 1:48 [TRAIN ] 20:52 step 2560/3000 loss 0.0042 (Step 2550 の行) 経過 5:07 上限まで 1:43 [TRAIN ] 20:57 step 2600/3000 loss 0.0046 (Step 2600 の行) 経過 5:13 上限まで 1:38 [TRAIN ] 21:02 step 2650/3000 loss 0.0046 (Step 2600 の行) 経過 5:18 上限まで 1:33 [TRAIN ] 21:07 step 2690/3000 loss 0.0045 (Step 2650 の行) 経過 5:23 上限まで 1:28 [TRAIN ] 21:12 step 2730/3000 loss 0.0042 (Step 2700 の行) 経過 5:28 上限まで 1:23 [TRAIN ] 21:17 step 2780/3000 loss 0.0041 (Step 2750 の行) 経過 5:33 上限まで 1:18 [TRAIN ] 21:22 step 2820/3000 loss 0.0038 (Step 2800 の行) 経過 5:38 上限まで 1:13 [TRAIN ] 21:27 step 2860/3000 loss 0.0045 (Step 2850 の行) 経過 5:43 上限まで 1:08 [TRAIN ] 21:32 step 2900/3000 loss 0.0037 (Step 2900 の行) 経過 5:48 上限まで 1:03 [TRAIN ] 21:37 step 2950/3000 loss 0.0037 (Step 2900 の行) 経過 5:53 上限まで 0:58 [TRAIN ] 21:42 step 2990/3000 loss 0.0036 (Step 2950 の行) 経過 5:58 上限まで 0:53 [TRAIN ] 21:48 step 3000/3000 loss 0.0036 (Step 2950 の行) 経過 6:03 上限まで 0:48 (21808 秒) [TRAIN ] 学習が終わりました (最後のチェックポイント 2999, loss 0.0036) [WATCH ] Pod 側の見張りを 00:48 まで延ばした (上げる・持ち帰る間に消えないように) === 7a. Pod 側の一覧 (2999) (21:48:04) [LIST ] 32 ファイル 6335846833 バイト (train_state なし) → /Volumes/KIOXIA/r2r2r-data/lab/Log27/train/pod_files_2999.tsv (2 秒) === 7b. Hugging Face へ上げる (PixelstoMotors/pi05-mecharm-deep-even10-lora-v1・非公開) (21:48:06) [HF ] 上げるもの: 32 ファイル 6,335,846,833 バイト (train_state/ は上げない) HFUP ok PixelstoMotors/pi05-mecharm-deep-even10-lora-v1 に 32 ファイル 6,335,846,833 バイトを上げ, 名前・バイト数が全部合うのを確かめた (14 秒) === 7d. ログを持ち帰る (21:48:20) [LOG ] train_deep_even10_v1.log を持ち帰った → /Volumes/KIOXIA/r2r2r-data/lab/Log27/train/train_deep_even10_v1.log (1800 行) [LOG ] gpu_train.csv を持ち帰った → /Volumes/KIOXIA/r2r2r-data/lab/Log27/train/gpu_train.csv (729 行) [LOG ] train_run.log を持ち帰った → /Volumes/KIOXIA/r2r2r-data/lab/Log27/train/train_run.log (2 行) [LOG ] prep_deep.log を持ち帰った → /Volumes/KIOXIA/r2r2r-data/lab/Log27/train/prep_deep.log (165 行) [LOG ] watchdog.log は Pod にない (持ち帰れなかった) [LOG ] Pod 側のチェックポイントの一覧 → /Volumes/KIOXIA/r2r2r-data/lab/Log27/train/pod_checkpoints.tsv (240 ファイル) (28 秒) === Pod を消す (id arn304g7xu5wfk) (21:48:48) [STOP ] 削除 (Terminate) を送った (21:48:50) [STOP ] Pod (id arn304g7xu5wfk) が消えたことを API で確かめた [COST ] 借りていた時間 372 分 24 秒 (6.21 時間) × 1時間 0.59 ドル = 約 3.66 ドル (約 579 円. 1ドル=158円) GPU=NVIDIA A40 === 7e. KIOXIA へ Hugging Face から取る (21:48:51) [KIOXIA] Hugging Face (PixelstoMotors/pi05-mecharm-deep-even10-lora-v1) から取ります → /Volumes/KIOXIA/r2r2r-data/checkpoints/pi05_mecharm_deep_even10_lora_v1/.incoming_2999_20261109_214851_8808 Still waiting to acquire lock on /Volumes/KIOXIA/r2r2r-data/checkpoints/pi05_mecharm_deep_even10_lora_v1/.incoming_2999_20261109_214851_8808/.cache/huggingface/.gitignore.lock (elapsed: 0.1 seconds) Still waiting to acquire lock on /Volumes/KIOXIA/r2r2r-data/checkpoints/pi05_mecharm_deep_even10_lora_v1/.incoming_2999_20261109_214851_8808/.cache/huggingface/.gitignore.lock (elapsed: 0.1 seconds) path=/Volumes/KIOXIA/r2r2r-data/checkpoints/pi05_mecharm_deep_even10_lora_v1/.incoming_2999_20261109_214851_8808 [KIOXIA] /Volumes/KIOXIA/r2r2r-data/checkpoints/pi05_mecharm_deep_even10_lora_v1/2999 に置いた (32 ファイル 6335846833 バイト. Pod 側と名前・バイト数が全部合った). assets の正規化の値は測り直したものと sha256 が同じ (125 秒) === 由来 (train_provenance.json) を書く [PROV ] /Volumes/KIOXIA/r2r2r-data/lab/Log27/train/train_provenance.json [PROV ] /Volumes/KIOXIA/r2r2r-data/checkpoints/pi05_mecharm_deep_even10_lora_v1/2999/train_provenance.json [PROV ] hf: url=https://huggingface.co/PixelstoMotors/pi05-mecharm-deep-even10-lora-v1/commit/94ed63d4f688ba9d73873f4b7312fa5564350280 [PROV ] 由来: /Volumes/KIOXIA/r2r2r-data/lab/Log27/train/train_provenance.json ・ /Volumes/KIOXIA/r2r2r-data/checkpoints/pi05_mecharm_deep_even10_lora_v1/2999/train_provenance.json ・ Hugging Face (PixelstoMotors/pi05-mecharm-deep-even10-lora-v1) にも置いた === かかった時間 0. 送るものと Mac の道具を確かめる 1 秒 1. Pod を作る 3 秒 2. Pod が立ち上がるまで待つ 57 秒 3a. スクリプトを送る 75 秒 3b. huggingface_hub を入れ, 鍵を渡す 10 秒 3c. 続きの元は取らない (π0.5 base から. 重みは学習の最初に openpi が取る) 0 秒 3d. openpi を取る (215abfb) 20 秒 3e. uv sync 21 秒 3f. addon・設定・assets 7 秒 4a. tar を送る 133 秒 4b. pod_prep_deep.sh 3000 (変換・正規化の値を測り直す) 158 秒 5. 学習を始める 6 秒 6. 学習を見守る (5 分おき) 21808 秒 7a. Pod 側の一覧 (2999) 2 秒 7b. Hugging Face へ上げる (PixelstoMotors/pi05-mecharm-deep-even10-lora-v1・非公開) 14 秒 7d. ログを持ち帰る 28 秒 7e. KIOXIA へ Hugging Face から取る 125 秒 [RESULT] 成功: 学習が終わりました (step 2999, loss 0.0036). Hugging Face と KIOXIA に置き, Pod を消しました. 372 分 24 秒 (6.21 時間) × 1時間 0.59 ドル = 約 3.66 ドル (約 579 円. 1ドル=158円) [RESULT] まとめ → /Volumes/KIOXIA/r2r2r-data/lab/Log27/train/result_20261109_153622.txt