# フィジカルAI研究室を作る Log#27 | 浅いつかみの癖を残さず、π0.5 を50本で一から学習 - 記録: LOG480（フィジカルAI研究室） - 公開: 2026/11/9 - 記事のページ: https://pixelstomotors.com/record/retrain-pi05-from-base-on-50-even-demos - サイト: Pixels to Motors（ADAPT DESIGN / 島津 栄一） https://pixelstomotors.com/ - 要約: 位置ごとに10本そろえた深いお手本50本で、Physical Intelligence の π0.5 を一から学習し直すと、Runpod の A40 で3,000ステップに6.2時間・約579円かかった。loss は 0.1015 から 0.0036 まで下がった。 > AI が読むための記事の全文です（コード・実行ログを含む）。引用するときは上の「記事のページ」の住所を使ってください。 --- 位置ごとに10本そろえた深いお手本50本で、Physical Intelligence の [π0.5](https://www.pi.website/blog/pi05)（ロボット用の AI のモデル）を一から学習し直すと、[Runpod](https://www.runpod.io/) で借りた GPU（NVIDIA A40）で3,000ステップに6.2時間、費用は約579円でした。学習の誤差（loss）は 0.1015 から 0.0036 まで下がり、できたモデルは実機の小型アーム mechArm 270 で動かせる状態にしてあります。 この回でやったことで、次のことが変わりました。 - 良くなること：前のモデルは、浅くつかんだお手本が入ったデータから4回続けて学習を重ねたものでした。今回は π0.5 の元の重み（base）から学習したので、浅いつかみの癖を持ち越していません。置き場所ごとの本数の偏り（LOG479 で直したもの）の効き目を、ほかの癖と混ぜずに見られます - もたらされること：base から学習する流れ（`--model deep1`）が、1行で回せるようになりました。Pod を借り、データを送り、正規化の値を測り直し、学習し、モデルを Hugging Face と手元の SSD に置いて Pod を消すまでを、寝ている間に任せられます。次からは学習率の下げ方もステップ数に合わせた形（`deep1b`）で回せます - 時間と費用：借りていた時間は372分（準備 約8分・学習 5時間58分・片付け 約6分）。1ステップ約7.2秒、約579円（1時間 0.59 ドル・1ドル＝158円）。人がしたのは、出発点・安全範囲・学習率の下げ方など5つを選んだことだけでした 困りごとは、前のモデルの出どころでした。今までのモデルは、つかみが浅いお手本を含む19本（trim2）で学習した重み（3999）に、補正のデータで3回続けて学習を重ねたものです（corr → corr2 → corr3）。LOG479 で浅いお手本を外し、5か所を10本ずつにそろえたので、その50本で続きを学習すると、古い重みの癖と新しいデータの効き目が混ざってしまいます。 **新しいデータの効き目を確かめるために、続きではなく π0.5 base から学習し、正規化の値も50本で測り直しました。** 費用は続きの学習（1,500ステップ・約3時間）のほぼ倍ですが、約579円です。 やったことの全体は次のとおりです。 | 時刻 | やったこと | 結果 | |---|---|---| | 15:01〜16:00 | base から学習する `--model deep1` を lab_train.sh に足した（依頼133） | 偽の Pod でのテスト 234件・codex のレビュー 3回で合格 | | 15:36〜15:44 | A40 を借り、50本を送って LeRobot の形に変え、正規化の値を測り直した | 準備 約8分。正規化の値を手元に持ち帰り、指紋（sha256）が Pod と同じことを確かめた | | 15:44〜21:42 | 3,000ステップ学習した | loss 0.1015 → 0.0036。GPU のメモリは最大 41,283 MiB（46,068 MiB 中） | | 17:25〜17:59 | 実機推論の安全範囲を50本から作り、次の学習から学習率をステップ数に合わせて下げきる形にした（依頼135） | 肘 j2 の上限が +41.4° から +52.5° に広がった | | 21:08〜21:21 | 写り方の確かめの基準画像を trim2 にそろえた（依頼137） | 安全範囲は50本、写り方の基準は位置合わせと同じ trim2 | | 21:42〜21:51 | モデルを Hugging Face と KIOXIA に置き、Pod を消した | 32ファイル・約6.3GB。約579円 | [図解: LOG480_from_base_training.html](https://pixelstomotors.com/embeds/LOG480_from_base_training.html) 次は、このモデル（deep1）を位置1〜5 で実機推論し、前のモデル（corr3）と成功の数を比べます。 GPU を買うか借りるかの判断は、Runpod で実際に借りて学習させた費用をもとに、別のページにまとめています。 [ロボットAIの学習用GPU、買うか借りるか](https://pixelstomotors.com/gpu-buy-or-rent) この回の作業は、Claude（手順と数字の確かめを一緒に進めた AI）と進めました。何をどう学習するかの決めはこちらで行い、学習の仕組みの直しと学習の実行は [Claude Code](https://claude.com/claude-code)（プログラムを書く AI）に依頼箱を通して頼んでいます。ここから下は、依頼文と報告・コードの差分・端末の出力・学習のログ・測定データを全量で載せた作業ログです。数字は記録から数えたもので、見込みのものは見込みと書いています。鍵（アクセストークンなど）は、どこにも載せていません。Pod の IP とポートは、記録の時点で `:` に置き換えてあります。 ## 作業ログ ― 50本で π0.5 を base から学習し、実機推論の準備をそろえるまで ### この回で使う言葉 | 言葉 | 意味 | |---|---| | π0.5 base | Physical Intelligence が公開している π0.5 の元の重み。いろいろなロボットのデータで事前に学習してある。ここから自分のアームのお手本で学習し直す | | 続きの学習 | すでに自分のデータで学習した重みを出発点に、さらに学習を重ねること。前のデータの癖も一緒に引き継ぐ | | LoRA | 大きなモデルの重みは固定したまま、小さな追加の重みだけを学習するやり方。少ない GPU のメモリで学習できる。このシリーズの学習はすべて LoRA | | ステップ | 学習の1回分。お手本から32コマ（batch 32）を取り出して、重みを1回直す | | loss | お手本の動きと、モデルが出した動きのずれ。学習が進むと下がる。データが違うと値の大きさも変わるので、別のデータの loss とは大小を比べられない | | 正規化の値 | 関節の角度を、平均0・ばらつき1くらいにそろえるための平均と幅。お手本から測る。学習と推論で同じ値を使う | | 学習率の下げ方 | 重みを直す幅（学習率）を、学習の終わりに向けて小さくしていく予定。何ステップかけて下げきるかを decay_steps で決める | | 安全範囲 | 実機推論のときに、モデルが出した関節の角度をそのまま送ってよい範囲。お手本の関節の範囲の外側 10° まで。外れたら送らない | | 写り方の確かめ | 実機推論の前に、上カメラの画が学習のときと同じ写り方かを、お手本の1枚目と比べて確かめる仕組み | | 依頼箱 | Claude Code への依頼をファイルで置くと、Mac の見張り係が順番に渡して実行させる仕組み | [Runpod](https://www.runpod.io/) は、GPU を時間単位で借りられるクラウドのサービスです。借りたマシンを Pod と呼びます。このシリーズでは、学習のたびに Pod を借り、終わったら消しています。[Hugging Face](https://huggingface.co/) は、AI のモデルとデータを置いて配るサービスです。学習したモデルは非公開のリポジトリ（PixelstoMotors/pi05-mecharm-deep-even10-lora-v1）に置き、推論の Pod はそこから取ってきます。 ## 学習の出発点を、続きではなく π0.5 base にした 依頼を出す前に、出発点を3つから選びました。 | 出発点 | 中身 | 見込み | |---|---|---| | π0.5 base から新しく（選んだもの） | 50本だけで学習。正規化の値も50本で測り直す | A40 で3,000ステップ 約5.5時間・約1,000円の見込み（実際は6.2時間・約579円） | | trim の 3999 から続き | 今までと同じ続きの形。1,500ステップ 約3時間 | 3999 は浅いお手本入りの trim2 で学習した重み | | corr3 の 1499 から続き | いちばん新しいモデルに重ねる | 補正の積み重ねの上に乗せる | base を選びました。もうひとつ、記録の番号はフィジカルAI研究室の Log27 にしました（お手本をそろえた回は Log26）。 - ねらい：浅いお手本の癖と、置き場所の偏りを直したデータの効き目を、混ぜずに見る - 変わったこと：出発点が trim の 3999（浅いお手本入りの19本で学習）から、π0.5 base に。正規化の値も trim2 の19本から、今回の50本に - 見込める効果：位置1〜5 の成功の差が縮むか、深くつかめるかを、このデータだけの結果として実機で確かめられる。確かめるのは次の回の実機推論 見込みの費用（約1,000円）は、A40 の値段を高めに見積もっていました。実際の値段は1時間 0.59 ドルでした。 ## lab_train.sh に、base から学習する --model deep1 を足した（依頼133） 今までの lab_train.sh は、前のモデルからの続きの学習（corr・corr2・corr3）しかできませんでした。Claude Code に、base から学習するモデル `deep1` を足してもらい、そのまま本番を始めてもらいました。 | 項目 | deep1 の中身 | |---|---| | データ | mecharm_redblock_box_deep_even10_v1（50本・7,128コマ・tar 193,353,728 バイト） | | 設定 | pi05_mecharm_deep_even10_lora（trim の pi05_mecharm_trim_lora を写し、名前・データ名・ステップ数だけ替えた。LoRA・batch 32） | | 出発点 | π0.5 base（gs://openpi-assets/checkpoints/pi05_base/params） | | 正規化 | Pod で compute_norm_stats.py を回して測り直し、学習の前に手元へ持ち帰る | | ステップ | 3,000（保存は1,000ごと。最後は 2999） | | 置き場所 | KIOXIA の lab/Log27/train/ と checkpoints/pi05_mecharm_deep_even10_lora_v1/2999/。Hugging Face の非公開リポジトリ | | 由来の記録 | train_provenance.json に、base から始めたこと・データの tar の指紋・正規化の値の指紋 | Claude Code は、偽の Pod で流れを通すテストを18件足し（全234件が合格）、codex（外のレビューの AI）に3回見てもらっています。1回目の指摘のうち大事だったのは、正規化の値を持ち帰れなくても成功になってしまう点でした。2回試してだめならチェックポイントの中から取り直し、それでもなければ失敗にする形に直っています。 ## Runpod の A40 で3,000ステップ、6.2時間・約579円だった 学習のターミナルの記録から、かかった時間をまとめました。Pod の中の時刻は UTC（日本時間より9時間前）で記録されています。 | 段階 | 時間 | |---|---| | Pod を作り、立ち上がるまで | 1分 | | スクリプトを送り、openpi を取り、設定を入れる | 2分13秒 | | 50本の tar（193MB）を送る | 2分13秒 | | LeRobot の形に変え、正規化の値を測り直す | 2分38秒 | | 学習（3,000ステップ） | 5時間58分 | | 学習の終わりを確かめ、Hugging Face へ上げ、ログを持ち帰り、Pod を消す | 約6分 | | KIOXIA へ Hugging Face から取る（Pod を消したあと） | 2分5秒 | | 借りていた時間の合計 | 372分24秒（6.21時間）・約3.66 ドル（約579円） | 1ステップは約7.2秒でした（5時間58分 ÷ 3,000）。前に同じ A40 で測った値（RUNPOD_TRAIN.md の約6.6秒）より少し遅く、1,000ステップごとの保存の時間も入っています。GPU のメモリは最大 41,283 MiB（46,068 MiB 中）、使用率の平均は97%でした。 loss は次のように下がりました。 | ステップ | 0 | 500 | 1,000 | 1,500 | 2,000 | 2,500 | 2,950 | |---|---|---|---|---|---|---|---| | loss | 0.1015 | 0.0113 | 0.0072 | 0.0058 | 0.0048 | 0.0043 | 0.0036 | 前に base から trim2 の19本で4,000ステップ学習したとき（LOG355）は、1,000ステップで 0.0078、2,950ステップで 0.0043 でした。データが違うので、この差でどちらのモデルが良いかは言えません。2,500ステップを過ぎても、まだゆっくり下がっていました。 ## 実機推論の安全範囲を、学習した50本から作るようにした（依頼135） 学習の途中で、依頼133 の報告から2つ決めました。1つ目は実機推論の安全範囲です。今までは、どのモデルで推論しても、trim2 の19本の関節の範囲から作っていました。50本は肘 j2 が +42.5° まで上がるお手本を含み、trim2 から作った上限（+41.4°）を超えます。 | 関節 | 50本から作った範囲（deep1） | trim2 から作った範囲（今まで） | |---|---|---| | j0 | −71.7〜+54.7 | −71.7〜+54.7 | | j1 | −8.8〜+88.0 | −8.7〜+88.0 | | j2 | −44.5〜+52.5 | −45.7〜+41.4 | | j3 | −17.8〜+27.0 | −17.8〜+27.0 | | j4 | −86.6〜+10.4 | −90.0〜+10.4 | | j5 | −50.7〜−15.3 | −50.6〜−13.6 | - ねらい：お手本どおりの姿勢が、安全範囲で止められないようにする - 変わったこと：deep1 で推論するときだけ、安全範囲を50本から作る。j2 の上限が 11.1° 広がり、j4 の下限は 3.4°、j5 の上限は 1.7° 狭まった。ほかのモデルは今までどおり trim2 - 見込める効果：位置によって肘を高く上げるお手本の動きを、途中で止めずに送れる。初回の実機では、j2 が高く上がる場面を目で見る 2つ目は学習率の下げ方です。今回の設定は trim の写しなので、5,000ステップかけて学習率を下げきる予定のまま、3,000ステップで止まります。学習率は下がりきる手前で終わりました。次に回す学習からは、下げきるステップ数をステップ数に合わせます。今回のモデル（deep1）と見分けがつくよう、新しい形は `deep1b`（設定 pi05_mecharm_deep_even10_b_lora・Hugging Face は pi05-mecharm-deep-even10-b-lora-v1）として分けてあります。 ## 写り方の確かめの基準画像だけは、trim2 にそろえた（依頼137） 依頼135 の報告で、もう1つ見つかりました。実機推論の前の写り方の確かめは、安全範囲と同じフォルダの最初のお手本の画を基準にしていました。deep1 では、その基準が50本の最初の1本（2026-10-31 の補正入りの本）になります。上カメラの位置合わせ（camera_align.py）は trim2 を基準にしているので、写り方の確かめも trim2 にそろえました。 | 推論するモデル | 安全範囲 | 写り方の基準画像 | 上カメラの位置合わせ | |---|---|---|---| | deep1・deep1b | 50本（deep_even10_v1） | trim2 | trim2 | | trim・corr〜corr3 | trim2 | trim2 | trim2 | run_policy_client.py に `--framing-raw-dir` を足して、基準画像を安全範囲とは別に指定できるようにしています。 学習の仕組みの直しは、学習が回っている間に入れました。lab_train.sh は始めに中身を読み込んでいて、Pod に送るものも送り終えていたので、回っている学習には効きません。依頼135・137 の報告で、学習の記録がその間も5分おきに進んでいたことを確かめています。 ## 依頼133：base から学習する deep1 を足して本番を始めた ### 依頼133 の依頼文（依頼箱/済み/20261109_133_そろえた50本でπ0.5をbaseから新しくRunpodで学習する.md） ```markdown --- 作業フォルダ: /Users/shimazueiichi/r2r2r-scripts 上限時間: 150 --- # 依頼133：位置ごとに10本そろえた50本（mecharm_redblock_box_deep_even10_v1）で、π0.5 を base から新しく Runpod で学習する 実機は動かさない。エイイチは見ていない。記録の Log 番号は **Log27**（フィジカルAI研究室 Log#27）。 **エイイチの決め（2026-11-09）：続きの学習ではなく、π0.5 base から新しく LoRA で学習する。正規化の値もこの50本で測り直す。** 依頼131 は完了している（記録.log・claude_code_reports/20261109_131.txt「判定：合格」）。 ## 背景 今までのモデル（trim の 3999 → corr → corr2 → corr3）は、浅くつかんだお手本が入った trim2 から積み重ねてきた。今回は浅い本を外し、位置1〜5 を各10本にそろえたデータなので、その効き目を古い重みの癖と混ぜずに見るため base から学習する。 ## やること 1. lab_train.sh・lab_lib.sh に、base から学習するモデル `deep1` を足す（今の corr / corr2 / corr3 の流れは変えない） - データ：mecharm_redblock_box_deep_even10_v1（KIOXIA の datasets/ の tar。Pod へ送って LeRobot 形式に変換） - 設定名：pi05_mecharm_deep_even10_lora（中身は trim の pi05_mecharm_trim_lora と同じ LoRA・batch 32 で、データ名だけ違う。add_trim_config.py の書き方に合わせる） - 初期の重み：π0.5 base（RUNPOD_TRAIN.md 5. と同じ）。続きの元の Hugging Face のモデルは取らない - 正規化：Pod で compute_norm_stats.py をこの設定で回す（RUNPOD_TRAIN.md 4.）。値は持ち帰りに含める - ステップ：3,000（保存は 1,000 ごと。最後は 2999）。上限時間 --limit は 7 時間（A40 で 1 ステップ約 6.6 秒の実測から約 5.5 時間＋準備） - 持ち帰り先：KIOXIA の lab/Log27/ と checkpoints/pi05_mecharm_deep_even10_lora_v1/。Hugging Face は非公開の PixelstoMotors/pi05-mecharm-deep-even10-lora-v1（最後の 2999 を assets ごと） - 由来（train_provenance.json）に、base から始めたこと・データの tar の指紋・正規化の値の指紋を残す - lab_start.sh の --model にも deep1 を足し、学習のあとこのモデルで実機推論のサーバーを立てられるようにする（今日は立てない） 2. test_lab_train.sh（lab_train_fakes.sh の偽物）で deep1 の流れを通す。今の corr〜corr3 のテストも通ること 3. codex で差分を見てもらう（Pod を借りて消すところ・上限の見張り・鍵の扱いが今までと同じに働くか）。指摘を直して通るまで 4. 変えたファイルだけを指定して git add・commit（git add . は使わない） 5. 本番を始める：`./lab_train.sh --log Log27 --model deep1 --steps 3000 --limit 7`（引数の名前は今の lab_train.sh に合わせる）。依頼箱の上限で切れないよう、今までの学習と同じ形で切り離して動かす。Pod が借りられて SSH が通り、学習のログに最初の loss が出るところまで見届けてから報告する。借りられなかったときは、ほかの GPU を試す今の動き（fallback）のままでよい 6. 途中経過は2時間ごと（または準備・学習・持ち帰りの区切り）に知らせる。終わったら今の lab_train.sh と同じく、モデルを持ち帰って Pod を消し、通知する ## 報告 claude_code_reports/20261109_133.txt に：足したモデルの中身（設定・ステップ・置き場所）、2 のテストの結果、codex の回数と指摘、commit の番号、本番の Pod（GPU の型番・借り始めの時刻・1時間の値段）、最初の loss とそのステップ、終わる見込みの時刻と費用（円で。ドルは補足）、決めてほしいこと。 **報告の最後の1行は必ず次のどれか**：「判定：合格」／「判定：止めた（決めてほしいことあり）」／「判定：失敗」 ``` ### 依頼133 の報告（r2r2r-scripts/claude_code_reports/20261109_133.txt） ```text 依頼133 報告：位置ごとに10本そろえた50本（mecharm_redblock_box_deep_even10_v1）で π0.5 を base から新しく Runpod で学習する（Log27） ■ やったこと 1. lab_train.sh・lab_lib.sh・lab_start.sh に --model deep1 を足した（corr / corr2 / corr3 の流れは変えていない） - データ: mecharm_redblock_box_deep_even10_v1（KIOXIA datasets/ の tar 193,353,728 バイト・50本 / 7128コマ） - 設定: pi05_mecharm_deep_even10_lora（新しい pi05/openpi_addon/add_deep_config.py。add_trim_config.py と同じく trim の pi05_mecharm_trim_lora を写し、名前・データ名・num_train_steps だけ替える。LoRA・batch 32・学習率の予定 warmup 300 / peak 5e-5 / decay_steps 5,000 / decay_lr 5e-6・weight_loader は trim と同じ） - 初期の重み: π0.5 base（gs://openpi-assets/checkpoints/pi05_base/params）。3c で続きの元は取らない - 正規化: 新しい pi05/openpi_addon/pod_prep_deep.sh が Pod で tar を広げて 50本を確かめ、LeRobot 形式に変換し、compute_norm_stats.py --config-name pi05_mecharm_deep_even10_lora で測り直す。値は学習の前に lab/Log27/train/norm_stats.json へ持ち帰り、sha256 が Pod と同じかを確かめる（持ち帰れなければ学習のあとチェックポイントの assets から取り直す。sha256 が違えば成功にしない） - ステップ: 既定 3,000（保存は 1,000 ごと・最後は 2999）。exp 名 deep_even10_v1 - 持ち帰り先: KIOXIA lab/Log27/train/ と checkpoints/pi05_mecharm_deep_even10_lora_v1/2999/。Hugging Face は非公開の PixelstoMotors/pi05-mecharm-deep-even10-lora-v1（2999 を assets ごと） - 由来 train_provenance.json: 入力に「pi05_base（gs://…・続きの学習ではない）」、データに tar の sha256、config.init_params=base、config.norm_stats に正規化の値の sha256・Pod の場所・lab の写し - lab_start.sh / lab_lib.sh の --model に deep1 を足した（Hugging Face から /workspace/ckpt/deep_v1_2999 に取り、add_deep_config.py --steps 3000 と assets を入れて serve_policy.py --policy.config=pi05_mecharm_deep_even10_lora で起動）。今日は立てていない - 上限の引数は今の lab_train.sh の名前 --limit-hours を使った（依頼文の --limit はない） 2. テスト（Pod は借りない偽物 lab_train_fakes.sh） - test_lab_train.sh: OK 234 / NG 0（前は 216。deep1 の 18件を足した：成功の流れ・3c を飛ばす・pod_prep_deep.sh・正規化の値の持ち帰り・2999・由来・見張り 7+1 時間・sha256 が返らない・持ち帰れない→チェックポイントから・中身が違う→失敗・KIOXIA に前からある 2999 の正規化の値が違う・--init を渡すと借りずに止まる・--stop で記録の deep1 から戻す） - test_lab_start.sh: OK 161 / NG 0（前は 153。deep1 で起動・別モデルのサーバーなら止まる・フォルダから名前） - pi05/test_merge_and_config.py: 30件 OK（add_deep_config.py が trim と同じで名前・データ・ステップだけ違う・二重に入れない・違うステップで止まる、pod_prep_deep.sh の中身） - 手を入れる前の corr〜corr3 のテストは全部通ったまま 3. codex のレビュー 3回 - 1回目 必須2・推奨2: (必須) 4b の出力を $(...) に溜めていた → corr と同じく流しながら Mac 側にも写し（lab/Log27/train/prep_deep_mac.txt）を残す形に直した (必須) 正規化の値を持ち帰れなくても成功になる → 2回試し、だめならチェックポイントの assets から取り直し、それでもなければ終了コード 1・通知に出すよう直した (推奨) sha256 が違っても置けたことになる・前からある 2999 は見ていない → どちらも失敗にするよう直した (推奨) 4b の途中経過を Mac 側に残す → 上の写しで直した - 2回目: 直ったことを確認。残りは「親 PID にだけ TERM を送ると、前に出している ssh が返るまで片付けが遅れる」。codex 自身も corr を含む 3a〜4b に前からある共通の制約で今回の後退ではないとした - 3回目: 上を範囲の外（据え置き）とした判断を踏まえ「必須なし・推奨なし」 （各 ssh には時間の上限があり、Pod 側の見張りが上限 +60分で Pod を消すので、Pod が残り続けることはない） 4. commit: 8db7028（git add は9ファイルを1つずつ指定） lab_train.sh, lab_lib.sh, lab_start.sh, lab_train_fakes.sh, test_lab_train.sh, test_lab_start.sh, pi05/test_merge_and_config.py, pi05/openpi_addon/add_deep_config.py（新）, pi05/openpi_addon/pod_prep_deep.sh（新） 5. 本番 - 15:36:22 に `nohup ./lab_train.sh --log Log27 --model deep1 --steps 3000 --limit-hours 7` を新しいセッションで裏に回した（Log18・Log22 と同じ形。pid 8808） - Pod: id arn304g7xu5wfk・NVIDIA A40（46068 MiB）・1時間 0.59 ドル・15:36:26 に借り始め（ほかの GPU へは回っていない） - 15:37 に SSH が通った。Pod 側の見張りは 23:36 に Pod を消す（Mac 側の上限は 22:36） - Pod で 50本を確かめた。測り直した正規化の値は sha256 813fcb8cccd147c1d49e5c06e695587142d4535ff7c1f27b9fc94ea6aee66ce4 → lab/Log27/train/norm_stats.json に持ち帰り済み - 15:44:27 学習開始。最初の loss: 0.1015（Step 0 の行・15:54 の見守り）。15:59 に loss 0.0611（Step 50 の行・step 78/3000） - 速さ: 5分で 44 ステップ（1ステップ約 6.8 秒。RUNPOD_TRAIN.md の 6.6 秒とほぼ同じ） - 出力: /Volumes/KIOXIA/r2r2r-data/lab/Log27/train/lab_train_20261109_153622.txt（nohup 側の写し nohup_lab_train_20261109_153622.txt）。まとめは同じ所の result_*.txt ■ 終わる見込み - 学習が終わるのは 21:30 ごろ。Hugging Face へ上げる・ログを持ち帰る・Pod を消すまでで 21:35 前後、KIOXIA へ取り終わるのは 21:40 ごろ（Log22 では学習のあとの片付けが約 5分） - Mac 側の上限 22:36 まで 1時間ほど余裕がある - 費用の見込み: 約 6.0 時間 × 0.59 ドル = 約 3.5 ドル（約 560 円。1ドル=158円）。上限まで借りても 7 時間で約 4.1 ドル（約 650 円） - 終わったら lab_train.sh が Mac の通知を出し、result_*.txt にまとめる（成功・失敗とも Pod を消してから） ■ 決めてほしいこと 1. 実機推論の安全範囲: lab_start.sh は今どのモデルでも trim2 のお手本を --raw-dir にして、安全範囲を「関節範囲 ±10°」で作っている。今回の 50本は j2 の上限が 42.4°で、trim2 の 31.4°より 11°高い（trim2 からの範囲は 41.4°まで）。deep1 で動かすとき --raw-dir を deep_even10_v1 にするか、trim2 のままにするかを決めてほしい（今回は lab_start.sh のこの部分は変えていない） 2. 学習率の予定: 依頼のとおり trim と同じ（decay_steps 5,000）のまま 3,000 ステップで回している。3,000 で終わるので学習率は下げきる前（5e-6 まで下がらない）で止まる。次に回すとき decay_steps をステップ数に合わせるか 3. 2時間ごとの途中経過: この依頼の作業はここで終わるので、途中経過は lab_train.sh が5分おきに書くターミナルの記録（上の lab_train_*.txt）と、終わったときの Mac の通知・result_*.txt で見てほしい（私からの2時間ごとの知らせは出せない） 判定：合格 ``` ### 依頼133 の commit（8db7028）の説明と変えたファイル ```text commit 8db7028ab50d6b30b7bcc1395891f4668a5a1ce8 Date: Mon Nov 9 15:36:01 2026 +0900 位置ごとに 10 本そろえた mecharm_redblock_box_deep_even10_v1 (50 本) で π0.5 base から新しく学習する lab_train.sh --model deep1 を足す (設定 pi05_mecharm_deep_even10_lora は trim と同じ LoRA・batch 32・学習率の予定でデータ名とステップ数だけ違う. add_deep_config.py. 既定 3000 ステップ・3c で続きの元を取らない・4b は pod_prep_deep.sh で変換し compute_norm_stats.py で正規化の値を測り直して lab//train/norm_stats.json に持ち帰る (持ち帰れなければチェックポイントの assets から. sha256 が違えば成功にしない)・上げる先 PixelstoMotors/pi05-mecharm-deep-even10-lora-v1・KIOXIA は checkpoints/pi05_mecharm_deep_even10_lora_v1/2999). 由来に π0.5 base から始めたこと・データの tar と正規化の値の sha256 を残す. lab_start.sh / lab_lib.sh の --model に deep1 (/workspace/ckpt/deep_v1_2999) を足す. corr〜corr3 の流れは変えない. テストを足す (lab_train 234・lab_start 161・test_merge_and_config 30) (依頼 133. codex 3 回) Co-Authored-By: Claude Opus 5.5 lab_lib.sh | 48 ++++++++---- lab_start.sh | 3 + lab_train.sh | 139 ++++++++++++++++++++++++++++------- lab_train_fakes.sh | 31 ++++++-- pi05/openpi_addon/add_deep_config.py | 80 ++++++++++++++++++++ pi05/openpi_addon/pod_prep_deep.sh | 67 +++++++++++++++++ pi05/test_merge_and_config.py | 43 +++++++++++ test_lab_start.sh | 18 ++++- test_lab_train.sh | 44 +++++++++++ 9 files changed, 426 insertions(+), 47 deletions(-) ``` ### 依頼133 の commit（8db7028）の差分の全量 ```diff diff --git a/lab_lib.sh b/lab_lib.sh index e9a0db7..6e1e082 100644 --- a/lab_lib.sh +++ b/lab_lib.sh @@ -276,13 +276,16 @@ lab_cmd_addon() { # 3f. 何度流しても同じ結果 (install_addon. # 入っている設定と --steps / --init が違うと止まる. 推論では weight_loader は使わない) # corr2 (依頼 57) は corr_v1_1499 から trim2_corr_v4 で続きを学習したもの (設定 pi05_mecharm_trim_corr_v4_lora) # corr3 (依頼 97) は corr_v2_1499 から trim2_corr_v5 で続きを学習したもの (設定 pi05_mecharm_trim_corr_v5_lora) -LAB_MODELS="trim corr corr2 corr3" +# deep1 (依頼 133) は π0.5 base から deep_even10_v1 (位置 1〜5 を各 10 本) で新しく学習したもの (設定 pi05_mecharm_deep_even10_lora. +# 中身は trim と同じ LoRA・batch 32・学習率の予定. 正規化の値はこの 50 本で測り直したもの) +LAB_MODELS="trim corr corr2 corr3 deep1" LAB_CORR_STEPS=1500 LAB_CORR_INIT=/workspace/ckpt/3999/params LAB_CORR2_STEPS=1500 LAB_CORR2_INIT=/workspace/ckpt/corr_v1_1499/params LAB_CORR3_STEPS=1500 LAB_CORR3_INIT=/workspace/ckpt/corr_v2_1499/params +LAB_DEEP1_STEPS=3000 # 依頼 133. add_deep_config.py の --steps (推論では使わないが, 学習と同じ値で入れる) lab_model_def() { # lab_model_def trim|corr → MODEL_REPO MODEL_CKPT MODEL_CONFIG MODEL_DATA MODEL_DESC. 知らない名前なら 1 case "$1" in trim) @@ -309,25 +312,35 @@ lab_model_def() { # lab_model_def trim|corr → MODEL_REPO MODEL_CKPT MODEL_CONFIG=pi05_mecharm_trim_corr_v5_lora MODEL_DATA=mecharm_redblock_box_teach_v1_trim2_corr_v5 # 依頼 97: corr_v4b の 7 本 + 11-06 のゲームパッドの 4 本 MODEL_DESC="corr2 の 1499 から trim2_corr_v5 で続きを学習した 1499 (依頼 97)" ;; + deep1) + MODEL_REPO=PixelstoMotors/pi05-mecharm-deep-even10-lora-v1 + MODEL_CKPT=/workspace/ckpt/deep_v1_2999 + MODEL_CONFIG=pi05_mecharm_deep_even10_lora + MODEL_DATA=mecharm_redblock_box_deep_even10_v1 # 依頼 131: 位置 1〜5 を各 10 本 (50 本) + MODEL_DESC="π0.5 base から deep_even10_v1 で新しく学習した 2999 (依頼 133)" ;; *) return 1 ;; esac } -lab_cmd_addon_model() { # lab_cmd_addon_model trim|corr|corr2|corr3 → 3f (trim は lab_cmd_addon と同じ. corr〜corr3 は学習の設定と assets も足す) +lab_cmd_addon_model() { # lab_cmd_addon_model trim|corr|corr2|corr3|deep1 → 3f (trim は lab_cmd_addon と同じ. corr〜deep1 は学習の設定と assets も足す) lab_model_def "$1" || return 1 if [ "$1" = trim ]; then lab_cmd_addon; return; fi - local corr_args="--steps $LAB_CORR_STEPS --init $LAB_CORR_INIT" - [ "$1" = corr2 ] && corr_args="--variant v4 --steps $LAB_CORR2_STEPS --init $LAB_CORR2_INIT" - [ "$1" = corr3 ] && corr_args="--variant v5 --steps $LAB_CORR3_STEPS --init $LAB_CORR3_INIT" + local add="add_trim_corr_config.py /workspace/openpi --steps $LAB_CORR_STEPS --init $LAB_CORR_INIT" + [ "$1" = corr2 ] && add="add_trim_corr_config.py /workspace/openpi --variant v4 --steps $LAB_CORR2_STEPS --init $LAB_CORR2_INIT" + [ "$1" = corr3 ] && add="add_trim_corr_config.py /workspace/openpi --variant v5 --steps $LAB_CORR3_STEPS --init $LAB_CORR3_INIT" + [ "$1" = deep1 ] && add="add_deep_config.py /workspace/openpi --steps $LAB_DEEP1_STEPS" # 依頼 133 # assets は hf download が置いたモデルの assets/local/<データ名> を, openpi の assets/<設定>/local/ に写す - # (trim の 3f が assets/pi05_mecharm_trim_lora/local/ に写すのと同じ形. 中身は trim2 と同じ正規化の値) - echo "cd /workspace/openpi && python3 /workspace/r2r2r/pi05/openpi_addon/install_addon.py /workspace/openpi && python3 /workspace/r2r2r/add_trim_config.py /workspace/openpi && python3 /workspace/r2r2r/pi05/openpi_addon/add_trim_corr_config.py /workspace/openpi $corr_args && mkdir -p assets/$MODEL_CONFIG/local && cp -r $MODEL_CKPT/assets/local/$MODEL_DATA assets/$MODEL_CONFIG/local/" + # (trim の 3f が assets/pi05_mecharm_trim_lora/local/ に写すのと同じ形. corr〜corr3 は trim2 と同じ正規化の値, + # deep1 は deep_even10_v1 で測り直した値) + echo "cd /workspace/openpi && python3 /workspace/r2r2r/pi05/openpi_addon/install_addon.py /workspace/openpi && python3 /workspace/r2r2r/add_trim_config.py /workspace/openpi && python3 /workspace/r2r2r/pi05/openpi_addon/$add && mkdir -p assets/$MODEL_CONFIG/local && cp -r $MODEL_CKPT/assets/local/$MODEL_DATA assets/$MODEL_CONFIG/local/" } -# ---- lab_train.sh で作るモデル (依頼 57. --model corr | corr2. 依頼 97 で corr3) ---- +# ---- lab_train.sh で作るモデル (依頼 57. --model corr | corr2. 依頼 97 で corr3. 依頼 133 で deep1) ---- # 作るモデルの名前は lab_start.sh の --model と同じ. 続きの元 (TRAIN_FROM) も lab_model_def のモデル -LAB_TRAIN_MODELS="corr corr2 corr3" -lab_train_def() { # lab_train_def corr|corr2|corr3 → TRAIN_* (知らない名前なら 1) +# deep1 は続きの元がない (π0.5 base から. TRAIN_FROM・TRAIN_REPO_IN・TRAIN_CKPT_IN・TRAIN_FROM_KIOXIA は空, TRAIN_INIT=base) +LAB_TRAIN_MODELS="corr corr2 corr3 deep1" +lab_train_def() { # lab_train_def corr|corr2|corr3|deep1 → TRAIN_* (知らない名前なら 1) + TRAIN_STEPS=1500; TRAIN_PREP=pod_prep_trim_corr.sh # ステップ数の既定・4b で流す準備 (依頼 133 で deep1 だけ違う) case "$1" in corr) # 依頼 49: 3999 から trim2_corr_v3 で続きを学習した 1499 TRAIN_FROM=trim; TRAIN_VARIANT=v3 @@ -338,20 +351,29 @@ lab_train_def() { # lab_train_def corr|corr2|corr3 → TRAIN_* (知ら corr3) # 依頼 97: corr2 の 1499 から trim2_corr_v5 で続きを学習する TRAIN_FROM=corr2; TRAIN_VARIANT=v5 TRAIN_FROM_KIOXIA=pi05_mecharm_trim_corr_lora_v2/1499 ;; + deep1) # 依頼 133: π0.5 base から deep_even10_v1 で新しく学習する (正規化の値は Pod で測り直す) + TRAIN_FROM=""; TRAIN_VARIANT=deep + TRAIN_FROM_KIOXIA="" + TRAIN_STEPS=3000; TRAIN_PREP=pod_prep_deep.sh ;; *) return 1 ;; esac lab_model_def "$1" || return 1 TRAIN_DATA=$MODEL_DATA; TRAIN_CONFIG=$MODEL_CONFIG; TRAIN_REPO_OUT=$MODEL_REPO TRAIN_CKPT_NAME=${MODEL_REPO#*/pi05-}; TRAIN_CKPT_NAME=pi05_${TRAIN_CKPT_NAME//-/_} # 例 pi05_mecharm_trim_corr_lora_v2 TRAIN_EXP=trim_corr_${TRAIN_VARIANT} + if [ -z "$TRAIN_FROM" ]; then # deep1: 続きの元を取らない + TRAIN_EXP=deep_even10_v1 + TRAIN_REPO_IN=""; TRAIN_CKPT_IN=""; TRAIN_INIT=base + return 0 + fi lab_model_def "$TRAIN_FROM" || return 1 TRAIN_REPO_IN=$MODEL_REPO; TRAIN_CKPT_IN=$MODEL_CKPT; TRAIN_INIT=$MODEL_CKPT/params lab_model_def "$1" # MODEL_* は作るモデルのものに戻す } -lab_cmd_addon_train() { # lab_cmd_addon_train corr|corr2|corr3 → lab_train.sh の 3f - # corr は lab_cmd_addon と同じ (3999 を取っているので trim の assets も写す). corr2・corr3 は 3999 を取らないので - # addon と trim2 の設定だけを入れる (学習の設定と正規化の値は pod_prep_trim_corr.sh が置く) +lab_cmd_addon_train() { # lab_cmd_addon_train corr|corr2|corr3|deep1 → lab_train.sh の 3f + # corr は lab_cmd_addon と同じ (3999 を取っているので trim の assets も写す). corr2・corr3・deep1 は 3999 を取らないので + # addon と trim2 の設定だけを入れる (学習の設定と正規化の値は pod_prep_trim_corr.sh・pod_prep_deep.sh が置く) if [ "$1" = corr ]; then lab_cmd_addon; return; fi echo "cd /workspace/openpi && python3 /workspace/r2r2r/pi05/openpi_addon/install_addon.py /workspace/openpi && python3 /workspace/r2r2r/add_trim_config.py /workspace/openpi" } diff --git a/lab_start.sh b/lab_start.sh index 41ec59b..6b33165 100755 --- a/lab_start.sh +++ b/lab_start.sh @@ -15,6 +15,9 @@ # 依頼 57. lab_train.sh --model corr2 で作る. Pod の /workspace/ckpt/corr_v2_1499) # corr3 = corr2 の 1499 から trim2_corr_v5 で続きを学習した 1499 (PixelstoMotors/pi05-mecharm-trim-corr-lora-v3. # 依頼 97. lab_train.sh --model corr3 で作る. Pod の /workspace/ckpt/corr_v3_1499) +# deep1 = π0.5 base から deep_even10_v1 (位置 1〜5 を各 10 本の 50 本) で新しく学習した 2999 +# (PixelstoMotors/pi05-mecharm-deep-even10-lora-v1. 依頼 133. lab_train.sh --model deep1 で作る. +# Pod の /workspace/ckpt/deep_v1_2999. 正規化の値はこの 50 本で測り直したもの (モデルの assets にある)) # 取る場所・設定・assets・サーバーの起動・つなぎ直しの確かめが選んだモデルに合う. 安全範囲 (学習データ ±10°) # と位置合わせのお手本はどれも trim2 の raw_episodes (corr_v3・corr_v4 と合わせても範囲はほぼ同じで, trim2 の方が狭い. # corr_v5 は 11-06 の 184650 の j2 が trim2 の上限 +31.4° を最大 11° 超える) diff --git a/lab_train.sh b/lab_train.sh index 5e5efe3..d09309c 100755 --- a/lab_train.sh +++ b/lab_train.sh @@ -15,7 +15,12 @@ # (設定 pi05_mecharm_trim_corr_v4_lora. 3c で取るのは 3999 ではなく corr の 1499) # corr3 = corr2 の 1499 から trim2_corr_v5 で続きを学習 → PixelstoMotors/pi05-mecharm-trim-corr-lora-v3 # (依頼 97. 設定 pi05_mecharm_trim_corr_v5_lora. 3c で取るのは corr2 の 1499) -# --steps / --init / --exp-name 学習 (既定 1500 / 続きの元の params / trim_corr_v3・trim_corr_v4・trim_corr_v5) +# deep1 = π0.5 base から deep_even10_v1 (位置 1〜5 を各 10 本の 50 本) で新しく学習 +# → PixelstoMotors/pi05-mecharm-deep-even10-lora-v1 (依頼 133. 設定 pi05_mecharm_deep_even10_lora. +# 3c は飛ばす (続きの元を取らない). 4b は pod_prep_deep.sh で, 正規化の値を Pod で測り直し, +# lab//train/norm_stats.json に持ち帰って sha256 を由来に残す) +# --steps / --init / --exp-name 学習 (既定 1500 (deep1 は 3000) / 続きの元の params (deep1 は base) / +# trim_corr_v3・trim_corr_v4・trim_corr_v5・deep_even10_v1) # 続きの元: corr は /workspace/ckpt/3999/params, corr2 は /workspace/ckpt/corr_v1_1499/params, # corr3 は /workspace/ckpt/corr_v2_1499/params # --poll-min 進みを見る間 (既定 5 分) @@ -24,7 +29,7 @@ # 1. Pod を作る (名前 p2m-train-. lab_start.sh の p2m- とは別). 記録は lab//train/lab_pod.json # 2. SSH が通るまで待ち, Pod 側に上限の見張りを置く (上限 + 60 分で, Runpod が Pod に入れる Pod 用の鍵で自分を消す) # 3. 準備 (lab_start.sh 3a〜3f と同じ. サーバーは起動しない) -# 4. tar を /workspace/data/ へ送り, pod_prep_trim_corr.sh +# 4. tar を /workspace/data/ へ送り, pod_prep_trim_corr.sh (deep1 は pod_prep_deep.sh ) # 5. 学習を setsid nohup で始める (Mac とのつながりが切れても止まらない). ログ /workspace/logs/train_.log, # GPU の様子は 30 秒おきに /workspace/logs/gpu_train.csv # 6. 数分おきにステップ・loss を出す @@ -49,7 +54,7 @@ ORIG_ARGS=("$@") # 由来に残す (依頼 54) # shellcheck source=lab_lib.sh . "$HERE/lab_lib.sh" -GPU=A40; FALLBACK=1; LIMIT_HOURS=5; STEPS=1500; INIT=""; EXP=""; MODEL=corr # INIT・EXP の既定は --model から +GPU=A40; FALLBACK=1; LIMIT_HOURS=5; STEPS=""; INIT=""; EXP=""; MODEL=corr # STEPS・INIT・EXP の既定は --model から POLL_MIN=5; STALL_MIN=30; LOG=""; DRY_RUN=0; MODE=train while [ $# -gt 0 ]; do case "$1" in @@ -72,6 +77,7 @@ done [ -n "$LOG" ] || { echo "[ERROR] --log Log17 のように Log 番号を付けてください" >&2; exit 2; } lab_train_def "$MODEL" || { echo "[ERROR] --model は ${LAB_TRAIN_MODELS// / か }: $MODEL" >&2; exit 2; } [ -n "$INIT" ] || INIT=$TRAIN_INIT +[ -n "$STEPS" ] || STEPS=$TRAIN_STEPS # corr〜corr3 は 1500, deep1 は 3000 (依頼 133) [ -n "$EXP" ] || EXP=$TRAIN_EXP num() { awk -v x="$1" 'BEGIN { exit !(x ~ /^[0-9]+(\.[0-9]+)?$/ && x > 0) }'; } for v in LIMIT_HOURS POLL_MIN STALL_MIN; do num "${!v}" || { echo "[ERROR] $v は 0 より大きい数: ${!v}" >&2; exit 2; }; done @@ -95,17 +101,19 @@ if [ $DRY_RUN = 1 ]; then fi CKPT_ROOT="$DATA_ROOT/checkpoints" ADD_TRIM="$DATA_ROOT/lab/Log11/pod/add_trim_config.py" -use_model() { # use_model corr|corr2|corr3 → 作るモデルで決まる値 (lab_lib.sh の lab_train_def. 依頼 57 / 97) +use_model() { # use_model corr|corr2|corr3|deep1 → 作るモデルで決まる値 (lab_lib.sh の lab_train_def. 依頼 57 / 97 / 133) lab_train_def "$1" || return 1 DATA_NAME=$TRAIN_DATA # corr: ..._trim2_corr_v3 / corr2: ..._trim2_corr_v4b (依頼 58) / corr3: ..._trim2_corr_v5 TAR="$DATA_ROOT/datasets/$DATA_NAME/${DATA_NAME}_raw_episodes.tar" CONFIG=$TRAIN_CONFIG # corr: pi05_mecharm_trim_corr_lora / corr2: ..._v4_lora / corr3: ..._v5_lora - HF_REPO_IN=$TRAIN_REPO_IN # 続きの元 (corr: 3999 / corr2: corr の 1499 / corr3: corr2 の 1499. lab_start.sh と同じ) + HF_REPO_IN=$TRAIN_REPO_IN # 続きの元 (corr: 3999 / corr2: corr の 1499 / corr3: corr2 の 1499. lab_start.sh と同じ. deep1 は空) HF_REPO_OUT=$TRAIN_REPO_OUT # 上げる先 (非公開). 名前は checkpoints/ のフォルダ名に合わせる CKPT_NAME=$TRAIN_CKPT_NAME # KIOXIA の checkpoints/<これ>/<ステップ>/ - CKPT_IN=$TRAIN_CKPT_IN # 3c で続きの元を置く場所 - CKPT_IN_KIOXIA="$CKPT_ROOT/$TRAIN_FROM_KIOXIA" # HF_REPO_IN と同じもの (由来の指紋を取る. 依頼 54) + CKPT_IN=$TRAIN_CKPT_IN # 3c で続きの元を置く場所 (deep1 は空) + CKPT_IN_KIOXIA="" # HF_REPO_IN と同じもの (由来の指紋を取る. 依頼 54. deep1 は空) + [ -n "$TRAIN_FROM_KIOXIA" ] && CKPT_IN_KIOXIA="$CKPT_ROOT/$TRAIN_FROM_KIOXIA" PREP_VARIANT=$TRAIN_VARIANT # pod_prep_trim_corr.sh の 3 つ目 (v3 / v4 / v5) + PREP_SH=$TRAIN_PREP # 4b で流す準備 (deep1 は pod_prep_deep.sh. 依頼 133) } use_model "$MODEL" POD_EXP_DIR=/workspace/openpi/checkpoints/$CONFIG/$EXP @@ -114,6 +122,9 @@ GPU_LOG=/workspace/logs/gpu_train.csv RUN_LOG=/workspace/logs/train_run.log EXIT_FILE=/workspace/logs/train_exit_code PREP_LOG=/workspace/logs/prep_trim_corr.log +[ "$PREP_SH" = pod_prep_deep.sh ] && PREP_LOG=/workspace/logs/prep_deep.log +NORM_SHA=""; NORM_POD=""; NORM_LOCAL="" # deep1: Pod で測り直した正規化の値 (sha256・Pod の openpi からの相対・持ち帰った先. 依頼 133) +TAR_SHA="" # deep1: 送る tar の sha256 (由来. 依頼 133) LAST_STEP=$((STEPS - 1)) # openpi は 0 から数え, 最後のステップ (num_train_steps - 1) を必ず保存する # ---- 書く前に KIOXIA のマウントを確かめる ---- @@ -158,17 +169,22 @@ PROV_PROCESS=""; PROV_ERROR=""; PROV_CKPT_DIR=""; PROV_HF=0; PROV_DONE=0 OPENPI_JSON='[]' # 3f のあとに Pod の openpi のコミットと変更の有無を入れる iso_of() { [ -n "${1:-}" ] && [ "$1" != 0 ] && date -r "$1" '+%Y-%m-%dT%H:%M:%S%z' | sed -E 's/([0-9]{2})$/:\1/'; } prov_inputs() { # 入力 (学習データ・続きの元 (corr は 3999・corr2 は corr の 1499・corr3 は corr2 の 1499)・設定を足すスクリプト) を spec の inputs の形で出す + # deep1 (依頼 133): 続きの元は π0.5 base (gs://. KIOXIA に原本がないので指紋はない). データには tar の sha256 も付ける local init_ck="" - case "$INIT" in "$CKPT_IN"/*) init_ck="$CKPT_IN_KIOXIA" ;; esac + [ -n "$CKPT_IN" ] && case "$INIT" in "$CKPT_IN"/*) init_ck="$CKPT_IN_KIOXIA" ;; esac jq -n --arg data "$DATA_NAME" --arg data_dir "$DATA_ROOT/datasets/$DATA_NAME" \ --arg init "$INIT" --arg init_ck "$init_ck" --arg init_id "${CKPT_IN_KIOXIA#"$CKPT_ROOT"/}" --arg repo_in "$HF_REPO_IN" \ - --arg add_trim "$ADD_TRIM" \ - '{inputs: ([{kind: "dataset", id: $data, path: $data_dir}] - + [{kind: "checkpoint", id: (if $init_ck == "" then (if $init == "" then null else $init end) else $init_id end), + --arg add_trim "$ADD_TRIM" --arg tar "${TAR#"$DATA_ROOT"/}" --arg tar_sha "$TAR_SHA" --arg base "$PI05_BASE" \ + '{inputs: ([{kind: "dataset", id: $data, path: $data_dir} + + (if $tar_sha == "" then {} else {tar: {path: $tar, sha256: $tar_sha}} end)] + + (if $init == "base" then [{kind: "checkpoint", id: "pi05_base", path: null, hf_repo: null, pod_path: null, url: $base, + note: "π0.5 base から新しく学習 (続きの学習ではない)"}] + else [{kind: "checkpoint", id: (if $init_ck == "" then (if $init == "" then null else $init end) else $init_id end), path: (if $init_ck == "" then null else $init_ck end), - hf_repo: (if $init_ck == "" then null else $repo_in end), pod_path: (if $init == "" then null else $init end)}] + hf_repo: (if $init_ck == "" then null else $repo_in end), pod_path: (if $init == "" then null else $init end)}] end) + [{kind: "config", id: "add_trim_config.py", path: $add_trim}])}' } +PI05_BASE=gs://openpi-assets/checkpoints/pi05_base/params # --init base の重み (install_addon.py の weight_loader と同じ) PROV_CODE_ARGS=(--repo-dir "${LAB_PROV_REPO:-$HERE}" --code-path lab_train.sh --code-path lab_lib.sh --code-path provenance.py --code-path pi05) prov_prepare() { # 始める前にコードと入力の指紋を測って PROV_START に持つ (終わったときに測ると, 途中で変えたものになる) local out @@ -199,7 +215,7 @@ prov_spec() { # 由来の spec (lab_train.sh が知っているこ --arg hf "${HF_MSG:-}" --arg kx "${KIOXIA_MSG:-}" --arg prov_hf "$PROV_HF" \ --arg pod "$POD_ID" --arg gpu "$GPU_USED" --arg cph "$COST_PER_HR" --arg yen "$YEN_PER_USD" \ --arg t0 "$(iso_of "$POD_T0")" --arg t1 "$(iso_of "$t_end")" --arg tt "$(iso_of "$TRAIN_T0")" --arg rented "$rented" \ - --arg cost_line "${COST_LINE:-}" \ + --arg cost_line "${COST_LINE:-}" --arg norm_sha "$NORM_SHA" --arg norm_pod "$NORM_POD" --arg norm_local "$NORM_LOCAL" \ 'def num: if . == "" then null else tonumber end; def str: if . == "" then null else . end; {producer: {script: $script, argv: $argv, dry_run: ($dry == "1")}, external_code: $openpi, @@ -207,8 +223,10 @@ prov_spec() { # 由来の spec (lab_train.sh が知っているこ code: ($start.code // null), outputs: (if $ck_dir == "" then [] else [{kind: "checkpoint", id: ($ckpt_name + "/" + $step), path: $ck_dir, hf_repo: (if $prov_hf == "1" then $repo_out else null end)}] end), # 指紋 files-v2 は由来のファイルを数えない (依頼 56) - config: {name: $config, exp_name: $exp, num_train_steps: ($steps | num), init_params: ($init | str), data_name: $data, - limit_hours: ($limit | num), checkpoint_step: ($step | str)}, + config: ({name: $config, exp_name: $exp, num_train_steps: ($steps | num), init_params: ($init | str), data_name: $data, + limit_hours: ($limit | num), checkpoint_step: ($step | str)} + + (if $norm_sha == "" then {} else {norm_stats: {source: "Pod で compute_norm_stats.py を回して測り直した (依頼 133)", + sha256: $norm_sha, pod_path: ("/workspace/openpi/" + $norm_pod), lab_copy: ($norm_local | str)}} end)), compute: {provider: "runpod", pod_id: ($pod | str), gpu: ($gpu | str), started_at: ($t0 | str), train_started_at: ($tt | str), ended_at: ($t1 | str), duration_sec: ($rented | num), cost_per_hr_usd: ($cph | num), cost_usd: (if $rented == "" then null else (($rented | tonumber) / 3600 * ($cph | tonumber) * 100 | round / 100) end), @@ -465,12 +483,34 @@ compare_with_pod() { # compare_with_pod フォルダ → Pod 側の一覧 head -20 <<<"$d" | sed 's/^/ /' return 1 } +norm_ok() { # norm_ok チェックポイントのフォルダ → deep1 でなければ 0. assets の正規化の値が Pod で測り直したもの (sha256) と同じなら 0 (依頼 133) + [ -z "$NORM_SHA" ] && return 0 + local nf="$1/assets/local/$DATA_NAME/norm_stats.json" + [ -f "$nf" ] && [ "$(shasum -a 256 "$nf" | cut -d' ' -f1)" = "$NORM_SHA" ] +} +norm_from_ckpt() { # deep1: 4b で持ち帰れなかった正規化の値を, KIOXIA に置いたチェックポイントの assets から lab//train/ に写す (codex R1) + [ -n "$NORM_SHA" ] && [ -z "$NORM_LOCAL" ] || return 0 + local d="$CKPT_ROOT/$CKPT_NAME/${CKPT_STEP:-}" f + if [ -n "${CKPT_STEP:-}" ] && norm_ok "$d" && f=$(dest_of norm_stats.json) && cp "$d/assets/local/$DATA_NAME/norm_stats.json" "$f" \ + && [ "$(shasum -a 256 "$f" | cut -d' ' -f1)" = "$NORM_SHA" ]; then + NORM_LOCAL=$f + echo "[NORM ] 正規化の値をチェックポイントの assets から持ち帰った → $NORM_LOCAL" + save_state '. + {norm_local:$l}' --arg l "$NORM_LOCAL" + return 0 + fi + echo "[NORM ] 正規化の値を lab/ に持ち帰れなかった" + return 1 +} place_ckpt() { # place_ckpt hf|scp → KIOXIA の checkpoints//<ステップ>/ に置いて確かめる. KIOXIA_MSG に結果 local how="$1" parent="$CKPT_ROOT/$CKPT_NAME" dest tmp n bytes dest="$parent/$CKPT_STEP" if [ ! -d "$DATA_ROOT" ]; then KIOXIA_MSG="KIOXIA がマウントされていない (置けなかった)"; return 1; fi if [ -e "$dest" ] || [ -L "$dest" ]; then # 前からあるものは上書き・移動・削除しない. 中身が同じかだけを見る - if compare_with_pod "$dest"; then KIOXIA_MSG="$dest はもとからあり, Pod 側と全部同じ (書き換えていない)"; return 0; fi + if compare_with_pod "$dest"; then + # deep1: 名前・バイト数が同じでも, 正規化の値の中身が違えば合ったとしない (codex R1) + norm_ok "$dest" || { KIOXIA_MSG="$dest はもとからあり, 名前・バイト数は Pod 側と同じだが assets の正規化の値が測り直したもの (sha256 ${NORM_SHA:0:12}…) と違う・ない. 上書きしなかった"; return 1; } + KIOXIA_MSG="$dest はもとからあり, Pod 側と全部同じ (書き換えていない)"; return 0 + fi KIOXIA_MSG="$dest がもとからあり, Pod 側と違う. 上書きしなかった"; return 1 fi mkdir -p "$parent" || { KIOXIA_MSG="$parent を作れない"; return 1; } @@ -499,6 +539,10 @@ place_ckpt() { # place_ckpt hf|scp → KIOXIA の checkpoints/&1 | tee $PREP_LOG" \ - || wrap_fail prep "pod_prep_trim_corr.sh に失敗 (上の出力を見る)" +if [ "$PREP_SH" = pod_prep_deep.sh ]; then # deep1 (依頼 133): 正規化の値を Pod で測り直す + stage "4b. pod_prep_deep.sh $STEPS (変換・正規化の値を測り直す)" + # 出力は corr と同じくその場で画面とターミナルの記録に流し, Mac 側にも写しを残して NORM の行を読む (codex R1: $(...) に溜めない) + PREP_OUT=$(dest_of prep_deep_mac.txt) + LAB_TO=$LAB_TO_LONG pssh "set -o pipefail; bash /workspace/r2r2r/pi05/openpi_addon/pod_prep_deep.sh $STEPS 2>&1 | tee $PREP_LOG" | tee "$PREP_OUT" \ + || wrap_fail prep "pod_prep_deep.sh に失敗 (上の出力を見る)" + nl=$(grep -E '^NORM sha256=[0-9a-f]{64} path=assets/[A-Za-z0-9_./-]+/norm_stats\.json$' "$PREP_OUT" | tail -1) + [ -n "$nl" ] || wrap_fail prep "pod_prep_deep.sh から正規化の値の sha256 が返らない" + NORM_SHA=$(sed -E 's/^NORM sha256=([0-9a-f]{64}) .*/\1/' <<<"$nl"); NORM_POD=${nl##* path=} + # 測り直した値を学習の前に持ち帰る (学習が落ちても残る). 持ち帰ったものの sha256 が Pod と同じかを確かめる. + # 持ち帰れなければ, 学習のあとにチェックポイントの assets から取り直す (norm_from_ckpt. codex R1) + NORM_LOCAL=$(dest_of norm_stats.json) + for _ in 1 2; do + if pod_scp "root@$POD_IP:/workspace/openpi/$NORM_POD" "$NORM_LOCAL" >/dev/null \ + && [ "$(shasum -a 256 "$NORM_LOCAL" | cut -d' ' -f1)" = "$NORM_SHA" ]; then + echo "[NORM ] 正規化の値 (sha256 ${NORM_SHA:0:12}…) を持ち帰った → $NORM_LOCAL"; break + fi + rm -f "$NORM_LOCAL" + done + if [ ! -f "$NORM_LOCAL" ]; then + echo "[WARN ] 正規化の値を持ち帰れなかった (または sha256 が違う). 学習のあとチェックポイントの assets から取り直す" + NORM_LOCAL="" + fi + save_state '. + {norm_sha256:$s, norm_pod:$p, norm_local:$l}' --arg s "$NORM_SHA" --arg p "$NORM_POD" --arg l "$NORM_LOCAL" +else + stage "4b. pod_prep_trim_corr.sh $STEPS $INIT $PREP_VARIANT" + LAB_TO=$LAB_TO_LONG pssh "set -o pipefail; bash /workspace/r2r2r/pi05/openpi_addon/pod_prep_trim_corr.sh $STEPS $INIT $PREP_VARIANT 2>&1 | tee $PREP_LOG" \ + || wrap_fail prep "pod_prep_trim_corr.sh に失敗 (上の出力を見る)" +fi stage_done # ---- 5. 学習を始める ---- diff --git a/lab_train_fakes.sh b/lab_train_fakes.sh index 8b8e861..7b86d96 100644 --- a/lab_train_fakes.sh +++ b/lab_train_fakes.sh @@ -28,6 +28,9 @@ # hf_same Hugging Face にもう同じ中身がある (上げ直さない. 由来も足さない. 前の回の由来がある. 依頼 54) # hf_prov_hang 由来を Hugging Face へ足すのが返らない (LAB_PROV_TO を小さくして使う. codex R2) # openpi_partial Pod の openpi のコミットの行だけ出て ssh が切れる (codex R1) +# norm_missing deep1: pod_prep_deep.sh が正規化の値の sha256 の行を出さずに終わる (依頼 133) +# norm_fetch_fail deep1: 測り直した正規化の値を 4b で持ち帰れない (学習は続け, あとでチェックポイントの assets から取る. 依頼 133) +# norm_bad deep1: Pod が言う正規化の値の sha256 と, 持ち帰ったもの・チェックポイントの assets の中身が違う (依頼 133) LAB_DRY_ROOT=${LAB_DRY_ROOT:-$(mktemp -d /tmp/lab_train_dry.XXXXXX)} LAB_FAKE_SCENARIO=${LAB_FAKE_SCENARIO:-success} @@ -37,6 +40,7 @@ if [ -d /Volumes/KIOXIA/r2r2r-data ]; then for f in "/Volumes/KIOXIA/r2r2r-data/datasets/mecharm_redblock_box_teach_v1_trim2_corr_v3/mecharm_redblock_box_teach_v1_trim2_corr_v3_raw_episodes.tar" \ "/Volumes/KIOXIA/r2r2r-data/datasets/mecharm_redblock_box_teach_v1_trim2_corr_v4b/mecharm_redblock_box_teach_v1_trim2_corr_v4b_raw_episodes.tar" \ "/Volumes/KIOXIA/r2r2r-data/datasets/mecharm_redblock_box_teach_v1_trim2_corr_v5/mecharm_redblock_box_teach_v1_trim2_corr_v5_raw_episodes.tar" \ + "/Volumes/KIOXIA/r2r2r-data/datasets/mecharm_redblock_box_deep_even10_v1/mecharm_redblock_box_deep_even10_v1_raw_episodes.tar" \ /Volumes/KIOXIA/r2r2r-data/lab/Log11/pod/add_trim_config.py "$RUNPOD_KEY_FILE" "$HF_TOKEN_FILE"; do echo "[DRY ] 本物の $(basename "$f"): $([ -s "$f" ] && echo ある || echo ない)" done @@ -48,7 +52,7 @@ RUNPOD_KEY_FILE="$LAB_DRY_ROOT/keys/runpod_api_key.txt"; HF_TOKEN_FILE="$LAB_DRY POD_SSH_KEY="$LAB_DRY_ROOT/keys/id_ed25519" mkdir -p "$LAB_DRY_ROOT/keys" "$LAB_DRY_ROOT/fake" "$DATA_ROOT/lab/Log11/pod" \ "$DATA_ROOT/datasets/mecharm_redblock_box_teach_v1_trim2_corr_v3" "$DATA_ROOT/datasets/mecharm_redblock_box_teach_v1_trim2_corr_v4b" \ - "$DATA_ROOT/datasets/mecharm_redblock_box_teach_v1_trim2_corr_v5" + "$DATA_ROOT/datasets/mecharm_redblock_box_teach_v1_trim2_corr_v5" "$DATA_ROOT/datasets/mecharm_redblock_box_deep_even10_v1" [ -s "$RUNPOD_KEY_FILE" ] || echo "DRY-RUNPOD-SECRET-KEY" > "$RUNPOD_KEY_FILE" [ -s "$HF_TOKEN_FILE" ] || echo "hf_DRYSECRETTOKEN" > "$HF_TOKEN_FILE" [ -s "$POD_SSH_KEY" ] || echo "dry-ssh-key" > "$POD_SSH_KEY" @@ -58,10 +62,16 @@ for v in v3 v4b v5; do # --model corr は corr_v3, corr2 は corr_v4b (依 TARF="$DATA_ROOT/datasets/mecharm_redblock_box_teach_v1_trim2_corr_$v/mecharm_redblock_box_teach_v1_trim2_corr_${v}_raw_episodes.tar" [ -f "$TARF" ] || echo "dry tar $v" > "$TARF" done +TARF="$DATA_ROOT/datasets/mecharm_redblock_box_deep_even10_v1/mecharm_redblock_box_deep_even10_v1_raw_episodes.tar" # deep1 (依頼 133) +[ -f "$TARF" ] || echo "dry tar deep_even10_v1" > "$TARF" # 続きの元 (由来の指紋を取る. 依頼 54): corr は 3999, corr2 は corr の 1499 (依頼 57. lab_train_def が決めた選んだモデルのものだけ. # corr の 1499 を置くと, corr の学習で置く先と重なる) -FAKE_INIT="$DATA_ROOT/checkpoints/${TRAIN_FROM_KIOXIA:-pi05_mecharm_trim_lora_v1/3999}" -[ -e "$FAKE_INIT" ] || { mkdir -p "$FAKE_INIT/params"; echo '{"dry": 3999}' > "$FAKE_INIT/_CHECKPOINT_METADATA"; head -c 100 /dev/zero > "$FAKE_INIT/params/x"; } +# deep1 (依頼 133) は続きの元がない (π0.5 base から) ので置かない +if [ "${TRAIN_INIT:-}" != base ]; then + FAKE_INIT="$DATA_ROOT/checkpoints/${TRAIN_FROM_KIOXIA:-pi05_mecharm_trim_lora_v1/3999}" + [ -e "$FAKE_INIT" ] || { mkdir -p "$FAKE_INIT/params"; echo '{"dry": 3999}' > "$FAKE_INIT/_CHECKPOINT_METADATA"; head -c 100 /dev/zero > "$FAKE_INIT/params/x"; } +fi +FAKE_NORM_SHA=c1448b9e612668eedc320cc285053fcee62dc9afb40e435c70948441d486b8e2 # 0 が 2120 バイト (fake_ckpt_files の norm_stats.json と同じ) touch "$LAB_DRY_ROOT/calls.log" "$LAB_DRY_ROOT/ssh.log" "$LAB_DRY_ROOT/scp.log" CREATE_WAIT=0; SSH_WAIT=0; LOGIN_WAIT=0; GONE_WAIT=0; FETCH_WAIT=0 @@ -123,15 +133,15 @@ pod_ssh() { printf '%s\n----\n' "$*" >> "$LAB_DRY_ROOT/ssh.log" cat > /dev/null [ -f "$LAB_DRY_ROOT/fake/deleted" ] && return 255 - local n + local n done_line="STATUS run=0 exit=0 age=4 bar=${STEPS:-1500} logstep=$(( ${STEPS:-1500} - 100 )) loss=0.0215 ckpt=$(( ${STEPS:-1500} - 1 )) cm=1" case "$*" in *": lt=watchdog"*) [ "$LAB_FAKE_SCENARIO" = wd_missing ] && echo "WATCHDOG run=1 pod=1 key=0 ctl=0" || echo "WATCHDOG run=1 pod=1 key=1 ctl=1" ;; *": lt=start_train"*) [ "$LAB_FAKE_SCENARIO" = late_start ] && sleep 9; echo STARTED ;; *": lt=status"*) n=$(fake_n status) - if [ "$LAB_FAKE_SCENARIO" = late_start ]; then echo "STATUS run=0 exit=0 age=4 bar=1500 logstep=1400 loss=0.0215 ckpt=1499 cm=1"; return 0; fi + if [ "$LAB_FAKE_SCENARIO" = late_start ]; then echo "$done_line"; return 0; fi if [ "$LAB_FAKE_SCENARIO" = late_finish ]; then # 確かめている間に上限を過ぎ, 学習も終わる - sleep 9; echo "STATUS run=0 exit=0 age=4 bar=1500 logstep=1400 loss=0.0215 ckpt=1499 cm=1"; return 0 + sleep 9; echo "$done_line"; return 0 fi case "$LAB_FAKE_SCENARIO:$n" in crash:1|killed:1|stall:1|timeout:*|*:1) @@ -139,7 +149,7 @@ pod_ssh() { else echo "STATUS run=1 exit=- age=3 bar=$((n * 100)) logstep=$((n * 100)) loss=0.0412 ckpt=- cm=0"; fi ;; crash:*) echo "STATUS run=0 exit=1 age=5 bar=650 logstep=600 loss=0.0399 ckpt=- cm=0" ;; killed:*) echo "STATUS run=0 exit=- age=5 bar=650 logstep=600 loss=0.0399 ckpt=- cm=0" ;; - *) echo "STATUS run=0 exit=0 age=4 bar=1500 logstep=1400 loss=0.0215 ckpt=1499 cm=1" ;; + *) echo "$done_line" ;; esac ;; *": lt=list_ckpt"*|*": lt=ls_all"*) if [ "$LAB_FAKE_SCENARIO" = list_partial ] && [[ "$*" == *"lt=list_ckpt"* ]]; then @@ -154,6 +164,10 @@ pod_ssh() { [ "$LAB_FAKE_SCENARIO" = hf_same ] && { echo "HFUP same PixelstoMotors/pi05-mecharm-trim-corr-lora-v1 にはもう同じ中身がある (5 ファイル). 上げ直さない"; return 0; } echo "HFUP ok PixelstoMotors/pi05-mecharm-trim-corr-lora-v1 に 5 ファイルを上げ, 名前・バイト数が全部合うのを確かめた" ;; *"pod_prep_trim_corr.sh"*) echo "== 5. 準備できました (ドライラン)" ;; + *"pod_prep_deep.sh"*) # 依頼 133: 正規化の値を測り直し, sha256 の行を出す + echo "== 4. 正規化の値をこの 50 本で測り直す (ドライラン)" + [ "$LAB_FAKE_SCENARIO" = norm_missing ] || echo "NORM sha256=$([ "$LAB_FAKE_SCENARIO" = norm_bad ] && printf 'a%.0s' {1..64} || echo $FAKE_NORM_SHA) path=assets/pi05_mecharm_deep_even10_lora/local/mecharm_redblock_box_deep_even10_v1/norm_stats.json" + echo "== 5. 準備できました (ドライラン)" ;; *": lt=openpi_rev"*) [ "$LAB_FAKE_SCENARIO" = openpi_unknown ] && { echo "fatal: not a git repository"; return 128; } echo "OPENPI 215abfb1234567890abcdef1234567890abcdef0" @@ -171,6 +185,9 @@ pod_scp() { case "$*" in *":/workspace/openpi/checkpoints/"*) fake_make_ckpt "$last" ;; *":/workspace/logs/watchdog.log"*) return 1 ;; # 見張りが働いていなければ, ないのがふつう + *":/workspace/openpi/assets/"*"/norm_stats.json "*) # deep1: 測り直した正規化の値 (依頼 133) + [ "$LAB_FAKE_SCENARIO" = norm_fetch_fail ] && return 1 + head -c 2120 /dev/zero > "$last" ;; *" root@"*":/workspace/logs/"*|"root@"*":/workspace/logs/"*) echo "dry log line (${1##*/})" > "$last" ;; esac return 0 diff --git a/pi05/openpi_addon/add_deep_config.py b/pi05/openpi_addon/add_deep_config.py new file mode 100755 index 0000000..f3409c1 --- /dev/null +++ b/pi05/openpi_addon/add_deep_config.py @@ -0,0 +1,80 @@ +#!/usr/bin/env python3 +"""openpi config.py に deep_even10_v1 (位置 1〜5 を各 10 本にそろえた 50 本) 用の学習設定 pi05_mecharm_deep_even10_lora を足す +(2026-11-09 依頼 133. lab_train.sh --model deep1). + +π0.5 base から新しく学習する (続きの学習ではない). 中身は trim の pi05_mecharm_trim_lora と同じ LoRA・batch 32・ +学習率の予定 (warmup 300 / peak 5e-5 / decay_steps 5_000 / decay_lr 5e-6)・weight_loader (π0.5 base) で, +次の 3 か所だけ変える (add_trim_config.py と同じ方式): + name pi05_mecharm_trim_lora -> pi05_mecharm_deep_even10_lora + repo_id local/mecharm_redblock_box_teach_v1_trim2 -> local/mecharm_redblock_box_deep_even10_v1 + num_train_steps 4_000 -> --steps (依頼 133 は 3000) +正規化の値は写さず, Pod で compute_norm_stats.py をこの設定で回して測り直す (pod_prep_deep.sh). + +install_addon.py と add_trim_config.py (lab/Log11/pod/) を実行した後に 1 回実行: + python3 add_deep_config.py /workspace/openpi --steps 3000 +何度実行しても同じ結果 (二重に入れない). 入っている設定と --steps が違うときは止まる. +""" +import argparse +import re +import sys +from pathlib import Path + +NAME = "pi05_mecharm_deep_even10_lora" +REPO_ID = "local/mecharm_redblock_box_deep_even10_v1" +MARK_A = " # >>> r2r2r mecharm deep_even10_v1 (train config)\n" +MARK_B = " # <<< r2r2r mecharm deep_even10_v1 (train config)\n" +TRIM_A = " # >>> r2r2r mecharm teach_v1_trim2 (train config)\n" +TRIM_B = " # <<< r2r2r mecharm teach_v1_trim2 (train config)\n" +BASE_LOADER = 'weight_loader=weight_loaders.CheckpointWeightLoader("gs://openpi-assets/checkpoints/pi05_base/params"),' +# trim と同じであることを確かめる値 (写したあとも変わっていないこと) +SAME_AS_TRIM = ("batch_size=32,", "warmup_steps=300,", "peak_lr=5e-5,", "decay_steps=5_000,", "decay_lr=5e-6,", BASE_LOADER) + + +def make_block(s, steps): + assert s.count(TRIM_A) == 1 and s.count(TRIM_B) == 1, "先に install_addon.py と add_trim_config.py を実行する" + block = s[s.index(TRIM_A):s.index(TRIM_B) + len(TRIM_B)] + reps = [ + (TRIM_A, MARK_A), + (TRIM_B, MARK_B), + ('name="pi05_mecharm_trim_lora"', f'name="{NAME}"'), + ('repo_id="local/mecharm_redblock_box_teach_v1_trim2"', f'repo_id="{REPO_ID}"'), + ("num_train_steps=4_000,", f"num_train_steps={steps},"), + ] + for a, b in reps: + assert block.count(a) == 1, f"見つからない/重複: {a!r}" + block = block.replace(a, b) + for x in SAME_AS_TRIM: + assert block.count(x) == 1, f"trim の設定に見つからない: {x!r}" + return block + + +def main(argv=None): + ap = argparse.ArgumentParser() + ap.add_argument("openpi_root") + ap.add_argument("--steps", type=int, required=True, help="学習のステップ数 (num_train_steps. 依頼 133 は 3000)") + args = ap.parse_args(argv) + if args.steps <= 0: + sys.exit("[ERROR] --steps は 1 以上") + cfg = Path(args.openpi_root).resolve() / "src" / "openpi" / "training" / "config.py" + s = cfg.read_text() + if MARK_A in s: + have = s[s.index(MARK_A):] + have = have[:have.index(MARK_B)] + m = re.search(r"num_train_steps=(\d+),", have) + if not (f'name="{NAME}"' in have and f'repo_id="{REPO_ID}"' in have and all(x in have for x in SAME_AS_TRIM)): + sys.exit("[ERROR] 入っている設定の名前・データ名・学習の値が違う. openpi の config.py を git checkout してから入れ直す") + if not (m and int(m.group(1)) == args.steps): + sys.exit("[ERROR] 別の --steps で入っている. openpi の config.py を git checkout してから入れ直す") + print("already installed (config.py は変更なし)") + return 0 + if f'name="{NAME}"' in s: + sys.exit(f"[ERROR] {NAME} がもう別の形で入っている. openpi の config.py を git checkout してから入れ直す") + block = make_block(s, args.steps) + j = s.index(TRIM_B) + len(TRIM_B) + cfg.write_text(s[:j] + block + s[j:]) + print(f"installed: {NAME} (repo_id={REPO_ID}, {args.steps} steps, init=π0.5 base)") + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/pi05/openpi_addon/pod_prep_deep.sh b/pi05/openpi_addon/pod_prep_deep.sh new file mode 100755 index 0000000..41048a6 --- /dev/null +++ b/pi05/openpi_addon/pod_prep_deep.sh @@ -0,0 +1,67 @@ +#!/usr/bin/env bash +# pod_prep_deep.sh — Pod で deep_even10_v1 (位置 1〜5 を各 10 本にそろえた 50 本) の学習の準備をする (2026-11-09 依頼 133). +# 学習は回さない. lab_train.sh --model deep1 の 4b が呼ぶ. +# +# 前提: lab_train.sh 3a〜3f で Pod を準備してある (openpi・install_addon.py・add_trim_config.py). 続きの元のモデルは取らない +# (π0.5 base から. 重みは学習の最初に openpi が gs://openpi-assets から取る) +# Mac から次の 2 つを送ってある: +# /workspace/data/mecharm_redblock_box_deep_even10_v1_raw_episodes.tar +# /workspace/r2r2r/pi05/ (add_deep_config.py と convert_to_lerobot.py を使う) +# +# bash /workspace/r2r2r/pi05/openpi_addon/pod_prep_deep.sh +# 例: bash ... 3000 +# +# すること (何度流しても同じ結果): +# 1. tar を /workspace/data/<データ名>/ に広げ, エピソードの本数を確かめる (Mac で作ったときは 50 本) +# 2. openpi の config.py に pi05_mecharm_deep_even10_lora を足す (--steps) +# 3. LeRobot 形式に変換 (repo_id local/<データ名>) +# 4. 正規化の値をこの 50 本で測り直す (RUNPOD_TRAIN.md 4. の compute_norm_stats.py をこの設定で). +# 置く場所は openpi が学習で読む assets/<設定の名前>/local/<データ名>/norm_stats.json. +# 最後に「NORM sha256=<64 桁> path=」の1行を出す (lab_train.sh が由来に残し, 持ち帰る) +# 5. 学習を始めるコマンドを出す (流さない) +set -euo pipefail + +STEPS="${1:?学習のステップ数 (例 3000)}" +NAME=mecharm_redblock_box_deep_even10_v1 +CONFIG=pi05_mecharm_deep_even10_lora +EXP=deep_even10_v1 +N_MAC=50 +TAR=/workspace/data/${NAME}_raw_episodes.tar +DATA=/workspace/data/$NAME +OPENPI=/workspace/openpi +ADDON=/workspace/r2r2r/pi05/openpi_addon +NORM_DST=assets/$CONFIG/local/$NAME/norm_stats.json # $OPENPI からの相対 + +[ -f "$TAR" ] || { echo "[ERROR] tar がない: $TAR" >&2; exit 1; } +# shellcheck disable=SC1091 +[ -f "$HOME/.local/bin/env" ] && source "$HOME/.local/bin/env" + +echo "== 1. tar を広げる → $DATA" +# 前に広げたものへ上書きしない (pod_prep_trim_corr.sh と同じ. 空の一時フォルダに広げて確かめてから入れ替える) +TMP="$DATA.tmp.$$" +rm -rf "$TMP" +mkdir -p "$TMP" +tar xf "$TAR" -C "$TMP" +N=$(find "$TMP/raw_episodes" -mindepth 2 -maxdepth 2 -name episode.npz | wc -l | tr -d ' ') +[ "$N" = "$N_MAC" ] || { echo "[ERROR] tar のエピソードが $N 本 (Mac で作ったときは $N_MAC 本): $TAR" >&2; rm -rf "$TMP"; exit 1; } +rm -rf "$DATA" +mv "$TMP" "$DATA" +echo " エピソード $N 本 (Mac で作ったときは $N_MAC 本)" + +cd "$OPENPI" +echo "== 2. 学習設定 $CONFIG を足す (steps=$STEPS. π0.5 base から)" +python3 "$ADDON/add_deep_config.py" "$OPENPI" --steps "$STEPS" + +echo "== 3. LeRobot 形式に変換 (local/$NAME)" +uv run python /workspace/r2r2r/pi05/convert_to_lerobot.py --raw-dir "$DATA/raw_episodes" --repo-id "local/$NAME" + +echo "== 4. 正規化の値をこの $N 本で測り直す (compute_norm_stats.py --config-name $CONFIG)" +rm -f "$NORM_DST" # 前の回の値を残さない (測れなかったときに古い値で学習しない) +uv run scripts/compute_norm_stats.py --config-name "$CONFIG" +[ -s "$NORM_DST" ] || { echo "[ERROR] 正規化の値ができていない: $OPENPI/$NORM_DST" >&2; exit 1; } +SHA=$(sha256sum "$NORM_DST" | cut -d' ' -f1) +echo " $OPENPI/$NORM_DST (sha256 ${SHA:0:12}…)" +echo "NORM sha256=$SHA path=$NORM_DST" + +echo "== 5. 準備できました. 学習はこのコマンドで始める (このスクリプトでは流さない):" +echo " cd $OPENPI && XLA_PYTHON_CLIENT_MEM_FRACTION=0.9 uv run scripts/train.py $CONFIG --exp-name=$EXP --overwrite 2>&1 | tee /workspace/logs/train_$EXP.log" diff --git a/pi05/test_merge_and_config.py b/pi05/test_merge_and_config.py index eb4d17d..31cc80c 100644 --- a/pi05/test_merge_and_config.py +++ b/pi05/test_merge_and_config.py @@ -36,6 +36,7 @@ sys.path.insert(0, str(_HERE)) sys.path.insert(0, str(_HERE / "openpi_addon")) import add_trim_corr_config as atc # noqa: E402 +import add_deep_config as adc # noqa: E402 import build_correction_episodes as bce # noqa: E402 import merge_raw_episodes as mre # noqa: E402 @@ -333,6 +334,48 @@ class TestConfigV5(unittest.TestCase): self.assertEqual(cfg.read_text(), text) +class TestConfigDeep(unittest.TestCase): + """依頼 133: deep_even10_v1 の設定は trim と同じで, 名前・データ名・ステップ数だけ違う. π0.5 base から.""" + def test_same_as_trim_except_data_and_steps(self): + with tempfile.TemporaryDirectory() as td, mock.patch("sys.stdout"): + cfg = fake_config(Path(td)) + self.assertEqual(atc.main([td, "--variant", "v5", "--steps", "1500", "--init", + "/workspace/ckpt/corr_v2_1499/params"]), 0) # corr の設定と並べて入る + self.assertEqual(adc.main([td, "--steps", "3000"]), 0) + c = configs(cfg) + new, trim = c["pi05_mecharm_deep_even10_lora"], c["pi05_mecharm_trim_lora"] + self.assertIn("local/mecharm_redblock_box_deep_even10_v1", new["data"]) + self.assertEqual(new["num_train_steps"], "3000") + self.assertIn("gs://openpi-assets/checkpoints/pi05_base/params", new["weight_loader"]) + for k in set(trim) - {"name", "data", "num_train_steps"}: + self.assertEqual(new[k], trim[k], k) + self.assertEqual(new["data"].replace("deep_even10_v1", "teach_v1_trim2"), trim["data"]) + self.assertEqual(trim["num_train_steps"], "4000") # trim2 の設定は変えない + text = cfg.read_text() + self.assertEqual(adc.main([td, "--steps", "3000"]), 0) # 二重に入れない + self.assertEqual(cfg.read_text(), text) + with self.assertRaises(SystemExit): + adc.main([td, "--steps", "2000"]) # 違うステップ数では止まる + self.assertEqual(cfg.read_text(), text) + + def test_requires_trim_first(self): + with tempfile.TemporaryDirectory() as td: + cfg = Path(td) / "src" / "openpi" / "training" / "config.py" + cfg.parent.mkdir(parents=True) + cfg.write_text("_CONFIGS = []\n") + with self.assertRaises(AssertionError): + adc.main([td, "--steps", "3000"]) + + def test_pod_prep_deep_computes_norm(self): + sh = _HERE / "openpi_addon" / "pod_prep_deep.sh" + self.assertEqual(subprocess.run(["bash", "-n", str(sh)]).returncode, 0) + code = "\n".join(ln for ln in sh.read_text().splitlines() if not ln.lstrip().startswith("#")) + for x in ('uv run scripts/compute_norm_stats.py --config-name "$CONFIG"', "CONFIG=pi05_mecharm_deep_even10_lora", + "NAME=mecharm_redblock_box_deep_even10_v1", "NORM_DST=assets/$CONFIG/local/$NAME/norm_stats.json", + 'echo "NORM sha256=$SHA path=$NORM_DST"', 'add_deep_config.py" "$OPENPI" --steps "$STEPS"'): + self.assertIn(x, code) + + class TestPodPrep(unittest.TestCase): """依頼 48: pod_prep_trim_corr.sh は正規化の値を計算し直さず, 3999 の学習の trim2 の値を置く.""" SH = _HERE / "openpi_addon" / "pod_prep_trim_corr.sh" diff --git a/test_lab_start.sh b/test_lab_start.sh index c481953..394bc5c 100755 --- a/test_lab_start.sh +++ b/test_lab_start.sh @@ -412,13 +412,27 @@ check "corr2 のサーバーが動いている Pod に --model corr3 → 止ま rm -rf "$T" check "フォルダからモデルの名前 (corr3)" '[ "$(. "$HERE/lab_lib.sh"; lab_model_of_dir /workspace/ckpt/corr_v3_1499)" = corr3 ]' +echo "== 34d. --model deep1 (依頼 133) で新しく作る → π0.5 base から deep_even10_v1 で学習したモデルを取り, deep の設定・assets を足して起動" +setup; run --model deep1 +check "終了コード 0" '[ $RC = 0 ]' +check "deep1 のモデルを /workspace/ckpt/deep_v1_2999 に取る (ほかのモデルは取らない)" 'has ssh.log "hf download PixelstoMotors/pi05-mecharm-deep-even10-lora-v1 --local-dir /workspace/ckpt/deep_v1_2999" && ! has ssh.log "hf download PixelstoMotors/pi05-mecharm-trim-lora-v1 " && ! has ssh.log "hf download PixelstoMotors/pi05-mecharm-trim-corr-lora-v"' +check "3f で学習と同じ deep の設定 (3000 ステップ) を足し, モデルの assets (測り直した正規化の値) を写す" 'has ssh.log "add_deep_config.py /workspace/openpi --steps 3000" && ! has ssh.log "add_trim_corr_config.py" && has ssh.log "mkdir -p assets/pi05_mecharm_deep_even10_lora/local && cp -r /workspace/ckpt/deep_v1_2999/assets/local/mecharm_redblock_box_deep_even10_v1 assets/pi05_mecharm_deep_even10_lora/local/"' +check "deep1 の設定とフォルダでサーバーを起動" 'has ssh.log "--policy.config=pi05_mecharm_deep_even10_lora --policy.dir=/workspace/ckpt/deep_v1_2999 2>&1"' +check "クライアントに --model deep1 を渡す" 'has py.log "--model deep1 (PixelstoMotors/pi05-mecharm-deep-even10-lora-v1)"' +check "記録に model=deep1" '[ "$(jq -r .model "$T/data/lab/LogT/pod/lab_pod.json")" = deep1 ]' +rm -rf "$T" +setup; state $ID_OLD; pod 200 "$RUNNING"; probe 1 1 1 1 1; serve_dir /workspace/ckpt/corr_v3_1499; run --model deep1 +check "corr3 のサーバーが動いている Pod に --model deep1 → 止まり, --model corr3 を案内" '[ $RC = 1 ] && has out.txt "別のモデル (--model corr3)で動いています" && ! has ssh.log "tmux new" && ! has ssh.log "pkill"' +rm -rf "$T" +check "フォルダからモデルの名前 (deep1)" '[ "$(. "$HERE/lab_lib.sh"; lab_model_of_dir /workspace/ckpt/deep_v1_2999)" = deep1 ]' + echo "== 35. --model に知らない名前 → 何もせずに止まる" setup; run --model foo -check "終了コード 2・使える名前を出す・API を呼ばない" '[ $RC = 2 ] && has out.txt "--model は trim か corr か corr2 か corr3" && [ ! -s "$T/calls.log" ]' +check "終了コード 2・使える名前を出す・API を呼ばない" '[ $RC = 2 ] && has out.txt "--model は trim か corr か corr2 か corr3 か deep1" && [ ! -s "$T/calls.log" ]' rm -rf "$T" setup (cd "$T" && PATH="$T/bin:$PATH" perl -e 'alarm 10; exec @ARGV' ./lab_start.sh --log LogT --dry-run --model < /dev/null > "$T/out.txt" 2>&1); RC=$? -check "--model だけで値がない → 回り続けずに終了コード 2 (codex R1)" '[ $RC = 2 ] && has out.txt "--model には trim か corr か corr2 か corr3 を付ける"' +check "--model だけで値がない → 回り続けずに終了コード 2 (codex R1)" '[ $RC = 2 ] && has out.txt "--model には trim か corr か corr2 か corr3 か deep1 を付ける"' rm -rf "$T" for a in "--gpu" "--episode" "--log" "--gpu --dry-run" "--episode --log LogT" "--log --dry-run"; do setup diff --git a/test_lab_train.sh b/test_lab_train.sh index d0f84fd..e0de0fc 100755 --- a/test_lab_train.sh +++ b/test_lab_train.sh @@ -237,6 +237,50 @@ TD="$T/data/lab/LogT/train" check "--stop (--model なし): 記録の model=corr3 の設定・exp でログと一覧を取る" '[ $(n_call DELETE) = 1 ] && [ -f "$TD/train_trim_corr_v5.log" ] && has ssh.log "cd /workspace/openpi/checkpoints/pi05_mecharm_trim_corr_v5_lora/trim_corr_v5" && [ "$(jq -r .config.data_name "$TD/train_provenance.json")" = mecharm_redblock_box_teach_v1_trim2_corr_v5 ]' rm -rf "$T" +echo "== 12d. --model deep1 (依頼 133): π0.5 base から deep_even10_v1 で新しく学習する (正規化の値は Pod で測り直す)" +run success --model deep1 --limit-hours 7 +CKD="$T/data/checkpoints/pi05_mecharm_deep_even10_lora_v1" +NSHA=c1448b9e612668eedc320cc285053fcee62dc9afb40e435c70948441d486b8e2 +check "終了コード 0・画面に --model deep1 (π0.5 base → 上げる先)・steps 3000 (既定)・上限 7 時間" '[ $RC = 0 ] && has out.txt "--model deep1 (π0.5 base → PixelstoMotors/pi05-mecharm-deep-even10-lora-v1)" && has out.txt "steps=3000 init=base 上限 7 時間"' +check "3c は続きの元を取らない (hf download を Pod で流さない)" '! has ssh.log "hf download" && has out.txt "3c. 続きの元は取らない (π0.5 base から"' +check "3f は addon と trim2 の設定だけ (3999 の assets を写さない)" 'has ssh.log "add_trim_config.py /workspace/openpi" && ! has ssh.log "cp -r /workspace/ckpt/"' +check "deep_even10_v1 の tar だけを送り, pod_prep_deep.sh 3000 (pod_prep_trim_corr.sh は流さない)" 'has scp.log "mecharm_redblock_box_deep_even10_v1_raw_episodes.tar root@" && ! has scp.log "corr_v" && has ssh.log "bash /workspace/r2r2r/pi05/openpi_addon/pod_prep_deep.sh 3000 2>&1 | tee /workspace/logs/prep_deep.log" && ! has ssh.log "pod_prep_trim_corr.sh"' +check "測り直した正規化の値を学習の前に持ち帰る (lab//train/norm_stats.json・sha256 が Pod と同じ)" '[ "$(shasum -a 256 "$TD/norm_stats.json" | cut -d" " -f1)" = $NSHA ] && [ $(line_of out.txt "[NORM ] 正規化の値") -lt $(line_of out.txt "=== 5. 学習を始める") ] && [ "$(jq -r .norm_sha256 "$TD/lab_pod.json")" = $NSHA ]' +check "学習は pi05_mecharm_deep_even10_lora --exp-name=deep_even10_v1" 'has ssh.log "uv run scripts/train.py pi05_mecharm_deep_even10_lora --exp-name=deep_even10_v1 --overwrite > /workspace/logs/train_deep_even10_v1.log"' +check "最後は 2999 を Hugging Face の pi05-mecharm-deep-even10-lora-v1 へ上げ, KIOXIA の checkpoints/pi05_mecharm_deep_even10_lora_v1/2999 に assets ごと置く" 'has ssh.log "hf_upload_ckpt.py --dir /workspace/openpi/checkpoints/pi05_mecharm_deep_even10_lora/deep_even10_v1/2999 --repo PixelstoMotors/pi05-mecharm-deep-even10-lora-v1" && [ -f "$CKD/2999/_CHECKPOINT_METADATA" ] && [ -f "$CKD/2999/assets/local/mecharm_redblock_box_deep_even10_v1/norm_stats.json" ] && has out.txt "assets の正規化の値は測り直したものと sha256 が同じ"' +check "記録 lab_pod.json に model・設定・exp・init・steps" '[ "$(jq -r "[.model,.config,.exp,.init,.steps] | join(\" \")" "$TD/lab_pod.json")" = "deep1 pi05_mecharm_deep_even10_lora deep_even10_v1 base 3000" ]' +PD="$TD/train_provenance.json" +check "由来: base から (pi05_base・gs://)・データの tar の sha256・正規化の値の sha256" '[ "$(jq -r ".inputs[1] | [.kind,.id,.url,.path] | join(\" \")" "$PD")" = "checkpoint pi05_base gs://openpi-assets/checkpoints/pi05_base/params " ] && [ "$(jq -r ".config.init_params" "$PD")" = base ] && [ "$(jq -r ".inputs[0].tar | [.path,.sha256] | join(\" \")" "$PD")" = "datasets/mecharm_redblock_box_deep_even10_v1/mecharm_redblock_box_deep_even10_v1_raw_episodes.tar $(shasum -a 256 "$T/data/datasets/mecharm_redblock_box_deep_even10_v1/mecharm_redblock_box_deep_even10_v1_raw_episodes.tar" | cut -d" " -f1)" ] && [ "$(jq -r ".config.norm_stats.sha256" "$PD")" = $NSHA ]' +check "由来: 出力・チェックポイント側と Hugging Face にも同じもの" '[ "$(jq -r ".outputs[0] | [.path,.hf_repo] | join(\" \")" "$PD")" = "checkpoints/pi05_mecharm_deep_even10_lora_v1/2999 PixelstoMotors/pi05-mecharm-deep-even10-lora-v1" ] && cmp -s "$PD" "$CKD/2999/train_provenance.json" && has calls.log "hf upload PixelstoMotors/pi05-mecharm-deep-even10-lora-v1 train_provenance.json"' +check "続きの元のチェックポイント (3999 など) を偽物でも置かない・鍵を出さない" '[ ! -e "$T/data/checkpoints/pi05_mecharm_trim_lora_v1" ] && no_secret' +check "見張りは上限 7 時間 + 60 分" '[ "$(jq -r ".watchdog_epoch - .created_epoch" "$TD/lab_pod.json")" = 28800 ]' +rm -rf "$T" +run norm_missing --model deep1 +check "deep1: 正規化の値の sha256 が返らない → 学習を始めずログを持ち帰って消す" '[ $RC = 1 ] && has notify.log "正規化の値の sha256 が返らない" && ! has ssh.log "lt=start_train" && [ $(n_call DELETE) = 1 ] && [ -f "$TD/prep_deep.log" ]' +rm -rf "$T" +run norm_fetch_fail --model deep1 +check "deep1: 4b で正規化の値を持ち帰れなくても学習は続け, KIOXIA のチェックポイントの assets から持ち帰る (codex R1)" '[ $RC = 0 ] && has out.txt "正規化の値を持ち帰れなかった" && has out.txt "正規化の値をチェックポイントの assets から持ち帰った" && [ "$(shasum -a 256 "$TD/norm_stats.json" | cut -d" " -f1)" = c1448b9e612668eedc320cc285053fcee62dc9afb40e435c70948441d486b8e2 ] && [ "$(jq -r ".config.norm_stats.lab_copy" "$TD/train_provenance.json")" = "$TD/norm_stats.json" ] && [ "$(jq -r .norm_local "$TD/lab_pod.json")" = "$TD/norm_stats.json" ]' +rm -rf "$T" +run norm_bad --model deep1 +check "deep1: 正規化の値の中身が Pod の sha256 と違う → 学習のあと KIOXIA に置けたとせず失敗で通知 (codex R1)" '[ $RC = 1 ] && [ ! -e "$TD/norm_stats.json" ] && has notify.log "assets の正規化の値が測り直したもの" && has notify.log "測り直した正規化の値を lab/ に持ち帰れなかった" && [ $(n_call DELETE) = 1 ]' +rm -rf "$T" +PRE='D="$T/data/checkpoints/pi05_mecharm_deep_even10_lora_v1/2999"; while IFS=$(printf "\t") read -r f n; do mkdir -p "$D/$(dirname "$f")"; head -c "$n" /dev/zero > "$D/$f"; done <<< "$(printf "_CHECKPOINT_METADATA\t426\nassets/local/mecharm_redblock_box_deep_even10_v1/norm_stats.json\t2120\nparams/_METADATA\t33052\nparams/d/0123abcd\t1048576\nparams/manifest.ocdbt\t118\n")"; head -c 2120 /dev/zero | tr "\0" x > "$D/assets/local/mecharm_redblock_box_deep_even10_v1/norm_stats.json"' +run success --model deep1 +PRE="" +check "deep1: KIOXIA に 2999 がもとからあり, 名前・バイト数は同じでも正規化の値の中身が違う → 合ったとせず上書きもしない (codex R1)" '[ $RC = 1 ] && has notify.log "名前・バイト数は Pod 側と同じだが assets の正規化の値が測り直したもの" && [ "$(head -c 1 "$T/data/checkpoints/pi05_mecharm_deep_even10_lora_v1/2999/assets/local/mecharm_redblock_box_deep_even10_v1/norm_stats.json")" = x ]' +rm -rf "$T" +rm -rf "$T" +run success --model deep1 --init /workspace/ckpt/3999/params +check "deep1 に --init で続きの元を渡す → Pod を借りずに止まる" '[ $RC = 1 ] && has out.txt "π0.5 base から学習する (--init /workspace/ckpt/3999/params は使えない)" && [ $(n_call "POST /pods") = 0 ]' +rm -rf "$T" +T=$(mktemp -d); mkdir -p "$T/data/lab/LogT/train" "$T/fake" +jq -n '{id:"dryrunpod0001", kind:"train", model:"deep1", config:"pi05_mecharm_deep_even10_lora", gpu:"NVIDIA A40", costPerHr:0.44, created_epoch:1793449178, created_at:"2026-10-31 21:19:38", exp:"deep_even10_v1", steps:3000, init:"base", train_started_epoch:1793449300, norm_sha256:"c1448b9e612668eedc320cc285053fcee62dc9afb40e435c70948441d486b8e2", norm_pod:"assets/pi05_mecharm_deep_even10_lora/local/mecharm_redblock_box_deep_even10_v1/norm_stats.json", norm_local:""}' > "$T/data/lab/LogT/train/lab_pod.json" +echo "NVIDIA A40" > "$T/fake/created" +(cd "$HERE" && LAB_DRY_ROOT="$T" ./lab_train.sh --log LogT --dry-run --stop < /dev/null > "$T/out.txt" 2>&1); RC=$? +TD="$T/data/lab/LogT/train" +check "--stop (--model なし): 記録の model=deep1 の設定・exp・準備のログでログと一覧を取り, 由来に正規化の値の sha256" '[ $(n_call DELETE) = 1 ] && [ -f "$TD/train_deep_even10_v1.log" ] && [ -f "$TD/prep_deep.log" ] && has ssh.log "cd /workspace/openpi/checkpoints/pi05_mecharm_deep_even10_lora/deep_even10_v1" && [ "$(jq -r .config.data_name "$TD/train_provenance.json")" = mecharm_redblock_box_deep_even10_v1 ] && [ "$(jq -r .config.norm_stats.sha256 "$TD/train_provenance.json")" = c1448b9e612668eedc320cc285053fcee62dc9afb40e435c70948441d486b8e2 ]' +rm -rf "$T" + echo "== 13. 見守る途中で止められても Pod を残さない (TERM. Ctrl-C・HUP も同じ扱い)" T=$(mktemp -d) (cd "$HERE" && LAB_DRY_ROOT="$T" LAB_FAKE_SCENARIO=timeout LAB_DRY_POLL_SEC=1 ./lab_train.sh --log LogT --dry-run --limit-hours 1 < /dev/null > "$T/out.txt" 2>&1) & ``` ## 学習の記録 ― Runpod の A40 で3,000ステップ（KIOXIA の lab/Log27/train/） ここの記録は、lab_train.sh が持ち帰ったものをそのまま載せています。Pod の中の時刻は UTC です。ターミナルの記録には、色を付ける制御の文字（ESC）が混ざっている行があります。 ### 学習のまとめ（result_20261109_153622.txt） ```text lab_train.sh Log27 2026-11-09 21:50:59 結果: 成功: 学習が終わりました (step 2999, loss 0.0036). Hugging Face と KIOXIA に置き, Pod を消しました. 372 分 24 秒 (6.21 時間) × 1時間 0.59 ドル = 約 3.66 ドル (約 579 円. 1ドル=158円) 学習: pi05_mecharm_deep_even10_lora --exp-name=deep_even10_v1 最後のチェックポイント 2999 最後の loss 0.0036 (Step 2950) Hugging Face (PixelstoMotors/pi05-mecharm-deep-even10-lora-v1): ok PixelstoMotors/pi05-mecharm-deep-even10-lora-v1 に 32 ファイル 6,335,846,833 バイトを上げ, 名前・バイト数が全部合うのを確かめた KIOXIA: /Volumes/KIOXIA/r2r2r-data/checkpoints/pi05_mecharm_deep_even10_lora_v1/2999 に置いた (32 ファイル 6335846833 バイト. Pod 側と名前・バイト数が全部合った). assets の正規化の値は測り直したものと sha256 が同じ ログを持ち帰った → /Volumes/KIOXIA/r2r2r-data/lab/Log27/train Pod (id arn304g7xu5wfk) は消した (API で確かめた) 借りていた時間 372 分 24 秒 (6.21 時間) × 1時間 0.59 ドル = 約 3.66 ドル (約 579 円. 1ドル=158円) 正規化の値 (Pod で測り直した. sha256 813fcb8cccd147c1d49e5c06e695587142d4535ff7c1f27b9fc94ea6aee66ce4): /Volumes/KIOXIA/r2r2r-data/lab/Log27/train/norm_stats.json 由来: /Volumes/KIOXIA/r2r2r-data/lab/Log27/train/train_provenance.json ・ /Volumes/KIOXIA/r2r2r-data/checkpoints/pi05_mecharm_deep_even10_lora_v1/2999/train_provenance.json ・ Hugging Face (PixelstoMotors/pi05-mecharm-deep-even10-lora-v1) にも置いた ターミナルの記録: /Volumes/KIOXIA/r2r2r-data/lab/Log27/train/lab_train_20261109_153622.txt ``` ### 学習のターミナルの記録（lab_train_20261109_153622.txt） ```text [LOG ] ターミナルの出力を残します → /Volumes/KIOXIA/r2r2r-data/lab/Log27/train/lab_train_20261109_153622.txt [LAB ] 2026-11-09 15:36:22 学習 GPU=NVIDIA A40 (借りられなければ NVIDIA RTX A6000 NVIDIA L40S) --model deep1 (π0.5 base → PixelstoMotors/pi05-mecharm-deep-even10-lora-v1) データ mecharm_redblock_box_deep_even10_v1 pi05_mecharm_deep_even10_lora --exp-name=deep_even10_v1 steps=3000 init=base 上限 7 時間 記録 /Volumes/KIOXIA/r2r2r-data/lab/Log27/train === 0. 送るものと Mac の道具を確かめる (15:36:22) [DATA ] tar の sha256 c7097302c784… (由来に残す) [DATA ] tar 193353728 バイト (/Volumes/KIOXIA/r2r2r-data/datasets/mecharm_redblock_box_deep_even10_v1/mecharm_redblock_box_deep_even10_v1_raw_episodes.tar) [PROV ] 始める時点のコード (8db7028) と入力 3 つの指紋を測った [MAC ] 終わるまでスリープしないようにした (caffeinate pid 8863). ふたは閉じない (1 秒) === 1. Pod を作る (15:36:23) [POD ] NVIDIA A40 を借ります (空きがなければ 30 秒おきに 5 回まで) [POD ] 作った: id arn304g7xu5wfk GPU=NVIDIA A40 1時間 0.59 ドル (記録 /Volumes/KIOXIA/r2r2r-data/lab/Log27/train/lab_pod.json) [POD ] 上限: 11-09 22:36 (7 時間) を過ぎたら, ログを持ち帰って消します (3 秒) === 2. Pod が立ち上がるまで待つ (15:36:26) [POD ] SSH の IP とポートを API から受け取った (:) [POD ] 入れた: 59541e1c2271 NVIDIA A40, 46068 MiB [WATCH ] Pod 側の見張りを置いた: 11-09 23:36 を過ぎたら Pod が自分を消す (Pod 用の鍵. Mac の鍵は送っていない) (57 秒) === 3a. スクリプトを送る (15:37:23) (75 秒) === 3b. huggingface_hub を入れ, 鍵を渡す (15:38:38) WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager, possibly rendering your system unusable. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv. Use the --root-user-action option if you know what you are doing and want to suppress this warning. Hint: The `hf-cli` skill is not installed. Run `hf skills add -g` to teach your AI agents how to use the `hf` CLI. ✓ Logged in user: PixelstoMotors (10 秒) === 3c. 続きの元は取らない (π0.5 base から. 重みは学習の最初に openpi が取る) (15:38:48) (0 秒) === 3d. openpi を取る (215abfb) (15:38:48) 215abfb docs(droid): fix config search instruction (#1023) (20 秒) === 3e. uv sync (15:39:08) downloading uv 0.13.7 x86_64-unknown-linux-gnu WARN: The following commands are shadowed by other commands in your PATH: uv uvx Using CPython 3.11.13 interpreter at: /usr/bin/python3.11 Creating virtual environment at: .venv Resolved 279 packages in 4ms Building openpi-client @ file:///workspace/openpi/packages/openpi-client Updating https://github.com/huggingface/lerobot (0cf864870cf29f4738d3ade893e6fd13fbd7cdb5) Building openpi @ file:///workspace/openpi Downloading hf-transfer (3.4MiB) Downloading torchvision (7.1MiB) Downloading imageio-ffmpeg (28.1MiB) Downloading opencv-python (60.1MiB) Downloading virtualenv (5.8MiB) Downloading pyarrow (40.3MiB) Downloading matplotlib (8.2MiB) Downloading h5py (4.3MiB) Downloading diffusers (3.4MiB) Downloading nvidia-nvjitlink-cu12 (18.8MiB) Downloading lxml (4.7MiB) Downloading opencv-python-headless (47.7MiB) Downloading polars (34.6MiB) Downloading cmake (26.6MiB) Downloading hf-xet (4.9MiB) Downloading pillow (4.4MiB) Downloading nvidia-cuda-nvrtc-cu12 (22.6MiB) Downloading dm-control (37.5MiB) Downloading ruff (11.0MiB) Downloading nvidia-cublas-cu12 (374.9MiB) Downloading nvidia-curand-cu12 (53.7MiB) Downloading mujoco (4.4MiB) Downloading nvidia-cuda-cupti-cu12 (8.5MiB) Downloading nvidia-nccl-cu12 (192.0MiB) Downloading nvidia-cusparselt-cu12 (149.5MiB) Downloading nvidia-cusparse-cu12 (206.5MiB) Downloading nvidia-cusolver-cu12 (150.9MiB) Downloading pyopengl (3.0MiB) Downloading wandb (20.4MiB) Downloading fonttools (4.7MiB) Downloading numpy (17.4MiB) Downloading labmaze (4.7MiB) Downloading numcodecs (8.4MiB) Downloading nvidia-cufft-cu12 (190.9MiB) Downloading rerun-sdk (68.1MiB) Downloading numba (3.7MiB) Downloading transformers (10.3MiB) Downloading jaxlib (100.2MiB) Downloading pandas (12.5MiB) Downloading nvidia-cuda-nvcc-cu12 (38.7MiB) Downloading jax-cuda12-plugin (15.9MiB) Downloading jax-cuda12-pjrt (99.8MiB) Downloading av (34.7MiB) Downloading triton (148.5MiB) Downloading llvmlite (40.4MiB) Downloading torch (783.1MiB) Downloading scipy (35.9MiB) Downloading tensorstore (17.5MiB) Downloading nvidia-cudnn-cu12 (544.5MiB) Downloading sympy (6.0MiB) Downloaded pyopengl Downloading debugpy (3.0MiB) Downloaded diffusers Downloaded hf-transfer Downloading tokenizers (2.9MiB) Downloading jax (2.3MiB) Downloaded numba Downloading widgetsnbextension (2.1MiB) Downloaded h5py Downloading ml-dtypes (2.1MiB) Downloaded pillow Downloaded mujoco Downloading networkx (1.9MiB) Downloading pydantic-core (1.9MiB) Downloaded fonttools Downloaded labmaze Downloaded lxml Downloading aiohttp (1.6MiB) Downloading jedi (1.5MiB) Downloading kiwisolver (1.4MiB) Downloaded hf-xet Downloading torchcodec (1.3MiB) Downloaded virtualenv Downloading sentencepiece (1.2MiB) Downloaded jax Downloading pygments (1.2MiB) Downloaded widgetsnbextension Downloading setuptools (1.1MiB) Downloaded sympy Downloaded debugpy Downloading pymunk (1.0MiB) Downloading nvidia-cufile-cu12 (1.1MiB) Downloaded kiwisolver Downloaded tokenizers Downloaded torchcodec Downloaded ml-dtypes Downloaded aiohttp Downloaded networkx Downloaded pydantic-core Downloaded jedi Downloaded torchvision Downloaded sentencepiece Downloaded pygments Downloaded setuptools Downloaded pymunk Downloaded nvidia-cufile-cu12 Building evdev==1.9.2 Downloaded nvidia-cuda-cupti-cu12 Downloaded matplotlib Downloaded numcodecs Building antlr4-python3-runtime==4.9.3 Downloaded transformers Downloaded ruff Downloaded pandas Built openpi-client @ file:///workspace/openpi/packages/openpi-client Built openpi @ file:///workspace/openpi Downloaded jax-cuda12-plugin Built antlr4-python3-runtime==4.9.3 Downloaded numpy Downloaded tensorstore Downloaded nvidia-nvjitlink-cu12 Downloaded wandb Downloaded nvidia-cuda-nvrtc-cu12 Downloaded imageio-ffmpeg Downloaded cmake Built evdev==1.9.2 Downloaded polars Downloaded av Downloaded scipy Downloaded nvidia-cuda-nvcc-cu12 Downloaded dm-control Downloaded pyarrow Downloaded llvmlite Updated https://github.com/huggingface/lerobot (0cf864870cf29f4738d3ade893e6fd13fbd7cdb5) Building lerobot @ git+https://github.com/huggingface/lerobot@0cf864870cf29f4738d3ade893e6fd13fbd7cdb5 Downloaded opencv-python-headless Downloaded nvidia-curand-cu12 Downloaded opencv-python Built lerobot @ git+https://github.com/huggingface/lerobot@0cf864870cf29f4738d3ade893e6fd13fbd7cdb5 Downloaded rerun-sdk Downloaded jax-cuda12-pjrt Downloaded jaxlib Downloaded nvidia-cusparselt-cu12 Downloaded triton Downloaded nvidia-cusolver-cu12 Downloaded nvidia-nccl-cu12 Downloaded nvidia-cufft-cu12 Downloaded nvidia-cusparse-cu12 Downloaded nvidia-cublas-cu12 Downloaded nvidia-cudnn-cu12 Downloaded torch Prepared 240 packages in 15.21s Installed 240 packages in 1.87s + absl-py==2.3.0 + aiohappyeyeballs==2.6.1 + aiohttp==3.12.4 + aiosignal==1.3.2 + annotated-types==0.7.0 + antlr4-python3-runtime==4.9.3 + asttokens==3.0.0 + attrs==25.3.0 + augmax==0.4.1 + av==17.0.0 + beartype==0.19.0 + beautifulsoup4==4.13.4 + blinker==1.9.0 + cachetools==5.5.2 + certifi==2025.4.26 + cffi==1.17.1 + cfgv==3.4.0 + charset-normalizer==3.4.2 + chex==0.1.90 + click==8.2.1 + cloudpickle==3.1.1 + cmake==4.0.2 + comm==0.2.2 + contourpy==1.3.2 + crc32c==2.7.1 + cycler==0.12.1 + datasets==3.6.0 + debugpy==1.8.14 + decorator==5.2.1 + deepdiff==8.5.0 + diffusers==0.33.1 + dill==0.3.8 + distlib==0.3.9 + dm-control==1.0.14 + dm-env==1.6 + dm-tree==0.1.9 + docker-pycreds==0.4.0 + docstring-parser==0.16 + donfig==0.8.1.post1 + draccus==0.10.0 + einops==0.8.1 + equinox==0.12.2 + etils==1.12.2 + evdev==1.9.2 + executing==2.2.0 + farama-notifications==0.0.4 + filelock==3.18.0 + flask==3.1.1 + flatbuffers==25.2.10 + flax==0.10.2 + fonttools==4.58.1 + frozenlist==1.6.0 + fsspec==2025.3.0 + gcsfs==2025.3.0 + gdown==5.2.0 + gitdb==4.0.12 + gitpython==3.1.44 + glfw==2.9.0 + google-api-core==2.24.2 + google-auth==2.40.2 + google-auth-oauthlib==1.2.2 + google-cloud-core==2.4.3 + google-cloud-storage==3.1.0 + google-crc32c==1.7.1 + google-resumable-media==2.7.2 + googleapis-common-protos==1.70.0 + gym-aloha==0.1.1 + gymnasium==0.29.1 + h5py==3.13.0 + hf-transfer==0.1.9 + hf-xet==1.1.2 + huggingface-hub==0.32.3 + humanize==4.12.3 + identify==2.6.12 + idna==3.10 + imageio==2.37.0 + imageio-ffmpeg==0.6.0 + importlib-metadata==8.7.0 + importlib-resources==6.5.2 + iniconfig==2.1.0 + inquirerpy==0.3.4 + ipykernel==6.29.5 + ipython==9.2.0 + ipython-pygments-lexers==1.1.1 + ipywidgets==8.1.7 + itsdangerous==2.2.0 + jax==0.5.3 + jax-cuda12-pjrt==0.5.3 + jax-cuda12-plugin==0.5.3 + jaxlib==0.5.3 + jaxtyping==0.2.36 + jedi==0.19.2 + jinja2==3.1.6 + jsonlines==4.0.0 + jupyter-client==8.6.3 + jupyter-core==5.8.1 + jupyterlab-widgets==3.0.15 + kiwisolver==1.4.8 + labmaze==1.0.6 + lerobot==0.1.0 (from git+https://github.com/huggingface/lerobot@0cf864870cf29f4738d3ade893e6fd13fbd7cdb5) + llvmlite==0.44.0 + lxml==5.4.0 + markdown-it-py==3.0.0 + markupsafe==3.0.2 + matplotlib==3.10.3 + matplotlib-inline==0.1.7 + mdurl==0.1.2 + mergedeep==1.3.4 + ml-collections==1.0.0 + ml-dtypes==0.4.1 + mpmath==1.3.0 + msgpack==1.1.0 + mujoco==2.3.7 + multidict==6.4.4 + multiprocess==0.70.16 + mypy-extensions==1.1.0 + nest-asyncio==1.6.0 + networkx==3.5 + nodeenv==1.9.1 + numba==0.61.2 + numcodecs==0.16.1 + numpy==1.26.4 + numpydantic==1.6.9 + nvidia-cublas-cu12==12.6.4.1 + nvidia-cuda-cupti-cu12==12.6.80 + nvidia-cuda-nvcc-cu12==12.9.41 + nvidia-cuda-nvrtc-cu12==12.6.77 + nvidia-cuda-runtime-cu12==12.6.77 + nvidia-cudnn-cu12==9.5.1.17 + nvidia-cufft-cu12==11.3.0.4 + nvidia-cufile-cu12==1.11.1.6 + nvidia-curand-cu12==10.3.7.77 + nvidia-cusolver-cu12==11.7.1.2 + nvidia-cusparse-cu12==12.5.4.2 + nvidia-cusparselt-cu12==0.6.3 + nvidia-ml-py==12.575.51 + nvidia-nccl-cu12==2.26.2 + nvidia-nvjitlink-cu12==12.6.85 + nvidia-nvtx-cu12==12.6.77 + oauthlib==3.2.2 + omegaconf==2.3.0 + opencv-python==4.11.0.86 + opencv-python-headless==4.11.0.86 + openpi==0.1.0 (from file:///workspace/openpi) + openpi-client==0.1.0 (from file:///workspace/openpi/packages/openpi-client) + opt-einsum==3.4.0 + optax==0.2.4 + orbax-checkpoint==0.11.13 + orderly-set==5.4.1 + packaging==25.0 + pandas==2.2.3 + parso==0.8.4 + pexpect==4.9.0 + pfzy==0.3.4 + pillow==11.2.1 + platformdirs==4.3.8 + pluggy==1.6.0 + polars==1.30.0 + pre-commit==4.2.0 + prompt-toolkit==3.0.51 + propcache==0.3.1 + proto-plus==1.26.1 + protobuf==4.25.8 + psutil==7.0.0 + ptyprocess==0.7.0 + pure-eval==0.2.3 + pyarrow==20.0.0 + pyasn1==0.6.1 + pyasn1-modules==0.4.2 + pycparser==2.22 + pydantic==2.11.5 + pydantic-core==2.33.2 + pygments==2.19.1 + pymunk==7.0.0 + pynput==1.8.1 + pynvml==12.0.0 + pyopengl==3.1.9 + pyparsing==3.2.3 + pysocks==1.7.1 + pytest==8.3.5 + python-dateutil==2.9.0.post0 + python-xlib==0.33 + pytz==2025.2 + pyyaml==6.0.2 + pyyaml-include==1.4.1 + pyzmq==26.4.0 + regex==2024.11.6 + requests==2.32.3 + requests-oauthlib==2.0.0 + rerun-sdk==0.23.1 + rich==14.0.0 + rsa==4.9.1 + ruff==0.11.12 + safetensors==0.5.3 + scipy==1.15.3 + sentencepiece==0.2.0 + sentry-sdk==2.29.1 + setproctitle==1.3.6 + setuptools==80.9.0 + shtab==1.7.2 + simplejson==3.20.1 + six==1.17.0 + smmap==5.0.2 + soupsieve==2.7 + stack-data==0.6.3 + sympy==1.14.0 + tensorstore==0.1.74 + termcolor==3.1.0 + tokenizers==0.21.1 + toml==0.10.2 + toolz==1.0.0 + torch==2.7.1 + torchcodec==0.4.0 + torchvision==0.22.1 + tornado==6.5.1 + tqdm==4.67.1 + tqdm-loggable==0.2 + traitlets==5.14.3 + transformers==4.53.2 + treescope==0.1.9 + triton==3.3.1 + typeguard==4.4.2 + typing-extensions==4.13.2 + typing-inspect==0.9.0 + typing-inspection==0.4.1 + tyro==0.9.22 + tzdata==2025.2 + urllib3==2.4.0 + virtualenv==20.31.2 + wadler-lindig==0.1.6 + wandb==0.19.11 + wcwidth==0.2.13 + websockets==15.0.1 + werkzeug==3.1.3 + widgetsnbextension==4.0.14 + wrapt==1.14.1 + xxhash==3.5.0 + yarl==1.20.0 + zarr==3.0.8 + zipp==3.22.0 (21 秒) === 3f. addon・設定・assets (15:39:29) installed: policies/mecharm_policy.py + config.py (pi05_mecharm_lora) installed: pi05_mecharm_trim_lora (repo_id=local/mecharm_redblock_box_teach_v1_trim2, 4,000 steps) [PROV ] openpi 215abfb217dbac7d5f1273282331b9b1866c0479 (2 ファイル変更・addon と設定を入れたあと) (7 秒) === 4a. tar を送る (15:39:36) -rwx------ 1 root root 193353728 Nov 9 06:41 /workspace/data/mecharm_redblock_box_deep_even10_v1_raw_episodes.tar (133 秒) === 4b. pod_prep_deep.sh 3000 (変換・正規化の値を測り直す) (15:41:49) == 1. tar を広げる → /workspace/data/mecharm_redblock_box_deep_even10_v1 エピソード 50 本 (Mac で作ったときは 50 本) == 2. 学習設定 pi05_mecharm_deep_even10_lora を足す (steps=3000. π0.5 base から) installed: pi05_mecharm_deep_even10_lora (repo_id=local/mecharm_redblock_box_deep_even10_v1, 3000 steps, init=π0.5 base) == 3. LeRobot 形式に変換 (local/mecharm_redblock_box_deep_even10_v1) Map: 0%| | 0/118 [00:00&1 | tee /workspace/logs/train_deep_even10_v1.log [NORM ] 正規化の値 (sha256 813fcb8cccd1…) を持ち帰った → /Volumes/KIOXIA/r2r2r-data/lab/Log27/train/norm_stats.json (158 秒) === 5. 学習を始める (15:44:27) [TRAIN ] 学習を始めました (Pod の中で setsid nohup. Mac とのつながりが切れても続く) ログ /workspace/logs/train_deep_even10_v1.log GPU /workspace/logs/gpu_train.csv (30 秒おき) (6 秒) === 6. 学習を見守る (5 分おき) (15:44:33) [TRAIN ] 15:44 step -/3000 loss - 経過 0:00 上限まで 6:51 ======================================================================== 学習は Pod の中で回っています. ここまで来れば寝てかまいません. このターミナルは閉じない・Mac のふたは閉じない (電源につないだまま). 終わると通知が出ます. まとめは /Volumes/KIOXIA/r2r2r-data/lab/Log27/train/result_*.txt ======================================================================== [TRAIN ] 15:49 step -/3000 loss - 経過 0:05 上限まで 6:46 [TRAIN ] 15:54 step 34/3000 loss 0.1015 (Step 0 の行) 経過 0:10 上限まで 6:41 [TRAIN ] 15:59 step 78/3000 loss 0.0611 (Step 50 の行) 経過 0:15 上限まで 6:36 [TRAIN ] 16:04 step 120/3000 loss 0.0422 (Step 100 の行) 経過 0:20 上限まで 6:31 [TRAIN ] 16:09 step 162/3000 loss 0.0296 (Step 150 の行) 経過 0:25 上限まで 6:26 [TRAIN ] 16:14 step 206/3000 loss 0.0213 (Step 200 の行) 経過 0:30 上限まで 6:21 [TRAIN ] 16:19 step 248/3000 loss 0.0213 (Step 200 の行) 経過 0:35 上限まで 6:16 [TRAIN ] 16:24 step 292/3000 loss 0.0186 (Step 250 の行) 経過 0:40 上限まで 6:11 [TRAIN ] 16:30 step 334/3000 loss 0.0169 (Step 300 の行) 経過 0:45 上限まで 6:06 [TRAIN ] 16:35 step 376/3000 loss 0.0143 (Step 350 の行) 経過 0:50 上限まで 6:01 [TRAIN ] 16:40 step 420/3000 loss 0.0124 (Step 400 の行) 経過 0:55 上限まで 5:56 [TRAIN ] 16:45 step 462/3000 loss 0.0126 (Step 450 の行) 経過 1:00 上限まで 5:51 [TRAIN ] 16:50 step 506/3000 loss 0.0113 (Step 500 の行) 経過 1:05 上限まで 5:46 [TRAIN ] 16:55 step 548/3000 loss 0.0113 (Step 500 の行) 経過 1:10 上限まで 5:41 [TRAIN ] 17:00 step 590/3000 loss 0.0116 (Step 550 の行) 経過 1:15 上限まで 5:36 [TRAIN ] 17:05 step 634/3000 loss 0.0111 (Step 600 の行) 経過 1:20 上限まで 5:31 [TRAIN ] 17:10 step 676/3000 loss 0.0100 (Step 650 の行) 経過 1:25 上限まで 5:26 [TRAIN ] 17:15 step 720/3000 loss 0.0086 (Step 700 の行) 経過 1:30 上限まで 5:21 [TRAIN ] 17:20 step 762/3000 loss 0.0089 (Step 750 の行) 経過 1:35 上限まで 5:15 [TRAIN ] 17:25 step 806/3000 loss 0.0083 (Step 800 の行) 経過 1:40 上限まで 5:10 [TRAIN ] 17:30 step 848/3000 loss 0.0083 (Step 800 の行) 経過 1:46 上限まで 5:05 [TRAIN ] 17:35 step 890/3000 loss 0.0083 (Step 850 の行) 経過 1:51 上限まで 5:00 [TRAIN ] 17:40 step 934/3000 loss 0.0080 (Step 900 の行) 経過 1:56 上限まで 4:55 [TRAIN ] 17:45 step 976/3000 loss 0.0075 (Step 950 の行) 経過 2:01 上限まで 4:50 [TRAIN ] 17:50 step 1020/3000 loss 0.0072 (Step 1000 の行) 経過 2:06 上限まで 4:45 [TRAIN ] 17:55 step 1060/3000 loss 0.0069 (Step 1050 の行) 経過 2:11 上限まで 4:40 [TRAIN ] 18:00 step 1100/3000 loss 0.0075 (Step 1100 の行) 経過 2:16 上限まで 4:35 [TRAIN ] 18:05 step 1150/3000 loss 0.0075 (Step 1100 の行) 経過 2:21 上限まで 4:30 [TRAIN ] 18:10 step 1190/3000 loss 0.0066 (Step 1150 の行) 経過 2:26 上限まで 4:25 [TRAIN ] 18:16 step 1230/3000 loss 0.0066 (Step 1200 の行) 経過 2:31 上限まで 4:20 [TRAIN ] 18:21 step 1280/3000 loss 0.0062 (Step 1250 の行) 経過 2:36 上限まで 4:15 [TRAIN ] 18:26 step 1320/3000 loss 0.0064 (Step 1300 の行) 経過 2:41 上限まで 4:10 [TRAIN ] 18:31 step 1360/3000 loss 0.0063 (Step 1350 の行) 経過 2:46 上限まで 4:05 [TRAIN ] 18:36 step 1400/3000 loss 0.0061 (Step 1400 の行) 経過 2:51 上限まで 4:00 [TRAIN ] 18:41 step 1450/3000 loss 0.0061 (Step 1400 の行) 経過 2:56 上限まで 3:55 [TRAIN ] 18:46 step 1490/3000 loss 0.0058 (Step 1450 の行) 経過 3:01 上限まで 3:50 [TRAIN ] 18:51 step 1530/3000 loss 0.0058 (Step 1500 の行) 経過 3:06 上限まで 3:45 [TRAIN ] 18:56 step 1580/3000 loss 0.0059 (Step 1550 の行) 経過 3:11 上限まで 3:40 [TRAIN ] 19:01 step 1620/3000 loss 0.0058 (Step 1600 の行) 経過 3:16 上限まで 3:35 [TRAIN ] 19:06 step 1660/3000 loss 0.0058 (Step 1650 の行) 経過 3:21 上限まで 3:29 [TRAIN ] 19:11 step 1700/3000 loss 0.0056 (Step 1700 の行) 経過 3:26 上限まで 3:24 [TRAIN ] 19:16 step 1750/3000 loss 0.0056 (Step 1700 の行) 経過 3:32 上限まで 3:19 [TRAIN ] 19:21 step 1790/3000 loss 0.0054 (Step 1750 の行) 経過 3:37 上限まで 3:14 [TRAIN ] 19:26 step 1830/3000 loss 0.0056 (Step 1800 の行) 経過 3:42 上限まで 3:09 [TRAIN ] 19:31 step 1880/3000 loss 0.0055 (Step 1850 の行) 経過 3:47 上限まで 3:04 [TRAIN ] 19:36 step 1920/3000 loss 0.0053 (Step 1900 の行) 経過 3:52 上限まで 2:59 [TRAIN ] 19:41 step 1960/3000 loss 0.0053 (Step 1950 の行) 経過 3:57 上限まで 2:54 [TRAIN ] 19:46 step 2000/3000 loss 0.0048 (Step 2000 の行) 経過 4:02 上限まで 2:49 [TRAIN ] 19:51 step 2050/3000 loss 0.0048 (Step 2000 の行) 経過 4:07 上限まで 2:44 [TRAIN ] 19:56 step 2090/3000 loss 0.0055 (Step 2050 の行) 経過 4:12 上限まで 2:39 [TRAIN ] 20:02 step 2130/3000 loss 0.0047 (Step 2100 の行) 経過 4:17 上限まで 2:34 [TRAIN ] 20:07 step 2180/3000 loss 0.0046 (Step 2150 の行) 経過 4:22 上限まで 2:29 [TRAIN ] 20:12 step 2220/3000 loss 0.0047 (Step 2200 の行) 経過 4:27 上限まで 2:24 [TRAIN ] 20:17 step 2260/3000 loss 0.0046 (Step 2250 の行) 経過 4:32 上限まで 2:19 [TRAIN ] 20:22 step 2300/3000 loss 0.0047 (Step 2300 の行) 経過 4:37 上限まで 2:14 [TRAIN ] 20:27 step 2350/3000 loss 0.0047 (Step 2300 の行) 経過 4:42 上限まで 2:09 [TRAIN ] 20:32 step 2390/3000 loss 0.0041 (Step 2350 の行) 経過 4:47 上限まで 2:04 [TRAIN ] 20:37 step 2430/3000 loss 0.0044 (Step 2400 の行) 経過 4:52 上限まで 1:59 [TRAIN ] 20:42 step 2480/3000 loss 0.0043 (Step 2450 の行) 経過 4:57 上限まで 1:54 [TRAIN ] 20:47 step 2520/3000 loss 0.0043 (Step 2500 の行) 経過 5:02 上限まで 1:48 [TRAIN ] 20:52 step 2560/3000 loss 0.0042 (Step 2550 の行) 経過 5:07 上限まで 1:43 [TRAIN ] 20:57 step 2600/3000 loss 0.0046 (Step 2600 の行) 経過 5:13 上限まで 1:38 [TRAIN ] 21:02 step 2650/3000 loss 0.0046 (Step 2600 の行) 経過 5:18 上限まで 1:33 [TRAIN ] 21:07 step 2690/3000 loss 0.0045 (Step 2650 の行) 経過 5:23 上限まで 1:28 [TRAIN ] 21:12 step 2730/3000 loss 0.0042 (Step 2700 の行) 経過 5:28 上限まで 1:23 [TRAIN ] 21:17 step 2780/3000 loss 0.0041 (Step 2750 の行) 経過 5:33 上限まで 1:18 [TRAIN ] 21:22 step 2820/3000 loss 0.0038 (Step 2800 の行) 経過 5:38 上限まで 1:13 [TRAIN ] 21:27 step 2860/3000 loss 0.0045 (Step 2850 の行) 経過 5:43 上限まで 1:08 [TRAIN ] 21:32 step 2900/3000 loss 0.0037 (Step 2900 の行) 経過 5:48 上限まで 1:03 [TRAIN ] 21:37 step 2950/3000 loss 0.0037 (Step 2900 の行) 経過 5:53 上限まで 0:58 [TRAIN ] 21:42 step 2990/3000 loss 0.0036 (Step 2950 の行) 経過 5:58 上限まで 0:53 [TRAIN ] 21:48 step 3000/3000 loss 0.0036 (Step 2950 の行) 経過 6:03 上限まで 0:48 (21808 秒) [TRAIN ] 学習が終わりました (最後のチェックポイント 2999, loss 0.0036) [WATCH ] Pod 側の見張りを 00:48 まで延ばした (上げる・持ち帰る間に消えないように) === 7a. Pod 側の一覧 (2999) (21:48:04) [LIST ] 32 ファイル 6335846833 バイト (train_state なし) → /Volumes/KIOXIA/r2r2r-data/lab/Log27/train/pod_files_2999.tsv (2 秒) === 7b. Hugging Face へ上げる (PixelstoMotors/pi05-mecharm-deep-even10-lora-v1・非公開) (21:48:06) [HF ] 上げるもの: 32 ファイル 6,335,846,833 バイト (train_state/ は上げない) HFUP ok PixelstoMotors/pi05-mecharm-deep-even10-lora-v1 に 32 ファイル 6,335,846,833 バイトを上げ, 名前・バイト数が全部合うのを確かめた (14 秒) === 7d. ログを持ち帰る (21:48:20) [LOG ] train_deep_even10_v1.log を持ち帰った → /Volumes/KIOXIA/r2r2r-data/lab/Log27/train/train_deep_even10_v1.log (1800 行) [LOG ] gpu_train.csv を持ち帰った → /Volumes/KIOXIA/r2r2r-data/lab/Log27/train/gpu_train.csv (729 行) [LOG ] train_run.log を持ち帰った → /Volumes/KIOXIA/r2r2r-data/lab/Log27/train/train_run.log (2 行) [LOG ] prep_deep.log を持ち帰った → /Volumes/KIOXIA/r2r2r-data/lab/Log27/train/prep_deep.log (165 行) [LOG ] watchdog.log は Pod にない (持ち帰れなかった) [LOG ] Pod 側のチェックポイントの一覧 → /Volumes/KIOXIA/r2r2r-data/lab/Log27/train/pod_checkpoints.tsv (240 ファイル) (28 秒) === Pod を消す (id arn304g7xu5wfk) (21:48:48) [STOP ] 削除 (Terminate) を送った (21:48:50) [STOP ] Pod (id arn304g7xu5wfk) が消えたことを API で確かめた [COST ] 借りていた時間 372 分 24 秒 (6.21 時間) × 1時間 0.59 ドル = 約 3.66 ドル (約 579 円. 1ドル=158円) GPU=NVIDIA A40 === 7e. KIOXIA へ Hugging Face から取る (21:48:51) [KIOXIA] Hugging Face (PixelstoMotors/pi05-mecharm-deep-even10-lora-v1) から取ります → /Volumes/KIOXIA/r2r2r-data/checkpoints/pi05_mecharm_deep_even10_lora_v1/.incoming_2999_20261109_214851_8808 Still waiting to acquire lock on /Volumes/KIOXIA/r2r2r-data/checkpoints/pi05_mecharm_deep_even10_lora_v1/.incoming_2999_20261109_214851_8808/.cache/huggingface/.gitignore.lock (elapsed: 0.1 seconds) Still waiting to acquire lock on /Volumes/KIOXIA/r2r2r-data/checkpoints/pi05_mecharm_deep_even10_lora_v1/.incoming_2999_20261109_214851_8808/.cache/huggingface/.gitignore.lock (elapsed: 0.1 seconds) path=/Volumes/KIOXIA/r2r2r-data/checkpoints/pi05_mecharm_deep_even10_lora_v1/.incoming_2999_20261109_214851_8808 [KIOXIA] /Volumes/KIOXIA/r2r2r-data/checkpoints/pi05_mecharm_deep_even10_lora_v1/2999 に置いた (32 ファイル 6335846833 バイト. Pod 側と名前・バイト数が全部合った). assets の正規化の値は測り直したものと sha256 が同じ (125 秒) === 由来 (train_provenance.json) を書く [PROV ] /Volumes/KIOXIA/r2r2r-data/lab/Log27/train/train_provenance.json [PROV ] /Volumes/KIOXIA/r2r2r-data/checkpoints/pi05_mecharm_deep_even10_lora_v1/2999/train_provenance.json [PROV ] hf: url=https://huggingface.co/PixelstoMotors/pi05-mecharm-deep-even10-lora-v1/commit/94ed63d4f688ba9d73873f4b7312fa5564350280 [PROV ] 由来: /Volumes/KIOXIA/r2r2r-data/lab/Log27/train/train_provenance.json ・ /Volumes/KIOXIA/r2r2r-data/checkpoints/pi05_mecharm_deep_even10_lora_v1/2999/train_provenance.json ・ Hugging Face (PixelstoMotors/pi05-mecharm-deep-even10-lora-v1) にも置いた === かかった時間 0. 送るものと Mac の道具を確かめる 1 秒 1. Pod を作る 3 秒 2. Pod が立ち上がるまで待つ 57 秒 3a. スクリプトを送る 75 秒 3b. huggingface_hub を入れ, 鍵を渡す 10 秒 3c. 続きの元は取らない (π0.5 base から. 重みは学習の最初に openpi が取る) 0 秒 3d. openpi を取る (215abfb) 20 秒 3e. uv sync 21 秒 3f. addon・設定・assets 7 秒 4a. tar を送る 133 秒 4b. pod_prep_deep.sh 3000 (変換・正規化の値を測り直す) 158 秒 5. 学習を始める 6 秒 6. 学習を見守る (5 分おき) 21808 秒 7a. Pod 側の一覧 (2999) 2 秒 7b. Hugging Face へ上げる (PixelstoMotors/pi05-mecharm-deep-even10-lora-v1・非公開) 14 秒 7d. ログを持ち帰る 28 秒 7e. KIOXIA へ Hugging Face から取る 125 秒 [RESULT] 成功: 学習が終わりました (step 2999, loss 0.0036). Hugging Face と KIOXIA に置き, Pod を消しました. 372 分 24 秒 (6.21 時間) × 1時間 0.59 ドル = 約 3.66 ドル (約 579 円. 1ドル=158円) [RESULT] まとめ → /Volumes/KIOXIA/r2r2r-data/lab/Log27/train/result_20261109_153622.txt ``` ### Pod の記録（lab_pod.json） ```json { "id": "arn304g7xu5wfk", "kind": "train", "model": "deep1", "gpu": "NVIDIA A40", "log": "Log27", "costPerHr": 0.59, "created_epoch": 1794206186, "created_at": "2026-11-09 15:36:26", "config": "pi05_mecharm_deep_even10_lora", "exp": "deep_even10_v1", "steps": 3000, "limit_sec": 25200, "limit_at": "2026-11-09 22:36:26", "init": "base", "limit_hours": "7", "prov_start": { "code": { "repo": "r2r2r-scripts", "commit": "8db7028ab50d6b30b7bcc1395891f4668a5a1ce8", "dirty": false, "dirty_files": [], "untracked_backups": [ "pi05/build_demo_episodes.py.bak_pre_LOG352_20261019" ], "diff_sha256": null, "measured_at": "2026-11-09T15:36:23+09:00", "paths": [ "lab_train.sh", "lab_lib.sh", "provenance.py", "pi05" ] }, "inputs": [ { "kind": "dataset", "id": "mecharm_redblock_box_deep_even10_v1", "path": "datasets/mecharm_redblock_box_deep_even10_v1", "fingerprint": "files-v2:9701f85562a810d1ac863d732b2494d4dacc224c88e0a3099a523c78825f40ad", "tar": { "path": "datasets/mecharm_redblock_box_deep_even10_v1/mecharm_redblock_box_deep_even10_v1_raw_episodes.tar", "sha256": "c7097302c7846be914c3ef61263bf443b0e6945f24770efd545e3813089747c8" }, "measured_at": "2026-11-09T15:36:23+09:00" }, { "kind": "checkpoint", "id": "pi05_base", "path": null, "fingerprint": null, "hf_repo": null, "pod_path": null, "url": "gs://openpi-assets/checkpoints/pi05_base/params", "note": "π0.5 base から新しく学習 (続きの学習ではない)", "measured_at": "2026-11-09T15:36:23+09:00" }, { "kind": "config", "id": "add_trim_config.py", "path": "lab/Log11/pod/add_trim_config.py", "fingerprint": "files-v2:3d7a5ce690b205880da44fa4b20de1b5905b7fb96a91580413ecb5cd660db44c", "sha256": "94059deb462623946344a5285f7ebd69c8041b5d820ed28fd4d383f7922f38a9", "measured_at": "2026-11-09T15:36:23+09:00" } ] }, "watchdog_epoch": 1794234986, "watchdog": "WATCHDOG run=1 pod=1 key=1 ctl=1", "openpi": [ { "name": "openpi", "commit": "215abfb217dbac7d5f1273282331b9b1866c0479", "dirty": true, "dirty_files": [ "src/openpi/training/config.py", "src/openpi/policies/mecharm_policy.py" ], "expected_commit": "215abfb" } ], "norm_sha256": "813fcb8cccd147c1d49e5c06e695587142d4535ff7c1f27b9fc94ea6aee66ce4", "norm_pod": "assets/pi05_mecharm_deep_even10_lora/local/mecharm_redblock_box_deep_even10_v1/norm_stats.json", "norm_local": "/Volumes/KIOXIA/r2r2r-data/lab/Log27/train/norm_stats.json", "train_started_epoch": 1794206673, "train_started_at": "2026-11-09 15:44:33", "terminated_epoch": 1794228530, "terminated_at": "2026-11-09 21:48:50", "rented_sec": 22344, "gone_confirmed": true } ``` ### 由来の記録（train_provenance.json） ```json { "schema_version": "provenance/1", "created_at": "2026-11-09T21:50:56+09:00", "producer": { "script": "lab_train.sh", "argv": [ "--log", "Log27", "--model", "deep1", "--steps", "3000", "--limit-hours", "7" ], "machine": "shimadzueiichinoMacBook-Pro.local", "platform": "Darwin 25.6.0", "python": "3.12.12", "dry_run": false }, "code": { "repo": "r2r2r-scripts", "commit": "8db7028ab50d6b30b7bcc1395891f4668a5a1ce8", "dirty": false, "dirty_files": [], "untracked_backups": [ "pi05/build_demo_episodes.py.bak_pre_LOG352_20261019" ], "diff_sha256": null, "measured_at": "2026-11-09T15:36:23+09:00", "paths": [ "lab_train.sh", "lab_lib.sh", "provenance.py", "pi05" ] }, "external_code": [ { "name": "openpi", "commit": "215abfb217dbac7d5f1273282331b9b1866c0479", "dirty": true, "dirty_files": [ "src/openpi/training/config.py", "src/openpi/policies/mecharm_policy.py" ], "expected_commit": "215abfb" } ], "inputs": [ { "kind": "dataset", "id": "mecharm_redblock_box_deep_even10_v1", "path": "datasets/mecharm_redblock_box_deep_even10_v1", "fingerprint": "files-v2:9701f85562a810d1ac863d732b2494d4dacc224c88e0a3099a523c78825f40ad", "tar": { "path": "datasets/mecharm_redblock_box_deep_even10_v1/mecharm_redblock_box_deep_even10_v1_raw_episodes.tar", "sha256": "c7097302c7846be914c3ef61263bf443b0e6945f24770efd545e3813089747c8" }, "measured_at": "2026-11-09T15:36:23+09:00" }, { "kind": "checkpoint", "id": "pi05_base", "path": null, "fingerprint": null, "hf_repo": null, "pod_path": null, "url": "gs://openpi-assets/checkpoints/pi05_base/params", "note": "π0.5 base から新しく学習 (続きの学習ではない)", "measured_at": "2026-11-09T15:36:23+09:00" }, { "kind": "config", "id": "add_trim_config.py", "path": "lab/Log11/pod/add_trim_config.py", "fingerprint": "files-v2:3d7a5ce690b205880da44fa4b20de1b5905b7fb96a91580413ecb5cd660db44c", "sha256": "94059deb462623946344a5285f7ebd69c8041b5d820ed28fd4d383f7922f38a9", "measured_at": "2026-11-09T15:36:23+09:00" } ], "outputs": [ { "kind": "checkpoint", "id": "pi05_mecharm_deep_even10_lora_v1/2999", "path": "checkpoints/pi05_mecharm_deep_even10_lora_v1/2999", "fingerprint": "files-v2:761d82c6043bc4492883f7eabb900dae8e458a004c7309159600ef3e0497e850", "hf_repo": "PixelstoMotors/pi05-mecharm-deep-even10-lora-v1" } ], "config": { "name": "pi05_mecharm_deep_even10_lora", "exp_name": "deep_even10_v1", "num_train_steps": 3000, "init_params": "base", "data_name": "mecharm_redblock_box_deep_even10_v1", "limit_hours": 7, "checkpoint_step": "2999", "norm_stats": { "source": "Pod で compute_norm_stats.py を回して測り直した (依頼 133)", "sha256": "813fcb8cccd147c1d49e5c06e695587142d4535ff7c1f27b9fc94ea6aee66ce4", "pod_path": "/workspace/openpi/assets/pi05_mecharm_deep_even10_lora/local/mecharm_redblock_box_deep_even10_v1/norm_stats.json", "lab_copy": "/Volumes/KIOXIA/r2r2r-data/lab/Log27/train/norm_stats.json" } }, "compute": { "provider": "runpod", "pod_id": "arn304g7xu5wfk", "gpu": "NVIDIA A40", "started_at": "2026-11-09T15:36:26+09:00", "train_started_at": "2026-11-09T15:44:33+09:00", "ended_at": "2026-11-09T21:48:50+09:00", "duration_sec": 22344, "cost_per_hr_usd": 0.59, "cost_usd": 3.66, "yen_per_usd": 158, "cost_line": "借りていた時間 372 分 24 秒 (6.21 時間) × 1時間 0.59 ドル = 約 3.66 ドル (約 579 円. 1ドル=158円)" }, "numbering": { "series": "フィジカルAI研究室", "series_log": "Log27", "article_no": "" }, "status": { "process": "completed", "task": "not_applicable" }, "notes": { "hugging_face": "ok PixelstoMotors/pi05-mecharm-deep-even10-lora-v1 に 32 ファイル 6,335,846,833 バイトを上げ, 名前・バイト数が全部合うのを確かめた", "kioxia": "/Volumes/KIOXIA/r2r2r-data/checkpoints/pi05_mecharm_deep_even10_lora_v1/2999 に置いた (32 ファイル 6335846833 バイト. Pod 側と名前・バイト数が全部合った). assets の正規化の値は測り直したものと sha256 が同じ" } } ``` ### 測り直した正規化の値（norm_stats.json） ```json { "norm_stats": { "state": { "mean": [ 2.1063313484191895, 46.03452682495117, 4.411215782165527, 3.1770036220550537, -50.12554168701172, -31.57465934753418, 0.43327710032463074 ], "std": [ 32.542823791503906, 20.7677059173584, 11.630378723144531, 4.709203720092773, 20.20290184020996, 3.3034868240356445, 0.49552804231643677 ], "q01": [ -61.328340319061276, 1.4899800171852111, -33.8316815612793, -4.052775578594208, -75.5969573038578, -40.60079463157654, 0.0 ], "q99": [ 44.55244774932862, 87.87266738977432, 30.300980303955072, 16.343755064582826, -0.26367661064863057, -26.275863413238525, 0.9998 ] }, "actions": { "mean": [ 0.26515433192253113, 0.0839180126786232, -0.02417420595884323, 0.03403187915682793, -0.15184271335601807, -0.040091510862112045, 0.4332770109176636 ], "std": [ 9.135751724243164, 9.129268646240234, 3.5192644596099854, 1.4412405490875244, 9.11519718170166, 0.7829089164733887, 0.49552804231643677 ], "q01": [ -21.13576020050049, -22.60392705345154, -11.015750971221925, -4.7546747798919675, -25.785808544158936, -2.6870099975585937, 0.0 ], "q99": [ 27.837298337936403, 23.288965936660766, 9.811358605575563, 3.7456359291076673, 25.752579555511474, 2.849487951660157, 0.9998 ] } } } ``` ### LeRobot の形への変換と正規化の値の測り直しのログ（prep_deep.log） ```text == 1. tar を広げる → /workspace/data/mecharm_redblock_box_deep_even10_v1 エピソード 50 本 (Mac で作ったときは 50 本) == 2. 学習設定 pi05_mecharm_deep_even10_lora を足す (steps=3000. π0.5 base から) installed: pi05_mecharm_deep_even10_lora (repo_id=local/mecharm_redblock_box_deep_even10_v1, 3000 steps, init=π0.5 base) == 3. LeRobot 形式に変換 (local/mecharm_redblock_box_deep_even10_v1) Map: 0%| | 0/118 [00:00&1 | tee /workspace/logs/train_deep_even10_v1.log ``` ### 学習のログ（train_deep_even10_v1.log） ```text 06:44:38.992 [I] Running on: 59541e1c2271 (1728:train.py:196) INFO:2026-11-09 06:44:39,482:jax._src.xla_bridge:925: Unable to initialize backend 'rocm': module 'jaxlib.xla_extension' has no attribute 'GpuAllocatorConfig' 06:44:39.482 [I] Unable to initialize backend 'rocm': module 'jaxlib.xla_extension' has no attribute 'GpuAllocatorConfig' (1728:xla_bridge.py:925) INFO:2026-11-09 06:44:39,483:jax._src.xla_bridge:925: Unable to initialize backend 'tpu': INTERNAL: Failed to open libtpu.so: libtpu.so: cannot open shared object file: No such file or directory 06:44:39.483 [I] Unable to initialize backend 'tpu': INTERNAL: Failed to open libtpu.so: libtpu.so: cannot open shared object file: No such file or directory (1728:xla_bridge.py:925) 06:44:40.530 [I] Created BasePyTreeCheckpointHandler: use_ocdbt=True, use_zarr3=False, pytree_metadata_options=PyTreeMetadataOptions(support_rich_types=False), array_metadata_store= (1728:base_pytree_checkpoint_handler.py:334) 06:44:40.530 [I] Created BasePyTreeCheckpointHandler: use_ocdbt=True, use_zarr3=False, pytree_metadata_options=PyTreeMetadataOptions(support_rich_types=False), array_metadata_store= (1728:base_pytree_checkpoint_handler.py:334) 06:44:40.530 [I] [thread=MainThread] Failed to get flag value for EXPERIMENTAL_ORBAX_USE_DISTRIBUTED_PROCESS_ID. (1728:multihost.py:390) 06:44:40.531 [I] [process=0][thread=MainThread] CheckpointManager init: checkpointers=None, item_names=None, item_handlers={'assets': , 'train_state': , 'params': }, handler_registry=None (1728:checkpoint_manager.py:620) 06:44:40.531 [I] Deferred registration for item: "assets". Adding handler `` for item "assets" and save args `` and restore args `` to `_handler_registry`. (1728:composite_checkpoint_handler.py:234) 06:44:40.531 [I] Deferred registration for item: "train_state". Adding handler `` for item "train_state" and save args `` and restore args `` to `_handler_registry`. (1728:composite_checkpoint_handler.py:234) 06:44:40.531 [I] Deferred registration for item: "params". Adding handler `` for item "params" and save args `` and restore args `` to `_handler_registry`. (1728:composite_checkpoint_handler.py:234) 06:44:40.531 [I] Deferred registration for item: "metrics". Adding handler `` for item "metrics" and save args `` and restore args `` to `_handler_registry`. (1728:composite_checkpoint_handler.py:234) 06:44:40.531 [I] Initialized registry DefaultCheckpointHandlerRegistry({('assets', ): , ('assets', ): , ('train_state', ): , ('train_state', ): , ('params', ): , ('params', ): , ('metrics', ): , ('metrics', ): }). (1728:composite_checkpoint_handler.py:502) 06:44:40.532 [I] orbax-checkpoint version: 0.11.13 (1728:abstract_checkpointer.py:35) 06:44:40.532 [I] [process=0][thread=MainThread] Using barrier_sync_fn: . at 0x7bbfd97e3ba0> timeout: 7200 secs and primary_host=0 for async checkpoint writes (1728:async_checkpointer.py:170) 06:44:40.532 [I] Found 0 checkpoint steps in /workspace/openpi/checkpoints/pi05_mecharm_deep_even10_lora/deep_even10_v1 (1728:checkpoint_manager.py:1701) 06:44:40.532 [I] [process=0][thread=MainThread] CheckpointManager created, primary_host=0, CheckpointManagerOptions=CheckpointManagerOptions(save_interval_steps=1, max_to_keep=1, keep_time_interval=None, keep_period=1000, should_keep_fn=None, best_fn=None, best_mode='max', keep_checkpoints_without_metrics=True, step_prefix=None, step_format_fixed_length=None, step_name_format=None, create=False, cleanup_tmp_directories=False, save_on_steps=frozenset(), single_host_load_and_broadcast=False, todelete_subdir=None, enable_background_delete=False, read_only=False, enable_async_checkpointing=True, async_options=AsyncOptions(timeout_secs=7200, barrier_sync_fn=None, post_finalization_callback=None, create_directories_asynchronously=True), multiprocessing_options=MultiprocessingOptions(primary_host=0, active_processes=None, barrier_sync_key_prefix=None), should_save_fn=None, file_options=FileOptions(path_permission_mode=None), save_root_metadata=True, temporary_path_class=None, save_decision_policy=None, prevent_write_metrics=False), root_directory=/workspace/openpi/checkpoints/pi05_mecharm_deep_even10_lora/deep_even10_v1: (1728:checkpoint_manager.py:801) 06:44:40.639 [I] Loaded norm stats from /workspace/openpi/assets/pi05_mecharm_deep_even10_lora/local/mecharm_redblock_box_deep_even10_v1 (1728:config.py:197) 06:44:40.640 [I] data_config: DataConfig(repo_id='local/mecharm_redblock_box_deep_even10_v1', asset_id='local/mecharm_redblock_box_deep_even10_v1', norm_stats={'state': NormStats(mean=array([ 2.10633135, 46.03452682, 4.41121578, 3.17700362, -50.12554169, -31.57465935, 0.4332771 ]), std=array([32.54282379, 20.76770592, 11.63037872, 4.70920372, 20.20290184, 3.30348682, 0.49552804]), q01=array([-61.32834032, 1.48998002, -33.83168156, -4.05277558, -75.5969573 , -40.60079463, 0. ]), q99=array([ 44.55244775, 87.87266739, 30.3009803 , 16.34375506, -0.26367661, -26.27586341, 0.9998 ])), 'actions': NormStats(mean=array([ 0.26515433, 0.08391801, -0.02417421, 0.03403188, -0.15184271, -0.04009151, 0.43327701]), std=array([9.13575172, 9.12926865, 3.51926446, 1.44124055, 9.11519718, 0.78290892, 0.49552804]), q01=array([-21.1357602 , -22.60392705, -11.01575097, -4.75467478, -25.78580854, -2.68701 , 0. ]), q99=array([27.83729834, 23.28896594, 9.81135861, 3.74563593, 25.75257956, 2.84948795, 0.9998 ]))}, repack_transforms=Group(inputs=[RepackTransform(structure={'observation/image': 'image', 'observation/state': 'state', 'actions': 'actions', 'prompt': 'prompt'})], outputs=()), data_transforms=Group(inputs=(MecharmInputs(model_type=), DeltaActions(mask=(True, True, True, True, True, True, False))), outputs=(AbsoluteActions(mask=(True, True, True, True, True, True, False)), MecharmOutputs())), model_transforms=Group(inputs=[InjectDefaultPrompt(prompt=None), ResizeImages(height=224, width=224), TokenizePrompt(tokenizer=, discrete_state_input=True), PadStatesAndActions(model_action_dim=32)], outputs=()), use_quantile_norm=True, action_sequence_keys=('actions',), prompt_from_task=True, rlds_data_dir=None, action_space=None, datasets=()) (1728:data_loader.py:243) 06:44:40.922 [I] local_batch_size: 32 (1728:data_loader.py:324) 06:44:56.278 [I] Initialized data loader: [0].images['base_0_rgb']: (32, 224, 224, 3)@float32 [0].images['left_wrist_0_rgb']: (32, 224, 224, 3)@float32 [0].images['right_wrist_0_rgb']: (32, 224, 224, 3)@float32 [0].image_masks['base_0_rgb']: (32,)@bool [0].image_masks['left_wrist_0_rgb']: (32,)@bool [0].image_masks['right_wrist_0_rgb']: (32,)@bool [0].state: (32, 32)@float32 [0].tokenized_prompt: (32, 200)@int32 [0].tokenized_prompt_mask: (32, 200)@bool [1]: (32, 10, 32)@float32 (1728:train.py:227) 06:44:57.542 [I] Downloading gs://openpi-assets/checkpoints/pi05_base/params to /root/.cache/openpi/openpi-assets/checkpoints/pi05_base/params (1728:download.py:86) 06:44:57.542 [W] gsutil not found, falling back to gcsfs. This may fail if GCP credentials are not configured correctly. (1728:download.py:111) 06:45:24.528 [I] Progress on: -/12441721931 rate:- remaining:? elapsed:00:00 postfix:- (1728:tqdm_logging.py:145) 06:45:34.571 [I] Progress on: 5.99GiB/11.6GiB rate:648,602,439.3iB/s remaining:00:09 elapsed:00:10 postfix:- (1728:tqdm_logging.py:145) 06:45:44.611 [I] Progress on: 11.4GiB/11.6GiB rate:434,047,087.2iB/s remaining:00:00 elapsed:00:20 postfix:- (1728:tqdm_logging.py:145) 06:45:46.615 [I] Progress on: 11.6GiB/11.6GiB rate:371,273,818.2iB/s remaining:00:00 elapsed:00:22 postfix:- (1728:tqdm_logging.py:145) 06:45:46.622 [I] Created BasePyTreeCheckpointHandler: use_ocdbt=True, use_zarr3=False, pytree_metadata_options=PyTreeMetadataOptions(support_rich_types=False), array_metadata_store= (1728:base_pytree_checkpoint_handler.py:334) 06:45:46.676 [I] Restoring checkpoint from /root/.cache/openpi/openpi-assets/checkpoints/pi05_base/params. (1728:checkpointer.py:298) 06:45:54.128 [I] [process=0] /jax/checkpoint/read/bytes_per_sec: 1.7 GiB/s (total bytes: 12.5 GiB) (time elapsed: 7 seconds) (per-host) (1728:base_pytree_checkpoint_handler.py:114) 06:45:54.129 [I] Finished restoring checkpoint in 7.45 seconds from /root/.cache/openpi/openpi-assets/checkpoints/pi05_base/params. (1728:checkpointer.py:309) 06:46:11.201 [I] Initialized train state: ['PaliGemma']['img']['Transformer']['encoder_norm']['bias'].value: (1152,)@float32 ['PaliGemma']['img']['Transformer']['encoder_norm']['scale'].value: (1152,)@float32 ['PaliGemma']['img']['Transformer']['encoderblock']['LayerNorm_0']['bias'].value: (27, 1152)@float32 ['PaliGemma']['img']['Transformer']['encoderblock']['LayerNorm_0']['scale'].value: (27, 1152)@float32 ['PaliGemma']['img']['Transformer']['encoderblock']['LayerNorm_1']['bias'].value: (27, 1152)@float32 ['PaliGemma']['img']['Transformer']['encoderblock']['LayerNorm_1']['scale'].value: (27, 1152)@float32 ['PaliGemma']['img']['Transformer']['encoderblock']['MlpBlock_0']['Dense_0']['bias'].value: (27, 4304)@float32 ['PaliGemma']['img']['Transformer']['encoderblock']['MlpBlock_0']['Dense_0']['kernel'].value: (27, 1152, 4304)@float32 ['PaliGemma']['img']['Transformer']['encoderblock']['MlpBlock_0']['Dense_1']['bias'].value: (27, 1152)@float32 ['PaliGemma']['img']['Transformer']['encoderblock']['MlpBlock_0']['Dense_1']['kernel'].value: (27, 4304, 1152)@float32 ['PaliGemma']['img']['Transformer']['encoderblock']['MultiHeadDotProductAttention_0']['key']['bias'].value: (27, 16, 72)@float32 ['PaliGemma']['img']['Transformer']['encoderblock']['MultiHeadDotProductAttention_0']['key']['kernel'].value: (27, 1152, 16, 72)@float32 ['PaliGemma']['img']['Transformer']['encoderblock']['MultiHeadDotProductAttention_0']['out']['bias'].value: (27, 1152)@float32 ['PaliGemma']['img']['Transformer']['encoderblock']['MultiHeadDotProductAttention_0']['out']['kernel'].value: (27, 16, 72, 1152)@float32 ['PaliGemma']['img']['Transformer']['encoderblock']['MultiHeadDotProductAttention_0']['query']['bias'].value: (27, 16, 72)@float32 ['PaliGemma']['img']['Transformer']['encoderblock']['MultiHeadDotProductAttention_0']['query']['kernel'].value: (27, 1152, 16, 72)@float32 ['PaliGemma']['img']['Transformer']['encoderblock']['MultiHeadDotProductAttention_0']['value']['bias'].value: (27, 16, 72)@float32 ['PaliGemma']['img']['Transformer']['encoderblock']['MultiHeadDotProductAttention_0']['value']['kernel'].value: (27, 1152, 16, 72)@float32 ['PaliGemma']['img']['embedding']['bias'].value: (1152,)@float32 ['PaliGemma']['img']['embedding']['kernel'].value: (14, 14, 3, 1152)@float32 ['PaliGemma']['img']['head']['bias'].value: (2048,)@float32 ['PaliGemma']['img']['head']['kernel'].value: (1152, 2048)@float32 ['PaliGemma']['img']['pos_embedding'].value: (1, 256, 1152)@float32 ['PaliGemma']['llm']['embedder']['input_embedding'].value: (257152, 2048)@bfloat16 ['PaliGemma']['llm']['final_norm']['scale'].value: (2048,)@bfloat16 ['PaliGemma']['llm']['final_norm_1']['Dense_0']['bias'].value: (3072,)@bfloat16 ['PaliGemma']['llm']['final_norm_1']['Dense_0']['kernel'].value: (1024, 3072)@bfloat16 ['PaliGemma']['llm']['layers']['attn']['attn_vec_einsum']['lora_a'].value: (18, 8, 256, 16)@float32 ['PaliGemma']['llm']['layers']['attn']['attn_vec_einsum']['lora_b'].value: (18, 8, 16, 2048)@float32 ['PaliGemma']['llm']['layers']['attn']['attn_vec_einsum']['w'].value: (18, 8, 256, 2048)@bfloat16 ['PaliGemma']['llm']['layers']['attn']['attn_vec_einsum_1']['lora_a'].value: (18, 8, 256, 32)@float32 ['PaliGemma']['llm']['layers']['attn']['attn_vec_einsum_1']['lora_b'].value: (18, 8, 32, 1024)@float32 ['PaliGemma']['llm']['layers']['attn']['attn_vec_einsum_1']['w'].value: (18, 8, 256, 1024)@bfloat16 ['PaliGemma']['llm']['layers']['attn']['kv_einsum']['lora_a'].value: (18, 2, 1, 2048, 16)@float32 ['PaliGemma']['llm']['layers']['attn']['kv_einsum']['lora_b'].value: (18, 2, 1, 16, 256)@float32 ['PaliGemma']['llm']['layers']['attn']['kv_einsum']['w'].value: (18, 2, 1, 2048, 256)@bfloat16 ['PaliGemma']['llm']['layers']['attn']['kv_einsum_1']['lora_a'].value: (18, 2, 1, 1024, 32)@float32 ['PaliGemma']['llm']['layers']['attn']['kv_einsum_1']['lora_b'].value: (18, 2, 1, 32, 256)@float32 ['PaliGemma']['llm']['layers']['attn']['kv_einsum_1']['w'].value: (18, 2, 1, 1024, 256)@bfloat16 ['PaliGemma']['llm']['layers']['attn']['q_einsum']['lora_a'].value: (18, 8, 2048, 16)@float32 ['PaliGemma']['llm']['layers']['attn']['q_einsum']['lora_b'].value: (18, 8, 16, 256)@float32 ['PaliGemma']['llm']['layers']['attn']['q_einsum']['w'].value: (18, 8, 2048, 256)@bfloat16 ['PaliGemma']['llm']['layers']['attn']['q_einsum_1']['lora_a'].value: (18, 8, 1024, 32)@float32 ['PaliGemma']['llm']['layers']['attn']['q_einsum_1']['lora_b'].value: (18, 8, 32, 256)@float32 ['PaliGemma']['llm']['layers']['attn']['q_einsum_1']['w'].value: (18, 8, 1024, 256)@bfloat16 ['PaliGemma']['llm']['layers']['mlp']['gating_einsum'].value: (18, 2, 2048, 16384)@bfloat16 ['PaliGemma']['llm']['layers']['mlp']['gating_einsum_lora_a'].value: (18, 2, 2048, 16)@float32 ['PaliGemma']['llm']['layers']['mlp']['gating_einsum_lora_b'].value: (18, 2, 16, 16384)@float32 ['PaliGemma']['llm']['layers']['mlp']['linear'].value: (18, 16384, 2048)@bfloat16 ['PaliGemma']['llm']['layers']['mlp']['linear_lora_a'].value: (18, 16384, 16)@float32 ['PaliGemma']['llm']['layers']['mlp']['linear_lora_b'].value: (18, 16, 2048)@float32 ['PaliGemma']['llm']['layers']['mlp_1']['gating_einsum'].value: (18, 2, 1024, 4096)@bfloat16 ['PaliGemma']['llm']['layers']['mlp_1']['gating_einsum_lora_a'].value: (18, 2, 1024, 32)@float32 ['PaliGemma']['llm']['layers']['mlp_1']['gating_einsum_lora_b'].value: (18, 2, 32, 4096)@float32 ['PaliGemma']['llm']['layers']['mlp_1']['linear'].value: (18, 4096, 1024)@bfloat16 ['PaliGemma']['llm']['layers']['mlp_1']['linear_lora_a'].value: (18, 4096, 32)@float32 ['PaliGemma']['llm']['layers']['mlp_1']['linear_lora_b'].value: (18, 32, 1024)@float32 ['PaliGemma']['llm']['layers']['pre_attention_norm']['scale'].value: (18, 2048)@bfloat16 ['PaliGemma']['llm']['layers']['pre_attention_norm_1']['Dense_0']['bias'].value: (18, 3072)@bfloat16 ['PaliGemma']['llm']['layers']['pre_attention_norm_1']['Dense_0']['kernel'].value: (18, 1024, 3072)@bfloat16 ['PaliGemma']['llm']['layers']['pre_ffw_norm']['scale'].value: (18, 2048)@bfloat16 ['PaliGemma']['llm']['layers']['pre_ffw_norm_1']['Dense_0']['bias'].value: (18, 3072)@bfloat16 ['PaliGemma']['llm']['layers']['pre_ffw_norm_1']['Dense_0']['kernel'].value: (18, 1024, 3072)@bfloat16 ['action_in_proj']['bias'].value: (1024,)@float32 ['action_in_proj']['kernel'].value: (32, 1024)@float32 ['action_out_proj']['bias'].value: (32,)@float32 ['action_out_proj']['kernel'].value: (1024, 32)@float32 ['time_mlp_in']['bias'].value: (1024,)@float32 ['time_mlp_in']['kernel'].value: (1024, 1024)@float32 ['time_mlp_out']['bias'].value: (1024,)@float32 ['time_mlp_out']['kernel'].value: (1024, 1024)@float32 (1728:train.py:238) 06:46:11.211 [I] Progress on: -/3000 rate:- remaining:? elapsed:00:00 postfix:- (1728:tqdm_logging.py:145) Step 0: grad_norm=0.8543, loss=0.1015, param_norm=1803.7705 06:50:45.728 [I] Progress on: -/3000 rate:- remaining:? elapsed:04:34 postfix:- (1728:tqdm_logging.py:145) 06:51:00.084 [I] Progress on: 4.00it/3.00kit rate:41.9s/it remaining:34:53:50 elapsed:04:48 postfix:- (1728:tqdm_logging.py:145) 06:51:14.071 [I] Progress on: 6.00it/3.00kit rate:21.7s/it remaining:18:04:40 elapsed:05:02 postfix:- (1728:tqdm_logging.py:145) 06:51:28.067 [I] Progress on: 8.00it/3.00kit rate:13.8s/it remaining:11:26:14 elapsed:05:16 postfix:- (1728:tqdm_logging.py:145) 06:51:42.099 [I] Progress on: 10.0it/3.00kit rate:10.2s/it remaining:8:29:21 elapsed:05:30 postfix:- (1728:tqdm_logging.py:145) 06:51:56.145 [I] Progress on: 12.0it/3.00kit rate:8.6s/it remaining:7:06:36 elapsed:05:44 postfix:- (1728:tqdm_logging.py:145) 06:52:10.225 [I] Progress on: 14.0it/3.00kit rate:7.8s/it remaining:6:27:18 elapsed:05:59 postfix:- (1728:tqdm_logging.py:145) 06:52:24.316 [I] Progress on: 16.0it/3.00kit rate:7.4s/it remaining:6:08:18 elapsed:06:13 postfix:- (1728:tqdm_logging.py:145) 06:52:38.390 [I] Progress on: 18.0it/3.00kit rate:7.2s/it remaining:5:58:41 elapsed:06:27 postfix:- (1728:tqdm_logging.py:145) 06:52:52.460 [I] Progress on: 20.0it/3.00kit rate:7.1s/it remaining:5:53:51 elapsed:06:41 postfix:- (1728:tqdm_logging.py:145) 06:53:06.544 [I] Progress on: 22.0it/3.00kit rate:7.1s/it remaining:5:51:31 elapsed:06:55 postfix:- (1728:tqdm_logging.py:145) 06:53:20.649 [I] Progress on: 24.0it/3.00kit rate:7.1s/it remaining:5:50:31 elapsed:07:09 postfix:- (1728:tqdm_logging.py:145) 06:53:34.755 [I] Progress on: 26.0it/3.00kit rate:7.1s/it remaining:5:49:56 elapsed:07:23 postfix:- (1728:tqdm_logging.py:145) 06:53:48.854 [I] Progress on: 28.0it/3.00kit rate:7.1s/it remaining:5:49:24 elapsed:07:37 postfix:- (1728:tqdm_logging.py:145) 06:54:02.968 [I] Progress on: 30.0it/3.00kit rate:7.1s/it remaining:5:49:16 elapsed:07:51 postfix:- (1728:tqdm_logging.py:145) 06:54:17.068 [I] Progress on: 32.0it/3.00kit rate:7.1s/it remaining:5:48:52 elapsed:08:05 postfix:- (1728:tqdm_logging.py:145) 06:54:31.173 [I] Progress on: 34.0it/3.00kit rate:7.1s/it remaining:5:48:36 elapsed:08:19 postfix:- (1728:tqdm_logging.py:145) 06:54:45.294 [I] Progress on: 36.0it/3.00kit rate:7.1s/it remaining:5:48:34 elapsed:08:34 postfix:- (1728:tqdm_logging.py:145) 06:54:59.380 [I] Progress on: 38.0it/3.00kit rate:7.0s/it remaining:5:47:59 elapsed:08:48 postfix:- (1728:tqdm_logging.py:145) 06:55:13.482 [I] Progress on: 40.0it/3.00kit rate:7.0s/it remaining:5:47:47 elapsed:09:02 postfix:- (1728:tqdm_logging.py:145) 06:55:27.611 [I] Progress on: 42.0it/3.00kit rate:7.1s/it remaining:5:47:54 elapsed:09:16 postfix:- (1728:tqdm_logging.py:145) 06:55:41.739 [I] Progress on: 44.0it/3.00kit rate:7.1s/it remaining:5:47:50 elapsed:09:30 postfix:- (1728:tqdm_logging.py:145) 06:55:55.877 [I] Progress on: 46.0it/3.00kit rate:7.1s/it remaining:5:47:45 elapsed:09:44 postfix:- (1728:tqdm_logging.py:145) 06:56:10.005 [I] Progress on: 48.0it/3.00kit rate:7.1s/it remaining:5:47:29 elapsed:09:58 postfix:- (1728:tqdm_logging.py:145) 06:56:24.151 [I] Progress on: 50.0it/3.00kit rate:7.1s/it remaining:5:47:30 elapsed:10:12 postfix:- (1728:tqdm_logging.py:145) Step 50: grad_norm=0.3387, loss=0.0611, param_norm=1803.7704 06:56:38.069 [I] Progress on: 50.0it/3.00kit rate:7.1s/it remaining:5:47:30 elapsed:10:26 postfix:- (1728:tqdm_logging.py:145) 06:56:52.642 [I] Progress on: 54.0it/3.00kit rate:6.8s/it remaining:5:33:48 elapsed:10:41 postfix:- (1728:tqdm_logging.py:145) 06:57:06.824 [I] Progress on: 56.0it/3.00kit rate:6.9s/it remaining:5:40:55 elapsed:10:55 postfix:- (1728:tqdm_logging.py:145) 06:57:20.958 [I] Progress on: 58.0it/3.00kit rate:7.0s/it remaining:5:43:41 elapsed:11:09 postfix:- (1728:tqdm_logging.py:145) 06:57:35.129 [I] Progress on: 60.0it/3.00kit rate:7.0s/it remaining:5:45:19 elapsed:11:23 postfix:- (1728:tqdm_logging.py:145) 06:57:49.290 [I] Progress on: 62.0it/3.00kit rate:7.1s/it remaining:5:45:56 elapsed:11:38 postfix:- (1728:tqdm_logging.py:145) 06:58:03.442 [I] Progress on: 64.0it/3.00kit rate:7.1s/it remaining:5:46:00 elapsed:11:52 postfix:- (1728:tqdm_logging.py:145) 06:58:17.617 [I] Progress on: 66.0it/3.00kit rate:7.1s/it remaining:5:46:07 elapsed:12:06 postfix:- (1728:tqdm_logging.py:145) 06:58:31.789 [I] Progress on: 68.0it/3.00kit rate:7.1s/it remaining:5:46:04 elapsed:12:20 postfix:- (1728:tqdm_logging.py:145) 06:58:45.965 [I] Progress on: 70.0it/3.00kit rate:7.1s/it remaining:5:45:58 elapsed:12:34 postfix:- (1728:tqdm_logging.py:145) 06:59:00.128 [I] Progress on: 72.0it/3.00kit rate:7.1s/it remaining:5:45:38 elapsed:12:48 postfix:- (1728:tqdm_logging.py:145) 06:59:14.293 [I] Progress on: 74.0it/3.00kit rate:7.1s/it remaining:5:45:25 elapsed:13:03 postfix:- (1728:tqdm_logging.py:145) 06:59:28.466 [I] Progress on: 76.0it/3.00kit rate:7.1s/it remaining:5:45:17 elapsed:13:17 postfix:- (1728:tqdm_logging.py:145) 06:59:42.649 [I] Progress on: 78.0it/3.00kit rate:7.1s/it remaining:5:45:11 elapsed:13:31 postfix:- (1728:tqdm_logging.py:145) 06:59:56.836 [I] Progress on: 80.0it/3.00kit rate:7.1s/it remaining:5:45:04 elapsed:13:45 postfix:- (1728:tqdm_logging.py:145) 07:00:11.028 [I] Progress on: 82.0it/3.00kit rate:7.1s/it remaining:5:45:00 elapsed:13:59 postfix:- (1728:tqdm_logging.py:145) 07:00:25.183 [I] Progress on: 84.0it/3.00kit rate:7.1s/it remaining:5:44:24 elapsed:14:13 postfix:- (1728:tqdm_logging.py:145) 07:00:39.355 [I] Progress on: 86.0it/3.00kit rate:7.1s/it remaining:5:44:07 elapsed:14:28 postfix:- (1728:tqdm_logging.py:145) 07:00:53.521 [I] Progress on: 88.0it/3.00kit rate:7.1s/it remaining:5:43:49 elapsed:14:42 postfix:- (1728:tqdm_logging.py:145) 07:01:07.697 [I] Progress on: 90.0it/3.00kit rate:7.1s/it remaining:5:43:44 elapsed:14:56 postfix:- (1728:tqdm_logging.py:145) 07:01:21.831 [I] Progress on: 92.0it/3.00kit rate:7.1s/it remaining:5:43:02 elapsed:15:10 postfix:- (1728:tqdm_logging.py:145) 07:01:35.980 [I] Progress on: 94.0it/3.00kit rate:7.1s/it remaining:5:42:41 elapsed:15:24 postfix:- (1728:tqdm_logging.py:145) 07:01:50.132 [I] Progress on: 96.0it/3.00kit rate:7.1s/it remaining:5:42:28 elapsed:15:38 postfix:- (1728:tqdm_logging.py:145) 07:02:04.282 [I] Progress on: 98.0it/3.00kit rate:7.1s/it remaining:5:42:12 elapsed:15:53 postfix:- (1728:tqdm_logging.py:145) 07:02:18.441 [I] Progress on: 100it/3.00kit rate:7.1s/it remaining:5:42:02 elapsed:16:07 postfix:- (1728:tqdm_logging.py:145) Step 100: grad_norm=0.2040, loss=0.0422, param_norm=1803.7709 07:02:32.238 [I] Progress on: 100it/3.00kit rate:7.1s/it remaining:5:42:02 elapsed:16:21 postfix:- (1728:tqdm_logging.py:145) 07:02:46.850 [I] Progress on: 104it/3.00kit rate:6.8s/it remaining:5:27:58 elapsed:16:35 postfix:- (1728:tqdm_logging.py:145) 07:03:01.010 [I] Progress on: 106it/3.00kit rate:6.9s/it remaining:5:34:43 elapsed:16:49 postfix:- (1728:tqdm_logging.py:145) 07:03:15.144 [I] Progress on: 108it/3.00kit rate:7.0s/it remaining:5:37:39 elapsed:17:03 postfix:- (1728:tqdm_logging.py:145) 07:03:29.301 [I] Progress on: 110it/3.00kit rate:7.0s/it remaining:5:39:17 elapsed:17:18 postfix:- (1728:tqdm_logging.py:145) 07:03:43.490 [I] Progress on: 112it/3.00kit rate:7.1s/it remaining:5:40:18 elapsed:17:32 postfix:- (1728:tqdm_logging.py:145) 07:03:57.660 [I] Progress on: 114it/3.00kit rate:7.1s/it remaining:5:40:03 elapsed:17:46 postfix:- (1728:tqdm_logging.py:145) 07:04:11.855 [I] Progress on: 116it/3.00kit rate:7.1s/it remaining:5:40:32 elapsed:18:00 postfix:- (1728:tqdm_logging.py:145) 07:04:26.003 [I] Progress on: 118it/3.00kit rate:7.1s/it remaining:5:40:04 elapsed:18:14 postfix:- (1728:tqdm_logging.py:145) 07:04:40.162 [I] Progress on: 120it/3.00kit rate:7.1s/it remaining:5:39:50 elapsed:18:28 postfix:- (1728:tqdm_logging.py:145) 07:04:54.353 [I] Progress on: 122it/3.00kit rate:7.1s/it remaining:5:39:59 elapsed:18:43 postfix:- (1728:tqdm_logging.py:145) 07:05:08.513 [I] Progress on: 124it/3.00kit rate:7.1s/it remaining:5:39:32 elapsed:18:57 postfix:- (1728:tqdm_logging.py:145) 07:05:22.665 [I] Progress on: 126it/3.00kit rate:7.1s/it remaining:5:39:09 elapsed:19:11 postfix:- (1728:tqdm_logging.py:145) 07:05:36.821 [I] Progress on: 128it/3.00kit rate:7.1s/it remaining:5:38:51 elapsed:19:25 postfix:- (1728:tqdm_logging.py:145) 07:05:50.992 [I] Progress on: 130it/3.00kit rate:7.1s/it remaining:5:38:47 elapsed:19:39 postfix:- (1728:tqdm_logging.py:145) 07:06:05.175 [I] Progress on: 132it/3.00kit rate:7.1s/it remaining:5:38:45 elapsed:19:53 postfix:- (1728:tqdm_logging.py:145) 07:06:19.345 [I] Progress on: 134it/3.00kit rate:7.1s/it remaining:5:38:28 elapsed:20:08 postfix:- (1728:tqdm_logging.py:145) 07:06:33.524 [I] Progress on: 136it/3.00kit rate:7.1s/it remaining:5:38:19 elapsed:20:22 postfix:- (1728:tqdm_logging.py:145) 07:06:47.687 [I] Progress on: 138it/3.00kit rate:7.1s/it remaining:5:37:56 elapsed:20:36 postfix:- (1728:tqdm_logging.py:145) 07:07:01.867 [I] Progress on: 140it/3.00kit rate:7.1s/it remaining:5:37:51 elapsed:20:50 postfix:- (1728:tqdm_logging.py:145) 07:07:16.031 [I] Progress on: 142it/3.00kit rate:7.1s/it remaining:5:37:30 elapsed:21:04 postfix:- (1728:tqdm_logging.py:145) 07:07:30.191 [I] Progress on: 144it/3.00kit rate:7.1s/it remaining:5:37:07 elapsed:21:18 postfix:- (1728:tqdm_logging.py:145) 07:07:44.352 [I] Progress on: 146it/3.00kit rate:7.1s/it remaining:5:36:50 elapsed:21:33 postfix:- (1728:tqdm_logging.py:145) 07:07:58.514 [I] Progress on: 148it/3.00kit rate:7.1s/it remaining:5:36:33 elapsed:21:47 postfix:- (1728:tqdm_logging.py:145) 07:08:12.671 [I] Progress on: 150it/3.00kit rate:7.1s/it remaining:5:36:15 elapsed:22:01 postfix:- (1728:tqdm_logging.py:145) Step 150: grad_norm=0.1744, loss=0.0296, param_norm=1803.7731 07:08:26.480 [I] Progress on: 150it/3.00kit rate:7.1s/it remaining:5:36:15 elapsed:22:15 postfix:- (1728:tqdm_logging.py:145) 07:08:41.113 [I] Progress on: 154it/3.00kit rate:6.8s/it remaining:5:22:32 elapsed:22:29 postfix:- (1728:tqdm_logging.py:145) 07:08:55.255 [I] Progress on: 156it/3.00kit rate:6.9s/it remaining:5:28:52 elapsed:22:44 postfix:- (1728:tqdm_logging.py:145) 07:09:09.403 [I] Progress on: 158it/3.00kit rate:7.0s/it remaining:5:31:54 elapsed:22:58 postfix:- (1728:tqdm_logging.py:145) 07:09:23.576 [I] Progress on: 160it/3.00kit rate:7.0s/it remaining:5:33:35 elapsed:23:12 postfix:- (1728:tqdm_logging.py:145) 07:09:37.722 [I] Progress on: 162it/3.00kit rate:7.1s/it remaining:5:34:00 elapsed:23:26 postfix:- (1728:tqdm_logging.py:145) 07:09:51.866 [I] Progress on: 164it/3.00kit rate:7.1s/it remaining:5:34:01 elapsed:23:40 postfix:- (1728:tqdm_logging.py:145) 07:10:06.043 [I] Progress on: 166it/3.00kit rate:7.1s/it remaining:5:34:19 elapsed:23:54 postfix:- (1728:tqdm_logging.py:145) 07:10:20.195 [I] Progress on: 168it/3.00kit rate:7.1s/it remaining:5:34:02 elapsed:24:08 postfix:- (1728:tqdm_logging.py:145) 07:10:34.354 [I] Progress on: 170it/3.00kit rate:7.1s/it remaining:5:33:52 elapsed:24:23 postfix:- (1728:tqdm_logging.py:145) 07:10:48.507 [I] Progress on: 172it/3.00kit rate:7.1s/it remaining:5:33:37 elapsed:24:37 postfix:- (1728:tqdm_logging.py:145) 07:11:02.655 [I] Progress on: 174it/3.00kit rate:7.1s/it remaining:5:33:18 elapsed:24:51 postfix:- (1728:tqdm_logging.py:145) 07:11:16.815 [I] Progress on: 176it/3.00kit rate:7.1s/it remaining:5:33:11 elapsed:25:05 postfix:- (1728:tqdm_logging.py:145) 07:11:30.969 [I] Progress on: 178it/3.00kit rate:7.1s/it remaining:5:32:56 elapsed:25:19 postfix:- (1728:tqdm_logging.py:145) 07:11:45.109 [I] Progress on: 180it/3.00kit rate:7.1s/it remaining:5:32:30 elapsed:25:33 postfix:- (1728:tqdm_logging.py:145) 07:11:59.283 [I] Progress on: 182it/3.00kit rate:7.1s/it remaining:5:32:33 elapsed:25:48 postfix:- (1728:tqdm_logging.py:145) 07:12:13.441 [I] Progress on: 184it/3.00kit rate:7.1s/it remaining:5:32:18 elapsed:26:02 postfix:- (1728:tqdm_logging.py:145) 07:12:27.583 [I] Progress on: 186it/3.00kit rate:7.1s/it remaining:5:31:51 elapsed:26:16 postfix:- (1728:tqdm_logging.py:145) 07:12:41.747 [I] Progress on: 188it/3.00kit rate:7.1s/it remaining:5:31:46 elapsed:26:30 postfix:- (1728:tqdm_logging.py:145) 07:12:55.902 [I] Progress on: 190it/3.00kit rate:7.1s/it remaining:5:31:29 elapsed:26:44 postfix:- (1728:tqdm_logging.py:145) 07:13:10.054 [I] Progress on: 192it/3.00kit rate:7.1s/it remaining:5:31:13 elapsed:26:58 postfix:- (1728:tqdm_logging.py:145) 07:13:24.199 [I] Progress on: 194it/3.00kit rate:7.1s/it remaining:5:30:52 elapsed:27:12 postfix:- (1728:tqdm_logging.py:145) 07:13:38.389 [I] Progress on: 196it/3.00kit rate:7.1s/it remaining:5:31:05 elapsed:27:27 postfix:- (1728:tqdm_logging.py:145) 07:13:52.549 [I] Progress on: 198it/3.00kit rate:7.1s/it remaining:5:30:43 elapsed:27:41 postfix:- (1728:tqdm_logging.py:145) 07:14:06.708 [I] Progress on: 200it/3.00kit rate:7.1s/it remaining:5:30:25 elapsed:27:55 postfix:- (1728:tqdm_logging.py:145) Step 200: grad_norm=0.1406, loss=0.0213, param_norm=1803.7776 07:14:20.525 [I] Progress on: 200it/3.00kit rate:7.1s/it remaining:5:30:25 elapsed:28:09 postfix:- (1728:tqdm_logging.py:145) 07:14:35.159 [I] Progress on: 204it/3.00kit rate:6.8s/it remaining:5:16:55 elapsed:28:23 postfix:- (1728:tqdm_logging.py:145) 07:14:49.316 [I] Progress on: 206it/3.00kit rate:6.9s/it remaining:5:23:18 elapsed:28:38 postfix:- (1728:tqdm_logging.py:145) 07:15:03.471 [I] Progress on: 208it/3.00kit rate:7.0s/it remaining:5:26:19 elapsed:28:52 postfix:- (1728:tqdm_logging.py:145) 07:15:17.656 [I] Progress on: 210it/3.00kit rate:7.1s/it remaining:5:28:00 elapsed:29:06 postfix:- (1728:tqdm_logging.py:145) 07:15:31.802 [I] Progress on: 212it/3.00kit rate:7.1s/it remaining:5:28:15 elapsed:29:20 postfix:- (1728:tqdm_logging.py:145) 07:15:45.963 [I] Progress on: 214it/3.00kit rate:7.1s/it remaining:5:28:22 elapsed:29:34 postfix:- (1728:tqdm_logging.py:145) 07:16:00.122 [I] Progress on: 216it/3.00kit rate:7.1s/it remaining:5:28:16 elapsed:29:48 postfix:- (1728:tqdm_logging.py:145) 07:16:14.279 [I] Progress on: 218it/3.00kit rate:7.1s/it remaining:5:28:09 elapsed:30:03 postfix:- (1728:tqdm_logging.py:145) 07:16:28.436 [I] Progress on: 220it/3.00kit rate:7.1s/it remaining:5:27:53 elapsed:30:17 postfix:- (1728:tqdm_logging.py:145) 07:16:43.178 [I] Progress on: 222it/3.00kit rate:7.3s/it remaining:5:35:48 elapsed:30:31 postfix:- (1728:tqdm_logging.py:145) 07:16:56.727 [I] Progress on: 224it/3.00kit rate:7.0s/it remaining:5:25:36 elapsed:30:45 postfix:- (1728:tqdm_logging.py:145) 07:17:10.889 [I] Progress on: 226it/3.00kit rate:7.1s/it remaining:5:26:23 elapsed:30:59 postfix:- (1728:tqdm_logging.py:145) 07:17:25.038 [I] Progress on: 228it/3.00kit rate:7.1s/it remaining:5:26:30 elapsed:31:13 postfix:- (1728:tqdm_logging.py:145) 07:17:39.193 [I] Progress on: 230it/3.00kit rate:7.1s/it remaining:5:26:30 elapsed:31:27 postfix:- (1728:tqdm_logging.py:145) 07:17:53.349 [I] Progress on: 232it/3.00kit rate:7.1s/it remaining:5:26:20 elapsed:31:42 postfix:- (1728:tqdm_logging.py:145) 07:18:07.525 [I] Progress on: 234it/3.00kit rate:7.1s/it remaining:5:26:27 elapsed:31:56 postfix:- (1728:tqdm_logging.py:145) 07:18:21.679 [I] Progress on: 236it/3.00kit rate:7.1s/it remaining:5:26:07 elapsed:32:10 postfix:- (1728:tqdm_logging.py:145) 07:18:35.829 [I] Progress on: 238it/3.00kit rate:7.1s/it remaining:5:25:44 elapsed:32:24 postfix:- (1728:tqdm_logging.py:145) 07:18:49.997 [I] Progress on: 240it/3.00kit rate:7.1s/it remaining:5:25:42 elapsed:32:38 postfix:- (1728:tqdm_logging.py:145) 07:19:04.154 [I] Progress on: 242it/3.00kit rate:7.1s/it remaining:5:25:27 elapsed:32:52 postfix:- (1728:tqdm_logging.py:145) 07:19:18.316 [I] Progress on: 244it/3.00kit rate:7.1s/it remaining:5:25:17 elapsed:33:07 postfix:- (1728:tqdm_logging.py:145) 07:19:32.461 [I] Progress on: 246it/3.00kit rate:7.1s/it remaining:5:24:50 elapsed:33:21 postfix:- (1728:tqdm_logging.py:145) 07:19:46.639 [I] Progress on: 248it/3.00kit rate:7.1s/it remaining:5:24:54 elapsed:33:35 postfix:- (1728:tqdm_logging.py:145) 07:20:00.788 [I] Progress on: 250it/3.00kit rate:7.1s/it remaining:5:24:27 elapsed:33:49 postfix:- (1728:tqdm_logging.py:145) Step 250: grad_norm=0.1348, loss=0.0186, param_norm=1803.7841 07:20:14.582 [I] Progress on: 250it/3.00kit rate:7.1s/it remaining:5:24:27 elapsed:34:03 postfix:- (1728:tqdm_logging.py:145) 07:20:29.164 [I] Progress on: 254it/3.00kit rate:6.8s/it remaining:5:10:47 elapsed:34:17 postfix:- (1728:tqdm_logging.py:145) 07:20:43.332 [I] Progress on: 256it/3.00kit rate:6.9s/it remaining:5:17:21 elapsed:34:32 postfix:- (1728:tqdm_logging.py:145) 07:20:57.496 [I] Progress on: 258it/3.00kit rate:7.0s/it remaining:5:20:26 elapsed:34:46 postfix:- (1728:tqdm_logging.py:145) 07:21:11.651 [I] Progress on: 260it/3.00kit rate:7.0s/it remaining:5:21:44 elapsed:35:00 postfix:- (1728:tqdm_logging.py:145) 07:21:25.802 [I] Progress on: 262it/3.00kit rate:7.1s/it remaining:5:22:13 elapsed:35:14 postfix:- (1728:tqdm_logging.py:145) 07:21:39.965 [I] Progress on: 264it/3.00kit rate:7.1s/it remaining:5:22:29 elapsed:35:28 postfix:- (1728:tqdm_logging.py:145) 07:21:54.120 [I] Progress on: 266it/3.00kit rate:7.1s/it remaining:5:22:25 elapsed:35:42 postfix:- (1728:tqdm_logging.py:145) 07:22:08.255 [I] Progress on: 268it/3.00kit rate:7.1s/it remaining:5:22:00 elapsed:35:57 postfix:- (1728:tqdm_logging.py:145) 07:22:22.434 [I] Progress on: 270it/3.00kit rate:7.1s/it remaining:5:22:11 elapsed:36:11 postfix:- (1728:tqdm_logging.py:145) 07:22:36.592 [I] Progress on: 272it/3.00kit rate:7.1s/it remaining:5:21:57 elapsed:36:25 postfix:- (1728:tqdm_logging.py:145) 07:22:50.749 [I] Progress on: 274it/3.00kit rate:7.1s/it remaining:5:21:42 elapsed:36:39 postfix:- (1728:tqdm_logging.py:145) 07:23:04.874 [I] Progress on: 276it/3.00kit rate:7.1s/it remaining:5:21:04 elapsed:36:53 postfix:- (1728:tqdm_logging.py:145) 07:23:19.012 [I] Progress on: 278it/3.00kit rate:7.1s/it remaining:5:20:47 elapsed:37:07 postfix:- (1728:tqdm_logging.py:145) 07:23:33.132 [I] Progress on: 280it/3.00kit rate:7.1s/it remaining:5:20:20 elapsed:37:21 postfix:- (1728:tqdm_logging.py:145) 07:23:47.302 [I] Progress on: 282it/3.00kit rate:7.1s/it remaining:5:20:32 elapsed:37:36 postfix:- (1728:tqdm_logging.py:145) 07:24:01.457 [I] Progress on: 284it/3.00kit rate:7.1s/it remaining:5:20:22 elapsed:37:50 postfix:- (1728:tqdm_logging.py:145) 07:24:15.592 [I] Progress on: 286it/3.00kit rate:7.1s/it remaining:5:19:55 elapsed:38:04 postfix:- (1728:tqdm_logging.py:145) 07:24:29.761 [I] Progress on: 288it/3.00kit rate:7.1s/it remaining:5:19:57 elapsed:38:18 postfix:- (1728:tqdm_logging.py:145) 07:24:43.906 [I] Progress on: 290it/3.00kit rate:7.1s/it remaining:5:19:34 elapsed:38:32 postfix:- (1728:tqdm_logging.py:145) 07:24:58.058 [I] Progress on: 292it/3.00kit rate:7.1s/it remaining:5:19:22 elapsed:38:46 postfix:- (1728:tqdm_logging.py:145) 07:25:12.197 [I] Progress on: 294it/3.00kit rate:7.1s/it remaining:5:18:59 elapsed:39:00 postfix:- (1728:tqdm_logging.py:145) 07:25:26.336 [I] Progress on: 296it/3.00kit rate:7.1s/it remaining:5:18:41 elapsed:39:15 postfix:- (1728:tqdm_logging.py:145) 07:25:40.482 [I] Progress on: 298it/3.00kit rate:7.1s/it remaining:5:18:32 elapsed:39:29 postfix:- (1728:tqdm_logging.py:145) 07:25:54.626 [I] Progress on: 300it/3.00kit rate:7.1s/it remaining:5:18:16 elapsed:39:43 postfix:- (1728:tqdm_logging.py:145) Step 300: grad_norm=0.1203, loss=0.0169, param_norm=1803.7925 07:26:08.423 [I] Progress on: 300it/3.00kit rate:7.1s/it remaining:5:18:16 elapsed:39:57 postfix:- (1728:tqdm_logging.py:145) 07:26:23.007 [I] Progress on: 304it/3.00kit rate:6.8s/it remaining:5:05:00 elapsed:40:11 postfix:- (1728:tqdm_logging.py:145) 07:26:37.149 [I] Progress on: 306it/3.00kit rate:6.9s/it remaining:5:11:16 elapsed:40:25 postfix:- (1728:tqdm_logging.py:145) 07:26:51.293 [I] Progress on: 308it/3.00kit rate:7.0s/it remaining:5:14:13 elapsed:40:40 postfix:- (1728:tqdm_logging.py:145) 07:27:05.439 [I] Progress on: 310it/3.00kit rate:7.0s/it remaining:5:15:32 elapsed:40:54 postfix:- (1728:tqdm_logging.py:145) 07:27:19.574 [I] Progress on: 312it/3.00kit rate:7.1s/it remaining:5:15:56 elapsed:41:08 postfix:- (1728:tqdm_logging.py:145) 07:27:33.722 [I] Progress on: 314it/3.00kit rate:7.1s/it remaining:5:16:10 elapsed:41:22 postfix:- (1728:tqdm_logging.py:145) 07:27:47.865 [I] Progress on: 316it/3.00kit rate:7.1s/it remaining:5:16:07 elapsed:41:36 postfix:- (1728:tqdm_logging.py:145) 07:28:02.005 [I] Progress on: 318it/3.00kit rate:7.1s/it remaining:5:15:56 elapsed:41:50 postfix:- (1728:tqdm_logging.py:145) 07:28:16.138 [I] Progress on: 320it/3.00kit rate:7.1s/it remaining:5:15:40 elapsed:42:04 postfix:- (1728:tqdm_logging.py:145) 07:28:30.284 [I] Progress on: 322it/3.00kit rate:7.1s/it remaining:5:15:31 elapsed:42:19 postfix:- (1728:tqdm_logging.py:145) 07:28:44.439 [I] Progress on: 324it/3.00kit rate:7.1s/it remaining:5:15:27 elapsed:42:33 postfix:- (1728:tqdm_logging.py:145) 07:28:58.586 [I] Progress on: 326it/3.00kit rate:7.1s/it remaining:5:15:13 elapsed:42:47 postfix:- (1728:tqdm_logging.py:145) 07:29:12.737 [I] Progress on: 328it/3.00kit rate:7.1s/it remaining:5:15:00 elapsed:43:01 postfix:- (1728:tqdm_logging.py:145) 07:29:26.889 [I] Progress on: 330it/3.00kit rate:7.1s/it remaining:5:14:49 elapsed:43:15 postfix:- (1728:tqdm_logging.py:145) 07:29:41.026 [I] Progress on: 332it/3.00kit rate:7.1s/it remaining:5:14:25 elapsed:43:29 postfix:- (1728:tqdm_logging.py:145) 07:29:55.183 [I] Progress on: 334it/3.00kit rate:7.1s/it remaining:5:14:20 elapsed:43:43 postfix:- (1728:tqdm_logging.py:145) 07:30:09.349 [I] Progress on: 336it/3.00kit rate:7.1s/it remaining:5:14:17 elapsed:43:58 postfix:- (1728:tqdm_logging.py:145) 07:30:23.490 [I] Progress on: 338it/3.00kit rate:7.1s/it remaining:5:13:50 elapsed:44:12 postfix:- (1728:tqdm_logging.py:145) 07:30:37.635 [I] Progress on: 340it/3.00kit rate:7.1s/it remaining:5:13:33 elapsed:44:26 postfix:- (1728:tqdm_logging.py:145) 07:30:51.781 [I] Progress on: 342it/3.00kit rate:7.1s/it remaining:5:13:18 elapsed:44:40 postfix:- (1728:tqdm_logging.py:145) 07:31:05.930 [I] Progress on: 344it/3.00kit rate:7.1s/it remaining:5:13:04 elapsed:44:54 postfix:- (1728:tqdm_logging.py:145) 07:31:20.076 [I] Progress on: 346it/3.00kit rate:7.1s/it remaining:5:12:50 elapsed:45:08 postfix:- (1728:tqdm_logging.py:145) 07:31:34.217 [I] Progress on: 348it/3.00kit rate:7.1s/it remaining:5:12:31 elapsed:45:23 postfix:- (1728:tqdm_logging.py:145) 07:31:48.368 [I] Progress on: 350it/3.00kit rate:7.1s/it remaining:5:12:22 elapsed:45:37 postfix:- (1728:tqdm_logging.py:145) Step 350: grad_norm=0.0930, loss=0.0143, param_norm=1803.8036 07:32:02.141 [I] Progress on: 350it/3.00kit rate:7.1s/it remaining:5:12:22 elapsed:45:50 postfix:- (1728:tqdm_logging.py:145) 07:32:16.718 [I] Progress on: 354it/3.00kit rate:6.8s/it remaining:4:59:15 elapsed:46:05 postfix:- (1728:tqdm_logging.py:145) 07:32:30.853 [I] Progress on: 356it/3.00kit rate:6.9s/it remaining:5:05:25 elapsed:46:19 postfix:- (1728:tqdm_logging.py:145) 07:32:45.011 [I] Progress on: 358it/3.00kit rate:7.0s/it remaining:5:08:33 elapsed:46:33 postfix:- (1728:tqdm_logging.py:145) 07:32:59.159 [I] Progress on: 360it/3.00kit rate:7.0s/it remaining:5:09:50 elapsed:46:47 postfix:- (1728:tqdm_logging.py:145) 07:33:13.331 [I] Progress on: 362it/3.00kit rate:7.1s/it remaining:5:10:36 elapsed:47:02 postfix:- (1728:tqdm_logging.py:145) 07:33:27.498 [I] Progress on: 364it/3.00kit rate:7.1s/it remaining:5:10:48 elapsed:47:16 postfix:- (1728:tqdm_logging.py:145) 07:33:41.649 [I] Progress on: 366it/3.00kit rate:7.1s/it remaining:5:10:34 elapsed:47:30 postfix:- (1728:tqdm_logging.py:145) 07:33:55.816 [I] Progress on: 368it/3.00kit rate:7.1s/it remaining:5:10:32 elapsed:47:44 postfix:- (1728:tqdm_logging.py:145) 07:34:09.954 [I] Progress on: 370it/3.00kit rate:7.1s/it remaining:5:10:04 elapsed:47:58 postfix:- (1728:tqdm_logging.py:145) 07:34:24.091 [I] Progress on: 372it/3.00kit rate:7.1s/it remaining:5:09:43 elapsed:48:12 postfix:- (1728:tqdm_logging.py:145) 07:34:38.260 [I] Progress on: 374it/3.00kit rate:7.1s/it remaining:5:09:49 elapsed:48:27 postfix:- (1728:tqdm_logging.py:145) 07:34:52.420 [I] Progress on: 376it/3.00kit rate:7.1s/it remaining:5:09:36 elapsed:48:41 postfix:- (1728:tqdm_logging.py:145) 07:35:06.574 [I] Progress on: 378it/3.00kit rate:7.1s/it remaining:5:09:19 elapsed:48:55 postfix:- (1728:tqdm_logging.py:145) 07:35:20.714 [I] Progress on: 380it/3.00kit rate:7.1s/it remaining:5:08:55 elapsed:49:09 postfix:- (1728:tqdm_logging.py:145) 07:35:34.892 [I] Progress on: 382it/3.00kit rate:7.1s/it remaining:5:09:02 elapsed:49:23 postfix:- (1728:tqdm_logging.py:145) 07:35:49.046 [I] Progress on: 384it/3.00kit rate:7.1s/it remaining:5:08:43 elapsed:49:37 postfix:- (1728:tqdm_logging.py:145) 07:36:03.192 [I] Progress on: 386it/3.00kit rate:7.1s/it remaining:5:08:20 elapsed:49:51 postfix:- (1728:tqdm_logging.py:145) 07:36:17.333 [I] Progress on: 388it/3.00kit rate:7.1s/it remaining:5:07:58 elapsed:50:06 postfix:- (1728:tqdm_logging.py:145) 07:36:31.451 [I] Progress on: 390it/3.00kit rate:7.1s/it remaining:5:07:24 elapsed:50:20 postfix:- (1728:tqdm_logging.py:145) 07:36:45.617 [I] Progress on: 392it/3.00kit rate:7.1s/it remaining:5:07:32 elapsed:50:34 postfix:- (1728:tqdm_logging.py:145) 07:36:59.769 [I] Progress on: 394it/3.00kit rate:7.1s/it remaining:5:07:22 elapsed:50:48 postfix:- (1728:tqdm_logging.py:145) 07:37:13.909 [I] Progress on: 396it/3.00kit rate:7.1s/it remaining:5:07:00 elapsed:51:02 postfix:- (1728:tqdm_logging.py:145) 07:37:28.032 [I] Progress on: 398it/3.00kit rate:7.1s/it remaining:5:06:30 elapsed:51:16 postfix:- (1728:tqdm_logging.py:145) 07:37:42.139 [I] Progress on: 400it/3.00kit rate:7.1s/it remaining:5:05:59 elapsed:51:30 postfix:- (1728:tqdm_logging.py:145) Step 400: grad_norm=0.0826, loss=0.0124, param_norm=1803.8142 07:37:55.913 [I] Progress on: 400it/3.00kit rate:7.1s/it remaining:5:05:59 elapsed:51:44 postfix:- (1728:tqdm_logging.py:145) 07:38:10.465 [I] Progress on: 404it/3.00kit rate:6.8s/it remaining:4:53:13 elapsed:51:59 postfix:- (1728:tqdm_logging.py:145) 07:38:24.613 [I] Progress on: 406it/3.00kit rate:6.9s/it remaining:4:59:29 elapsed:52:13 postfix:- (1728:tqdm_logging.py:145) 07:38:38.765 [I] Progress on: 408it/3.00kit rate:7.0s/it remaining:5:02:34 elapsed:52:27 postfix:- (1728:tqdm_logging.py:145) 07:38:52.895 [I] Progress on: 410it/3.00kit rate:7.0s/it remaining:5:03:37 elapsed:52:41 postfix:- (1728:tqdm_logging.py:145) 07:39:07.034 [I] Progress on: 412it/3.00kit rate:7.1s/it remaining:5:04:08 elapsed:52:55 postfix:- (1728:tqdm_logging.py:145) 07:39:21.185 [I] Progress on: 414it/3.00kit rate:7.1s/it remaining:5:04:26 elapsed:53:09 postfix:- (1728:tqdm_logging.py:145) 07:39:35.340 [I] Progress on: 416it/3.00kit rate:7.1s/it remaining:5:04:28 elapsed:53:24 postfix:- (1728:tqdm_logging.py:145) 07:39:49.495 [I] Progress on: 418it/3.00kit rate:7.1s/it remaining:5:04:23 elapsed:53:38 postfix:- (1728:tqdm_logging.py:145) 07:40:03.639 [I] Progress on: 420it/3.00kit rate:7.1s/it remaining:5:04:04 elapsed:53:52 postfix:- (1728:tqdm_logging.py:145) 07:40:17.783 [I] Progress on: 422it/3.00kit rate:7.1s/it remaining:5:03:49 elapsed:54:06 postfix:- (1728:tqdm_logging.py:145) 07:40:31.947 [I] Progress on: 424it/3.00kit rate:7.1s/it remaining:5:03:47 elapsed:54:20 postfix:- (1728:tqdm_logging.py:145) 07:40:46.097 [I] Progress on: 426it/3.00kit rate:7.1s/it remaining:5:03:31 elapsed:54:34 postfix:- (1728:tqdm_logging.py:145) 07:41:00.260 [I] Progress on: 428it/3.00kit rate:7.1s/it remaining:5:03:23 elapsed:54:49 postfix:- (1728:tqdm_logging.py:145) 07:41:14.403 [I] Progress on: 430it/3.00kit rate:7.1s/it remaining:5:02:58 elapsed:55:03 postfix:- (1728:tqdm_logging.py:145) 07:41:28.561 [I] Progress on: 432it/3.00kit rate:7.1s/it remaining:5:02:48 elapsed:55:17 postfix:- (1728:tqdm_logging.py:145) 07:41:42.704 [I] Progress on: 434it/3.00kit rate:7.1s/it remaining:5:02:30 elapsed:55:31 postfix:- (1728:tqdm_logging.py:145) 07:41:56.859 [I] Progress on: 436it/3.00kit rate:7.1s/it remaining:5:02:19 elapsed:55:45 postfix:- (1728:tqdm_logging.py:145) 07:42:10.992 [I] Progress on: 438it/3.00kit rate:7.1s/it remaining:5:01:53 elapsed:55:59 postfix:- (1728:tqdm_logging.py:145) 07:42:25.132 [I] Progress on: 440it/3.00kit rate:7.1s/it remaining:5:01:38 elapsed:56:13 postfix:- (1728:tqdm_logging.py:145) 07:42:39.273 [I] Progress on: 442it/3.00kit rate:7.1s/it remaining:5:01:24 elapsed:56:28 postfix:- (1728:tqdm_logging.py:145) 07:42:53.984 [I] Progress on: 444it/3.00kit rate:7.2s/it remaining:5:08:26 elapsed:56:42 postfix:- (1728:tqdm_logging.py:145) 07:43:07.598 [I] Progress on: 446it/3.00kit rate:7.0s/it remaining:4:59:58 elapsed:56:56 postfix:- (1728:tqdm_logging.py:145) 07:43:21.753 [I] Progress on: 448it/3.00kit rate:7.1s/it remaining:5:00:32 elapsed:57:10 postfix:- (1728:tqdm_logging.py:145) 07:43:35.909 [I] Progress on: 450it/3.00kit rate:7.1s/it remaining:5:00:43 elapsed:57:24 postfix:- (1728:tqdm_logging.py:145) Step 450: grad_norm=0.0762, loss=0.0126, param_norm=1803.8242 07:43:49.712 [I] Progress on: 450it/3.00kit rate:7.1s/it remaining:5:00:43 elapsed:57:38 postfix:- (1728:tqdm_logging.py:145) 07:44:04.268 [I] Progress on: 454it/3.00kit rate:6.8s/it remaining:4:47:54 elapsed:57:53 postfix:- (1728:tqdm_logging.py:145) 07:44:18.419 [I] Progress on: 456it/3.00kit rate:6.9s/it remaining:4:53:56 elapsed:58:07 postfix:- (1728:tqdm_logging.py:145) 07:44:32.564 [I] Progress on: 458it/3.00kit rate:7.0s/it remaining:4:56:42 elapsed:58:21 postfix:- (1728:tqdm_logging.py:145) 07:44:46.712 [I] Progress on: 460it/3.00kit rate:7.0s/it remaining:4:57:57 elapsed:58:35 postfix:- (1728:tqdm_logging.py:145) 07:45:00.859 [I] Progress on: 462it/3.00kit rate:7.1s/it remaining:4:58:25 elapsed:58:49 postfix:- (1728:tqdm_logging.py:145) 07:45:15.016 [I] Progress on: 464it/3.00kit rate:7.1s/it remaining:4:58:40 elapsed:59:03 postfix:- (1728:tqdm_logging.py:145) 07:45:29.172 [I] Progress on: 466it/3.00kit rate:7.1s/it remaining:4:58:38 elapsed:59:17 postfix:- (1728:tqdm_logging.py:145) 07:45:43.317 [I] Progress on: 468it/3.00kit rate:7.1s/it remaining:4:58:24 elapsed:59:32 postfix:- (1728:tqdm_logging.py:145) 07:45:57.454 [I] Progress on: 470it/3.00kit rate:7.1s/it remaining:4:58:04 elapsed:59:46 postfix:- (1728:tqdm_logging.py:145) 07:46:11.603 [I] Progress on: 472it/3.00kit rate:7.1s/it remaining:4:57:55 elapsed:1:00:00 postfix:- (1728:tqdm_logging.py:145) 07:46:25.742 [I] Progress on: 474it/3.00kit rate:7.1s/it remaining:4:57:38 elapsed:1:00:14 postfix:- (1728:tqdm_logging.py:145) 07:46:39.897 [I] Progress on: 476it/3.00kit rate:7.1s/it remaining:4:57:36 elapsed:1:00:28 postfix:- (1728:tqdm_logging.py:145) 07:46:54.047 [I] Progress on: 478it/3.00kit rate:7.1s/it remaining:4:57:24 elapsed:1:00:42 postfix:- (1728:tqdm_logging.py:145) 07:47:08.203 [I] Progress on: 480it/3.00kit rate:7.1s/it remaining:4:57:11 elapsed:1:00:56 postfix:- (1728:tqdm_logging.py:145) 07:47:22.366 [I] Progress on: 482it/3.00kit rate:7.1s/it remaining:4:57:06 elapsed:1:01:11 postfix:- (1728:tqdm_logging.py:145) 07:47:36.480 [I] Progress on: 484it/3.00kit rate:7.1s/it remaining:4:56:24 elapsed:1:01:25 postfix:- (1728:tqdm_logging.py:145) 07:47:50.593 [I] Progress on: 486it/3.00kit rate:7.1s/it remaining:4:55:56 elapsed:1:01:39 postfix:- (1728:tqdm_logging.py:145) 07:48:04.710 [I] Progress on: 488it/3.00kit rate:7.1s/it remaining:4:55:38 elapsed:1:01:53 postfix:- (1728:tqdm_logging.py:145) 07:48:18.807 [I] Progress on: 490it/3.00kit rate:7.1s/it remaining:4:55:08 elapsed:1:02:07 postfix:- (1728:tqdm_logging.py:145) 07:48:32.937 [I] Progress on: 492it/3.00kit rate:7.1s/it remaining:4:55:07 elapsed:1:02:21 postfix:- (1728:tqdm_logging.py:145) 07:48:47.054 [I] Progress on: 494it/3.00kit rate:7.1s/it remaining:4:54:51 elapsed:1:02:35 postfix:- (1728:tqdm_logging.py:145) 07:49:01.155 [I] Progress on: 496it/3.00kit rate:7.1s/it remaining:4:54:25 elapsed:1:02:49 postfix:- (1728:tqdm_logging.py:145) 07:49:15.284 [I] Progress on: 498it/3.00kit rate:7.1s/it remaining:4:54:24 elapsed:1:03:04 postfix:- (1728:tqdm_logging.py:145) 07:49:29.417 [I] Progress on: 500it/3.00kit rate:7.1s/it remaining:4:54:17 elapsed:1:03:18 postfix:- (1728:tqdm_logging.py:145) Step 500: grad_norm=0.0669, loss=0.0113, param_norm=1803.8344 07:49:43.166 [I] Progress on: 500it/3.00kit rate:7.1s/it remaining:4:54:17 elapsed:1:03:31 postfix:- (1728:tqdm_logging.py:145) 07:49:57.687 [I] Progress on: 504it/3.00kit rate:6.8s/it remaining:4:41:33 elapsed:1:03:46 postfix:- (1728:tqdm_logging.py:145) 07:50:11.801 [I] Progress on: 506it/3.00kit rate:6.9s/it remaining:4:47:24 elapsed:1:04:00 postfix:- (1728:tqdm_logging.py:145) 07:50:25.929 [I] Progress on: 508it/3.00kit rate:7.0s/it remaining:4:50:19 elapsed:1:04:14 postfix:- (1728:tqdm_logging.py:145) 07:50:40.057 [I] Progress on: 510it/3.00kit rate:7.0s/it remaining:4:51:36 elapsed:1:04:28 postfix:- (1728:tqdm_logging.py:145) 07:50:54.170 [I] Progress on: 512it/3.00kit rate:7.0s/it remaining:4:51:59 elapsed:1:04:42 postfix:- (1728:tqdm_logging.py:145) 07:51:08.288 [I] Progress on: 514it/3.00kit rate:7.1s/it remaining:4:52:06 elapsed:1:04:57 postfix:- (1728:tqdm_logging.py:145) 07:51:22.408 [I] Progress on: 516it/3.00kit rate:7.1s/it remaining:4:52:04 elapsed:1:05:11 postfix:- (1728:tqdm_logging.py:145) 07:51:36.550 [I] Progress on: 518it/3.00kit rate:7.1s/it remaining:4:52:10 elapsed:1:05:25 postfix:- (1728:tqdm_logging.py:145) 07:51:50.696 [I] Progress on: 520it/3.00kit rate:7.1s/it remaining:4:52:09 elapsed:1:05:39 postfix:- (1728:tqdm_logging.py:145) 07:52:04.850 [I] Progress on: 522it/3.00kit rate:7.1s/it remaining:4:52:05 elapsed:1:05:53 postfix:- (1728:tqdm_logging.py:145) 07:52:19.013 [I] Progress on: 524it/3.00kit rate:7.1s/it remaining:4:52:03 elapsed:1:06:07 postfix:- (1728:tqdm_logging.py:145) 07:52:33.157 [I] Progress on: 526it/3.00kit rate:7.1s/it remaining:4:51:42 elapsed:1:06:21 postfix:- (1728:tqdm_logging.py:145) 07:52:47.290 [I] Progress on: 528it/3.00kit rate:7.1s/it remaining:4:51:19 elapsed:1:06:36 postfix:- (1728:tqdm_logging.py:145) 07:53:01.426 [I] Progress on: 530it/3.00kit rate:7.1s/it remaining:4:51:03 elapsed:1:06:50 postfix:- (1728:tqdm_logging.py:145) 07:53:15.567 [I] Progress on: 532it/3.00kit rate:7.1s/it remaining:4:50:52 elapsed:1:07:04 postfix:- (1728:tqdm_logging.py:145) 07:53:29.713 [I] Progress on: 534it/3.00kit rate:7.1s/it remaining:4:50:40 elapsed:1:07:18 postfix:- (1728:tqdm_logging.py:145) 07:53:43.828 [I] Progress on: 536it/3.00kit rate:7.1s/it remaining:4:50:08 elapsed:1:07:32 postfix:- (1728:tqdm_logging.py:145) 07:53:57.959 [I] Progress on: 538it/3.00kit rate:7.1s/it remaining:4:49:55 elapsed:1:07:46 postfix:- (1728:tqdm_logging.py:145) 07:54:12.070 [I] Progress on: 540it/3.00kit rate:7.1s/it remaining:4:49:28 elapsed:1:08:00 postfix:- (1728:tqdm_logging.py:145) 07:54:26.178 [I] Progress on: 542it/3.00kit rate:7.1s/it remaining:4:49:06 elapsed:1:08:14 postfix:- (1728:tqdm_logging.py:145) 07:54:40.307 [I] Progress on: 544it/3.00kit rate:7.1s/it remaining:4:49:03 elapsed:1:08:29 postfix:- (1728:tqdm_logging.py:145) 07:54:54.422 [I] Progress on: 546it/3.00kit rate:7.1s/it remaining:4:48:45 elapsed:1:08:43 postfix:- (1728:tqdm_logging.py:145) 07:55:08.528 [I] Progress on: 548it/3.00kit rate:7.1s/it remaining:4:48:23 elapsed:1:08:57 postfix:- (1728:tqdm_logging.py:145) 07:55:22.647 [I] Progress on: 550it/3.00kit rate:7.1s/it remaining:4:48:14 elapsed:1:09:11 postfix:- (1728:tqdm_logging.py:145) Step 550: grad_norm=0.0705, loss=0.0116, param_norm=1803.8448 07:55:36.411 [I] Progress on: 550it/3.00kit rate:7.1s/it remaining:4:48:14 elapsed:1:09:25 postfix:- (1728:tqdm_logging.py:145) 07:55:51.008 [I] Progress on: 554it/3.00kit rate:6.8s/it remaining:4:36:28 elapsed:1:09:39 postfix:- (1728:tqdm_logging.py:145) 07:56:05.123 [I] Progress on: 556it/3.00kit rate:6.9s/it remaining:4:41:51 elapsed:1:09:53 postfix:- (1728:tqdm_logging.py:145) 07:56:19.258 [I] Progress on: 558it/3.00kit rate:7.0s/it remaining:4:44:46 elapsed:1:10:08 postfix:- (1728:tqdm_logging.py:145) 07:56:33.374 [I] Progress on: 560it/3.00kit rate:7.0s/it remaining:4:45:49 elapsed:1:10:22 postfix:- (1728:tqdm_logging.py:145) 07:56:47.506 [I] Progress on: 562it/3.00kit rate:7.0s/it remaining:4:46:23 elapsed:1:10:36 postfix:- (1728:tqdm_logging.py:145) 07:57:01.643 [I] Progress on: 564it/3.00kit rate:7.1s/it remaining:4:46:34 elapsed:1:10:50 postfix:- (1728:tqdm_logging.py:145) 07:57:15.758 [I] Progress on: 566it/3.00kit rate:7.1s/it remaining:4:46:18 elapsed:1:11:04 postfix:- (1728:tqdm_logging.py:145) 07:57:29.871 [I] Progress on: 568it/3.00kit rate:7.1s/it remaining:4:46:02 elapsed:1:11:18 postfix:- (1728:tqdm_logging.py:145) 07:57:43.997 [I] Progress on: 570it/3.00kit rate:7.1s/it remaining:4:45:56 elapsed:1:11:32 postfix:- (1728:tqdm_logging.py:145) 07:57:58.119 [I] Progress on: 572it/3.00kit rate:7.1s/it remaining:4:45:45 elapsed:1:11:46 postfix:- (1728:tqdm_logging.py:145) 07:58:12.261 [I] Progress on: 574it/3.00kit rate:7.1s/it remaining:4:45:43 elapsed:1:12:01 postfix:- (1728:tqdm_logging.py:145) 07:58:26.391 [I] Progress on: 576it/3.00kit rate:7.1s/it remaining:4:45:27 elapsed:1:12:15 postfix:- (1728:tqdm_logging.py:145) 07:58:40.499 [I] Progress on: 578it/3.00kit rate:7.1s/it remaining:4:44:59 elapsed:1:12:29 postfix:- (1728:tqdm_logging.py:145) 07:58:54.650 [I] Progress on: 580it/3.00kit rate:7.1s/it remaining:4:45:02 elapsed:1:12:43 postfix:- (1728:tqdm_logging.py:145) 07:59:08.777 [I] Progress on: 582it/3.00kit rate:7.1s/it remaining:4:44:45 elapsed:1:12:57 postfix:- (1728:tqdm_logging.py:145) 07:59:22.886 [I] Progress on: 584it/3.00kit rate:7.1s/it remaining:4:44:17 elapsed:1:13:11 postfix:- (1728:tqdm_logging.py:145) 07:59:37.028 [I] Progress on: 586it/3.00kit rate:7.1s/it remaining:4:44:17 elapsed:1:13:25 postfix:- (1728:tqdm_logging.py:145) 07:59:51.167 [I] Progress on: 588it/3.00kit rate:7.1s/it remaining:4:44:09 elapsed:1:13:39 postfix:- (1728:tqdm_logging.py:145) 08:00:05.302 [I] Progress on: 590it/3.00kit rate:7.1s/it remaining:4:43:52 elapsed:1:13:54 postfix:- (1728:tqdm_logging.py:145) 08:00:19.464 [I] Progress on: 592it/3.00kit rate:7.1s/it remaining:4:43:53 elapsed:1:14:08 postfix:- (1728:tqdm_logging.py:145) 08:00:33.593 [I] Progress on: 594it/3.00kit rate:7.1s/it remaining:4:43:31 elapsed:1:14:22 postfix:- (1728:tqdm_logging.py:145) 08:00:47.702 [I] Progress on: 596it/3.00kit rate:7.1s/it remaining:4:42:57 elapsed:1:14:36 postfix:- (1728:tqdm_logging.py:145) 08:01:01.844 [I] Progress on: 598it/3.00kit rate:7.1s/it remaining:4:42:56 elapsed:1:14:50 postfix:- (1728:tqdm_logging.py:145) 08:01:15.996 [I] Progress on: 600it/3.00kit rate:7.1s/it remaining:4:42:53 elapsed:1:15:04 postfix:- (1728:tqdm_logging.py:145) Step 600: grad_norm=0.0696, loss=0.0111, param_norm=1803.8556 08:01:29.757 [I] Progress on: 600it/3.00kit rate:7.1s/it remaining:4:42:53 elapsed:1:15:18 postfix:- (1728:tqdm_logging.py:145) 08:01:44.320 [I] Progress on: 604it/3.00kit rate:6.8s/it remaining:4:30:46 elapsed:1:15:33 postfix:- (1728:tqdm_logging.py:145) 08:01:58.445 [I] Progress on: 606it/3.00kit rate:6.9s/it remaining:4:36:14 elapsed:1:15:47 postfix:- (1728:tqdm_logging.py:145) 08:02:12.601 [I] Progress on: 608it/3.00kit rate:7.0s/it remaining:4:39:07 elapsed:1:16:01 postfix:- (1728:tqdm_logging.py:145) 08:02:26.759 [I] Progress on: 610it/3.00kit rate:7.0s/it remaining:4:40:26 elapsed:1:16:15 postfix:- (1728:tqdm_logging.py:145) 08:02:40.904 [I] Progress on: 612it/3.00kit rate:7.1s/it remaining:4:40:49 elapsed:1:16:29 postfix:- (1728:tqdm_logging.py:145) 08:02:55.063 [I] Progress on: 614it/3.00kit rate:7.1s/it remaining:4:41:02 elapsed:1:16:43 postfix:- (1728:tqdm_logging.py:145) 08:03:09.211 [I] Progress on: 616it/3.00kit rate:7.1s/it remaining:4:40:57 elapsed:1:16:57 postfix:- (1728:tqdm_logging.py:145) 08:03:23.353 [I] Progress on: 618it/3.00kit rate:7.1s/it remaining:4:40:40 elapsed:1:17:12 postfix:- (1728:tqdm_logging.py:145) 08:03:37.511 [I] Progress on: 620it/3.00kit rate:7.1s/it remaining:4:40:34 elapsed:1:17:26 postfix:- (1728:tqdm_logging.py:145) 08:03:51.666 [I] Progress on: 622it/3.00kit rate:7.1s/it remaining:4:40:24 elapsed:1:17:40 postfix:- (1728:tqdm_logging.py:145) 08:04:05.831 [I] Progress on: 624it/3.00kit rate:7.1s/it remaining:4:40:19 elapsed:1:17:54 postfix:- (1728:tqdm_logging.py:145) 08:04:19.981 [I] Progress on: 626it/3.00kit rate:7.1s/it remaining:4:40:00 elapsed:1:18:08 postfix:- (1728:tqdm_logging.py:145) 08:04:34.119 [I] Progress on: 628it/3.00kit rate:7.1s/it remaining:4:39:35 elapsed:1:18:22 postfix:- (1728:tqdm_logging.py:145) 08:04:48.251 [I] Progress on: 630it/3.00kit rate:7.1s/it remaining:4:39:11 elapsed:1:18:37 postfix:- (1728:tqdm_logging.py:145) 08:05:02.380 [I] Progress on: 632it/3.00kit rate:7.1s/it remaining:4:38:53 elapsed:1:18:51 postfix:- (1728:tqdm_logging.py:145) 08:05:16.514 [I] Progress on: 634it/3.00kit rate:7.1s/it remaining:4:38:41 elapsed:1:19:05 postfix:- (1728:tqdm_logging.py:145) 08:05:30.625 [I] Progress on: 636it/3.00kit rate:7.1s/it remaining:4:38:13 elapsed:1:19:19 postfix:- (1728:tqdm_logging.py:145) 08:05:44.746 [I] Progress on: 638it/3.00kit rate:7.1s/it remaining:4:37:58 elapsed:1:19:33 postfix:- (1728:tqdm_logging.py:145) 08:05:58.891 [I] Progress on: 640it/3.00kit rate:7.1s/it remaining:4:37:58 elapsed:1:19:47 postfix:- (1728:tqdm_logging.py:145) 08:06:13.022 [I] Progress on: 642it/3.00kit rate:7.1s/it remaining:4:37:44 elapsed:1:20:01 postfix:- (1728:tqdm_logging.py:145) 08:06:27.158 [I] Progress on: 644it/3.00kit rate:7.1s/it remaining:4:37:30 elapsed:1:20:15 postfix:- (1728:tqdm_logging.py:145) 08:06:41.284 [I] Progress on: 646it/3.00kit rate:7.1s/it remaining:4:37:12 elapsed:1:20:30 postfix:- (1728:tqdm_logging.py:145) 08:06:55.416 [I] Progress on: 648it/3.00kit rate:7.1s/it remaining:4:37:00 elapsed:1:20:44 postfix:- (1728:tqdm_logging.py:145) 08:07:09.528 [I] Progress on: 650it/3.00kit rate:7.1s/it remaining:4:36:33 elapsed:1:20:58 postfix:- (1728:tqdm_logging.py:145) Step 650: grad_norm=0.0620, loss=0.0100, param_norm=1803.8658 08:07:23.283 [I] Progress on: 650it/3.00kit rate:7.1s/it remaining:4:36:33 elapsed:1:21:12 postfix:- (1728:tqdm_logging.py:145) 08:07:37.836 [I] Progress on: 654it/3.00kit rate:6.8s/it remaining:4:24:50 elapsed:1:21:26 postfix:- (1728:tqdm_logging.py:145) 08:07:51.959 [I] Progress on: 656it/3.00kit rate:6.9s/it remaining:4:30:19 elapsed:1:21:40 postfix:- (1728:tqdm_logging.py:145) 08:08:06.084 [I] Progress on: 658it/3.00kit rate:7.0s/it remaining:4:32:56 elapsed:1:21:54 postfix:- (1728:tqdm_logging.py:145) 08:08:20.206 [I] Progress on: 660it/3.00kit rate:7.0s/it remaining:4:34:03 elapsed:1:22:08 postfix:- (1728:tqdm_logging.py:145) 08:08:34.316 [I] Progress on: 662it/3.00kit rate:7.0s/it remaining:4:34:21 elapsed:1:22:23 postfix:- (1728:tqdm_logging.py:145) 08:08:48.442 [I] Progress on: 664it/3.00kit rate:7.1s/it remaining:4:34:32 elapsed:1:22:37 postfix:- (1728:tqdm_logging.py:145) 08:09:03.055 [I] Progress on: 666it/3.00kit rate:7.2s/it remaining:4:40:14 elapsed:1:22:51 postfix:- (1728:tqdm_logging.py:145) 08:09:16.676 [I] Progress on: 668it/3.00kit rate:7.0s/it remaining:4:33:03 elapsed:1:23:05 postfix:- (1728:tqdm_logging.py:145) 08:09:30.796 [I] Progress on: 670it/3.00kit rate:7.0s/it remaining:4:33:30 elapsed:1:23:19 postfix:- (1728:tqdm_logging.py:145) 08:09:44.914 [I] Progress on: 672it/3.00kit rate:7.1s/it remaining:4:33:37 elapsed:1:23:33 postfix:- (1728:tqdm_logging.py:145) 08:09:59.033 [I] Progress on: 674it/3.00kit rate:7.1s/it remaining:4:33:33 elapsed:1:23:47 postfix:- (1728:tqdm_logging.py:145) 08:10:13.155 [I] Progress on: 676it/3.00kit rate:7.1s/it remaining:4:33:22 elapsed:1:24:01 postfix:- (1728:tqdm_logging.py:145) 08:10:27.263 [I] Progress on: 678it/3.00kit rate:7.1s/it remaining:4:33:04 elapsed:1:24:16 postfix:- (1728:tqdm_logging.py:145) 08:10:41.370 [I] Progress on: 680it/3.00kit rate:7.1s/it remaining:4:32:44 elapsed:1:24:30 postfix:- (1728:tqdm_logging.py:145) 08:10:55.515 [I] Progress on: 682it/3.00kit rate:7.1s/it remaining:4:32:49 elapsed:1:24:44 postfix:- (1728:tqdm_logging.py:145) 08:11:09.609 [I] Progress on: 684it/3.00kit rate:7.1s/it remaining:4:32:16 elapsed:1:24:58 postfix:- (1728:tqdm_logging.py:145) 08:11:23.739 [I] Progress on: 686it/3.00kit rate:7.1s/it remaining:4:32:13 elapsed:1:25:12 postfix:- (1728:tqdm_logging.py:145) 08:11:37.840 [I] Progress on: 688it/3.00kit rate:7.1s/it remaining:4:31:50 elapsed:1:25:26 postfix:- (1728:tqdm_logging.py:145) 08:11:51.969 [I] Progress on: 690it/3.00kit rate:7.1s/it remaining:4:31:48 elapsed:1:25:40 postfix:- (1728:tqdm_logging.py:145) 08:12:06.059 [I] Progress on: 692it/3.00kit rate:7.1s/it remaining:4:31:18 elapsed:1:25:54 postfix:- (1728:tqdm_logging.py:145) 08:12:20.181 [I] Progress on: 694it/3.00kit rate:7.1s/it remaining:4:31:12 elapsed:1:26:08 postfix:- (1728:tqdm_logging.py:145) 08:12:34.295 [I] Progress on: 696it/3.00kit rate:7.1s/it remaining:4:30:58 elapsed:1:26:23 postfix:- (1728:tqdm_logging.py:145) 08:12:48.397 [I] Progress on: 698it/3.00kit rate:7.1s/it remaining:4:30:37 elapsed:1:26:37 postfix:- (1728:tqdm_logging.py:145) 08:13:02.515 [I] Progress on: 700it/3.00kit rate:7.1s/it remaining:4:30:30 elapsed:1:26:51 postfix:- (1728:tqdm_logging.py:145) Step 700: grad_norm=0.0589, loss=0.0086, param_norm=1803.8752 08:13:16.264 [I] Progress on: 700it/3.00kit rate:7.1s/it remaining:4:30:30 elapsed:1:27:05 postfix:- (1728:tqdm_logging.py:145) 08:13:30.781 [I] Progress on: 704it/3.00kit rate:6.8s/it remaining:4:18:51 elapsed:1:27:19 postfix:- (1728:tqdm_logging.py:145) 08:13:44.901 [I] Progress on: 706it/3.00kit rate:6.9s/it remaining:4:24:22 elapsed:1:27:33 postfix:- (1728:tqdm_logging.py:145) 08:13:59.025 [I] Progress on: 708it/3.00kit rate:7.0s/it remaining:4:26:58 elapsed:1:27:47 postfix:- (1728:tqdm_logging.py:145) 08:14:13.135 [I] Progress on: 710it/3.00kit rate:7.0s/it remaining:4:28:02 elapsed:1:28:01 postfix:- (1728:tqdm_logging.py:145) 08:14:27.260 [I] Progress on: 712it/3.00kit rate:7.0s/it remaining:4:28:33 elapsed:1:28:16 postfix:- (1728:tqdm_logging.py:145) 08:14:41.373 [I] Progress on: 714it/3.00kit rate:7.0s/it remaining:4:28:33 elapsed:1:28:30 postfix:- (1728:tqdm_logging.py:145) 08:14:55.564 [I] Progress on: 716it/3.00kit rate:7.1s/it remaining:4:29:18 elapsed:1:28:44 postfix:- (1728:tqdm_logging.py:145) 08:15:09.643 [I] Progress on: 718it/3.00kit rate:7.1s/it remaining:4:28:21 elapsed:1:28:58 postfix:- (1728:tqdm_logging.py:145) 08:15:23.766 [I] Progress on: 720it/3.00kit rate:7.1s/it remaining:4:28:12 elapsed:1:29:12 postfix:- (1728:tqdm_logging.py:145) 08:15:37.889 [I] Progress on: 722it/3.00kit rate:7.1s/it remaining:4:28:01 elapsed:1:29:26 postfix:- (1728:tqdm_logging.py:145) 08:15:52.004 [I] Progress on: 724it/3.00kit rate:7.1s/it remaining:4:27:44 elapsed:1:29:40 postfix:- (1728:tqdm_logging.py:145) 08:16:06.118 [I] Progress on: 726it/3.00kit rate:7.1s/it remaining:4:27:28 elapsed:1:29:54 postfix:- (1728:tqdm_logging.py:145) 08:16:20.241 [I] Progress on: 728it/3.00kit rate:7.1s/it remaining:4:27:17 elapsed:1:30:09 postfix:- (1728:tqdm_logging.py:145) 08:16:34.364 [I] Progress on: 730it/3.00kit rate:7.1s/it remaining:4:27:05 elapsed:1:30:23 postfix:- (1728:tqdm_logging.py:145) 08:16:48.484 [I] Progress on: 732it/3.00kit rate:7.1s/it remaining:4:26:53 elapsed:1:30:37 postfix:- (1728:tqdm_logging.py:145) 08:17:02.598 [I] Progress on: 734it/3.00kit rate:7.1s/it remaining:4:26:33 elapsed:1:30:51 postfix:- (1728:tqdm_logging.py:145) 08:17:16.728 [I] Progress on: 736it/3.00kit rate:7.1s/it remaining:4:26:27 elapsed:1:31:05 postfix:- (1728:tqdm_logging.py:145) 08:17:30.834 [I] Progress on: 738it/3.00kit rate:7.1s/it remaining:4:26:02 elapsed:1:31:19 postfix:- (1728:tqdm_logging.py:145) 08:17:44.947 [I] Progress on: 740it/3.00kit rate:7.1s/it remaining:4:25:47 elapsed:1:31:33 postfix:- (1728:tqdm_logging.py:145) 08:17:59.066 [I] Progress on: 742it/3.00kit rate:7.1s/it remaining:4:25:34 elapsed:1:31:47 postfix:- (1728:tqdm_logging.py:145) 08:18:13.164 [I] Progress on: 744it/3.00kit rate:7.1s/it remaining:4:25:08 elapsed:1:32:01 postfix:- (1728:tqdm_logging.py:145) 08:18:27.283 [I] Progress on: 746it/3.00kit rate:7.1s/it remaining:4:25:03 elapsed:1:32:16 postfix:- (1728:tqdm_logging.py:145) 08:18:41.398 [I] Progress on: 748it/3.00kit rate:7.1s/it remaining:4:24:50 elapsed:1:32:30 postfix:- (1728:tqdm_logging.py:145) 08:18:55.498 [I] Progress on: 750it/3.00kit rate:7.1s/it remaining:4:24:29 elapsed:1:32:44 postfix:- (1728:tqdm_logging.py:145) Step 750: grad_norm=0.0588, loss=0.0089, param_norm=1803.8850 08:19:09.264 [I] Progress on: 750it/3.00kit rate:7.1s/it remaining:4:24:29 elapsed:1:32:58 postfix:- (1728:tqdm_logging.py:145) 08:19:23.817 [I] Progress on: 754it/3.00kit rate:6.8s/it remaining:4:13:26 elapsed:1:33:12 postfix:- (1728:tqdm_logging.py:145) 08:19:37.931 [I] Progress on: 756it/3.00kit rate:6.9s/it remaining:4:18:38 elapsed:1:33:26 postfix:- (1728:tqdm_logging.py:145) 08:19:52.055 [I] Progress on: 758it/3.00kit rate:7.0s/it remaining:4:21:11 elapsed:1:33:40 postfix:- (1728:tqdm_logging.py:145) 08:20:06.166 [I] Progress on: 760it/3.00kit rate:7.0s/it remaining:4:22:12 elapsed:1:33:54 postfix:- (1728:tqdm_logging.py:145) 08:20:20.288 [I] Progress on: 762it/3.00kit rate:7.0s/it remaining:4:22:38 elapsed:1:34:09 postfix:- (1728:tqdm_logging.py:145) 08:20:34.397 [I] Progress on: 764it/3.00kit rate:7.0s/it remaining:4:22:38 elapsed:1:34:23 postfix:- (1728:tqdm_logging.py:145) 08:20:48.531 [I] Progress on: 766it/3.00kit rate:7.1s/it remaining:4:22:46 elapsed:1:34:37 postfix:- (1728:tqdm_logging.py:145) 08:21:02.648 [I] Progress on: 768it/3.00kit rate:7.1s/it remaining:4:22:32 elapsed:1:34:51 postfix:- (1728:tqdm_logging.py:145) 08:21:16.759 [I] Progress on: 770it/3.00kit rate:7.1s/it remaining:4:22:14 elapsed:1:35:05 postfix:- (1728:tqdm_logging.py:145) 08:21:30.880 [I] Progress on: 772it/3.00kit rate:7.1s/it remaining:4:22:03 elapsed:1:35:19 postfix:- (1728:tqdm_logging.py:145) 08:21:44.992 [I] Progress on: 774it/3.00kit rate:7.1s/it remaining:4:21:46 elapsed:1:35:33 postfix:- (1728:tqdm_logging.py:145) 08:21:59.125 [I] Progress on: 776it/3.00kit rate:7.1s/it remaining:4:21:44 elapsed:1:35:47 postfix:- (1728:tqdm_logging.py:145) 08:22:13.253 [I] Progress on: 778it/3.00kit rate:7.1s/it remaining:4:21:32 elapsed:1:36:02 postfix:- (1728:tqdm_logging.py:145) 08:22:27.363 [I] Progress on: 780it/3.00kit rate:7.1s/it remaining:4:21:08 elapsed:1:36:16 postfix:- (1728:tqdm_logging.py:145) 08:22:41.470 [I] Progress on: 782it/3.00kit rate:7.1s/it remaining:4:20:47 elapsed:1:36:30 postfix:- (1728:tqdm_logging.py:145) 08:22:55.598 [I] Progress on: 784it/3.00kit rate:7.1s/it remaining:4:20:43 elapsed:1:36:44 postfix:- (1728:tqdm_logging.py:145) 08:23:09.706 [I] Progress on: 786it/3.00kit rate:7.1s/it remaining:4:20:22 elapsed:1:36:58 postfix:- (1728:tqdm_logging.py:145) 08:23:23.833 [I] Progress on: 788it/3.00kit rate:7.1s/it remaining:4:20:14 elapsed:1:37:12 postfix:- (1728:tqdm_logging.py:145) 08:23:37.947 [I] Progress on: 790it/3.00kit rate:7.1s/it remaining:4:19:54 elapsed:1:37:26 postfix:- (1728:tqdm_logging.py:145) 08:23:52.058 [I] Progress on: 792it/3.00kit rate:7.1s/it remaining:4:19:39 elapsed:1:37:40 postfix:- (1728:tqdm_logging.py:145) 08:24:06.173 [I] Progress on: 794it/3.00kit rate:7.1s/it remaining:4:19:26 elapsed:1:37:54 postfix:- (1728:tqdm_logging.py:145) 08:24:20.288 [I] Progress on: 796it/3.00kit rate:7.1s/it remaining:4:19:12 elapsed:1:38:09 postfix:- (1728:tqdm_logging.py:145) 08:24:34.420 [I] Progress on: 798it/3.00kit rate:7.1s/it remaining:4:19:09 elapsed:1:38:23 postfix:- (1728:tqdm_logging.py:145) 08:24:48.534 [I] Progress on: 800it/3.00kit rate:7.1s/it remaining:4:18:49 elapsed:1:38:37 postfix:- (1728:tqdm_logging.py:145) Step 800: grad_norm=0.0571, loss=0.0083, param_norm=1803.8943 08:25:02.298 [I] Progress on: 800it/3.00kit rate:7.1s/it remaining:4:18:49 elapsed:1:38:51 postfix:- (1728:tqdm_logging.py:145) 08:25:16.857 [I] Progress on: 804it/3.00kit rate:6.8s/it remaining:4:07:56 elapsed:1:39:05 postfix:- (1728:tqdm_logging.py:145) 08:25:30.950 [I] Progress on: 806it/3.00kit rate:6.9s/it remaining:4:12:46 elapsed:1:39:19 postfix:- (1728:tqdm_logging.py:145) 08:25:45.083 [I] Progress on: 808it/3.00kit rate:7.0s/it remaining:4:15:24 elapsed:1:39:33 postfix:- (1728:tqdm_logging.py:145) 08:25:59.196 [I] Progress on: 810it/3.00kit rate:7.0s/it remaining:4:16:21 elapsed:1:39:47 postfix:- (1728:tqdm_logging.py:145) 08:26:13.310 [I] Progress on: 812it/3.00kit rate:7.0s/it remaining:4:16:41 elapsed:1:40:02 postfix:- (1728:tqdm_logging.py:145) 08:26:27.454 [I] Progress on: 814it/3.00kit rate:7.1s/it remaining:4:17:01 elapsed:1:40:16 postfix:- (1728:tqdm_logging.py:145) 08:26:41.575 [I] Progress on: 816it/3.00kit rate:7.1s/it remaining:4:16:54 elapsed:1:40:30 postfix:- (1728:tqdm_logging.py:145) 08:26:55.682 [I] Progress on: 818it/3.00kit rate:7.1s/it remaining:4:16:36 elapsed:1:40:44 postfix:- (1728:tqdm_logging.py:145) 08:27:09.807 [I] Progress on: 820it/3.00kit rate:7.1s/it remaining:4:16:29 elapsed:1:40:58 postfix:- (1728:tqdm_logging.py:145) 08:27:23.920 [I] Progress on: 822it/3.00kit rate:7.1s/it remaining:4:16:11 elapsed:1:41:12 postfix:- (1728:tqdm_logging.py:145) 08:27:38.024 [I] Progress on: 824it/3.00kit rate:7.1s/it remaining:4:15:51 elapsed:1:41:26 postfix:- (1728:tqdm_logging.py:145) 08:27:52.136 [I] Progress on: 826it/3.00kit rate:7.1s/it remaining:4:15:39 elapsed:1:41:40 postfix:- (1728:tqdm_logging.py:145) 08:28:06.266 [I] Progress on: 828it/3.00kit rate:7.1s/it remaining:4:15:37 elapsed:1:41:55 postfix:- (1728:tqdm_logging.py:145) 08:28:20.382 [I] Progress on: 830it/3.00kit rate:7.1s/it remaining:4:15:22 elapsed:1:42:09 postfix:- (1728:tqdm_logging.py:145) 08:28:34.498 [I] Progress on: 832it/3.00kit rate:7.1s/it remaining:4:15:06 elapsed:1:42:23 postfix:- (1728:tqdm_logging.py:145) 08:28:48.607 [I] Progress on: 834it/3.00kit rate:7.1s/it remaining:4:14:47 elapsed:1:42:37 postfix:- (1728:tqdm_logging.py:145) 08:29:02.719 [I] Progress on: 836it/3.00kit rate:7.1s/it remaining:4:14:31 elapsed:1:42:51 postfix:- (1728:tqdm_logging.py:145) 08:29:16.819 [I] Progress on: 838it/3.00kit rate:7.1s/it remaining:4:14:09 elapsed:1:43:05 postfix:- (1728:tqdm_logging.py:145) 08:29:30.963 [I] Progress on: 840it/3.00kit rate:7.1s/it remaining:4:14:18 elapsed:1:43:19 postfix:- (1728:tqdm_logging.py:145) 08:29:45.085 [I] Progress on: 842it/3.00kit rate:7.1s/it remaining:4:14:03 elapsed:1:43:33 postfix:- (1728:tqdm_logging.py:145) 08:29:59.204 [I] Progress on: 844it/3.00kit rate:7.1s/it remaining:4:13:45 elapsed:1:43:47 postfix:- (1728:tqdm_logging.py:145) 08:30:13.329 [I] Progress on: 846it/3.00kit rate:7.1s/it remaining:4:13:31 elapsed:1:44:02 postfix:- (1728:tqdm_logging.py:145) 08:30:27.439 [I] Progress on: 848it/3.00kit rate:7.1s/it remaining:4:13:11 elapsed:1:44:16 postfix:- (1728:tqdm_logging.py:145) 08:30:41.540 [I] Progress on: 850it/3.00kit rate:7.1s/it remaining:4:12:48 elapsed:1:44:30 postfix:- (1728:tqdm_logging.py:145) Step 850: grad_norm=0.0576, loss=0.0083, param_norm=1803.9048 08:30:55.314 [I] Progress on: 850it/3.00kit rate:7.1s/it remaining:4:12:48 elapsed:1:44:44 postfix:- (1728:tqdm_logging.py:145) 08:31:09.859 [I] Progress on: 854it/3.00kit rate:6.8s/it remaining:4:02:08 elapsed:1:44:58 postfix:- (1728:tqdm_logging.py:145) 08:31:23.974 [I] Progress on: 856it/3.00kit rate:6.9s/it remaining:4:07:08 elapsed:1:45:12 postfix:- (1728:tqdm_logging.py:145) 08:31:38.091 [I] Progress on: 858it/3.00kit rate:7.0s/it remaining:4:09:26 elapsed:1:45:26 postfix:- (1728:tqdm_logging.py:145) 08:31:52.211 [I] Progress on: 860it/3.00kit rate:7.0s/it remaining:4:10:31 elapsed:1:45:40 postfix:- (1728:tqdm_logging.py:145) 08:32:06.331 [I] Progress on: 862it/3.00kit rate:7.0s/it remaining:4:10:56 elapsed:1:45:55 postfix:- (1728:tqdm_logging.py:145) 08:32:20.454 [I] Progress on: 864it/3.00kit rate:7.1s/it remaining:4:11:02 elapsed:1:46:09 postfix:- (1728:tqdm_logging.py:145) 08:32:34.569 [I] Progress on: 866it/3.00kit rate:7.1s/it remaining:4:10:58 elapsed:1:46:23 postfix:- (1728:tqdm_logging.py:145) 08:32:48.667 [I] Progress on: 868it/3.00kit rate:7.1s/it remaining:4:10:36 elapsed:1:46:37 postfix:- (1728:tqdm_logging.py:145) 08:33:02.785 [I] Progress on: 870it/3.00kit rate:7.1s/it remaining:4:10:28 elapsed:1:46:51 postfix:- (1728:tqdm_logging.py:145) 08:33:16.926 [I] Progress on: 872it/3.00kit rate:7.1s/it remaining:4:10:32 elapsed:1:47:05 postfix:- (1728:tqdm_logging.py:145) 08:33:31.063 [I] Progress on: 874it/3.00kit rate:7.1s/it remaining:4:10:21 elapsed:1:47:19 postfix:- (1728:tqdm_logging.py:145) 08:33:45.179 [I] Progress on: 876it/3.00kit rate:7.1s/it remaining:4:10:00 elapsed:1:47:33 postfix:- (1728:tqdm_logging.py:145) 08:33:59.297 [I] Progress on: 878it/3.00kit rate:7.1s/it remaining:4:09:44 elapsed:1:47:48 postfix:- (1728:tqdm_logging.py:145) 08:34:13.395 [I] Progress on: 880it/3.00kit rate:7.1s/it remaining:4:09:17 elapsed:1:48:02 postfix:- (1728:tqdm_logging.py:145) 08:34:27.527 [I] Progress on: 882it/3.00kit rate:7.1s/it remaining:4:09:16 elapsed:1:48:16 postfix:- (1728:tqdm_logging.py:145) 08:34:41.626 [I] Progress on: 884it/3.00kit rate:7.1s/it remaining:4:08:49 elapsed:1:48:30 postfix:- (1728:tqdm_logging.py:145) 08:34:55.748 [I] Progress on: 886it/3.00kit rate:7.1s/it remaining:4:08:40 elapsed:1:48:44 postfix:- (1728:tqdm_logging.py:145) 08:35:10.428 [I] Progress on: 888it/3.00kit rate:7.2s/it remaining:4:14:23 elapsed:1:48:59 postfix:- (1728:tqdm_logging.py:145) 08:35:24.000 [I] Progress on: 890it/3.00kit rate:7.0s/it remaining:4:07:10 elapsed:1:49:12 postfix:- (1728:tqdm_logging.py:145) 08:35:38.093 [I] Progress on: 892it/3.00kit rate:7.0s/it remaining:4:07:20 elapsed:1:49:26 postfix:- (1728:tqdm_logging.py:145) 08:35:52.208 [I] Progress on: 894it/3.00kit rate:7.0s/it remaining:4:07:22 elapsed:1:49:40 postfix:- (1728:tqdm_logging.py:145) 08:36:06.329 [I] Progress on: 896it/3.00kit rate:7.1s/it remaining:4:07:19 elapsed:1:49:55 postfix:- (1728:tqdm_logging.py:145) 08:36:20.461 [I] Progress on: 898it/3.00kit rate:7.1s/it remaining:4:07:18 elapsed:1:50:09 postfix:- (1728:tqdm_logging.py:145) 08:36:34.579 [I] Progress on: 900it/3.00kit rate:7.1s/it remaining:4:07:02 elapsed:1:50:23 postfix:- (1728:tqdm_logging.py:145) Step 900: grad_norm=0.0554, loss=0.0080, param_norm=1803.9143 08:36:48.331 [I] Progress on: 900it/3.00kit rate:7.1s/it remaining:4:07:02 elapsed:1:50:37 postfix:- (1728:tqdm_logging.py:145) 08:37:02.850 [I] Progress on: 904it/3.00kit rate:6.8s/it remaining:3:56:22 elapsed:1:50:51 postfix:- (1728:tqdm_logging.py:145) 08:37:16.966 [I] Progress on: 906it/3.00kit rate:6.9s/it remaining:4:01:20 elapsed:1:51:05 postfix:- (1728:tqdm_logging.py:145) 08:37:31.089 [I] Progress on: 908it/3.00kit rate:7.0s/it remaining:4:03:43 elapsed:1:51:19 postfix:- (1728:tqdm_logging.py:145) 08:37:45.200 [I] Progress on: 910it/3.00kit rate:7.0s/it remaining:4:04:39 elapsed:1:51:33 postfix:- (1728:tqdm_logging.py:145) 08:37:59.316 [I] Progress on: 912it/3.00kit rate:7.0s/it remaining:4:05:01 elapsed:1:51:48 postfix:- (1728:tqdm_logging.py:145) 08:38:13.433 [I] Progress on: 914it/3.00kit rate:7.0s/it remaining:4:05:05 elapsed:1:52:02 postfix:- (1728:tqdm_logging.py:145) 08:38:27.576 [I] Progress on: 916it/3.00kit rate:7.1s/it remaining:4:05:16 elapsed:1:52:16 postfix:- (1728:tqdm_logging.py:145) 08:38:41.699 [I] Progress on: 918it/3.00kit rate:7.1s/it remaining:4:05:04 elapsed:1:52:30 postfix:- (1728:tqdm_logging.py:145) 08:38:55.823 [I] Progress on: 920it/3.00kit rate:7.1s/it remaining:4:04:50 elapsed:1:52:44 postfix:- (1728:tqdm_logging.py:145) 08:39:09.923 [I] Progress on: 922it/3.00kit rate:7.1s/it remaining:4:04:23 elapsed:1:52:58 postfix:- (1728:tqdm_logging.py:145) 08:39:24.059 [I] Progress on: 924it/3.00kit rate:7.1s/it remaining:4:04:22 elapsed:1:53:12 postfix:- (1728:tqdm_logging.py:145) 08:39:38.170 [I] Progress on: 926it/3.00kit rate:7.1s/it remaining:4:04:01 elapsed:1:53:26 postfix:- (1728:tqdm_logging.py:145) 08:39:52.270 [I] Progress on: 928it/3.00kit rate:7.1s/it remaining:4:03:38 elapsed:1:53:41 postfix:- (1728:tqdm_logging.py:145) 08:40:06.394 [I] Progress on: 930it/3.00kit rate:7.1s/it remaining:4:03:30 elapsed:1:53:55 postfix:- (1728:tqdm_logging.py:145) 08:40:20.519 [I] Progress on: 932it/3.00kit rate:7.1s/it remaining:4:03:20 elapsed:1:54:09 postfix:- (1728:tqdm_logging.py:145) 08:40:34.631 [I] Progress on: 934it/3.00kit rate:7.1s/it remaining:4:03:03 elapsed:1:54:23 postfix:- (1728:tqdm_logging.py:145) 08:40:48.731 [I] Progress on: 936it/3.00kit rate:7.1s/it remaining:4:02:40 elapsed:1:54:37 postfix:- (1728:tqdm_logging.py:145) 08:41:02.859 [I] Progress on: 938it/3.00kit rate:7.1s/it remaining:4:02:39 elapsed:1:54:51 postfix:- (1728:tqdm_logging.py:145) 08:41:16.973 [I] Progress on: 940it/3.00kit rate:7.1s/it remaining:4:02:22 elapsed:1:55:05 postfix:- (1728:tqdm_logging.py:145) 08:41:31.090 [I] Progress on: 942it/3.00kit rate:7.1s/it remaining:4:02:08 elapsed:1:55:19 postfix:- (1728:tqdm_logging.py:145) 08:41:45.206 [I] Progress on: 944it/3.00kit rate:7.1s/it remaining:4:01:54 elapsed:1:55:33 postfix:- (1728:tqdm_logging.py:145) 08:41:59.296 [I] Progress on: 946it/3.00kit rate:7.1s/it remaining:4:01:25 elapsed:1:55:48 postfix:- (1728:tqdm_logging.py:145) 08:42:13.409 [I] Progress on: 948it/3.00kit rate:7.1s/it remaining:4:01:15 elapsed:1:56:02 postfix:- (1728:tqdm_logging.py:145) 08:42:27.537 [I] Progress on: 950it/3.00kit rate:7.1s/it remaining:4:01:11 elapsed:1:56:16 postfix:- (1728:tqdm_logging.py:145) Step 950: grad_norm=0.0543, loss=0.0075, param_norm=1803.9233 08:42:41.286 [I] Progress on: 950it/3.00kit rate:7.1s/it remaining:4:01:11 elapsed:1:56:30 postfix:- (1728:tqdm_logging.py:145) 08:42:55.799 [I] Progress on: 954it/3.00kit rate:6.8s/it remaining:3:50:40 elapsed:1:56:44 postfix:- (1728:tqdm_logging.py:145) 08:43:09.916 [I] Progress on: 956it/3.00kit rate:6.9s/it remaining:3:55:30 elapsed:1:56:58 postfix:- (1728:tqdm_logging.py:145) 08:43:24.029 [I] Progress on: 958it/3.00kit rate:7.0s/it remaining:3:57:44 elapsed:1:57:12 postfix:- (1728:tqdm_logging.py:145) 08:43:38.162 [I] Progress on: 960it/3.00kit rate:7.0s/it remaining:3:58:55 elapsed:1:57:26 postfix:- (1728:tqdm_logging.py:145) 08:43:52.261 [I] Progress on: 962it/3.00kit rate:7.0s/it remaining:3:59:03 elapsed:1:57:41 postfix:- (1728:tqdm_logging.py:145) 08:44:06.385 [I] Progress on: 964it/3.00kit rate:7.1s/it remaining:3:59:13 elapsed:1:57:55 postfix:- (1728:tqdm_logging.py:145) 08:44:20.504 [I] Progress on: 966it/3.00kit rate:7.1s/it remaining:3:59:11 elapsed:1:58:09 postfix:- (1728:tqdm_logging.py:145) 08:44:34.624 [I] Progress on: 968it/3.00kit rate:7.1s/it remaining:3:59:02 elapsed:1:58:23 postfix:- (1728:tqdm_logging.py:145) 08:44:48.748 [I] Progress on: 970it/3.00kit rate:7.1s/it remaining:3:58:52 elapsed:1:58:37 postfix:- (1728:tqdm_logging.py:145) 08:45:02.879 [I] Progress on: 972it/3.00kit rate:7.1s/it remaining:3:58:44 elapsed:1:58:51 postfix:- (1728:tqdm_logging.py:145) 08:45:16.974 [I] Progress on: 974it/3.00kit rate:7.1s/it remaining:3:58:13 elapsed:1:59:05 postfix:- (1728:tqdm_logging.py:145) 08:45:31.109 [I] Progress on: 976it/3.00kit rate:7.1s/it remaining:3:58:12 elapsed:1:59:19 postfix:- (1728:tqdm_logging.py:145) 08:45:45.234 [I] Progress on: 978it/3.00kit rate:7.1s/it remaining:3:58:02 elapsed:1:59:34 postfix:- (1728:tqdm_logging.py:145) 08:45:59.363 [I] Progress on: 980it/3.00kit rate:7.1s/it remaining:3:57:50 elapsed:1:59:48 postfix:- (1728:tqdm_logging.py:145) 08:46:13.462 [I] Progress on: 982it/3.00kit rate:7.1s/it remaining:3:57:21 elapsed:2:00:02 postfix:- (1728:tqdm_logging.py:145) 08:46:27.578 [I] Progress on: 984it/3.00kit rate:7.1s/it remaining:3:57:07 elapsed:2:00:16 postfix:- (1728:tqdm_logging.py:145) 08:46:41.775 [I] Progress on: 986it/3.00kit rate:7.1s/it remaining:3:57:42 elapsed:2:00:30 postfix:- (1728:tqdm_logging.py:145) 08:46:55.847 [I] Progress on: 988it/3.00kit rate:7.1s/it remaining:3:56:47 elapsed:2:00:44 postfix:- (1728:tqdm_logging.py:145) 08:47:09.943 [I] Progress on: 990it/3.00kit rate:7.1s/it remaining:3:56:22 elapsed:2:00:58 postfix:- (1728:tqdm_logging.py:145) 08:47:24.056 [I] Progress on: 992it/3.00kit rate:7.1s/it remaining:3:56:06 elapsed:2:01:12 postfix:- (1728:tqdm_logging.py:145) 08:47:38.153 [I] Progress on: 994it/3.00kit rate:7.1s/it remaining:3:55:46 elapsed:2:01:26 postfix:- (1728:tqdm_logging.py:145) 08:47:52.270 [I] Progress on: 996it/3.00kit rate:7.1s/it remaining:3:55:36 elapsed:2:01:41 postfix:- (1728:tqdm_logging.py:145) 08:48:06.384 [I] Progress on: 998it/3.00kit rate:7.1s/it remaining:3:55:23 elapsed:2:01:55 postfix:- (1728:tqdm_logging.py:145) 08:48:20.505 [I] Progress on: 1.00kit/3.00kit rate:7.1s/it remaining:3:55:12 elapsed:2:02:09 postfix:- (1728:tqdm_logging.py:145) Step 1000: grad_norm=0.0518, loss=0.0072, param_norm=1803.9321 08:48:34.253 [I] Progress on: 1.00kit/3.00kit rate:7.1s/it remaining:3:55:12 elapsed:2:02:23 postfix:- (1728:tqdm_logging.py:145) 08:48:34.297 [I] [process=0][thread=MainThread][wait_until_finished] No Save Finalize thread to wait for. Returning. (1728:checkpoint_manager.py:1987) 08:48:34.298 [I] [process=0] Saving checkpoint at step 1000 (1728:checkpoint_manager.py:1408) 08:48:34.299 [I] [process=0] Started async saving checkpoint to /workspace/openpi/checkpoints/pi05_mecharm_deep_even10_lora/deep_even10_v1/1000. (1728:async_checkpointer.py:439) 08:48:34.300 [I] Using ThreadSafeKeyValueSignalingClient (1728:signaling_client.py:332) 08:48:34.303 [I] Creating tmp directory /workspace/openpi/checkpoints/pi05_mecharm_deep_even10_lora/deep_even10_v1/1000.orbax-checkpoint-tmp-0 (1728:atomicity.py:144) 08:48:34.319 [I] Wrote Metadata={'item_handlers': None, 'metrics': {}, 'performance_metrics': {}, 'init_timestamp_nsecs': 1794214114317702192, 'commit_timestamp_nsecs': None, 'custom_metadata': {}}, json={"item_handlers": null, "metrics": {}, "performance_metrics": {}, "init_timestamp_nsecs": 1794214114317702192, "commit_timestamp_nsecs": null, "custom_metadata": {}} to /workspace/openpi/checkpoints/pi05_mecharm_deep_even10_lora/deep_even10_v1/1000.orbax-checkpoint-tmp-0/_CHECKPOINT_METADATA (1728:checkpoint.py:186) 08:48:34.321 [I] Creating tmp directory /workspace/openpi/checkpoints/pi05_mecharm_deep_even10_lora/deep_even10_v1/1000.orbax-checkpoint-tmp-0/assets.orbax-checkpoint-tmp-1 (1728:atomicity.py:144) 08:48:34.322 [I] Creating tmp directory /workspace/openpi/checkpoints/pi05_mecharm_deep_even10_lora/deep_even10_v1/1000.orbax-checkpoint-tmp-0/params.orbax-checkpoint-tmp-2 (1728:atomicity.py:144) 08:48:34.323 [I] Creating tmp directory /workspace/openpi/checkpoints/pi05_mecharm_deep_even10_lora/deep_even10_v1/1000.orbax-checkpoint-tmp-0/train_state.orbax-checkpoint-tmp-3 (1728:atomicity.py:144) 08:48:34.343 [I] Transferring arrays to host memory with options: use_replica_parallel=True, enable_pinned_host_transfer=False (1728:replica_slices.py:341) 08:48:40.505 [I] Array name: 'params.PaliGemma.llm.layers.mlp.gating_einsum.value', global shape: (18, 2, 2048, 16384), write shape: (18, 2, 2048, 16384), chosen chunk shape: (18, 2, 2048, 8192) (1728:tensorstore_utils.py:408) 08:48:40.506 [I] Transferring arrays to host memory with options: use_replica_parallel=True, enable_pinned_host_transfer=False (1728:replica_slices.py:341) 08:48:40.565 [I] [process=0][thread=array_type_handler] Wrote 71 array_metadata.ArrayMetadata to /workspace/openpi/checkpoints/pi05_mecharm_deep_even10_lora/deep_even10_v1/1000.orbax-checkpoint-tmp-0/params.orbax-checkpoint-tmp-2/array_metadatas/process_0 (1728:array_metadata_store.py:198) 08:48:44.391 [I] [process=0] /jax/checkpoint/write/blocking_bytes_per_sec: 731.8 MiB/s (total bytes: 7.2 GiB) (time elapsed: 10 seconds) (per-host) (1728:base_pytree_checkpoint_handler.py:114) 08:48:44.396 [I] [process=0] /jax/checkpoint/write/blocking_bytes_per_sec: 353.4 MiB/s (total bytes: 3.5 GiB) (time elapsed: 10 seconds) (per-host) (1728:base_pytree_checkpoint_handler.py:114) 08:48:44.409 [I] [process=0][thread=async_save] Background save thread started. (1728:async_checkpointer.py:77) 08:48:44.409 [I] Finished blocking save in 10.11 seconds. Continuing to save asynchronously to /workspace/openpi/checkpoints/pi05_mecharm_deep_even10_lora/deep_even10_v1/1000. (1728:async_checkpointer.py:548) 08:48:44.411 [I] [process=0][thread=MainThread][step=1000] Starting CheckpointManager Save Finalize thread=save_finalize (1728:checkpoint_manager.py:1453) 08:48:44.416 [I] [process=0][thread=save_finalize] Waiting for background save thread=async_save. (1728:async_checkpointer.py:258) 08:48:44.418 [I] {'step': 1000, 'event_type': 'save', 'directory': '/workspace/openpi/checkpoints/pi05_mecharm_deep_even10_lora/deep_even10_v1', 'reached_preemption': False, 'preemption_received_at': None, 'synchronous': False, 'wait_for_prev_start_time': 1794214114.2973533, 'wait_for_prev_duration_secs': 0.0001914501190185547, 'checkpointer_blocking_start_time': 1794214114.2982178, 'checkpointer_blocking_duration_secs': 10.112950325012207, 'get_old_steps_start_time': 1794214124.4111974, 'get_old_steps_duration_secs': 8.106231689453125e-06, 'checkpoint_manager_blocking_start_time': 1794214114.2969413, 'checkpoint_manager_blocking_duration_secs': 10.1212158203125} (1728:standard_logger.py:34) 08:48:44.420 [I] Progress on: 1.00kit/3.00kit rate:12.1s/it remaining:6:43:35 elapsed:2:02:33 postfix:- (1728:tqdm_logging.py:145) 08:48:44.491 [I] [process=0][thread=array_type_handler] Wrote 105 array_metadata.ArrayMetadata to /workspace/openpi/checkpoints/pi05_mecharm_deep_even10_lora/deep_even10_v1/1000.orbax-checkpoint-tmp-0/train_state.orbax-checkpoint-tmp-3/array_metadatas/process_0 (1728:array_metadata_store.py:198) 08:48:45.825 [I] [process=0] /jax/checkpoint/write/bytes_per_sec: 640.7 MiB/s (total bytes: 7.2 GiB) (time elapsed: 11 seconds) (per-host) (1728:base_pytree_checkpoint_handler.py:114) 08:48:47.183 [I] [process=0] /jax/checkpoint/write/bytes_per_sec: 276.9 MiB/s (total bytes: 3.5 GiB) (time elapsed: 12 seconds) (per-host) (1728:base_pytree_checkpoint_handler.py:114) 08:48:47.183 [I] [process=0][thread=async_save] 5 Handler Commit operations completed. (1728:async_checkpointer.py:87) 08:48:47.184 [I] Read Metadata={'item_handlers': None, 'metrics': {}, 'performance_metrics': {}, 'init_timestamp_nsecs': 1794214114317702192, 'commit_timestamp_nsecs': None, 'custom_metadata': {}} from /workspace/openpi/checkpoints/pi05_mecharm_deep_even10_lora/deep_even10_v1/1000.orbax-checkpoint-tmp-0/_CHECKPOINT_METADATA (1728:checkpoint.py:226) 08:48:47.184 [I] Renaming /workspace/openpi/checkpoints/pi05_mecharm_deep_even10_lora/deep_even10_v1/1000.orbax-checkpoint-tmp-0/assets.orbax-checkpoint-tmp-1 to /workspace/openpi/checkpoints/pi05_mecharm_deep_even10_lora/deep_even10_v1/1000.orbax-checkpoint-tmp-0/assets (1728:atomicity.py:289) 08:48:47.187 [I] Updated Metadata={'item_handlers': {'assets': 'openpi.training.checkpoints.CallbackHandler', 'params': 'orbax.checkpoint._src.handlers.pytree_checkpoint_handler.PyTreeCheckpointHandler', 'train_state': 'orbax.checkpoint._src.handlers.pytree_checkpoint_handler.PyTreeCheckpointHandler'}, 'metrics': {}, 'performance_metrics': {}, 'init_timestamp_nsecs': 1794214114317702192, 'commit_timestamp_nsecs': None, 'custom_metadata': {}} to /workspace/openpi/checkpoints/pi05_mecharm_deep_even10_lora/deep_even10_v1/1000.orbax-checkpoint-tmp-0/_CHECKPOINT_METADATA (1728:checkpoint.py:245) 08:48:47.189 [I] [process=0][thread=async_save] Skipped cross-host ArrayMetadata validation because only one process is found: process_index=0. (1728:array_metadata_store.py:362) 08:48:47.194 [I] Renaming /workspace/openpi/checkpoints/pi05_mecharm_deep_even10_lora/deep_even10_v1/1000.orbax-checkpoint-tmp-0/train_state.orbax-checkpoint-tmp-3 to /workspace/openpi/checkpoints/pi05_mecharm_deep_even10_lora/deep_even10_v1/1000.orbax-checkpoint-tmp-0/train_state (1728:atomicity.py:289) 08:48:47.197 [I] [process=0][thread=async_save] Skipped cross-host ArrayMetadata validation because only one process is found: process_index=0. (1728:array_metadata_store.py:362) 08:48:47.201 [I] Renaming /workspace/openpi/checkpoints/pi05_mecharm_deep_even10_lora/deep_even10_v1/1000.orbax-checkpoint-tmp-0/params.orbax-checkpoint-tmp-2 to /workspace/openpi/checkpoints/pi05_mecharm_deep_even10_lora/deep_even10_v1/1000.orbax-checkpoint-tmp-0/params (1728:atomicity.py:289) 08:48:47.201 [I] Renaming /workspace/openpi/checkpoints/pi05_mecharm_deep_even10_lora/deep_even10_v1/1000.orbax-checkpoint-tmp-0 to /workspace/openpi/checkpoints/pi05_mecharm_deep_even10_lora/deep_even10_v1/1000 (1728:atomicity.py:289) 08:48:47.203 [I] [process=0][thread=async_save] Finished saving checkpoint (finalized tmp dir) to `/workspace/openpi/checkpoints/pi05_mecharm_deep_even10_lora/deep_even10_v1/1000`. (1728:atomicity.py:573) 08:48:47.203 [I] Finished asynchronous save (blocking + background) in 12.91 seconds to /workspace/openpi/checkpoints/pi05_mecharm_deep_even10_lora/deep_even10_v1/1000 (1728:async_checkpointer.py:412) 08:48:47.203 [I] [process=0][thread=async_save] Background save thread done. (1728:async_checkpointer.py:143) 08:48:47.203 [I] [process=0][thread=save_finalize] Done with waiting for background save thread=async_save. (1728:async_checkpointer.py:266) 08:48:47.203 [I] [process=0][thread=save_finalize] No errors found in background save thread=async_save. (1728:async_checkpointer.py:276) 08:48:47.204 [I] [process=0][thread=save_finalize][step=1000] CheckpointManager Save Finalize is syncing with other hosts... (1728:checkpoint_manager.py:2095) 08:48:47.204 [I] [process=0][thread=save_finalize][step=1000] CheckpointManager Save Finalize is done on all hosts. (1728:checkpoint_manager.py:2110) 08:48:58.859 [I] Progress on: 1.00kit/3.00kit rate:7.8s/it remaining:4:19:03 elapsed:2:02:47 postfix:- (1728:tqdm_logging.py:145) 08:49:12.953 [I] Progress on: 1.01kit/3.00kit rate:7.4s/it remaining:4:06:17 elapsed:2:03:01 postfix:- (1728:tqdm_logging.py:145) 08:49:27.067 [I] Progress on: 1.01kit/3.00kit rate:7.2s/it remaining:4:00:03 elapsed:2:03:15 postfix:- (1728:tqdm_logging.py:145) 08:49:41.207 [I] Progress on: 1.01kit/3.00kit rate:7.1s/it remaining:3:57:05 elapsed:2:03:29 postfix:- (1728:tqdm_logging.py:145) 08:49:55.309 [I] Progress on: 1.01kit/3.00kit rate:7.1s/it remaining:3:55:13 elapsed:2:03:44 postfix:- (1728:tqdm_logging.py:145) 08:50:09.448 [I] Progress on: 1.01kit/3.00kit rate:7.1s/it remaining:3:54:28 elapsed:2:03:58 postfix:- (1728:tqdm_logging.py:145) 08:50:23.587 [I] Progress on: 1.02kit/3.00kit rate:7.1s/it remaining:3:54:00 elapsed:2:04:12 postfix:- (1728:tqdm_logging.py:145) 08:50:37.713 [I] Progress on: 1.02kit/3.00kit rate:7.1s/it remaining:3:53:29 elapsed:2:04:26 postfix:- (1728:tqdm_logging.py:145) 08:50:51.849 [I] Progress on: 1.02kit/3.00kit rate:7.1s/it remaining:3:53:14 elapsed:2:04:40 postfix:- (1728:tqdm_logging.py:145) 08:51:05.996 [I] Progress on: 1.02kit/3.00kit rate:7.1s/it remaining:3:53:06 elapsed:2:04:54 postfix:- (1728:tqdm_logging.py:145) 08:51:20.152 [I] Progress on: 1.02kit/3.00kit rate:7.1s/it remaining:3:52:59 elapsed:2:05:08 postfix:- (1728:tqdm_logging.py:145) 08:51:34.270 [I] Progress on: 1.03kit/3.00kit rate:7.1s/it remaining:3:52:30 elapsed:2:05:23 postfix:- (1728:tqdm_logging.py:145) 08:51:48.430 [I] Progress on: 1.03kit/3.00kit rate:7.1s/it remaining:3:52:28 elapsed:2:05:37 postfix:- (1728:tqdm_logging.py:145) 08:52:02.572 [I] Progress on: 1.03kit/3.00kit rate:7.1s/it remaining:3:52:13 elapsed:2:05:51 postfix:- (1728:tqdm_logging.py:145) 08:52:16.720 [I] Progress on: 1.03kit/3.00kit rate:7.1s/it remaining:3:52:01 elapsed:2:06:05 postfix:- (1728:tqdm_logging.py:145) 08:52:30.857 [I] Progress on: 1.03kit/3.00kit rate:7.1s/it remaining:3:51:41 elapsed:2:06:19 postfix:- (1728:tqdm_logging.py:145) 08:52:44.976 [I] Progress on: 1.04kit/3.00kit rate:7.1s/it remaining:3:51:14 elapsed:2:06:33 postfix:- (1728:tqdm_logging.py:145) 08:52:59.108 [I] Progress on: 1.04kit/3.00kit rate:7.1s/it remaining:3:50:59 elapsed:2:06:47 postfix:- (1728:tqdm_logging.py:145) 08:53:13.227 [I] Progress on: 1.04kit/3.00kit rate:7.1s/it remaining:3:50:40 elapsed:2:07:02 postfix:- (1728:tqdm_logging.py:145) 08:53:27.356 [I] Progress on: 1.04kit/3.00kit rate:7.1s/it remaining:3:50:27 elapsed:2:07:16 postfix:- (1728:tqdm_logging.py:145) 08:53:41.485 [I] Progress on: 1.04kit/3.00kit rate:7.1s/it remaining:3:50:16 elapsed:2:07:30 postfix:- (1728:tqdm_logging.py:145) 08:53:55.610 [I] Progress on: 1.05kit/3.00kit rate:7.1s/it remaining:3:50:04 elapsed:2:07:44 postfix:- (1728:tqdm_logging.py:145) 08:54:09.711 [I] Progress on: 1.05kit/3.00kit rate:7.1s/it remaining:3:49:35 elapsed:2:07:58 postfix:- (1728:tqdm_logging.py:145) 08:54:23.835 [I] Progress on: 1.05kit/3.00kit rate:7.1s/it remaining:3:49:24 elapsed:2:08:12 postfix:- (1728:tqdm_logging.py:145) Step 1050: grad_norm=0.0482, loss=0.0069, param_norm=1803.9409 08:54:37.585 [I] Progress on: 1.05kit/3.00kit rate:7.1s/it remaining:3:49:24 elapsed:2:08:26 postfix:- (1728:tqdm_logging.py:145) 08:54:52.105 [I] Progress on: 1.05kit/3.00kit rate:6.8s/it remaining:3:39:28 elapsed:2:08:40 postfix:- (1728:tqdm_logging.py:145) 08:55:06.217 [I] Progress on: 1.06kit/3.00kit rate:6.9s/it remaining:3:44:02 elapsed:2:08:55 postfix:- (1728:tqdm_logging.py:145) 08:55:20.320 [I] Progress on: 1.06kit/3.00kit rate:7.0s/it remaining:3:46:01 elapsed:2:09:09 postfix:- (1728:tqdm_logging.py:145) 08:55:34.467 [I] Progress on: 1.06kit/3.00kit rate:7.0s/it remaining:3:47:14 elapsed:2:09:23 postfix:- (1728:tqdm_logging.py:145) 08:55:48.581 [I] Progress on: 1.06kit/3.00kit rate:7.0s/it remaining:3:47:28 elapsed:2:09:37 postfix:- (1728:tqdm_logging.py:145) 08:56:02.701 [I] Progress on: 1.06kit/3.00kit rate:7.1s/it remaining:3:47:30 elapsed:2:09:51 postfix:- (1728:tqdm_logging.py:145) 08:56:16.818 [I] Progress on: 1.07kit/3.00kit rate:7.1s/it remaining:3:47:24 elapsed:2:10:05 postfix:- (1728:tqdm_logging.py:145) 08:56:30.928 [I] Progress on: 1.07kit/3.00kit rate:7.1s/it remaining:3:47:08 elapsed:2:10:19 postfix:- (1728:tqdm_logging.py:145) 08:56:45.045 [I] Progress on: 1.07kit/3.00kit rate:7.1s/it remaining:3:46:58 elapsed:2:10:33 postfix:- (1728:tqdm_logging.py:145) 08:56:59.173 [I] Progress on: 1.07kit/3.00kit rate:7.1s/it remaining:3:46:51 elapsed:2:10:47 postfix:- (1728:tqdm_logging.py:145) 08:57:13.280 [I] Progress on: 1.07kit/3.00kit rate:7.1s/it remaining:3:46:31 elapsed:2:11:02 postfix:- (1728:tqdm_logging.py:145) 08:57:27.391 [I] Progress on: 1.08kit/3.00kit rate:7.1s/it remaining:3:46:16 elapsed:2:11:16 postfix:- (1728:tqdm_logging.py:145) 08:57:41.513 [I] Progress on: 1.08kit/3.00kit rate:7.1s/it remaining:3:46:03 elapsed:2:11:30 postfix:- (1728:tqdm_logging.py:145) 08:57:55.645 [I] Progress on: 1.08kit/3.00kit rate:7.1s/it remaining:3:45:59 elapsed:2:11:44 postfix:- (1728:tqdm_logging.py:145) 08:58:09.756 [I] Progress on: 1.08kit/3.00kit rate:7.1s/it remaining:3:45:38 elapsed:2:11:58 postfix:- (1728:tqdm_logging.py:145) 08:58:23.870 [I] Progress on: 1.08kit/3.00kit rate:7.1s/it remaining:3:45:21 elapsed:2:12:12 postfix:- (1728:tqdm_logging.py:145) 08:58:37.991 [I] Progress on: 1.09kit/3.00kit rate:7.1s/it remaining:3:45:08 elapsed:2:12:26 postfix:- (1728:tqdm_logging.py:145) 08:58:52.098 [I] Progress on: 1.09kit/3.00kit rate:7.1s/it remaining:3:44:48 elapsed:2:12:40 postfix:- (1728:tqdm_logging.py:145) 08:59:06.226 [I] Progress on: 1.09kit/3.00kit rate:7.1s/it remaining:3:44:41 elapsed:2:12:55 postfix:- (1728:tqdm_logging.py:145) 08:59:20.349 [I] Progress on: 1.09kit/3.00kit rate:7.1s/it remaining:3:44:32 elapsed:2:13:09 postfix:- (1728:tqdm_logging.py:145) 08:59:34.476 [I] Progress on: 1.09kit/3.00kit rate:7.1s/it remaining:3:44:19 elapsed:2:13:23 postfix:- (1728:tqdm_logging.py:145) 08:59:48.606 [I] Progress on: 1.10kit/3.00kit rate:7.1s/it remaining:3:44:10 elapsed:2:13:37 postfix:- (1728:tqdm_logging.py:145) 09:00:02.706 [I] Progress on: 1.10kit/3.00kit rate:7.1s/it remaining:3:43:44 elapsed:2:13:51 postfix:- (1728:tqdm_logging.py:145) 09:00:16.837 [I] Progress on: 1.10kit/3.00kit rate:7.1s/it remaining:3:43:35 elapsed:2:14:05 postfix:- (1728:tqdm_logging.py:145) Step 1100: grad_norm=0.0518, loss=0.0075, param_norm=1803.9497 09:00:30.606 [I] Progress on: 1.10kit/3.00kit rate:7.1s/it remaining:3:43:35 elapsed:2:14:19 postfix:- (1728:tqdm_logging.py:145) 09:00:45.154 [I] Progress on: 1.10kit/3.00kit rate:6.8s/it remaining:3:33:59 elapsed:2:14:33 postfix:- (1728:tqdm_logging.py:145) 09:00:59.266 [I] Progress on: 1.11kit/3.00kit rate:6.9s/it remaining:3:38:18 elapsed:2:14:48 postfix:- (1728:tqdm_logging.py:145) 09:01:13.390 [I] Progress on: 1.11kit/3.00kit rate:7.0s/it remaining:3:40:25 elapsed:2:15:02 postfix:- (1728:tqdm_logging.py:145) 09:01:28.056 [I] Progress on: 1.11kit/3.00kit rate:7.2s/it remaining:3:46:29 elapsed:2:15:16 postfix:- (1728:tqdm_logging.py:145) 09:01:41.620 [I] Progress on: 1.11kit/3.00kit rate:7.0s/it remaining:3:40:27 elapsed:2:15:30 postfix:- (1728:tqdm_logging.py:145) 09:01:55.736 [I] Progress on: 1.11kit/3.00kit rate:7.0s/it remaining:3:41:04 elapsed:2:15:44 postfix:- (1728:tqdm_logging.py:145) 09:02:09.880 [I] Progress on: 1.12kit/3.00kit rate:7.1s/it remaining:3:41:28 elapsed:2:15:58 postfix:- (1728:tqdm_logging.py:145) 09:02:24.002 [I] Progress on: 1.12kit/3.00kit rate:7.1s/it remaining:3:41:20 elapsed:2:16:12 postfix:- (1728:tqdm_logging.py:145) 09:02:38.145 [I] Progress on: 1.12kit/3.00kit rate:7.1s/it remaining:3:41:20 elapsed:2:16:26 postfix:- (1728:tqdm_logging.py:145) 09:02:52.254 [I] Progress on: 1.12kit/3.00kit rate:7.1s/it remaining:3:40:56 elapsed:2:16:41 postfix:- (1728:tqdm_logging.py:145) 09:03:06.378 [I] Progress on: 1.12kit/3.00kit rate:7.1s/it remaining:3:40:45 elapsed:2:16:55 postfix:- (1728:tqdm_logging.py:145) 09:03:20.480 [I] Progress on: 1.13kit/3.00kit rate:7.1s/it remaining:3:40:20 elapsed:2:17:09 postfix:- (1728:tqdm_logging.py:145) 09:03:34.609 [I] Progress on: 1.13kit/3.00kit rate:7.1s/it remaining:3:40:13 elapsed:2:17:23 postfix:- (1728:tqdm_logging.py:145) 09:03:48.741 [I] Progress on: 1.13kit/3.00kit rate:7.1s/it remaining:3:40:05 elapsed:2:17:37 postfix:- (1728:tqdm_logging.py:145) 09:04:02.869 [I] Progress on: 1.13kit/3.00kit rate:7.1s/it remaining:3:39:52 elapsed:2:17:51 postfix:- (1728:tqdm_logging.py:145) 09:04:16.987 [I] Progress on: 1.13kit/3.00kit rate:7.1s/it remaining:3:39:36 elapsed:2:18:05 postfix:- (1728:tqdm_logging.py:145) 09:04:31.104 [I] Progress on: 1.14kit/3.00kit rate:7.1s/it remaining:3:39:17 elapsed:2:18:19 postfix:- (1728:tqdm_logging.py:145) 09:04:45.220 [I] Progress on: 1.14kit/3.00kit rate:7.1s/it remaining:3:39:01 elapsed:2:18:34 postfix:- (1728:tqdm_logging.py:145) 09:04:59.348 [I] Progress on: 1.14kit/3.00kit rate:7.1s/it remaining:3:38:52 elapsed:2:18:48 postfix:- (1728:tqdm_logging.py:145) 09:05:13.467 [I] Progress on: 1.14kit/3.00kit rate:7.1s/it remaining:3:38:38 elapsed:2:19:02 postfix:- (1728:tqdm_logging.py:145) 09:05:27.597 [I] Progress on: 1.14kit/3.00kit rate:7.1s/it remaining:3:38:29 elapsed:2:19:16 postfix:- (1728:tqdm_logging.py:145) 09:05:41.740 [I] Progress on: 1.15kit/3.00kit rate:7.1s/it remaining:3:38:23 elapsed:2:19:30 postfix:- (1728:tqdm_logging.py:145) 09:05:55.862 [I] Progress on: 1.15kit/3.00kit rate:7.1s/it remaining:3:38:02 elapsed:2:19:44 postfix:- (1728:tqdm_logging.py:145) 09:06:09.970 [I] Progress on: 1.15kit/3.00kit rate:7.1s/it remaining:3:37:38 elapsed:2:19:58 postfix:- (1728:tqdm_logging.py:145) Step 1150: grad_norm=0.0504, loss=0.0066, param_norm=1803.9578 09:06:23.742 [I] Progress on: 1.15kit/3.00kit rate:7.1s/it remaining:3:37:38 elapsed:2:20:12 postfix:- (1728:tqdm_logging.py:145) 09:06:38.291 [I] Progress on: 1.15kit/3.00kit rate:6.8s/it remaining:3:28:30 elapsed:2:20:27 postfix:- (1728:tqdm_logging.py:145) 09:06:52.416 [I] Progress on: 1.16kit/3.00kit rate:6.9s/it remaining:3:32:45 elapsed:2:20:41 postfix:- (1728:tqdm_logging.py:145) 09:07:06.525 [I] Progress on: 1.16kit/3.00kit rate:7.0s/it remaining:3:34:35 elapsed:2:20:55 postfix:- (1728:tqdm_logging.py:145) 09:07:20.659 [I] Progress on: 1.16kit/3.00kit rate:7.0s/it remaining:3:35:32 elapsed:2:21:09 postfix:- (1728:tqdm_logging.py:145) 09:07:34.792 [I] Progress on: 1.16kit/3.00kit rate:7.0s/it remaining:3:35:53 elapsed:2:21:23 postfix:- (1728:tqdm_logging.py:145) 09:07:48.901 [I] Progress on: 1.16kit/3.00kit rate:7.1s/it remaining:3:35:46 elapsed:2:21:37 postfix:- (1728:tqdm_logging.py:145) 09:08:03.035 [I] Progress on: 1.17kit/3.00kit rate:7.1s/it remaining:3:35:45 elapsed:2:21:51 postfix:- (1728:tqdm_logging.py:145) 09:08:17.160 [I] Progress on: 1.17kit/3.00kit rate:7.1s/it remaining:3:35:33 elapsed:2:22:05 postfix:- (1728:tqdm_logging.py:145) 09:08:31.289 [I] Progress on: 1.17kit/3.00kit rate:7.1s/it remaining:3:35:24 elapsed:2:22:20 postfix:- (1728:tqdm_logging.py:145) 09:08:45.404 [I] Progress on: 1.17kit/3.00kit rate:7.1s/it remaining:3:35:04 elapsed:2:22:34 postfix:- (1728:tqdm_logging.py:145) 09:08:59.527 [I] Progress on: 1.17kit/3.00kit rate:7.1s/it remaining:3:34:52 elapsed:2:22:48 postfix:- (1728:tqdm_logging.py:145) 09:09:13.629 [I] Progress on: 1.18kit/3.00kit rate:7.1s/it remaining:3:34:30 elapsed:2:23:02 postfix:- (1728:tqdm_logging.py:145) 09:09:27.742 [I] Progress on: 1.18kit/3.00kit rate:7.1s/it remaining:3:34:14 elapsed:2:23:16 postfix:- (1728:tqdm_logging.py:145) 09:09:41.859 [I] Progress on: 1.18kit/3.00kit rate:7.1s/it remaining:3:34:02 elapsed:2:23:30 postfix:- (1728:tqdm_logging.py:145) 09:09:55.985 [I] Progress on: 1.18kit/3.00kit rate:7.1s/it remaining:3:33:55 elapsed:2:23:44 postfix:- (1728:tqdm_logging.py:145) 09:10:10.102 [I] Progress on: 1.18kit/3.00kit rate:7.1s/it remaining:3:33:40 elapsed:2:23:58 postfix:- (1728:tqdm_logging.py:145) 09:10:24.194 [I] Progress on: 1.19kit/3.00kit rate:7.1s/it remaining:3:33:13 elapsed:2:24:12 postfix:- (1728:tqdm_logging.py:145) 09:10:38.323 [I] Progress on: 1.19kit/3.00kit rate:7.1s/it remaining:3:33:06 elapsed:2:24:27 postfix:- (1728:tqdm_logging.py:145) 09:10:52.450 [I] Progress on: 1.19kit/3.00kit rate:7.1s/it remaining:3:32:59 elapsed:2:24:41 postfix:- (1728:tqdm_logging.py:145) 09:11:06.556 [I] Progress on: 1.19kit/3.00kit rate:7.1s/it remaining:3:32:37 elapsed:2:24:55 postfix:- (1728:tqdm_logging.py:145) 09:11:20.665 [I] Progress on: 1.19kit/3.00kit rate:7.1s/it remaining:3:32:22 elapsed:2:25:09 postfix:- (1728:tqdm_logging.py:145) 09:11:34.779 [I] Progress on: 1.20kit/3.00kit rate:7.1s/it remaining:3:32:08 elapsed:2:25:23 postfix:- (1728:tqdm_logging.py:145) 09:11:48.905 [I] Progress on: 1.20kit/3.00kit rate:7.1s/it remaining:3:31:59 elapsed:2:25:37 postfix:- (1728:tqdm_logging.py:145) 09:12:03.026 [I] Progress on: 1.20kit/3.00kit rate:7.1s/it remaining:3:31:48 elapsed:2:25:51 postfix:- (1728:tqdm_logging.py:145) Step 1200: grad_norm=0.0484, loss=0.0066, param_norm=1803.9659 09:12:16.769 [I] Progress on: 1.20kit/3.00kit rate:7.1s/it remaining:3:31:48 elapsed:2:26:05 postfix:- (1728:tqdm_logging.py:145) 09:12:31.292 [I] Progress on: 1.20kit/3.00kit rate:6.8s/it remaining:3:22:32 elapsed:2:26:20 postfix:- (1728:tqdm_logging.py:145) 09:12:45.410 [I] Progress on: 1.21kit/3.00kit rate:6.9s/it remaining:3:26:47 elapsed:2:26:34 postfix:- (1728:tqdm_logging.py:145) 09:12:59.525 [I] Progress on: 1.21kit/3.00kit rate:7.0s/it remaining:3:28:43 elapsed:2:26:48 postfix:- (1728:tqdm_logging.py:145) 09:13:13.643 [I] Progress on: 1.21kit/3.00kit rate:7.0s/it remaining:3:29:35 elapsed:2:27:02 postfix:- (1728:tqdm_logging.py:145) 09:13:27.764 [I] Progress on: 1.21kit/3.00kit rate:7.0s/it remaining:3:29:53 elapsed:2:27:16 postfix:- (1728:tqdm_logging.py:145) 09:13:41.896 [I] Progress on: 1.21kit/3.00kit rate:7.1s/it remaining:3:29:59 elapsed:2:27:30 postfix:- (1728:tqdm_logging.py:145) 09:13:56.011 [I] Progress on: 1.22kit/3.00kit rate:7.1s/it remaining:3:29:48 elapsed:2:27:44 postfix:- (1728:tqdm_logging.py:145) 09:14:10.128 [I] Progress on: 1.22kit/3.00kit rate:7.1s/it remaining:3:29:35 elapsed:2:27:58 postfix:- (1728:tqdm_logging.py:145) 09:14:24.257 [I] Progress on: 1.22kit/3.00kit rate:7.1s/it remaining:3:29:26 elapsed:2:28:13 postfix:- (1728:tqdm_logging.py:145) 09:14:38.378 [I] Progress on: 1.22kit/3.00kit rate:7.1s/it remaining:3:29:12 elapsed:2:28:27 postfix:- (1728:tqdm_logging.py:145) 09:14:52.492 [I] Progress on: 1.22kit/3.00kit rate:7.1s/it remaining:3:28:56 elapsed:2:28:41 postfix:- (1728:tqdm_logging.py:145) 09:15:06.614 [I] Progress on: 1.23kit/3.00kit rate:7.1s/it remaining:3:28:43 elapsed:2:28:55 postfix:- (1728:tqdm_logging.py:145) 09:15:20.756 [I] Progress on: 1.23kit/3.00kit rate:7.1s/it remaining:3:28:36 elapsed:2:29:09 postfix:- (1728:tqdm_logging.py:145) 09:15:34.884 [I] Progress on: 1.23kit/3.00kit rate:7.1s/it remaining:3:28:23 elapsed:2:29:23 postfix:- (1728:tqdm_logging.py:145) 09:15:48.985 [I] Progress on: 1.23kit/3.00kit rate:7.1s/it remaining:3:27:57 elapsed:2:29:37 postfix:- (1728:tqdm_logging.py:145) 09:16:03.095 [I] Progress on: 1.23kit/3.00kit rate:7.1s/it remaining:3:27:40 elapsed:2:29:51 postfix:- (1728:tqdm_logging.py:145) 09:16:17.217 [I] Progress on: 1.24kit/3.00kit rate:7.1s/it remaining:3:27:31 elapsed:2:30:06 postfix:- (1728:tqdm_logging.py:145) 09:16:31.353 [I] Progress on: 1.24kit/3.00kit rate:7.1s/it remaining:3:27:23 elapsed:2:30:20 postfix:- (1728:tqdm_logging.py:145) 09:16:45.482 [I] Progress on: 1.24kit/3.00kit rate:7.1s/it remaining:3:27:13 elapsed:2:30:34 postfix:- (1728:tqdm_logging.py:145) 09:16:59.591 [I] Progress on: 1.24kit/3.00kit rate:7.1s/it remaining:3:26:49 elapsed:2:30:48 postfix:- (1728:tqdm_logging.py:145) 09:17:13.711 [I] Progress on: 1.24kit/3.00kit rate:7.1s/it remaining:3:26:35 elapsed:2:31:02 postfix:- (1728:tqdm_logging.py:145) 09:17:27.834 [I] Progress on: 1.25kit/3.00kit rate:7.1s/it remaining:3:26:25 elapsed:2:31:16 postfix:- (1728:tqdm_logging.py:145) 09:17:41.960 [I] Progress on: 1.25kit/3.00kit rate:7.1s/it remaining:3:26:13 elapsed:2:31:30 postfix:- (1728:tqdm_logging.py:145) 09:17:56.072 [I] Progress on: 1.25kit/3.00kit rate:7.1s/it remaining:3:25:54 elapsed:2:31:44 postfix:- (1728:tqdm_logging.py:145) Step 1250: grad_norm=0.0472, loss=0.0062, param_norm=1803.9744 09:18:09.830 [I] Progress on: 1.25kit/3.00kit rate:7.1s/it remaining:3:25:54 elapsed:2:31:58 postfix:- (1728:tqdm_logging.py:145) 09:18:24.384 [I] Progress on: 1.25kit/3.00kit rate:6.8s/it remaining:3:17:04 elapsed:2:32:13 postfix:- (1728:tqdm_logging.py:145) 09:18:38.488 [I] Progress on: 1.26kit/3.00kit rate:6.9s/it remaining:3:21:00 elapsed:2:32:27 postfix:- (1728:tqdm_logging.py:145) 09:18:52.604 [I] Progress on: 1.26kit/3.00kit rate:7.0s/it remaining:3:22:52 elapsed:2:32:41 postfix:- (1728:tqdm_logging.py:145) 09:19:06.719 [I] Progress on: 1.26kit/3.00kit rate:7.0s/it remaining:3:23:40 elapsed:2:32:55 postfix:- (1728:tqdm_logging.py:145) 09:19:20.835 [I] Progress on: 1.26kit/3.00kit rate:7.0s/it remaining:3:23:56 elapsed:2:33:09 postfix:- (1728:tqdm_logging.py:145) 09:19:34.966 [I] Progress on: 1.26kit/3.00kit rate:7.1s/it remaining:3:24:03 elapsed:2:33:23 postfix:- (1728:tqdm_logging.py:145) 09:19:49.078 [I] Progress on: 1.27kit/3.00kit rate:7.1s/it remaining:3:23:51 elapsed:2:33:37 postfix:- (1728:tqdm_logging.py:145) 09:20:03.205 [I] Progress on: 1.27kit/3.00kit rate:7.1s/it remaining:3:23:46 elapsed:2:33:51 postfix:- (1728:tqdm_logging.py:145) 09:20:17.312 [I] Progress on: 1.27kit/3.00kit rate:7.1s/it remaining:3:23:28 elapsed:2:34:06 postfix:- (1728:tqdm_logging.py:145) 09:20:31.424 [I] Progress on: 1.27kit/3.00kit rate:7.1s/it remaining:3:23:13 elapsed:2:34:20 postfix:- (1728:tqdm_logging.py:145) 09:20:45.559 [I] Progress on: 1.27kit/3.00kit rate:7.1s/it remaining:3:23:07 elapsed:2:34:34 postfix:- (1728:tqdm_logging.py:145) 09:20:59.674 [I] Progress on: 1.28kit/3.00kit rate:7.1s/it remaining:3:22:51 elapsed:2:34:48 postfix:- (1728:tqdm_logging.py:145) 09:21:13.776 [I] Progress on: 1.28kit/3.00kit rate:7.1s/it remaining:3:22:30 elapsed:2:35:02 postfix:- (1728:tqdm_logging.py:145) 09:21:27.888 [I] Progress on: 1.28kit/3.00kit rate:7.1s/it remaining:3:22:14 elapsed:2:35:16 postfix:- (1728:tqdm_logging.py:145) 09:21:42.011 [I] Progress on: 1.28kit/3.00kit rate:7.1s/it remaining:3:22:04 elapsed:2:35:30 postfix:- (1728:tqdm_logging.py:145) 09:21:56.117 [I] Progress on: 1.28kit/3.00kit rate:7.1s/it remaining:3:21:48 elapsed:2:35:44 postfix:- (1728:tqdm_logging.py:145) 09:22:10.235 [I] Progress on: 1.29kit/3.00kit rate:7.1s/it remaining:3:21:37 elapsed:2:35:59 postfix:- (1728:tqdm_logging.py:145) 09:22:24.338 [I] Progress on: 1.29kit/3.00kit rate:7.1s/it remaining:3:21:17 elapsed:2:36:13 postfix:- (1728:tqdm_logging.py:145) 09:22:38.458 [I] Progress on: 1.29kit/3.00kit rate:7.1s/it remaining:3:21:06 elapsed:2:36:27 postfix:- (1728:tqdm_logging.py:145) 09:22:52.585 [I] Progress on: 1.29kit/3.00kit rate:7.1s/it remaining:3:20:59 elapsed:2:36:41 postfix:- (1728:tqdm_logging.py:145) 09:23:06.694 [I] Progress on: 1.29kit/3.00kit rate:7.1s/it remaining:3:20:41 elapsed:2:36:55 postfix:- (1728:tqdm_logging.py:145) 09:23:20.806 [I] Progress on: 1.30kit/3.00kit rate:7.1s/it remaining:3:20:24 elapsed:2:37:09 postfix:- (1728:tqdm_logging.py:145) 09:23:34.924 [I] Progress on: 1.30kit/3.00kit rate:7.1s/it remaining:3:20:11 elapsed:2:37:23 postfix:- (1728:tqdm_logging.py:145) 09:23:49.041 [I] Progress on: 1.30kit/3.00kit rate:7.1s/it remaining:3:19:58 elapsed:2:37:37 postfix:- (1728:tqdm_logging.py:145) Step 1300: grad_norm=0.0461, loss=0.0064, param_norm=1803.9818 09:24:02.800 [I] Progress on: 1.30kit/3.00kit rate:7.1s/it remaining:3:19:58 elapsed:2:37:51 postfix:- (1728:tqdm_logging.py:145) 09:24:17.366 [I] Progress on: 1.30kit/3.00kit rate:6.8s/it remaining:3:11:32 elapsed:2:38:06 postfix:- (1728:tqdm_logging.py:145) 09:24:31.466 [I] Progress on: 1.31kit/3.00kit rate:6.9s/it remaining:3:15:14 elapsed:2:38:20 postfix:- (1728:tqdm_logging.py:145) 09:24:45.588 [I] Progress on: 1.31kit/3.00kit rate:7.0s/it remaining:3:17:07 elapsed:2:38:34 postfix:- (1728:tqdm_logging.py:145) 09:24:59.706 [I] Progress on: 1.31kit/3.00kit rate:7.0s/it remaining:3:17:53 elapsed:2:38:48 postfix:- (1728:tqdm_logging.py:145) 09:25:13.815 [I] Progress on: 1.31kit/3.00kit rate:7.0s/it remaining:3:18:04 elapsed:2:39:02 postfix:- (1728:tqdm_logging.py:145) 09:25:27.918 [I] Progress on: 1.31kit/3.00kit rate:7.0s/it remaining:3:17:58 elapsed:2:39:16 postfix:- (1728:tqdm_logging.py:145) 09:25:42.053 [I] Progress on: 1.32kit/3.00kit rate:7.1s/it remaining:3:18:02 elapsed:2:39:30 postfix:- (1728:tqdm_logging.py:145) 09:25:56.158 [I] Progress on: 1.32kit/3.00kit rate:7.1s/it remaining:3:17:43 elapsed:2:39:44 postfix:- (1728:tqdm_logging.py:145) 09:26:10.305 [I] Progress on: 1.32kit/3.00kit rate:7.1s/it remaining:3:17:46 elapsed:2:39:59 postfix:- (1728:tqdm_logging.py:145) 09:26:24.412 [I] Progress on: 1.32kit/3.00kit rate:7.1s/it remaining:3:17:24 elapsed:2:40:13 postfix:- (1728:tqdm_logging.py:145) 09:26:38.522 [I] Progress on: 1.32kit/3.00kit rate:7.1s/it remaining:3:17:08 elapsed:2:40:27 postfix:- (1728:tqdm_logging.py:145) 09:26:52.649 [I] Progress on: 1.33kit/3.00kit rate:7.1s/it remaining:3:16:59 elapsed:2:40:41 postfix:- (1728:tqdm_logging.py:145) 09:27:06.762 [I] Progress on: 1.33kit/3.00kit rate:7.1s/it remaining:3:16:42 elapsed:2:40:55 postfix:- (1728:tqdm_logging.py:145) 09:27:20.861 [I] Progress on: 1.33kit/3.00kit rate:7.1s/it remaining:3:16:20 elapsed:2:41:09 postfix:- (1728:tqdm_logging.py:145) 09:27:35.545 [I] Progress on: 1.33kit/3.00kit rate:7.2s/it remaining:3:20:50 elapsed:2:41:24 postfix:- (1728:tqdm_logging.py:145) 09:27:49.134 [I] Progress on: 1.33kit/3.00kit rate:7.0s/it remaining:3:15:17 elapsed:2:41:37 postfix:- (1728:tqdm_logging.py:145) 09:28:03.241 [I] Progress on: 1.34kit/3.00kit rate:7.0s/it remaining:3:15:26 elapsed:2:41:52 postfix:- (1728:tqdm_logging.py:145) 09:28:17.356 [I] Progress on: 1.34kit/3.00kit rate:7.1s/it remaining:3:15:25 elapsed:2:42:06 postfix:- (1728:tqdm_logging.py:145) 09:28:31.466 [I] Progress on: 1.34kit/3.00kit rate:7.1s/it remaining:3:15:17 elapsed:2:42:20 postfix:- (1728:tqdm_logging.py:145) 09:28:45.588 [I] Progress on: 1.34kit/3.00kit rate:7.1s/it remaining:3:15:10 elapsed:2:42:34 postfix:- (1728:tqdm_logging.py:145) 09:28:59.699 [I] Progress on: 1.34kit/3.00kit rate:7.1s/it remaining:3:14:55 elapsed:2:42:48 postfix:- (1728:tqdm_logging.py:145) 09:29:13.825 [I] Progress on: 1.35kit/3.00kit rate:7.1s/it remaining:3:14:47 elapsed:2:43:02 postfix:- (1728:tqdm_logging.py:145) 09:29:27.934 [I] Progress on: 1.35kit/3.00kit rate:7.1s/it remaining:3:14:29 elapsed:2:43:16 postfix:- (1728:tqdm_logging.py:145) 09:29:42.031 [I] Progress on: 1.35kit/3.00kit rate:7.1s/it remaining:3:14:08 elapsed:2:43:30 postfix:- (1728:tqdm_logging.py:145) Step 1350: grad_norm=0.0502, loss=0.0063, param_norm=1803.9896 09:29:55.766 [I] Progress on: 1.35kit/3.00kit rate:7.1s/it remaining:3:14:08 elapsed:2:43:44 postfix:- (1728:tqdm_logging.py:145) 09:30:10.284 [I] Progress on: 1.35kit/3.00kit rate:6.8s/it remaining:3:05:38 elapsed:2:43:59 postfix:- (1728:tqdm_logging.py:145) 09:30:24.398 [I] Progress on: 1.36kit/3.00kit rate:6.9s/it remaining:3:09:29 elapsed:2:44:13 postfix:- (1728:tqdm_logging.py:145) 09:30:38.503 [I] Progress on: 1.36kit/3.00kit rate:7.0s/it remaining:3:11:09 elapsed:2:44:27 postfix:- (1728:tqdm_logging.py:145) 09:30:52.618 [I] Progress on: 1.36kit/3.00kit rate:7.0s/it remaining:3:11:55 elapsed:2:44:41 postfix:- (1728:tqdm_logging.py:145) 09:31:06.725 [I] Progress on: 1.36kit/3.00kit rate:7.0s/it remaining:3:12:09 elapsed:2:44:55 postfix:- (1728:tqdm_logging.py:145) 09:31:20.837 [I] Progress on: 1.36kit/3.00kit rate:7.0s/it remaining:3:12:08 elapsed:2:45:09 postfix:- (1728:tqdm_logging.py:145) 09:31:34.962 [I] Progress on: 1.37kit/3.00kit rate:7.1s/it remaining:3:12:07 elapsed:2:45:23 postfix:- (1728:tqdm_logging.py:145) 09:31:49.068 [I] Progress on: 1.37kit/3.00kit rate:7.1s/it remaining:3:11:53 elapsed:2:45:37 postfix:- (1728:tqdm_logging.py:145) 09:32:03.180 [I] Progress on: 1.37kit/3.00kit rate:7.1s/it remaining:3:11:41 elapsed:2:45:51 postfix:- (1728:tqdm_logging.py:145) 09:32:17.280 [I] Progress on: 1.37kit/3.00kit rate:7.1s/it remaining:3:11:21 elapsed:2:46:06 postfix:- (1728:tqdm_logging.py:145) 09:32:31.396 [I] Progress on: 1.37kit/3.00kit rate:7.1s/it remaining:3:11:11 elapsed:2:46:20 postfix:- (1728:tqdm_logging.py:145) 09:32:45.499 [I] Progress on: 1.38kit/3.00kit rate:7.1s/it remaining:3:10:54 elapsed:2:46:34 postfix:- (1728:tqdm_logging.py:145) 09:32:59.610 [I] Progress on: 1.38kit/3.00kit rate:7.1s/it remaining:3:10:43 elapsed:2:46:48 postfix:- (1728:tqdm_logging.py:145) 09:33:13.721 [I] Progress on: 1.38kit/3.00kit rate:7.1s/it remaining:3:10:27 elapsed:2:47:02 postfix:- (1728:tqdm_logging.py:145) 09:33:27.840 [I] Progress on: 1.38kit/3.00kit rate:7.1s/it remaining:3:10:17 elapsed:2:47:16 postfix:- (1728:tqdm_logging.py:145) 09:33:41.953 [I] Progress on: 1.38kit/3.00kit rate:7.1s/it remaining:3:10:03 elapsed:2:47:30 postfix:- (1728:tqdm_logging.py:145) 09:33:56.070 [I] Progress on: 1.39kit/3.00kit rate:7.1s/it remaining:3:09:50 elapsed:2:47:44 postfix:- (1728:tqdm_logging.py:145) 09:34:10.177 [I] Progress on: 1.39kit/3.00kit rate:7.1s/it remaining:3:09:33 elapsed:2:47:58 postfix:- (1728:tqdm_logging.py:145) 09:34:24.319 [I] Progress on: 1.39kit/3.00kit rate:7.1s/it remaining:3:09:32 elapsed:2:48:13 postfix:- (1728:tqdm_logging.py:145) 09:34:38.429 [I] Progress on: 1.39kit/3.00kit rate:7.1s/it remaining:3:09:13 elapsed:2:48:27 postfix:- (1728:tqdm_logging.py:145) 09:34:52.538 [I] Progress on: 1.39kit/3.00kit rate:7.1s/it remaining:3:08:55 elapsed:2:48:41 postfix:- (1728:tqdm_logging.py:145) 09:35:06.668 [I] Progress on: 1.40kit/3.00kit rate:7.1s/it remaining:3:08:48 elapsed:2:48:55 postfix:- (1728:tqdm_logging.py:145) 09:35:20.786 [I] Progress on: 1.40kit/3.00kit rate:7.1s/it remaining:3:08:31 elapsed:2:49:09 postfix:- (1728:tqdm_logging.py:145) 09:35:34.900 [I] Progress on: 1.40kit/3.00kit rate:7.1s/it remaining:3:08:13 elapsed:2:49:23 postfix:- (1728:tqdm_logging.py:145) Step 1400: grad_norm=0.0499, loss=0.0061, param_norm=1803.9980 09:35:48.655 [I] Progress on: 1.40kit/3.00kit rate:7.1s/it remaining:3:08:13 elapsed:2:49:37 postfix:- (1728:tqdm_logging.py:145) 09:36:03.201 [I] Progress on: 1.40kit/3.00kit rate:6.8s/it remaining:3:00:03 elapsed:2:49:51 postfix:- (1728:tqdm_logging.py:145) 09:36:17.367 [I] Progress on: 1.41kit/3.00kit rate:6.9s/it remaining:3:04:10 elapsed:2:50:06 postfix:- (1728:tqdm_logging.py:145) 09:36:31.423 [I] Progress on: 1.41kit/3.00kit rate:7.0s/it remaining:3:05:13 elapsed:2:50:20 postfix:- (1728:tqdm_logging.py:145) 09:36:45.534 [I] Progress on: 1.41kit/3.00kit rate:7.0s/it remaining:3:05:59 elapsed:2:50:34 postfix:- (1728:tqdm_logging.py:145) 09:36:59.651 [I] Progress on: 1.41kit/3.00kit rate:7.0s/it remaining:3:06:19 elapsed:2:50:48 postfix:- (1728:tqdm_logging.py:145) 09:37:13.752 [I] Progress on: 1.41kit/3.00kit rate:7.0s/it remaining:3:06:14 elapsed:2:51:02 postfix:- (1728:tqdm_logging.py:145) 09:37:27.856 [I] Progress on: 1.42kit/3.00kit rate:7.0s/it remaining:3:06:06 elapsed:2:51:16 postfix:- (1728:tqdm_logging.py:145) 09:37:41.974 [I] Progress on: 1.42kit/3.00kit rate:7.1s/it remaining:3:05:57 elapsed:2:51:30 postfix:- (1728:tqdm_logging.py:145) 09:37:56.078 [I] Progress on: 1.42kit/3.00kit rate:7.0s/it remaining:3:05:38 elapsed:2:51:44 postfix:- (1728:tqdm_logging.py:145) 09:38:10.212 [I] Progress on: 1.42kit/3.00kit rate:7.1s/it remaining:3:05:37 elapsed:2:51:58 postfix:- (1728:tqdm_logging.py:145) 09:38:24.310 [I] Progress on: 1.42kit/3.00kit rate:7.1s/it remaining:3:05:17 elapsed:2:52:13 postfix:- (1728:tqdm_logging.py:145) 09:38:38.452 [I] Progress on: 1.43kit/3.00kit rate:7.1s/it remaining:3:05:10 elapsed:2:52:27 postfix:- (1728:tqdm_logging.py:145) 09:38:52.550 [I] Progress on: 1.43kit/3.00kit rate:7.1s/it remaining:3:04:45 elapsed:2:52:41 postfix:- (1728:tqdm_logging.py:145) 09:39:06.661 [I] Progress on: 1.43kit/3.00kit rate:7.1s/it remaining:3:04:34 elapsed:2:52:55 postfix:- (1728:tqdm_logging.py:145) 09:39:20.780 [I] Progress on: 1.43kit/3.00kit rate:7.1s/it remaining:3:04:26 elapsed:2:53:09 postfix:- (1728:tqdm_logging.py:145) 09:39:34.895 [I] Progress on: 1.43kit/3.00kit rate:7.1s/it remaining:3:04:13 elapsed:2:53:23 postfix:- (1728:tqdm_logging.py:145) 09:39:49.032 [I] Progress on: 1.44kit/3.00kit rate:7.1s/it remaining:3:04:08 elapsed:2:53:37 postfix:- (1728:tqdm_logging.py:145) 09:40:03.134 [I] Progress on: 1.44kit/3.00kit rate:7.1s/it remaining:3:03:43 elapsed:2:53:51 postfix:- (1728:tqdm_logging.py:145) 09:40:17.258 [I] Progress on: 1.44kit/3.00kit rate:7.1s/it remaining:3:03:33 elapsed:2:54:06 postfix:- (1728:tqdm_logging.py:145) 09:40:31.367 [I] Progress on: 1.44kit/3.00kit rate:7.1s/it remaining:3:03:14 elapsed:2:54:20 postfix:- (1728:tqdm_logging.py:145) 09:40:45.479 [I] Progress on: 1.44kit/3.00kit rate:7.1s/it remaining:3:03:01 elapsed:2:54:34 postfix:- (1728:tqdm_logging.py:145) 09:40:59.599 [I] Progress on: 1.45kit/3.00kit rate:7.1s/it remaining:3:02:48 elapsed:2:54:48 postfix:- (1728:tqdm_logging.py:145) 09:41:13.725 [I] Progress on: 1.45kit/3.00kit rate:7.1s/it remaining:3:02:38 elapsed:2:55:02 postfix:- (1728:tqdm_logging.py:145) 09:41:27.838 [I] Progress on: 1.45kit/3.00kit rate:7.1s/it remaining:3:02:22 elapsed:2:55:16 postfix:- (1728:tqdm_logging.py:145) Step 1450: grad_norm=0.0573, loss=0.0058, param_norm=1804.0057 09:41:41.590 [I] Progress on: 1.45kit/3.00kit rate:7.1s/it remaining:3:02:22 elapsed:2:55:30 postfix:- (1728:tqdm_logging.py:145) 09:41:56.113 [I] Progress on: 1.45kit/3.00kit rate:6.8s/it remaining:2:54:22 elapsed:2:55:44 postfix:- (1728:tqdm_logging.py:145) 09:42:10.247 [I] Progress on: 1.46kit/3.00kit rate:6.9s/it remaining:2:58:05 elapsed:2:55:59 postfix:- (1728:tqdm_logging.py:145) 09:42:24.355 [I] Progress on: 1.46kit/3.00kit rate:7.0s/it remaining:2:59:35 elapsed:2:56:13 postfix:- (1728:tqdm_logging.py:145) 09:42:38.465 [I] Progress on: 1.46kit/3.00kit rate:7.0s/it remaining:3:00:14 elapsed:2:56:27 postfix:- (1728:tqdm_logging.py:145) 09:42:52.604 [I] Progress on: 1.46kit/3.00kit rate:7.0s/it remaining:3:00:37 elapsed:2:56:41 postfix:- (1728:tqdm_logging.py:145) 09:43:06.748 [I] Progress on: 1.46kit/3.00kit rate:7.1s/it remaining:3:00:44 elapsed:2:56:55 postfix:- (1728:tqdm_logging.py:145) 09:43:20.838 [I] Progress on: 1.47kit/3.00kit rate:7.1s/it remaining:3:00:19 elapsed:2:57:09 postfix:- (1728:tqdm_logging.py:145) 09:43:34.945 [I] Progress on: 1.47kit/3.00kit rate:7.1s/it remaining:3:00:06 elapsed:2:57:23 postfix:- (1728:tqdm_logging.py:145) 09:43:49.078 [I] Progress on: 1.47kit/3.00kit rate:7.1s/it remaining:3:00:01 elapsed:2:57:37 postfix:- (1728:tqdm_logging.py:145) 09:44:03.205 [I] Progress on: 1.47kit/3.00kit rate:7.1s/it remaining:2:59:50 elapsed:2:57:51 postfix:- (1728:tqdm_logging.py:145) 09:44:17.327 [I] Progress on: 1.47kit/3.00kit rate:7.1s/it remaining:2:59:36 elapsed:2:58:06 postfix:- (1728:tqdm_logging.py:145) 09:44:31.457 [I] Progress on: 1.48kit/3.00kit rate:7.1s/it remaining:2:59:23 elapsed:2:58:20 postfix:- (1728:tqdm_logging.py:145) 09:44:45.580 [I] Progress on: 1.48kit/3.00kit rate:7.1s/it remaining:2:59:08 elapsed:2:58:34 postfix:- (1728:tqdm_logging.py:145) 09:44:59.717 [I] Progress on: 1.48kit/3.00kit rate:7.1s/it remaining:2:59:00 elapsed:2:58:48 postfix:- (1728:tqdm_logging.py:145) 09:45:13.843 [I] Progress on: 1.48kit/3.00kit rate:7.1s/it remaining:2:58:44 elapsed:2:59:02 postfix:- (1728:tqdm_logging.py:145) 09:45:27.963 [I] Progress on: 1.48kit/3.00kit rate:7.1s/it remaining:2:58:26 elapsed:2:59:16 postfix:- (1728:tqdm_logging.py:145) 09:45:42.081 [I] Progress on: 1.49kit/3.00kit rate:7.1s/it remaining:2:58:10 elapsed:2:59:30 postfix:- (1728:tqdm_logging.py:145) 09:45:56.189 [I] Progress on: 1.49kit/3.00kit rate:7.1s/it remaining:2:57:50 elapsed:2:59:44 postfix:- (1728:tqdm_logging.py:145) 09:46:10.326 [I] Progress on: 1.49kit/3.00kit rate:7.1s/it remaining:2:57:44 elapsed:2:59:59 postfix:- (1728:tqdm_logging.py:145) 09:46:24.444 [I] Progress on: 1.49kit/3.00kit rate:7.1s/it remaining:2:57:28 elapsed:3:00:13 postfix:- (1728:tqdm_logging.py:145) 09:46:38.564 [I] Progress on: 1.49kit/3.00kit rate:7.1s/it remaining:2:57:13 elapsed:3:00:27 postfix:- (1728:tqdm_logging.py:145) 09:46:52.670 [I] Progress on: 1.50kit/3.00kit rate:7.1s/it remaining:2:56:53 elapsed:3:00:41 postfix:- (1728:tqdm_logging.py:145) 09:47:06.792 [I] Progress on: 1.50kit/3.00kit rate:7.1s/it remaining:2:56:42 elapsed:3:00:55 postfix:- (1728:tqdm_logging.py:145) 09:47:20.904 [I] Progress on: 1.50kit/3.00kit rate:7.1s/it remaining:2:56:26 elapsed:3:01:09 postfix:- (1728:tqdm_logging.py:145) Step 1500: grad_norm=0.0495, loss=0.0058, param_norm=1804.0128 09:47:34.675 [I] Progress on: 1.50kit/3.00kit rate:7.1s/it remaining:2:56:26 elapsed:3:01:23 postfix:- (1728:tqdm_logging.py:145) 09:47:49.222 [I] Progress on: 1.50kit/3.00kit rate:6.8s/it remaining:2:48:51 elapsed:3:01:38 postfix:- (1728:tqdm_logging.py:145) 09:48:03.336 [I] Progress on: 1.51kit/3.00kit rate:6.9s/it remaining:2:52:14 elapsed:3:01:52 postfix:- (1728:tqdm_logging.py:145) 09:48:17.455 [I] Progress on: 1.51kit/3.00kit rate:7.0s/it remaining:2:53:49 elapsed:3:02:06 postfix:- (1728:tqdm_logging.py:145) 09:48:31.556 [I] Progress on: 1.51kit/3.00kit rate:7.0s/it remaining:2:54:21 elapsed:3:02:20 postfix:- (1728:tqdm_logging.py:145) 09:48:45.673 [I] Progress on: 1.51kit/3.00kit rate:7.0s/it remaining:2:54:35 elapsed:3:02:34 postfix:- (1728:tqdm_logging.py:145) 09:48:59.785 [I] Progress on: 1.51kit/3.00kit rate:7.0s/it remaining:2:54:32 elapsed:3:02:48 postfix:- (1728:tqdm_logging.py:145) 09:49:13.891 [I] Progress on: 1.52kit/3.00kit rate:7.0s/it remaining:2:54:22 elapsed:3:03:02 postfix:- (1728:tqdm_logging.py:145) 09:49:27.999 [I] Progress on: 1.52kit/3.00kit rate:7.1s/it remaining:2:54:11 elapsed:3:03:16 postfix:- (1728:tqdm_logging.py:145) 09:49:42.117 [I] Progress on: 1.52kit/3.00kit rate:7.1s/it remaining:2:54:00 elapsed:3:03:30 postfix:- (1728:tqdm_logging.py:145) 09:49:56.238 [I] Progress on: 1.52kit/3.00kit rate:7.1s/it remaining:2:53:50 elapsed:3:03:45 postfix:- (1728:tqdm_logging.py:145) 09:50:10.358 [I] Progress on: 1.52kit/3.00kit rate:7.1s/it remaining:2:53:38 elapsed:3:03:59 postfix:- (1728:tqdm_logging.py:145) 09:50:24.475 [I] Progress on: 1.53kit/3.00kit rate:7.1s/it remaining:2:53:23 elapsed:3:04:13 postfix:- (1728:tqdm_logging.py:145) 09:50:38.585 [I] Progress on: 1.53kit/3.00kit rate:7.1s/it remaining:2:53:06 elapsed:3:04:27 postfix:- (1728:tqdm_logging.py:145) 09:50:52.714 [I] Progress on: 1.53kit/3.00kit rate:7.1s/it remaining:2:52:59 elapsed:3:04:41 postfix:- (1728:tqdm_logging.py:145) 09:51:06.829 [I] Progress on: 1.53kit/3.00kit rate:7.1s/it remaining:2:52:43 elapsed:3:04:55 postfix:- (1728:tqdm_logging.py:145) 09:51:20.949 [I] Progress on: 1.53kit/3.00kit rate:7.1s/it remaining:2:52:28 elapsed:3:05:09 postfix:- (1728:tqdm_logging.py:145) 09:51:35.069 [I] Progress on: 1.54kit/3.00kit rate:7.1s/it remaining:2:52:16 elapsed:3:05:23 postfix:- (1728:tqdm_logging.py:145) 09:51:49.180 [I] Progress on: 1.54kit/3.00kit rate:7.1s/it remaining:2:51:58 elapsed:3:05:37 postfix:- (1728:tqdm_logging.py:145) 09:52:03.293 [I] Progress on: 1.54kit/3.00kit rate:7.1s/it remaining:2:51:43 elapsed:3:05:52 postfix:- (1728:tqdm_logging.py:145) 09:52:17.408 [I] Progress on: 1.54kit/3.00kit rate:7.1s/it remaining:2:51:29 elapsed:3:06:06 postfix:- (1728:tqdm_logging.py:145) 09:52:31.530 [I] Progress on: 1.54kit/3.00kit rate:7.1s/it remaining:2:51:18 elapsed:3:06:20 postfix:- (1728:tqdm_logging.py:145) 09:52:45.643 [I] Progress on: 1.55kit/3.00kit rate:7.1s/it remaining:2:51:02 elapsed:3:06:34 postfix:- (1728:tqdm_logging.py:145) 09:52:59.764 [I] Progress on: 1.55kit/3.00kit rate:7.1s/it remaining:2:50:50 elapsed:3:06:48 postfix:- (1728:tqdm_logging.py:145) 09:53:13.899 [I] Progress on: 1.55kit/3.00kit rate:7.1s/it remaining:2:50:41 elapsed:3:07:02 postfix:- (1728:tqdm_logging.py:145) Step 1550: grad_norm=0.0472, loss=0.0059, param_norm=1804.0178 09:53:27.647 [I] Progress on: 1.55kit/3.00kit rate:7.1s/it remaining:2:50:41 elapsed:3:07:16 postfix:- (1728:tqdm_logging.py:145) 09:53:42.731 [I] Progress on: 1.55kit/3.00kit rate:6.9s/it remaining:2:47:05 elapsed:3:07:31 postfix:- (1728:tqdm_logging.py:145) 09:53:56.334 [I] Progress on: 1.56kit/3.00kit rate:6.9s/it remaining:2:45:53 elapsed:3:07:45 postfix:- (1728:tqdm_logging.py:145) 09:54:10.453 [I] Progress on: 1.56kit/3.00kit rate:7.0s/it remaining:2:47:47 elapsed:3:07:59 postfix:- (1728:tqdm_logging.py:145) 09:54:24.573 [I] Progress on: 1.56kit/3.00kit rate:7.0s/it remaining:2:48:35 elapsed:3:08:13 postfix:- (1728:tqdm_logging.py:145) 09:54:38.688 [I] Progress on: 1.56kit/3.00kit rate:7.0s/it remaining:2:48:50 elapsed:3:08:27 postfix:- (1728:tqdm_logging.py:145) 09:54:52.791 [I] Progress on: 1.56kit/3.00kit rate:7.1s/it remaining:2:48:46 elapsed:3:08:41 postfix:- (1728:tqdm_logging.py:145) 09:55:06.906 [I] Progress on: 1.57kit/3.00kit rate:7.1s/it remaining:2:48:38 elapsed:3:08:55 postfix:- (1728:tqdm_logging.py:145) 09:55:21.024 [I] Progress on: 1.57kit/3.00kit rate:7.1s/it remaining:2:48:26 elapsed:3:09:09 postfix:- (1728:tqdm_logging.py:145) 09:55:35.136 [I] Progress on: 1.57kit/3.00kit rate:7.1s/it remaining:2:48:11 elapsed:3:09:23 postfix:- (1728:tqdm_logging.py:145) 09:55:49.245 [I] Progress on: 1.57kit/3.00kit rate:7.1s/it remaining:2:47:56 elapsed:3:09:38 postfix:- (1728:tqdm_logging.py:145) 09:56:03.353 [I] Progress on: 1.57kit/3.00kit rate:7.1s/it remaining:2:47:39 elapsed:3:09:52 postfix:- (1728:tqdm_logging.py:145) 09:56:17.467 [I] Progress on: 1.58kit/3.00kit rate:7.1s/it remaining:2:47:25 elapsed:3:10:06 postfix:- (1728:tqdm_logging.py:145) 09:56:31.575 [I] Progress on: 1.58kit/3.00kit rate:7.1s/it remaining:2:47:11 elapsed:3:10:20 postfix:- (1728:tqdm_logging.py:145) 09:56:45.695 [I] Progress on: 1.58kit/3.00kit rate:7.1s/it remaining:2:47:01 elapsed:3:10:34 postfix:- (1728:tqdm_logging.py:145) 09:56:59.801 [I] Progress on: 1.58kit/3.00kit rate:7.1s/it remaining:2:46:44 elapsed:3:10:48 postfix:- (1728:tqdm_logging.py:145) 09:57:13.908 [I] Progress on: 1.58kit/3.00kit rate:7.1s/it remaining:2:46:29 elapsed:3:11:02 postfix:- (1728:tqdm_logging.py:145) 09:57:28.031 [I] Progress on: 1.59kit/3.00kit rate:7.1s/it remaining:2:46:18 elapsed:3:11:16 postfix:- (1728:tqdm_logging.py:145) 09:57:42.162 [I] Progress on: 1.59kit/3.00kit rate:7.1s/it remaining:2:46:11 elapsed:3:11:30 postfix:- (1728:tqdm_logging.py:145) 09:57:56.273 [I] Progress on: 1.59kit/3.00kit rate:7.1s/it remaining:2:45:53 elapsed:3:11:45 postfix:- (1728:tqdm_logging.py:145) 09:58:10.371 [I] Progress on: 1.59kit/3.00kit rate:7.1s/it remaining:2:45:33 elapsed:3:11:59 postfix:- (1728:tqdm_logging.py:145) 09:58:24.500 [I] Progress on: 1.59kit/3.00kit rate:7.1s/it remaining:2:45:25 elapsed:3:12:13 postfix:- (1728:tqdm_logging.py:145) 09:58:38.624 [I] Progress on: 1.60kit/3.00kit rate:7.1s/it remaining:2:45:13 elapsed:3:12:27 postfix:- (1728:tqdm_logging.py:145) 09:58:52.733 [I] Progress on: 1.60kit/3.00kit rate:7.1s/it remaining:2:44:55 elapsed:3:12:41 postfix:- (1728:tqdm_logging.py:145) 09:59:06.852 [I] Progress on: 1.60kit/3.00kit rate:7.1s/it remaining:2:44:42 elapsed:3:12:55 postfix:- (1728:tqdm_logging.py:145) Step 1600: grad_norm=0.0482, loss=0.0058, param_norm=1804.0236 09:59:20.605 [I] Progress on: 1.60kit/3.00kit rate:7.1s/it remaining:2:44:42 elapsed:3:13:09 postfix:- (1728:tqdm_logging.py:145) 09:59:35.172 [I] Progress on: 1.60kit/3.00kit rate:6.8s/it remaining:2:37:34 elapsed:3:13:23 postfix:- (1728:tqdm_logging.py:145) 09:59:49.290 [I] Progress on: 1.61kit/3.00kit rate:6.9s/it remaining:2:40:44 elapsed:3:13:38 postfix:- (1728:tqdm_logging.py:145) 10:00:03.482 [I] Progress on: 1.61kit/3.00kit rate:7.0s/it remaining:2:42:40 elapsed:3:13:52 postfix:- (1728:tqdm_logging.py:145) 10:00:17.603 [I] Progress on: 1.61kit/3.00kit rate:7.0s/it remaining:2:43:01 elapsed:3:14:06 postfix:- (1728:tqdm_logging.py:145) 10:00:31.728 [I] Progress on: 1.61kit/3.00kit rate:7.0s/it remaining:2:43:04 elapsed:3:14:20 postfix:- (1728:tqdm_logging.py:145) 10:00:45.841 [I] Progress on: 1.61kit/3.00kit rate:7.1s/it remaining:2:42:55 elapsed:3:14:34 postfix:- (1728:tqdm_logging.py:145) 10:00:59.956 [I] Progress on: 1.62kit/3.00kit rate:7.1s/it remaining:2:42:45 elapsed:3:14:48 postfix:- (1728:tqdm_logging.py:145) 10:01:14.060 [I] Progress on: 1.62kit/3.00kit rate:7.1s/it remaining:2:42:27 elapsed:3:15:02 postfix:- (1728:tqdm_logging.py:145) 10:01:28.170 [I] Progress on: 1.62kit/3.00kit rate:7.1s/it remaining:2:42:15 elapsed:3:15:16 postfix:- (1728:tqdm_logging.py:145) 10:01:42.294 [I] Progress on: 1.62kit/3.00kit rate:7.1s/it remaining:2:42:05 elapsed:3:15:31 postfix:- (1728:tqdm_logging.py:145) 10:01:56.405 [I] Progress on: 1.62kit/3.00kit rate:7.1s/it remaining:2:41:50 elapsed:3:15:45 postfix:- (1728:tqdm_logging.py:145) 10:02:10.535 [I] Progress on: 1.63kit/3.00kit rate:7.1s/it remaining:2:41:41 elapsed:3:15:59 postfix:- (1728:tqdm_logging.py:145) 10:02:24.651 [I] Progress on: 1.63kit/3.00kit rate:7.1s/it remaining:2:41:25 elapsed:3:16:13 postfix:- (1728:tqdm_logging.py:145) 10:02:38.767 [I] Progress on: 1.63kit/3.00kit rate:7.1s/it remaining:2:41:11 elapsed:3:16:27 postfix:- (1728:tqdm_logging.py:145) 10:02:52.871 [I] Progress on: 1.63kit/3.00kit rate:7.1s/it remaining:2:40:51 elapsed:3:16:41 postfix:- (1728:tqdm_logging.py:145) 10:03:06.995 [I] Progress on: 1.63kit/3.00kit rate:7.1s/it remaining:2:40:42 elapsed:3:16:55 postfix:- (1728:tqdm_logging.py:145) 10:03:21.112 [I] Progress on: 1.64kit/3.00kit rate:7.1s/it remaining:2:40:28 elapsed:3:17:09 postfix:- (1728:tqdm_logging.py:145) 10:03:35.223 [I] Progress on: 1.64kit/3.00kit rate:7.1s/it remaining:2:40:12 elapsed:3:17:24 postfix:- (1728:tqdm_logging.py:145) 10:03:49.332 [I] Progress on: 1.64kit/3.00kit rate:7.1s/it remaining:2:39:57 elapsed:3:17:38 postfix:- (1728:tqdm_logging.py:145) 10:04:03.451 [I] Progress on: 1.64kit/3.00kit rate:7.1s/it remaining:2:39:44 elapsed:3:17:52 postfix:- (1728:tqdm_logging.py:145) 10:04:17.570 [I] Progress on: 1.64kit/3.00kit rate:7.1s/it remaining:2:39:31 elapsed:3:18:06 postfix:- (1728:tqdm_logging.py:145) 10:04:31.690 [I] Progress on: 1.65kit/3.00kit rate:7.1s/it remaining:2:39:19 elapsed:3:18:20 postfix:- (1728:tqdm_logging.py:145) 10:04:45.827 [I] Progress on: 1.65kit/3.00kit rate:7.1s/it remaining:2:39:11 elapsed:3:18:34 postfix:- (1728:tqdm_logging.py:145) 10:04:59.929 [I] Progress on: 1.65kit/3.00kit rate:7.1s/it remaining:2:38:49 elapsed:3:18:48 postfix:- (1728:tqdm_logging.py:145) Step 1650: grad_norm=0.0454, loss=0.0058, param_norm=1804.0297 10:05:13.719 [I] Progress on: 1.65kit/3.00kit rate:7.1s/it remaining:2:38:49 elapsed:3:19:02 postfix:- (1728:tqdm_logging.py:145) 10:05:28.276 [I] Progress on: 1.65kit/3.00kit rate:6.8s/it remaining:2:31:58 elapsed:3:19:17 postfix:- (1728:tqdm_logging.py:145) 10:05:42.381 [I] Progress on: 1.66kit/3.00kit rate:6.9s/it remaining:2:34:56 elapsed:3:19:31 postfix:- (1728:tqdm_logging.py:145) 10:05:56.486 [I] Progress on: 1.66kit/3.00kit rate:7.0s/it remaining:2:36:15 elapsed:3:19:45 postfix:- (1728:tqdm_logging.py:145) 10:06:10.598 [I] Progress on: 1.66kit/3.00kit rate:7.0s/it remaining:2:36:48 elapsed:3:19:59 postfix:- (1728:tqdm_logging.py:145) 10:06:24.708 [I] Progress on: 1.66kit/3.00kit rate:7.0s/it remaining:2:36:58 elapsed:3:20:13 postfix:- (1728:tqdm_logging.py:145) 10:06:38.819 [I] Progress on: 1.66kit/3.00kit rate:7.0s/it remaining:2:36:54 elapsed:3:20:27 postfix:- (1728:tqdm_logging.py:145) 10:06:52.933 [I] Progress on: 1.67kit/3.00kit rate:7.1s/it remaining:2:36:44 elapsed:3:20:41 postfix:- (1728:tqdm_logging.py:145) 10:07:07.056 [I] Progress on: 1.67kit/3.00kit rate:7.1s/it remaining:2:36:37 elapsed:3:20:55 postfix:- (1728:tqdm_logging.py:145) 10:07:21.168 [I] Progress on: 1.67kit/3.00kit rate:7.1s/it remaining:2:36:23 elapsed:3:21:09 postfix:- (1728:tqdm_logging.py:145) 10:07:35.283 [I] Progress on: 1.67kit/3.00kit rate:7.1s/it remaining:2:36:10 elapsed:3:21:24 postfix:- (1728:tqdm_logging.py:145) 10:07:49.420 [I] Progress on: 1.67kit/3.00kit rate:7.1s/it remaining:2:36:03 elapsed:3:21:38 postfix:- (1728:tqdm_logging.py:145) 10:08:03.535 [I] Progress on: 1.68kit/3.00kit rate:7.1s/it remaining:2:35:46 elapsed:3:21:52 postfix:- (1728:tqdm_logging.py:145) 10:08:17.659 [I] Progress on: 1.68kit/3.00kit rate:7.1s/it remaining:2:35:34 elapsed:3:22:06 postfix:- (1728:tqdm_logging.py:145) 10:08:31.771 [I] Progress on: 1.68kit/3.00kit rate:7.1s/it remaining:2:35:17 elapsed:3:22:20 postfix:- (1728:tqdm_logging.py:145) 10:08:45.876 [I] Progress on: 1.68kit/3.00kit rate:7.1s/it remaining:2:34:57 elapsed:3:22:34 postfix:- (1728:tqdm_logging.py:145) 10:08:59.987 [I] Progress on: 1.68kit/3.00kit rate:7.1s/it remaining:2:34:44 elapsed:3:22:48 postfix:- (1728:tqdm_logging.py:145) 10:09:14.114 [I] Progress on: 1.69kit/3.00kit rate:7.1s/it remaining:2:34:35 elapsed:3:23:02 postfix:- (1728:tqdm_logging.py:145) 10:09:28.234 [I] Progress on: 1.69kit/3.00kit rate:7.1s/it remaining:2:34:21 elapsed:3:23:17 postfix:- (1728:tqdm_logging.py:145) 10:09:42.343 [I] Progress on: 1.69kit/3.00kit rate:7.1s/it remaining:2:34:04 elapsed:3:23:31 postfix:- (1728:tqdm_logging.py:145) 10:09:56.453 [I] Progress on: 1.69kit/3.00kit rate:7.1s/it remaining:2:33:48 elapsed:3:23:45 postfix:- (1728:tqdm_logging.py:145) 10:10:10.575 [I] Progress on: 1.69kit/3.00kit rate:7.1s/it remaining:2:33:36 elapsed:3:23:59 postfix:- (1728:tqdm_logging.py:145) 10:10:24.695 [I] Progress on: 1.70kit/3.00kit rate:7.1s/it remaining:2:33:23 elapsed:3:24:13 postfix:- (1728:tqdm_logging.py:145) 10:10:38.810 [I] Progress on: 1.70kit/3.00kit rate:7.1s/it remaining:2:33:09 elapsed:3:24:27 postfix:- (1728:tqdm_logging.py:145) 10:10:52.913 [I] Progress on: 1.70kit/3.00kit rate:7.1s/it remaining:2:32:49 elapsed:3:24:41 postfix:- (1728:tqdm_logging.py:145) Step 1700: grad_norm=0.0460, loss=0.0056, param_norm=1804.0359 10:11:06.654 [I] Progress on: 1.70kit/3.00kit rate:7.1s/it remaining:2:32:49 elapsed:3:24:55 postfix:- (1728:tqdm_logging.py:145) 10:11:21.160 [I] Progress on: 1.70kit/3.00kit rate:6.8s/it remaining:2:26:02 elapsed:3:25:09 postfix:- (1728:tqdm_logging.py:145) 10:11:35.273 [I] Progress on: 1.71kit/3.00kit rate:6.9s/it remaining:2:29:03 elapsed:3:25:24 postfix:- (1728:tqdm_logging.py:145) 10:11:49.378 [I] Progress on: 1.71kit/3.00kit rate:7.0s/it remaining:2:30:23 elapsed:3:25:38 postfix:- (1728:tqdm_logging.py:145) 10:12:03.494 [I] Progress on: 1.71kit/3.00kit rate:7.0s/it remaining:2:30:58 elapsed:3:25:52 postfix:- (1728:tqdm_logging.py:145) 10:12:17.603 [I] Progress on: 1.71kit/3.00kit rate:7.0s/it remaining:2:31:06 elapsed:3:26:06 postfix:- (1728:tqdm_logging.py:145) 10:12:31.712 [I] Progress on: 1.71kit/3.00kit rate:7.0s/it remaining:2:31:02 elapsed:3:26:20 postfix:- (1728:tqdm_logging.py:145) 10:12:45.833 [I] Progress on: 1.72kit/3.00kit rate:7.1s/it remaining:2:30:56 elapsed:3:26:34 postfix:- (1728:tqdm_logging.py:145) 10:12:59.942 [I] Progress on: 1.72kit/3.00kit rate:7.1s/it remaining:2:30:42 elapsed:3:26:48 postfix:- (1728:tqdm_logging.py:145) 10:13:14.066 [I] Progress on: 1.72kit/3.00kit rate:7.1s/it remaining:2:30:34 elapsed:3:27:02 postfix:- (1728:tqdm_logging.py:145) 10:13:28.181 [I] Progress on: 1.72kit/3.00kit rate:7.1s/it remaining:2:30:20 elapsed:3:27:16 postfix:- (1728:tqdm_logging.py:145) 10:13:42.285 [I] Progress on: 1.72kit/3.00kit rate:7.1s/it remaining:2:30:02 elapsed:3:27:31 postfix:- (1728:tqdm_logging.py:145) 10:13:56.409 [I] Progress on: 1.73kit/3.00kit rate:7.1s/it remaining:2:29:52 elapsed:3:27:45 postfix:- (1728:tqdm_logging.py:145) 10:14:10.534 [I] Progress on: 1.73kit/3.00kit rate:7.1s/it remaining:2:29:41 elapsed:3:27:59 postfix:- (1728:tqdm_logging.py:145) 10:14:24.636 [I] Progress on: 1.73kit/3.00kit rate:7.1s/it remaining:2:29:21 elapsed:3:28:13 postfix:- (1728:tqdm_logging.py:145) 10:14:38.745 [I] Progress on: 1.73kit/3.00kit rate:7.1s/it remaining:2:29:06 elapsed:3:28:27 postfix:- (1728:tqdm_logging.py:145) 10:14:52.870 [I] Progress on: 1.73kit/3.00kit rate:7.1s/it remaining:2:28:55 elapsed:3:28:41 postfix:- (1728:tqdm_logging.py:145) 10:15:06.987 [I] Progress on: 1.74kit/3.00kit rate:7.1s/it remaining:2:28:41 elapsed:3:28:55 postfix:- (1728:tqdm_logging.py:145) 10:15:21.108 [I] Progress on: 1.74kit/3.00kit rate:7.1s/it remaining:2:28:28 elapsed:3:29:09 postfix:- (1728:tqdm_logging.py:145) 10:15:35.213 [I] Progress on: 1.74kit/3.00kit rate:7.1s/it remaining:2:28:09 elapsed:3:29:24 postfix:- (1728:tqdm_logging.py:145) 10:15:49.342 [I] Progress on: 1.74kit/3.00kit rate:7.1s/it remaining:2:28:00 elapsed:3:29:38 postfix:- (1728:tqdm_logging.py:145) 10:16:03.449 [I] Progress on: 1.74kit/3.00kit rate:7.1s/it remaining:2:27:42 elapsed:3:29:52 postfix:- (1728:tqdm_logging.py:145) 10:16:17.559 [I] Progress on: 1.75kit/3.00kit rate:7.1s/it remaining:2:27:29 elapsed:3:30:06 postfix:- (1728:tqdm_logging.py:145) 10:16:31.677 [I] Progress on: 1.75kit/3.00kit rate:7.1s/it remaining:2:27:15 elapsed:3:30:20 postfix:- (1728:tqdm_logging.py:145) 10:16:45.784 [I] Progress on: 1.75kit/3.00kit rate:7.1s/it remaining:2:26:59 elapsed:3:30:34 postfix:- (1728:tqdm_logging.py:145) Step 1750: grad_norm=0.0466, loss=0.0054, param_norm=1804.0421 10:16:59.535 [I] Progress on: 1.75kit/3.00kit rate:7.1s/it remaining:2:26:59 elapsed:3:30:48 postfix:- (1728:tqdm_logging.py:145) 10:17:14.081 [I] Progress on: 1.75kit/3.00kit rate:6.8s/it remaining:2:20:32 elapsed:3:31:02 postfix:- (1728:tqdm_logging.py:145) 10:17:28.198 [I] Progress on: 1.76kit/3.00kit rate:6.9s/it remaining:2:23:24 elapsed:3:31:16 postfix:- (1728:tqdm_logging.py:145) 10:17:42.294 [I] Progress on: 1.76kit/3.00kit rate:7.0s/it remaining:2:24:32 elapsed:3:31:31 postfix:- (1728:tqdm_logging.py:145) 10:17:56.421 [I] Progress on: 1.76kit/3.00kit rate:7.0s/it remaining:2:25:09 elapsed:3:31:45 postfix:- (1728:tqdm_logging.py:145) 10:18:10.527 [I] Progress on: 1.76kit/3.00kit rate:7.0s/it remaining:2:25:14 elapsed:3:31:59 postfix:- (1728:tqdm_logging.py:145) 10:18:24.625 [I] Progress on: 1.76kit/3.00kit rate:7.0s/it remaining:2:25:07 elapsed:3:32:13 postfix:- (1728:tqdm_logging.py:145) 10:18:38.729 [I] Progress on: 1.77kit/3.00kit rate:7.0s/it remaining:2:24:58 elapsed:3:32:27 postfix:- (1728:tqdm_logging.py:145) 10:18:52.825 [I] Progress on: 1.77kit/3.00kit rate:7.0s/it remaining:2:24:43 elapsed:3:32:41 postfix:- (1728:tqdm_logging.py:145) 10:19:06.928 [I] Progress on: 1.77kit/3.00kit rate:7.1s/it remaining:2:24:31 elapsed:3:32:55 postfix:- (1728:tqdm_logging.py:145) 10:19:21.023 [I] Progress on: 1.77kit/3.00kit rate:7.0s/it remaining:2:24:15 elapsed:3:33:09 postfix:- (1728:tqdm_logging.py:145) 10:19:35.121 [I] Progress on: 1.77kit/3.00kit rate:7.0s/it remaining:2:24:01 elapsed:3:33:23 postfix:- (1728:tqdm_logging.py:145) 10:19:49.807 [I] Progress on: 1.78kit/3.00kit rate:7.2s/it remaining:2:27:21 elapsed:3:33:38 postfix:- (1728:tqdm_logging.py:145) 10:20:03.375 [I] Progress on: 1.78kit/3.00kit rate:7.0s/it remaining:2:23:07 elapsed:3:33:52 postfix:- (1728:tqdm_logging.py:145) 10:20:17.480 [I] Progress on: 1.78kit/3.00kit rate:7.0s/it remaining:2:23:12 elapsed:3:34:06 postfix:- (1728:tqdm_logging.py:145) 10:20:31.586 [I] Progress on: 1.78kit/3.00kit rate:7.1s/it remaining:2:23:08 elapsed:3:34:20 postfix:- (1728:tqdm_logging.py:145) 10:20:45.686 [I] Progress on: 1.78kit/3.00kit rate:7.1s/it remaining:2:22:58 elapsed:3:34:34 postfix:- (1728:tqdm_logging.py:145) 10:20:59.802 [I] Progress on: 1.79kit/3.00kit rate:7.1s/it remaining:2:22:50 elapsed:3:34:48 postfix:- (1728:tqdm_logging.py:145) 10:21:13.908 [I] Progress on: 1.79kit/3.00kit rate:7.1s/it remaining:2:22:35 elapsed:3:35:02 postfix:- (1728:tqdm_logging.py:145) 10:21:28.005 [I] Progress on: 1.79kit/3.00kit rate:7.1s/it remaining:2:22:18 elapsed:3:35:16 postfix:- (1728:tqdm_logging.py:145) 10:21:42.113 [I] Progress on: 1.79kit/3.00kit rate:7.1s/it remaining:2:22:06 elapsed:3:35:30 postfix:- (1728:tqdm_logging.py:145) 10:21:56.210 [I] Progress on: 1.79kit/3.00kit rate:7.1s/it remaining:2:21:48 elapsed:3:35:44 postfix:- (1728:tqdm_logging.py:145) 10:22:10.319 [I] Progress on: 1.80kit/3.00kit rate:7.1s/it remaining:2:21:33 elapsed:3:35:59 postfix:- (1728:tqdm_logging.py:145) 10:22:24.411 [I] Progress on: 1.80kit/3.00kit rate:7.1s/it remaining:2:21:14 elapsed:3:36:13 postfix:- (1728:tqdm_logging.py:145) 10:22:38.527 [I] Progress on: 1.80kit/3.00kit rate:7.1s/it remaining:2:21:03 elapsed:3:36:27 postfix:- (1728:tqdm_logging.py:145) Step 1800: grad_norm=0.0444, loss=0.0056, param_norm=1804.0483 10:22:52.274 [I] Progress on: 1.80kit/3.00kit rate:7.1s/it remaining:2:21:03 elapsed:3:36:41 postfix:- (1728:tqdm_logging.py:145) 10:23:06.806 [I] Progress on: 1.80kit/3.00kit rate:6.8s/it remaining:2:14:50 elapsed:3:36:55 postfix:- (1728:tqdm_logging.py:145) 10:23:20.921 [I] Progress on: 1.81kit/3.00kit rate:6.9s/it remaining:2:17:34 elapsed:3:37:09 postfix:- (1728:tqdm_logging.py:145) 10:23:35.021 [I] Progress on: 1.81kit/3.00kit rate:7.0s/it remaining:2:18:42 elapsed:3:37:23 postfix:- (1728:tqdm_logging.py:145) 10:23:49.141 [I] Progress on: 1.81kit/3.00kit rate:7.0s/it remaining:2:19:15 elapsed:3:37:37 postfix:- (1728:tqdm_logging.py:145) 10:24:03.260 [I] Progress on: 1.81kit/3.00kit rate:7.0s/it remaining:2:19:24 elapsed:3:37:52 postfix:- (1728:tqdm_logging.py:145) 10:24:17.349 [I] Progress on: 1.81kit/3.00kit rate:7.0s/it remaining:2:19:10 elapsed:3:38:06 postfix:- (1728:tqdm_logging.py:145) 10:24:31.472 [I] Progress on: 1.82kit/3.00kit rate:7.1s/it remaining:2:19:08 elapsed:3:38:20 postfix:- (1728:tqdm_logging.py:145) 10:24:45.565 [I] Progress on: 1.82kit/3.00kit rate:7.0s/it remaining:2:18:52 elapsed:3:38:34 postfix:- (1728:tqdm_logging.py:145) 10:24:59.689 [I] Progress on: 1.82kit/3.00kit rate:7.1s/it remaining:2:18:45 elapsed:3:38:48 postfix:- (1728:tqdm_logging.py:145) 10:25:13.791 [I] Progress on: 1.82kit/3.00kit rate:7.1s/it remaining:2:18:29 elapsed:3:39:02 postfix:- (1728:tqdm_logging.py:145) 10:25:27.911 [I] Progress on: 1.82kit/3.00kit rate:7.1s/it remaining:2:18:17 elapsed:3:39:16 postfix:- (1728:tqdm_logging.py:145) 10:25:42.036 [I] Progress on: 1.83kit/3.00kit rate:7.1s/it remaining:2:18:07 elapsed:3:39:30 postfix:- (1728:tqdm_logging.py:145) 10:25:56.150 [I] Progress on: 1.83kit/3.00kit rate:7.1s/it remaining:2:17:53 elapsed:3:39:44 postfix:- (1728:tqdm_logging.py:145) 10:26:10.256 [I] Progress on: 1.83kit/3.00kit rate:7.1s/it remaining:2:17:36 elapsed:3:39:59 postfix:- (1728:tqdm_logging.py:145) 10:26:24.372 [I] Progress on: 1.83kit/3.00kit rate:7.1s/it remaining:2:17:22 elapsed:3:40:13 postfix:- (1728:tqdm_logging.py:145) 10:26:38.478 [I] Progress on: 1.83kit/3.00kit rate:7.1s/it remaining:2:17:06 elapsed:3:40:27 postfix:- (1728:tqdm_logging.py:145) 10:26:52.590 [I] Progress on: 1.84kit/3.00kit rate:7.1s/it remaining:2:16:53 elapsed:3:40:41 postfix:- (1728:tqdm_logging.py:145) 10:27:06.695 [I] Progress on: 1.84kit/3.00kit rate:7.1s/it remaining:2:16:36 elapsed:3:40:55 postfix:- (1728:tqdm_logging.py:145) 10:27:20.815 [I] Progress on: 1.84kit/3.00kit rate:7.1s/it remaining:2:16:25 elapsed:3:41:09 postfix:- (1728:tqdm_logging.py:145) 10:27:34.934 [I] Progress on: 1.84kit/3.00kit rate:7.1s/it remaining:2:16:14 elapsed:3:41:23 postfix:- (1728:tqdm_logging.py:145) 10:27:49.037 [I] Progress on: 1.84kit/3.00kit rate:7.1s/it remaining:2:15:56 elapsed:3:41:37 postfix:- (1728:tqdm_logging.py:145) 10:28:03.159 [I] Progress on: 1.85kit/3.00kit rate:7.1s/it remaining:2:15:46 elapsed:3:41:51 postfix:- (1728:tqdm_logging.py:145) 10:28:17.275 [I] Progress on: 1.85kit/3.00kit rate:7.1s/it remaining:2:15:30 elapsed:3:42:06 postfix:- (1728:tqdm_logging.py:145) 10:28:31.389 [I] Progress on: 1.85kit/3.00kit rate:7.1s/it remaining:2:15:15 elapsed:3:42:20 postfix:- (1728:tqdm_logging.py:145) Step 1850: grad_norm=0.0454, loss=0.0055, param_norm=1804.0544 10:28:45.136 [I] Progress on: 1.85kit/3.00kit rate:7.1s/it remaining:2:15:15 elapsed:3:42:33 postfix:- (1728:tqdm_logging.py:145) 10:28:59.693 [I] Progress on: 1.85kit/3.00kit rate:6.8s/it remaining:2:09:19 elapsed:3:42:48 postfix:- (1728:tqdm_logging.py:145) 10:29:13.787 [I] Progress on: 1.86kit/3.00kit rate:6.9s/it remaining:2:11:47 elapsed:3:43:02 postfix:- (1728:tqdm_logging.py:145) 10:29:27.891 [I] Progress on: 1.86kit/3.00kit rate:7.0s/it remaining:2:12:55 elapsed:3:43:16 postfix:- (1728:tqdm_logging.py:145) 10:29:42.013 [I] Progress on: 1.86kit/3.00kit rate:7.0s/it remaining:2:13:25 elapsed:3:43:30 postfix:- (1728:tqdm_logging.py:145) 10:29:56.106 [I] Progress on: 1.86kit/3.00kit rate:7.0s/it remaining:2:13:24 elapsed:3:43:44 postfix:- (1728:tqdm_logging.py:145) 10:30:10.336 [I] Progress on: 1.86kit/3.00kit rate:7.1s/it remaining:2:13:50 elapsed:3:43:59 postfix:- (1728:tqdm_logging.py:145) 10:30:24.444 [I] Progress on: 1.87kit/3.00kit rate:7.1s/it remaining:2:13:26 elapsed:3:44:13 postfix:- (1728:tqdm_logging.py:145) 10:30:38.559 [I] Progress on: 1.87kit/3.00kit rate:7.1s/it remaining:2:13:10 elapsed:3:44:27 postfix:- (1728:tqdm_logging.py:145) 10:30:52.679 [I] Progress on: 1.87kit/3.00kit rate:7.1s/it remaining:2:12:56 elapsed:3:44:41 postfix:- (1728:tqdm_logging.py:145) 10:31:06.786 [I] Progress on: 1.87kit/3.00kit rate:7.1s/it remaining:2:12:39 elapsed:3:44:55 postfix:- (1728:tqdm_logging.py:145) 10:31:20.892 [I] Progress on: 1.87kit/3.00kit rate:7.1s/it remaining:2:12:22 elapsed:3:45:09 postfix:- (1728:tqdm_logging.py:145) 10:31:35.005 [I] Progress on: 1.88kit/3.00kit rate:7.1s/it remaining:2:12:10 elapsed:3:45:23 postfix:- (1728:tqdm_logging.py:145) 10:31:49.120 [I] Progress on: 1.88kit/3.00kit rate:7.1s/it remaining:2:11:58 elapsed:3:45:37 postfix:- (1728:tqdm_logging.py:145) 10:32:03.229 [I] Progress on: 1.88kit/3.00kit rate:7.1s/it remaining:2:11:42 elapsed:3:45:52 postfix:- (1728:tqdm_logging.py:145) 10:32:17.330 [I] Progress on: 1.88kit/3.00kit rate:7.1s/it remaining:2:11:25 elapsed:3:46:06 postfix:- (1728:tqdm_logging.py:145) 10:32:31.439 [I] Progress on: 1.88kit/3.00kit rate:7.1s/it remaining:2:11:11 elapsed:3:46:20 postfix:- (1728:tqdm_logging.py:145) 10:32:45.553 [I] Progress on: 1.89kit/3.00kit rate:7.1s/it remaining:2:10:58 elapsed:3:46:34 postfix:- (1728:tqdm_logging.py:145) 10:32:59.660 [I] Progress on: 1.89kit/3.00kit rate:7.1s/it remaining:2:10:43 elapsed:3:46:48 postfix:- (1728:tqdm_logging.py:145) 10:33:13.771 [I] Progress on: 1.89kit/3.00kit rate:7.1s/it remaining:2:10:31 elapsed:3:47:02 postfix:- (1728:tqdm_logging.py:145) 10:33:27.868 [I] Progress on: 1.89kit/3.00kit rate:7.1s/it remaining:2:10:11 elapsed:3:47:16 postfix:- (1728:tqdm_logging.py:145) 10:33:41.971 [I] Progress on: 1.89kit/3.00kit rate:7.1s/it remaining:2:09:57 elapsed:3:47:30 postfix:- (1728:tqdm_logging.py:145) 10:33:56.096 [I] Progress on: 1.90kit/3.00kit rate:7.1s/it remaining:2:09:49 elapsed:3:47:44 postfix:- (1728:tqdm_logging.py:145) 10:34:10.198 [I] Progress on: 1.90kit/3.00kit rate:7.1s/it remaining:2:09:34 elapsed:3:47:58 postfix:- (1728:tqdm_logging.py:145) 10:34:24.304 [I] Progress on: 1.90kit/3.00kit rate:7.1s/it remaining:2:09:18 elapsed:3:48:13 postfix:- (1728:tqdm_logging.py:145) Step 1900: grad_norm=0.0470, loss=0.0053, param_norm=1804.0609 10:34:38.081 [I] Progress on: 1.90kit/3.00kit rate:7.1s/it remaining:2:09:18 elapsed:3:48:26 postfix:- (1728:tqdm_logging.py:145) 10:34:52.624 [I] Progress on: 1.90kit/3.00kit rate:6.8s/it remaining:2:03:39 elapsed:3:48:41 postfix:- (1728:tqdm_logging.py:145) 10:35:06.732 [I] Progress on: 1.91kit/3.00kit rate:6.9s/it remaining:2:06:03 elapsed:3:48:55 postfix:- (1728:tqdm_logging.py:145) 10:35:20.862 [I] Progress on: 1.91kit/3.00kit rate:7.0s/it remaining:2:07:14 elapsed:3:49:09 postfix:- (1728:tqdm_logging.py:145) 10:35:34.971 [I] Progress on: 1.91kit/3.00kit rate:7.0s/it remaining:2:07:36 elapsed:3:49:23 postfix:- (1728:tqdm_logging.py:145) 10:35:49.078 [I] Progress on: 1.91kit/3.00kit rate:7.0s/it remaining:2:07:38 elapsed:3:49:37 postfix:- (1728:tqdm_logging.py:145) 10:36:03.170 [I] Progress on: 1.91kit/3.00kit rate:7.0s/it remaining:2:07:28 elapsed:3:49:51 postfix:- (1728:tqdm_logging.py:145) 10:36:17.310 [I] Progress on: 1.92kit/3.00kit rate:7.1s/it remaining:2:07:29 elapsed:3:50:06 postfix:- (1728:tqdm_logging.py:145) 10:36:31.479 [I] Progress on: 1.92kit/3.00kit rate:7.1s/it remaining:2:07:34 elapsed:3:50:20 postfix:- (1728:tqdm_logging.py:145) 10:36:45.597 [I] Progress on: 1.92kit/3.00kit rate:7.1s/it remaining:2:07:17 elapsed:3:50:34 postfix:- (1728:tqdm_logging.py:145) 10:36:59.644 [I] Progress on: 1.92kit/3.00kit rate:7.0s/it remaining:2:06:36 elapsed:3:50:48 postfix:- (1728:tqdm_logging.py:145) 10:37:13.765 [I] Progress on: 1.92kit/3.00kit rate:7.1s/it remaining:2:06:29 elapsed:3:51:02 postfix:- (1728:tqdm_logging.py:145) 10:37:27.879 [I] Progress on: 1.93kit/3.00kit rate:7.1s/it remaining:2:06:17 elapsed:3:51:16 postfix:- (1728:tqdm_logging.py:145) 10:37:41.986 [I] Progress on: 1.93kit/3.00kit rate:7.1s/it remaining:2:06:02 elapsed:3:51:30 postfix:- (1728:tqdm_logging.py:145) 10:37:56.101 [I] Progress on: 1.93kit/3.00kit rate:7.1s/it remaining:2:05:49 elapsed:3:51:44 postfix:- (1728:tqdm_logging.py:145) 10:38:10.217 [I] Progress on: 1.93kit/3.00kit rate:7.1s/it remaining:2:05:36 elapsed:3:51:59 postfix:- (1728:tqdm_logging.py:145) 10:38:24.311 [I] Progress on: 1.93kit/3.00kit rate:7.1s/it remaining:2:05:16 elapsed:3:52:13 postfix:- (1728:tqdm_logging.py:145) 10:38:38.404 [I] Progress on: 1.94kit/3.00kit rate:7.0s/it remaining:2:05:00 elapsed:3:52:27 postfix:- (1728:tqdm_logging.py:145) 10:38:52.532 [I] Progress on: 1.94kit/3.00kit rate:7.1s/it remaining:2:04:53 elapsed:3:52:41 postfix:- (1728:tqdm_logging.py:145) 10:39:06.641 [I] Progress on: 1.94kit/3.00kit rate:7.1s/it remaining:2:04:38 elapsed:3:52:55 postfix:- (1728:tqdm_logging.py:145) 10:39:20.755 [I] Progress on: 1.94kit/3.00kit rate:7.1s/it remaining:2:04:25 elapsed:3:53:09 postfix:- (1728:tqdm_logging.py:145) 10:39:34.884 [I] Progress on: 1.94kit/3.00kit rate:7.1s/it remaining:2:04:14 elapsed:3:53:23 postfix:- (1728:tqdm_logging.py:145) 10:39:48.992 [I] Progress on: 1.95kit/3.00kit rate:7.1s/it remaining:2:03:58 elapsed:3:53:37 postfix:- (1728:tqdm_logging.py:145) 10:40:03.096 [I] Progress on: 1.95kit/3.00kit rate:7.1s/it remaining:2:03:41 elapsed:3:53:51 postfix:- (1728:tqdm_logging.py:145) 10:40:17.204 [I] Progress on: 1.95kit/3.00kit rate:7.1s/it remaining:2:03:26 elapsed:3:54:05 postfix:- (1728:tqdm_logging.py:145) Step 1950: grad_norm=0.0417, loss=0.0053, param_norm=1804.0662 10:40:30.963 [I] Progress on: 1.95kit/3.00kit rate:7.1s/it remaining:2:03:26 elapsed:3:54:19 postfix:- (1728:tqdm_logging.py:145) 10:40:45.506 [I] Progress on: 1.95kit/3.00kit rate:6.8s/it remaining:1:57:59 elapsed:3:54:34 postfix:- (1728:tqdm_logging.py:145) 10:40:59.631 [I] Progress on: 1.96kit/3.00kit rate:6.9s/it remaining:2:00:21 elapsed:3:54:48 postfix:- (1728:tqdm_logging.py:145) 10:41:13.750 [I] Progress on: 1.96kit/3.00kit rate:7.0s/it remaining:2:01:24 elapsed:3:55:02 postfix:- (1728:tqdm_logging.py:145) 10:41:27.858 [I] Progress on: 1.96kit/3.00kit rate:7.0s/it remaining:2:01:44 elapsed:3:55:16 postfix:- (1728:tqdm_logging.py:145) 10:41:41.975 [I] Progress on: 1.96kit/3.00kit rate:7.0s/it remaining:2:01:48 elapsed:3:55:30 postfix:- (1728:tqdm_logging.py:145) 10:41:56.073 [I] Progress on: 1.96kit/3.00kit rate:7.0s/it remaining:2:01:39 elapsed:3:55:44 postfix:- (1728:tqdm_logging.py:145) 10:42:10.176 [I] Progress on: 1.97kit/3.00kit rate:7.0s/it remaining:2:01:27 elapsed:3:55:58 postfix:- (1728:tqdm_logging.py:145) 10:42:24.296 [I] Progress on: 1.97kit/3.00kit rate:7.1s/it remaining:2:01:20 elapsed:3:56:13 postfix:- (1728:tqdm_logging.py:145) 10:42:38.416 [I] Progress on: 1.97kit/3.00kit rate:7.1s/it remaining:2:01:08 elapsed:3:56:27 postfix:- (1728:tqdm_logging.py:145) 10:42:52.527 [I] Progress on: 1.97kit/3.00kit rate:7.1s/it remaining:2:00:53 elapsed:3:56:41 postfix:- (1728:tqdm_logging.py:145) 10:43:06.622 [I] Progress on: 1.97kit/3.00kit rate:7.1s/it remaining:2:00:34 elapsed:3:56:55 postfix:- (1728:tqdm_logging.py:145) 10:43:20.741 [I] Progress on: 1.98kit/3.00kit rate:7.1s/it remaining:2:00:24 elapsed:3:57:09 postfix:- (1728:tqdm_logging.py:145) 10:43:34.857 [I] Progress on: 1.98kit/3.00kit rate:7.1s/it remaining:2:00:10 elapsed:3:57:23 postfix:- (1728:tqdm_logging.py:145) 10:43:48.950 [I] Progress on: 1.98kit/3.00kit rate:7.1s/it remaining:1:59:52 elapsed:3:57:37 postfix:- (1728:tqdm_logging.py:145) 10:44:03.049 [I] Progress on: 1.98kit/3.00kit rate:7.1s/it remaining:1:59:37 elapsed:3:57:51 postfix:- (1728:tqdm_logging.py:145) 10:44:17.168 [I] Progress on: 1.98kit/3.00kit rate:7.1s/it remaining:1:59:27 elapsed:3:58:05 postfix:- (1728:tqdm_logging.py:145) 10:44:31.266 [I] Progress on: 1.99kit/3.00kit rate:7.1s/it remaining:1:59:09 elapsed:3:58:20 postfix:- (1728:tqdm_logging.py:145) 10:44:45.371 [I] Progress on: 1.99kit/3.00kit rate:7.1s/it remaining:1:58:55 elapsed:3:58:34 postfix:- (1728:tqdm_logging.py:145) 10:44:59.496 [I] Progress on: 1.99kit/3.00kit rate:7.1s/it remaining:1:58:47 elapsed:3:58:48 postfix:- (1728:tqdm_logging.py:145) 10:45:13.619 [I] Progress on: 1.99kit/3.00kit rate:7.1s/it remaining:1:58:36 elapsed:3:59:02 postfix:- (1728:tqdm_logging.py:145) 10:45:27.718 [I] Progress on: 1.99kit/3.00kit rate:7.1s/it remaining:1:58:16 elapsed:3:59:16 postfix:- (1728:tqdm_logging.py:145) 10:45:41.844 [I] Progress on: 2.00kit/3.00kit rate:7.1s/it remaining:1:58:06 elapsed:3:59:30 postfix:- (1728:tqdm_logging.py:145) 10:45:56.500 [I] Progress on: 2.00kit/3.00kit rate:7.2s/it remaining:2:00:35 elapsed:3:59:45 postfix:- (1728:tqdm_logging.py:145) 10:46:10.059 [I] Progress on: 2.00kit/3.00kit rate:7.0s/it remaining:1:57:02 elapsed:3:59:58 postfix:- (1728:tqdm_logging.py:145) Step 2000: grad_norm=0.0434, loss=0.0048, param_norm=1804.0712 10:46:23.852 [I] Progress on: 2.00kit/3.00kit rate:7.0s/it remaining:1:57:02 elapsed:4:00:12 postfix:- (1728:tqdm_logging.py:145) 10:46:23.882 [I] [process=0][thread=MainThread][wait_until_finished] No Save Finalize thread to wait for. Returning. (1728:checkpoint_manager.py:1987) 10:46:23.884 [I] [process=0] Saving checkpoint at step 2000 (1728:checkpoint_manager.py:1408) 10:46:23.885 [I] [process=0] Started async saving checkpoint to /workspace/openpi/checkpoints/pi05_mecharm_deep_even10_lora/deep_even10_v1/2000. (1728:async_checkpointer.py:439) 10:46:23.906 [I] Creating tmp directory /workspace/openpi/checkpoints/pi05_mecharm_deep_even10_lora/deep_even10_v1/2000.orbax-checkpoint-tmp-4 (1728:atomicity.py:144) 10:46:23.925 [I] Creating tmp directory /workspace/openpi/checkpoints/pi05_mecharm_deep_even10_lora/deep_even10_v1/2000.orbax-checkpoint-tmp-4/assets.orbax-checkpoint-tmp-5 (1728:atomicity.py:144) 10:46:23.925 [I] Creating tmp directory /workspace/openpi/checkpoints/pi05_mecharm_deep_even10_lora/deep_even10_v1/2000.orbax-checkpoint-tmp-4/params.orbax-checkpoint-tmp-6 (1728:atomicity.py:144) 10:46:23.925 [I] Creating tmp directory /workspace/openpi/checkpoints/pi05_mecharm_deep_even10_lora/deep_even10_v1/2000.orbax-checkpoint-tmp-4/train_state.orbax-checkpoint-tmp-7 (1728:atomicity.py:144) 10:46:23.949 [I] Transferring arrays to host memory with options: use_replica_parallel=True, enable_pinned_host_transfer=False (1728:replica_slices.py:341) 10:46:29.339 [I] Transferring arrays to host memory with options: use_replica_parallel=True, enable_pinned_host_transfer=False (1728:replica_slices.py:341) 10:46:29.347 [I] Array name: 'params.PaliGemma.llm.layers.mlp.gating_einsum.value', global shape: (18, 2, 2048, 16384), write shape: (18, 2, 2048, 16384), chosen chunk shape: (18, 2, 2048, 8192) (1728:tensorstore_utils.py:408) 10:46:29.402 [I] [process=0][thread=array_type_handler] Wrote 71 array_metadata.ArrayMetadata to /workspace/openpi/checkpoints/pi05_mecharm_deep_even10_lora/deep_even10_v1/2000.orbax-checkpoint-tmp-4/params.orbax-checkpoint-tmp-6/array_metadatas/process_0 (1728:array_metadata_store.py:198) 10:46:32.880 [I] [process=0] /jax/checkpoint/write/blocking_bytes_per_sec: 821.2 MiB/s (total bytes: 7.2 GiB) (time elapsed: 8 seconds) (per-host) (1728:base_pytree_checkpoint_handler.py:114) 10:46:32.885 [I] [process=0] /jax/checkpoint/write/blocking_bytes_per_sec: 396.9 MiB/s (total bytes: 3.5 GiB) (time elapsed: 8 seconds) (per-host) (1728:base_pytree_checkpoint_handler.py:114) 10:46:32.890 [I] [process=0][thread=async_save] Background save thread started. (1728:async_checkpointer.py:77) 10:46:32.890 [I] Finished blocking save in 9.01 seconds. Continuing to save asynchronously to /workspace/openpi/checkpoints/pi05_mecharm_deep_even10_lora/deep_even10_v1/2000. (1728:async_checkpointer.py:548) 10:46:32.891 [I] Preserving Checkpoint[step=1000 | time=2026-11-09 08:48:44.411183+00:00]: (Reason: on keep_period=1000). (1728:checkpoint_manager.py:1949) 10:46:32.892 [I] [process=0][thread=MainThread][step=2000] Starting CheckpointManager Save Finalize thread=save_finalize (1728:checkpoint_manager.py:1453) 10:46:32.893 [I] [process=0][thread=save_finalize] Waiting for background save thread=async_save. (1728:async_checkpointer.py:258) 10:46:32.894 [I] {'step': 2000, 'event_type': 'save', 'directory': '/workspace/openpi/checkpoints/pi05_mecharm_deep_even10_lora/deep_even10_v1', 'reached_preemption': False, 'preemption_received_at': None, 'synchronous': False, 'wait_for_prev_start_time': 1794221183.882874, 'wait_for_prev_duration_secs': 0.00022172927856445312, 'checkpointer_blocking_start_time': 1794221183.8842478, 'checkpointer_blocking_duration_secs': 9.006876945495605, 'get_old_steps_start_time': 1794221192.8911457, 'get_old_steps_duration_secs': 0.0008509159088134766, 'checkpoint_manager_blocking_start_time': 1794221183.8821158, 'checkpoint_manager_blocking_duration_secs': 9.011928796768188} (1728:standard_logger.py:34) 10:46:32.976 [I] [process=0][thread=array_type_handler] Wrote 105 array_metadata.ArrayMetadata to /workspace/openpi/checkpoints/pi05_mecharm_deep_even10_lora/deep_even10_v1/2000.orbax-checkpoint-tmp-4/train_state.orbax-checkpoint-tmp-7/array_metadatas/process_0 (1728:array_metadata_store.py:198) 10:46:35.022 [I] [process=0] /jax/checkpoint/write/bytes_per_sec: 663.2 MiB/s (total bytes: 7.2 GiB) (time elapsed: 11 seconds) (per-host) (1728:base_pytree_checkpoint_handler.py:114) 10:46:35.991 [I] [process=0] /jax/checkpoint/write/bytes_per_sec: 294.9 MiB/s (total bytes: 3.5 GiB) (time elapsed: 12 seconds) (per-host) (1728:base_pytree_checkpoint_handler.py:114) 10:46:35.991 [I] [process=0][thread=async_save] 5 Handler Commit operations completed. (1728:async_checkpointer.py:87) 10:46:35.992 [I] Renaming /workspace/openpi/checkpoints/pi05_mecharm_deep_even10_lora/deep_even10_v1/2000.orbax-checkpoint-tmp-4/assets.orbax-checkpoint-tmp-5 to /workspace/openpi/checkpoints/pi05_mecharm_deep_even10_lora/deep_even10_v1/2000.orbax-checkpoint-tmp-4/assets (1728:atomicity.py:289) 10:46:35.997 [I] [process=0][thread=async_save] Skipped cross-host ArrayMetadata validation because only one process is found: process_index=0. (1728:array_metadata_store.py:362) 10:46:36.004 [I] Renaming /workspace/openpi/checkpoints/pi05_mecharm_deep_even10_lora/deep_even10_v1/2000.orbax-checkpoint-tmp-4/train_state.orbax-checkpoint-tmp-7 to /workspace/openpi/checkpoints/pi05_mecharm_deep_even10_lora/deep_even10_v1/2000.orbax-checkpoint-tmp-4/train_state (1728:atomicity.py:289) 10:46:36.008 [I] [process=0][thread=async_save] Skipped cross-host ArrayMetadata validation because only one process is found: process_index=0. (1728:array_metadata_store.py:362) 10:46:36.014 [I] Renaming /workspace/openpi/checkpoints/pi05_mecharm_deep_even10_lora/deep_even10_v1/2000.orbax-checkpoint-tmp-4/params.orbax-checkpoint-tmp-6 to /workspace/openpi/checkpoints/pi05_mecharm_deep_even10_lora/deep_even10_v1/2000.orbax-checkpoint-tmp-4/params (1728:atomicity.py:289) 10:46:36.015 [I] Renaming /workspace/openpi/checkpoints/pi05_mecharm_deep_even10_lora/deep_even10_v1/2000.orbax-checkpoint-tmp-4 to /workspace/openpi/checkpoints/pi05_mecharm_deep_even10_lora/deep_even10_v1/2000 (1728:atomicity.py:289) 10:46:36.018 [I] [process=0][thread=async_save] Finished saving checkpoint (finalized tmp dir) to `/workspace/openpi/checkpoints/pi05_mecharm_deep_even10_lora/deep_even10_v1/2000`. (1728:atomicity.py:573) 10:46:36.018 [I] Finished asynchronous save (blocking + background) in 12.13 seconds to /workspace/openpi/checkpoints/pi05_mecharm_deep_even10_lora/deep_even10_v1/2000 (1728:async_checkpointer.py:412) 10:46:36.018 [I] [process=0][thread=async_save] Background save thread done. (1728:async_checkpointer.py:143) 10:46:36.019 [I] [process=0][thread=save_finalize] Done with waiting for background save thread=async_save. (1728:async_checkpointer.py:266) 10:46:36.019 [I] [process=0][thread=save_finalize] No errors found in background save thread=async_save. (1728:async_checkpointer.py:276) 10:46:36.019 [I] [process=0][thread=save_finalize][step=2000] CheckpointManager Save Finalize is syncing with other hosts... (1728:checkpoint_manager.py:2095) 10:46:36.019 [I] [process=0][thread=save_finalize][step=2000] CheckpointManager Save Finalize is done on all hosts. (1728:checkpoint_manager.py:2110) 10:46:40.313 [I] Progress on: 2.00kit/3.00kit rate:7.9s/it remaining:2:11:43 elapsed:4:00:29 postfix:- (1728:tqdm_logging.py:145) 10:46:54.365 [I] Progress on: 2.00kit/3.00kit rate:7.5s/it remaining:2:03:48 elapsed:4:00:43 postfix:- (1728:tqdm_logging.py:145) 10:47:08.437 [I] Progress on: 2.01kit/3.00kit rate:7.2s/it remaining:1:59:52 elapsed:4:00:57 postfix:- (1728:tqdm_logging.py:145) 10:47:22.534 [I] Progress on: 2.01kit/3.00kit rate:7.1s/it remaining:1:57:57 elapsed:4:01:11 postfix:- (1728:tqdm_logging.py:145) 10:47:36.665 [I] Progress on: 2.01kit/3.00kit rate:7.1s/it remaining:1:57:03 elapsed:4:01:25 postfix:- (1728:tqdm_logging.py:145) 10:47:50.782 [I] Progress on: 2.01kit/3.00kit rate:7.1s/it remaining:1:56:27 elapsed:4:01:39 postfix:- (1728:tqdm_logging.py:145) 10:48:04.895 [I] Progress on: 2.02kit/3.00kit rate:7.1s/it remaining:1:56:02 elapsed:4:01:53 postfix:- (1728:tqdm_logging.py:145) 10:48:19.031 [I] Progress on: 2.02kit/3.00kit rate:7.1s/it remaining:1:55:47 elapsed:4:02:07 postfix:- (1728:tqdm_logging.py:145) 10:48:33.145 [I] Progress on: 2.02kit/3.00kit rate:7.1s/it remaining:1:55:28 elapsed:4:02:21 postfix:- (1728:tqdm_logging.py:145) 10:48:47.257 [I] Progress on: 2.02kit/3.00kit rate:7.1s/it remaining:1:55:10 elapsed:4:02:36 postfix:- (1728:tqdm_logging.py:145) 10:49:01.388 [I] Progress on: 2.02kit/3.00kit rate:7.1s/it remaining:1:54:59 elapsed:4:02:50 postfix:- (1728:tqdm_logging.py:145) 10:49:15.512 [I] Progress on: 2.02kit/3.00kit rate:7.1s/it remaining:1:54:44 elapsed:4:03:04 postfix:- (1728:tqdm_logging.py:145) 10:49:29.628 [I] Progress on: 2.03kit/3.00kit rate:7.1s/it remaining:1:54:29 elapsed:4:03:18 postfix:- (1728:tqdm_logging.py:145) 10:49:43.757 [I] Progress on: 2.03kit/3.00kit rate:7.1s/it remaining:1:54:17 elapsed:4:03:32 postfix:- (1728:tqdm_logging.py:145) 10:49:57.890 [I] Progress on: 2.03kit/3.00kit rate:7.1s/it remaining:1:54:05 elapsed:4:03:46 postfix:- (1728:tqdm_logging.py:145) 10:50:12.021 [I] Progress on: 2.03kit/3.00kit rate:7.1s/it remaining:1:53:51 elapsed:4:04:00 postfix:- (1728:tqdm_logging.py:145) 10:50:26.147 [I] Progress on: 2.04kit/3.00kit rate:7.1s/it remaining:1:53:37 elapsed:4:04:14 postfix:- (1728:tqdm_logging.py:145) 10:50:40.286 [I] Progress on: 2.04kit/3.00kit rate:7.1s/it remaining:1:53:25 elapsed:4:04:29 postfix:- (1728:tqdm_logging.py:145) 10:50:54.408 [I] Progress on: 2.04kit/3.00kit rate:7.1s/it remaining:1:53:08 elapsed:4:04:43 postfix:- (1728:tqdm_logging.py:145) 10:51:08.538 [I] Progress on: 2.04kit/3.00kit rate:7.1s/it remaining:1:52:55 elapsed:4:04:57 postfix:- (1728:tqdm_logging.py:145) 10:51:22.660 [I] Progress on: 2.04kit/3.00kit rate:7.1s/it remaining:1:52:39 elapsed:4:05:11 postfix:- (1728:tqdm_logging.py:145) 10:51:36.773 [I] Progress on: 2.04kit/3.00kit rate:7.1s/it remaining:1:52:22 elapsed:4:05:25 postfix:- (1728:tqdm_logging.py:145) 10:51:50.918 [I] Progress on: 2.05kit/3.00kit rate:7.1s/it remaining:1:52:14 elapsed:4:05:39 postfix:- (1728:tqdm_logging.py:145) 10:52:05.033 [I] Progress on: 2.05kit/3.00kit rate:7.1s/it remaining:1:51:55 elapsed:4:05:53 postfix:- (1728:tqdm_logging.py:145) Step 2050: grad_norm=0.0472, loss=0.0055, param_norm=1804.0768 10:52:25.853 [I] Progress on: 2.05kit/3.00kit rate:7.1s/it remaining:1:51:45 elapsed:4:06:14 postfix:- (1728:tqdm_logging.py:145) 10:52:40.436 [I] Progress on: 2.05kit/3.00kit rate:6.8s/it remaining:1:46:54 elapsed:4:06:29 postfix:- (1728:tqdm_logging.py:145) 10:52:54.553 [I] Progress on: 2.06kit/3.00kit rate:6.9s/it remaining:1:48:53 elapsed:4:06:43 postfix:- (1728:tqdm_logging.py:145) 10:53:08.695 [I] Progress on: 2.06kit/3.00kit rate:7.0s/it remaining:1:49:52 elapsed:4:06:57 postfix:- (1728:tqdm_logging.py:145) 10:53:22.827 [I] Progress on: 2.06kit/3.00kit rate:7.0s/it remaining:1:50:10 elapsed:4:07:11 postfix:- (1728:tqdm_logging.py:145) 10:53:36.965 [I] Progress on: 2.06kit/3.00kit rate:7.1s/it remaining:1:50:13 elapsed:4:07:25 postfix:- (1728:tqdm_logging.py:145) 10:53:51.120 [I] Progress on: 2.06kit/3.00kit rate:7.1s/it remaining:1:50:12 elapsed:4:07:39 postfix:- (1728:tqdm_logging.py:145) 10:54:05.247 [I] Progress on: 2.07kit/3.00kit rate:7.1s/it remaining:1:49:57 elapsed:4:07:54 postfix:- (1728:tqdm_logging.py:145) 10:54:19.371 [I] Progress on: 2.07kit/3.00kit rate:7.1s/it remaining:1:49:42 elapsed:4:08:08 postfix:- (1728:tqdm_logging.py:145) 10:54:33.495 [I] Progress on: 2.07kit/3.00kit rate:7.1s/it remaining:1:49:28 elapsed:4:08:22 postfix:- (1728:tqdm_logging.py:145) 10:54:47.631 [I] Progress on: 2.07kit/3.00kit rate:7.1s/it remaining:1:49:16 elapsed:4:08:36 postfix:- (1728:tqdm_logging.py:145) 10:55:01.764 [I] Progress on: 2.07kit/3.00kit rate:7.1s/it remaining:1:49:02 elapsed:4:08:50 postfix:- (1728:tqdm_logging.py:145) 10:55:15.903 [I] Progress on: 2.08kit/3.00kit rate:7.1s/it remaining:1:48:50 elapsed:4:09:04 postfix:- (1728:tqdm_logging.py:145) 10:55:30.029 [I] Progress on: 2.08kit/3.00kit rate:7.1s/it remaining:1:48:34 elapsed:4:09:18 postfix:- (1728:tqdm_logging.py:145) 10:55:44.152 [I] Progress on: 2.08kit/3.00kit rate:7.1s/it remaining:1:48:18 elapsed:4:09:32 postfix:- (1728:tqdm_logging.py:145) 10:55:58.262 [I] Progress on: 2.08kit/3.00kit rate:7.1s/it remaining:1:48:00 elapsed:4:09:47 postfix:- (1728:tqdm_logging.py:145) 10:56:12.400 [I] Progress on: 2.08kit/3.00kit rate:7.1s/it remaining:1:47:50 elapsed:4:10:01 postfix:- (1728:tqdm_logging.py:145) 10:56:26.530 [I] Progress on: 2.09kit/3.00kit rate:7.1s/it remaining:1:47:37 elapsed:4:10:15 postfix:- (1728:tqdm_logging.py:145) 10:56:40.658 [I] Progress on: 2.09kit/3.00kit rate:7.1s/it remaining:1:47:23 elapsed:4:10:29 postfix:- (1728:tqdm_logging.py:145) 10:56:54.761 [I] Progress on: 2.09kit/3.00kit rate:7.1s/it remaining:1:47:04 elapsed:4:10:43 postfix:- (1728:tqdm_logging.py:145) 10:57:08.864 [I] Progress on: 2.09kit/3.00kit rate:7.1s/it remaining:1:46:46 elapsed:4:10:57 postfix:- (1728:tqdm_logging.py:145) 10:57:22.983 [I] Progress on: 2.09kit/3.00kit rate:7.1s/it remaining:1:46:34 elapsed:4:11:11 postfix:- (1728:tqdm_logging.py:145) 10:57:37.089 [I] Progress on: 2.10kit/3.00kit rate:7.1s/it remaining:1:46:18 elapsed:4:11:25 postfix:- (1728:tqdm_logging.py:145) 10:57:51.210 [I] Progress on: 2.10kit/3.00kit rate:7.1s/it remaining:1:46:06 elapsed:4:11:39 postfix:- (1728:tqdm_logging.py:145) 10:58:05.333 [I] Progress on: 2.10kit/3.00kit rate:7.1s/it remaining:1:45:53 elapsed:4:11:54 postfix:- (1728:tqdm_logging.py:145) Step 2100: grad_norm=0.0436, loss=0.0047, param_norm=1804.0822 10:58:19.083 [I] Progress on: 2.10kit/3.00kit rate:7.1s/it remaining:1:45:53 elapsed:4:12:07 postfix:- (1728:tqdm_logging.py:145) 10:58:33.636 [I] Progress on: 2.10kit/3.00kit rate:6.8s/it remaining:1:41:06 elapsed:4:12:22 postfix:- (1728:tqdm_logging.py:145) 10:58:47.727 [I] Progress on: 2.11kit/3.00kit rate:6.9s/it remaining:1:42:58 elapsed:4:12:36 postfix:- (1728:tqdm_logging.py:145) 10:59:01.848 [I] Progress on: 2.11kit/3.00kit rate:7.0s/it remaining:1:43:52 elapsed:4:12:50 postfix:- (1728:tqdm_logging.py:145) 10:59:15.947 [I] Progress on: 2.11kit/3.00kit rate:7.0s/it remaining:1:44:06 elapsed:4:13:04 postfix:- (1728:tqdm_logging.py:145) 10:59:30.039 [I] Progress on: 2.11kit/3.00kit rate:7.0s/it remaining:1:44:04 elapsed:4:13:18 postfix:- (1728:tqdm_logging.py:145) 10:59:44.157 [I] Progress on: 2.11kit/3.00kit rate:7.0s/it remaining:1:44:01 elapsed:4:13:32 postfix:- (1728:tqdm_logging.py:145) 10:59:58.262 [I] Progress on: 2.12kit/3.00kit rate:7.0s/it remaining:1:43:50 elapsed:4:13:47 postfix:- (1728:tqdm_logging.py:145) 11:00:12.371 [I] Progress on: 2.12kit/3.00kit rate:7.1s/it remaining:1:43:39 elapsed:4:14:01 postfix:- (1728:tqdm_logging.py:145) 11:00:26.482 [I] Progress on: 2.12kit/3.00kit rate:7.1s/it remaining:1:43:26 elapsed:4:14:15 postfix:- (1728:tqdm_logging.py:145) 11:00:40.583 [I] Progress on: 2.12kit/3.00kit rate:7.1s/it remaining:1:43:11 elapsed:4:14:29 postfix:- (1728:tqdm_logging.py:145) 11:00:54.687 [I] Progress on: 2.12kit/3.00kit rate:7.1s/it remaining:1:42:56 elapsed:4:14:43 postfix:- (1728:tqdm_logging.py:145) 11:01:08.792 [I] Progress on: 2.13kit/3.00kit rate:7.1s/it remaining:1:42:43 elapsed:4:14:57 postfix:- (1728:tqdm_logging.py:145) 11:01:22.897 [I] Progress on: 2.13kit/3.00kit rate:7.1s/it remaining:1:42:28 elapsed:4:15:11 postfix:- (1728:tqdm_logging.py:145) 11:01:37.002 [I] Progress on: 2.13kit/3.00kit rate:7.1s/it remaining:1:42:15 elapsed:4:15:25 postfix:- (1728:tqdm_logging.py:145) 11:01:51.103 [I] Progress on: 2.13kit/3.00kit rate:7.1s/it remaining:1:42:00 elapsed:4:15:39 postfix:- (1728:tqdm_logging.py:145) 11:02:05.215 [I] Progress on: 2.13kit/3.00kit rate:7.1s/it remaining:1:41:47 elapsed:4:15:54 postfix:- (1728:tqdm_logging.py:145) 11:02:19.325 [I] Progress on: 2.14kit/3.00kit rate:7.1s/it remaining:1:41:34 elapsed:4:16:08 postfix:- (1728:tqdm_logging.py:145) 11:02:33.420 [I] Progress on: 2.14kit/3.00kit rate:7.1s/it remaining:1:41:17 elapsed:4:16:22 postfix:- (1728:tqdm_logging.py:145) 11:02:47.516 [I] Progress on: 2.14kit/3.00kit rate:7.0s/it remaining:1:41:02 elapsed:4:16:36 postfix:- (1728:tqdm_logging.py:145) 11:03:01.634 [I] Progress on: 2.14kit/3.00kit rate:7.1s/it remaining:1:40:52 elapsed:4:16:50 postfix:- (1728:tqdm_logging.py:145) 11:03:15.724 [I] Progress on: 2.14kit/3.00kit rate:7.1s/it remaining:1:40:35 elapsed:4:17:04 postfix:- (1728:tqdm_logging.py:145) 11:03:29.818 [I] Progress on: 2.15kit/3.00kit rate:7.0s/it remaining:1:40:19 elapsed:4:17:18 postfix:- (1728:tqdm_logging.py:145) 11:03:43.920 [I] Progress on: 2.15kit/3.00kit rate:7.1s/it remaining:1:40:07 elapsed:4:17:32 postfix:- (1728:tqdm_logging.py:145) 11:03:58.031 [I] Progress on: 2.15kit/3.00kit rate:7.1s/it remaining:1:39:55 elapsed:4:17:46 postfix:- (1728:tqdm_logging.py:145) Step 2150: grad_norm=0.0452, loss=0.0046, param_norm=1804.0868 11:04:11.794 [I] Progress on: 2.15kit/3.00kit rate:7.1s/it remaining:1:39:55 elapsed:4:18:00 postfix:- (1728:tqdm_logging.py:145) 11:04:26.318 [I] Progress on: 2.15kit/3.00kit rate:6.8s/it remaining:1:35:25 elapsed:4:18:15 postfix:- (1728:tqdm_logging.py:145) 11:04:40.409 [I] Progress on: 2.16kit/3.00kit rate:6.9s/it remaining:1:37:11 elapsed:4:18:29 postfix:- (1728:tqdm_logging.py:145) 11:04:54.530 [I] Progress on: 2.16kit/3.00kit rate:7.0s/it remaining:1:38:03 elapsed:4:18:43 postfix:- (1728:tqdm_logging.py:145) 11:05:08.631 [I] Progress on: 2.16kit/3.00kit rate:7.0s/it remaining:1:38:15 elapsed:4:18:57 postfix:- (1728:tqdm_logging.py:145) 11:05:22.734 [I] Progress on: 2.16kit/3.00kit rate:7.0s/it remaining:1:38:16 elapsed:4:19:11 postfix:- (1728:tqdm_logging.py:145) 11:05:36.847 [I] Progress on: 2.16kit/3.00kit rate:7.0s/it remaining:1:38:11 elapsed:4:19:25 postfix:- (1728:tqdm_logging.py:145) 11:05:50.952 [I] Progress on: 2.17kit/3.00kit rate:7.1s/it remaining:1:38:00 elapsed:4:19:39 postfix:- (1728:tqdm_logging.py:145) 11:06:05.049 [I] Progress on: 2.17kit/3.00kit rate:7.0s/it remaining:1:37:45 elapsed:4:19:53 postfix:- (1728:tqdm_logging.py:145) 11:06:19.145 [I] Progress on: 2.17kit/3.00kit rate:7.0s/it remaining:1:37:31 elapsed:4:20:07 postfix:- (1728:tqdm_logging.py:145) 11:06:33.270 [I] Progress on: 2.17kit/3.00kit rate:7.1s/it remaining:1:37:23 elapsed:4:20:22 postfix:- (1728:tqdm_logging.py:145) 11:06:47.377 [I] Progress on: 2.17kit/3.00kit rate:7.1s/it remaining:1:37:07 elapsed:4:20:36 postfix:- (1728:tqdm_logging.py:145) 11:07:01.484 [I] Progress on: 2.18kit/3.00kit rate:7.1s/it remaining:1:36:52 elapsed:4:20:50 postfix:- (1728:tqdm_logging.py:145) 11:07:15.596 [I] Progress on: 2.18kit/3.00kit rate:7.1s/it remaining:1:36:39 elapsed:4:21:04 postfix:- (1728:tqdm_logging.py:145) 11:07:29.688 [I] Progress on: 2.18kit/3.00kit rate:7.1s/it remaining:1:36:21 elapsed:4:21:18 postfix:- (1728:tqdm_logging.py:145) 11:07:43.792 [I] Progress on: 2.18kit/3.00kit rate:7.1s/it remaining:1:36:08 elapsed:4:21:32 postfix:- (1728:tqdm_logging.py:145) 11:07:57.889 [I] Progress on: 2.18kit/3.00kit rate:7.1s/it remaining:1:35:53 elapsed:4:21:46 postfix:- (1728:tqdm_logging.py:145) 11:08:11.999 [I] Progress on: 2.19kit/3.00kit rate:7.1s/it remaining:1:35:41 elapsed:4:22:00 postfix:- (1728:tqdm_logging.py:145) 11:08:26.099 [I] Progress on: 2.19kit/3.00kit rate:7.1s/it remaining:1:35:26 elapsed:4:22:14 postfix:- (1728:tqdm_logging.py:145) 11:08:40.195 [I] Progress on: 2.19kit/3.00kit rate:7.1s/it remaining:1:35:11 elapsed:4:22:28 postfix:- (1728:tqdm_logging.py:145) 11:08:54.309 [I] Progress on: 2.19kit/3.00kit rate:7.1s/it remaining:1:35:00 elapsed:4:22:43 postfix:- (1728:tqdm_logging.py:145) 11:09:08.433 [I] Progress on: 2.19kit/3.00kit rate:7.1s/it remaining:1:34:49 elapsed:4:22:57 postfix:- (1728:tqdm_logging.py:145) 11:09:22.526 [I] Progress on: 2.20kit/3.00kit rate:7.1s/it remaining:1:34:30 elapsed:4:23:11 postfix:- (1728:tqdm_logging.py:145) 11:09:36.633 [I] Progress on: 2.20kit/3.00kit rate:7.1s/it remaining:1:34:16 elapsed:4:23:25 postfix:- (1728:tqdm_logging.py:145) 11:09:50.743 [I] Progress on: 2.20kit/3.00kit rate:7.1s/it remaining:1:34:03 elapsed:4:23:39 postfix:- (1728:tqdm_logging.py:145) Step 2200: grad_norm=0.0434, loss=0.0047, param_norm=1804.0909 11:10:04.480 [I] Progress on: 2.20kit/3.00kit rate:7.1s/it remaining:1:34:03 elapsed:4:23:53 postfix:- (1728:tqdm_logging.py:145) 11:10:18.992 [I] Progress on: 2.20kit/3.00kit rate:6.8s/it remaining:1:29:43 elapsed:4:24:07 postfix:- (1728:tqdm_logging.py:145) 11:10:33.108 [I] Progress on: 2.21kit/3.00kit rate:6.9s/it remaining:1:31:29 elapsed:4:24:21 postfix:- (1728:tqdm_logging.py:145) 11:10:47.227 [I] Progress on: 2.21kit/3.00kit rate:7.0s/it remaining:1:32:14 elapsed:4:24:36 postfix:- (1728:tqdm_logging.py:145) 11:11:01.336 [I] Progress on: 2.21kit/3.00kit rate:7.0s/it remaining:1:32:27 elapsed:4:24:50 postfix:- (1728:tqdm_logging.py:145) 11:11:15.439 [I] Progress on: 2.21kit/3.00kit rate:7.0s/it remaining:1:32:25 elapsed:4:25:04 postfix:- (1728:tqdm_logging.py:145) 11:11:29.564 [I] Progress on: 2.21kit/3.00kit rate:7.0s/it remaining:1:32:21 elapsed:4:25:18 postfix:- (1728:tqdm_logging.py:145) 11:11:43.674 [I] Progress on: 2.22kit/3.00kit rate:7.1s/it remaining:1:32:09 elapsed:4:25:32 postfix:- (1728:tqdm_logging.py:145) 11:11:57.794 [I] Progress on: 2.22kit/3.00kit rate:7.1s/it remaining:1:31:58 elapsed:4:25:46 postfix:- (1728:tqdm_logging.py:145) 11:12:12.404 [I] Progress on: 2.22kit/3.00kit rate:7.2s/it remaining:1:33:40 elapsed:4:26:01 postfix:- (1728:tqdm_logging.py:145) 11:12:26.010 [I] Progress on: 2.22kit/3.00kit rate:7.0s/it remaining:1:31:06 elapsed:4:26:14 postfix:- (1728:tqdm_logging.py:145) 11:12:40.129 [I] Progress on: 2.22kit/3.00kit rate:7.0s/it remaining:1:31:08 elapsed:4:26:28 postfix:- (1728:tqdm_logging.py:145) 11:12:54.240 [I] Progress on: 2.23kit/3.00kit rate:7.1s/it remaining:1:30:59 elapsed:4:26:43 postfix:- (1728:tqdm_logging.py:145) 11:13:08.357 [I] Progress on: 2.23kit/3.00kit rate:7.1s/it remaining:1:30:49 elapsed:4:26:57 postfix:- (1728:tqdm_logging.py:145) 11:13:22.472 [I] Progress on: 2.23kit/3.00kit rate:7.1s/it remaining:1:30:38 elapsed:4:27:11 postfix:- (1728:tqdm_logging.py:145) 11:13:36.573 [I] Progress on: 2.23kit/3.00kit rate:7.1s/it remaining:1:30:22 elapsed:4:27:25 postfix:- (1728:tqdm_logging.py:145) 11:13:50.664 [I] Progress on: 2.23kit/3.00kit rate:7.1s/it remaining:1:30:05 elapsed:4:27:39 postfix:- (1728:tqdm_logging.py:145) 11:14:04.779 [I] Progress on: 2.24kit/3.00kit rate:7.1s/it remaining:1:29:54 elapsed:4:27:53 postfix:- (1728:tqdm_logging.py:145) 11:14:18.895 [I] Progress on: 2.24kit/3.00kit rate:7.1s/it remaining:1:29:42 elapsed:4:28:07 postfix:- (1728:tqdm_logging.py:145) 11:14:33.010 [I] Progress on: 2.24kit/3.00kit rate:7.1s/it remaining:1:29:28 elapsed:4:28:21 postfix:- (1728:tqdm_logging.py:145) 11:14:47.111 [I] Progress on: 2.24kit/3.00kit rate:7.1s/it remaining:1:29:11 elapsed:4:28:35 postfix:- (1728:tqdm_logging.py:145) 11:15:01.233 [I] Progress on: 2.24kit/3.00kit rate:7.1s/it remaining:1:29:00 elapsed:4:28:50 postfix:- (1728:tqdm_logging.py:145) 11:15:15.359 [I] Progress on: 2.25kit/3.00kit rate:7.1s/it remaining:1:28:48 elapsed:4:29:04 postfix:- (1728:tqdm_logging.py:145) 11:15:29.454 [I] Progress on: 2.25kit/3.00kit rate:7.1s/it remaining:1:28:28 elapsed:4:29:18 postfix:- (1728:tqdm_logging.py:145) 11:15:43.542 [I] Progress on: 2.25kit/3.00kit rate:7.1s/it remaining:1:28:07 elapsed:4:29:32 postfix:- (1728:tqdm_logging.py:145) Step 2250: grad_norm=0.0419, loss=0.0046, param_norm=1804.0946 11:15:57.294 [I] Progress on: 2.25kit/3.00kit rate:7.1s/it remaining:1:28:07 elapsed:4:29:46 postfix:- (1728:tqdm_logging.py:145) 11:16:11.826 [I] Progress on: 2.25kit/3.00kit rate:6.8s/it remaining:1:24:06 elapsed:4:30:00 postfix:- (1728:tqdm_logging.py:145) 11:16:25.986 [I] Progress on: 2.26kit/3.00kit rate:6.9s/it remaining:1:25:55 elapsed:4:30:14 postfix:- (1728:tqdm_logging.py:145) 11:16:40.121 [I] Progress on: 2.26kit/3.00kit rate:7.0s/it remaining:1:26:36 elapsed:4:30:28 postfix:- (1728:tqdm_logging.py:145) 11:16:54.166 [I] Progress on: 2.26kit/3.00kit rate:7.0s/it remaining:1:26:29 elapsed:4:30:42 postfix:- (1728:tqdm_logging.py:145) 11:17:08.265 [I] Progress on: 2.26kit/3.00kit rate:7.0s/it remaining:1:26:29 elapsed:4:30:57 postfix:- (1728:tqdm_logging.py:145) 11:17:22.435 [I] Progress on: 2.26kit/3.00kit rate:7.1s/it remaining:1:26:34 elapsed:4:31:11 postfix:- (1728:tqdm_logging.py:145) 11:17:36.480 [I] Progress on: 2.27kit/3.00kit rate:7.0s/it remaining:1:26:08 elapsed:4:31:25 postfix:- (1728:tqdm_logging.py:145) 11:17:50.581 [I] Progress on: 2.27kit/3.00kit rate:7.0s/it remaining:1:25:57 elapsed:4:31:39 postfix:- (1728:tqdm_logging.py:145) 11:18:04.690 [I] Progress on: 2.27kit/3.00kit rate:7.1s/it remaining:1:25:46 elapsed:4:31:53 postfix:- (1728:tqdm_logging.py:145) 11:18:18.817 [I] Progress on: 2.27kit/3.00kit rate:7.1s/it remaining:1:25:37 elapsed:4:32:07 postfix:- (1728:tqdm_logging.py:145) 11:18:32.902 [I] Progress on: 2.27kit/3.00kit rate:7.0s/it remaining:1:25:18 elapsed:4:32:21 postfix:- (1728:tqdm_logging.py:145) 11:18:47.014 [I] Progress on: 2.28kit/3.00kit rate:7.1s/it remaining:1:25:06 elapsed:4:32:35 postfix:- (1728:tqdm_logging.py:145) 11:19:01.141 [I] Progress on: 2.28kit/3.00kit rate:7.1s/it remaining:1:24:56 elapsed:4:32:49 postfix:- (1728:tqdm_logging.py:145) 11:19:15.249 [I] Progress on: 2.28kit/3.00kit rate:7.1s/it remaining:1:24:41 elapsed:4:33:04 postfix:- (1728:tqdm_logging.py:145) 11:19:29.356 [I] Progress on: 2.28kit/3.00kit rate:7.1s/it remaining:1:24:26 elapsed:4:33:18 postfix:- (1728:tqdm_logging.py:145) 11:19:43.447 [I] Progress on: 2.28kit/3.00kit rate:7.1s/it remaining:1:24:08 elapsed:4:33:32 postfix:- (1728:tqdm_logging.py:145) 11:19:57.557 [I] Progress on: 2.29kit/3.00kit rate:7.1s/it remaining:1:23:55 elapsed:4:33:46 postfix:- (1728:tqdm_logging.py:145) 11:20:11.676 [I] Progress on: 2.29kit/3.00kit rate:7.1s/it remaining:1:23:44 elapsed:4:34:00 postfix:- (1728:tqdm_logging.py:145) 11:20:25.796 [I] Progress on: 2.29kit/3.00kit rate:7.1s/it remaining:1:23:30 elapsed:4:34:14 postfix:- (1728:tqdm_logging.py:145) 11:20:39.911 [I] Progress on: 2.29kit/3.00kit rate:7.1s/it remaining:1:23:16 elapsed:4:34:28 postfix:- (1728:tqdm_logging.py:145) 11:20:54.033 [I] Progress on: 2.29kit/3.00kit rate:7.1s/it remaining:1:23:04 elapsed:4:34:42 postfix:- (1728:tqdm_logging.py:145) 11:21:08.141 [I] Progress on: 2.30kit/3.00kit rate:7.1s/it remaining:1:22:48 elapsed:4:34:56 postfix:- (1728:tqdm_logging.py:145) 11:21:22.258 [I] Progress on: 2.30kit/3.00kit rate:7.1s/it remaining:1:22:35 elapsed:4:35:11 postfix:- (1728:tqdm_logging.py:145) 11:21:36.359 [I] Progress on: 2.30kit/3.00kit rate:7.1s/it remaining:1:22:19 elapsed:4:35:25 postfix:- (1728:tqdm_logging.py:145) Step 2300: grad_norm=0.0421, loss=0.0047, param_norm=1804.0990 11:21:50.098 [I] Progress on: 2.30kit/3.00kit rate:7.1s/it remaining:1:22:19 elapsed:4:35:38 postfix:- (1728:tqdm_logging.py:145) 11:22:04.611 [I] Progress on: 2.30kit/3.00kit rate:6.8s/it remaining:1:18:27 elapsed:4:35:53 postfix:- (1728:tqdm_logging.py:145) 11:22:18.718 [I] Progress on: 2.31kit/3.00kit rate:6.9s/it remaining:1:19:56 elapsed:4:36:07 postfix:- (1728:tqdm_logging.py:145) 11:22:32.840 [I] Progress on: 2.31kit/3.00kit rate:7.0s/it remaining:1:20:35 elapsed:4:36:21 postfix:- (1728:tqdm_logging.py:145) 11:22:46.950 [I] Progress on: 2.31kit/3.00kit rate:7.0s/it remaining:1:20:45 elapsed:4:36:35 postfix:- (1728:tqdm_logging.py:145) 11:23:01.051 [I] Progress on: 2.31kit/3.00kit rate:7.0s/it remaining:1:20:40 elapsed:4:36:49 postfix:- (1728:tqdm_logging.py:145) 11:23:15.171 [I] Progress on: 2.31kit/3.00kit rate:7.0s/it remaining:1:20:35 elapsed:4:37:03 postfix:- (1728:tqdm_logging.py:145) 11:23:29.291 [I] Progress on: 2.32kit/3.00kit rate:7.1s/it remaining:1:20:25 elapsed:4:37:18 postfix:- (1728:tqdm_logging.py:145) 11:23:43.394 [I] Progress on: 2.32kit/3.00kit rate:7.1s/it remaining:1:20:11 elapsed:4:37:32 postfix:- (1728:tqdm_logging.py:145) 11:23:57.505 [I] Progress on: 2.32kit/3.00kit rate:7.1s/it remaining:1:19:57 elapsed:4:37:46 postfix:- (1728:tqdm_logging.py:145) 11:24:11.627 [I] Progress on: 2.32kit/3.00kit rate:7.1s/it remaining:1:19:46 elapsed:4:38:00 postfix:- (1728:tqdm_logging.py:145) 11:24:25.750 [I] Progress on: 2.32kit/3.00kit rate:7.1s/it remaining:1:19:33 elapsed:4:38:14 postfix:- (1728:tqdm_logging.py:145) 11:24:39.857 [I] Progress on: 2.33kit/3.00kit rate:7.1s/it remaining:1:19:16 elapsed:4:38:28 postfix:- (1728:tqdm_logging.py:145) 11:24:53.964 [I] Progress on: 2.33kit/3.00kit rate:7.1s/it remaining:1:19:00 elapsed:4:38:42 postfix:- (1728:tqdm_logging.py:145) 11:25:08.087 [I] Progress on: 2.33kit/3.00kit rate:7.1s/it remaining:1:18:48 elapsed:4:38:56 postfix:- (1728:tqdm_logging.py:145) 11:25:22.214 [I] Progress on: 2.33kit/3.00kit rate:7.1s/it remaining:1:18:36 elapsed:4:39:11 postfix:- (1728:tqdm_logging.py:145) 11:25:36.301 [I] Progress on: 2.33kit/3.00kit rate:7.1s/it remaining:1:18:16 elapsed:4:39:25 postfix:- (1728:tqdm_logging.py:145) 11:25:50.412 [I] Progress on: 2.34kit/3.00kit rate:7.1s/it remaining:1:18:03 elapsed:4:39:39 postfix:- (1728:tqdm_logging.py:145) 11:26:04.522 [I] Progress on: 2.34kit/3.00kit rate:7.1s/it remaining:1:17:49 elapsed:4:39:53 postfix:- (1728:tqdm_logging.py:145) 11:26:18.694 [I] Progress on: 2.34kit/3.00kit rate:7.1s/it remaining:1:17:43 elapsed:4:40:07 postfix:- (1728:tqdm_logging.py:145) 11:26:32.807 [I] Progress on: 2.34kit/3.00kit rate:7.1s/it remaining:1:17:27 elapsed:4:40:21 postfix:- (1728:tqdm_logging.py:145) 11:26:46.892 [I] Progress on: 2.34kit/3.00kit rate:7.1s/it remaining:1:17:07 elapsed:4:40:35 postfix:- (1728:tqdm_logging.py:145) 11:27:00.977 [I] Progress on: 2.35kit/3.00kit rate:7.0s/it remaining:1:16:49 elapsed:4:40:49 postfix:- (1728:tqdm_logging.py:145) 11:27:15.078 [I] Progress on: 2.35kit/3.00kit rate:7.0s/it remaining:1:16:36 elapsed:4:41:03 postfix:- (1728:tqdm_logging.py:145) 11:27:29.201 [I] Progress on: 2.35kit/3.00kit rate:7.1s/it remaining:1:16:25 elapsed:4:41:17 postfix:- (1728:tqdm_logging.py:145) Step 2350: grad_norm=0.0401, loss=0.0041, param_norm=1804.1027 11:27:42.959 [I] Progress on: 2.35kit/3.00kit rate:7.1s/it remaining:1:16:25 elapsed:4:41:31 postfix:- (1728:tqdm_logging.py:145) 11:27:57.491 [I] Progress on: 2.35kit/3.00kit rate:6.8s/it remaining:1:12:51 elapsed:4:41:46 postfix:- (1728:tqdm_logging.py:145) 11:28:11.606 [I] Progress on: 2.36kit/3.00kit rate:6.9s/it remaining:1:14:12 elapsed:4:42:00 postfix:- (1728:tqdm_logging.py:145) 11:28:25.721 [I] Progress on: 2.36kit/3.00kit rate:7.0s/it remaining:1:14:45 elapsed:4:42:14 postfix:- (1728:tqdm_logging.py:145) 11:28:39.815 [I] Progress on: 2.36kit/3.00kit rate:7.0s/it remaining:1:14:50 elapsed:4:42:28 postfix:- (1728:tqdm_logging.py:145) 11:28:53.934 [I] Progress on: 2.36kit/3.00kit rate:7.0s/it remaining:1:14:50 elapsed:4:42:42 postfix:- (1728:tqdm_logging.py:145) 11:29:08.037 [I] Progress on: 2.36kit/3.00kit rate:7.0s/it remaining:1:14:40 elapsed:4:42:56 postfix:- (1728:tqdm_logging.py:145) 11:29:22.132 [I] Progress on: 2.37kit/3.00kit rate:7.0s/it remaining:1:14:27 elapsed:4:43:10 postfix:- (1728:tqdm_logging.py:145) 11:29:36.249 [I] Progress on: 2.37kit/3.00kit rate:7.1s/it remaining:1:14:17 elapsed:4:43:25 postfix:- (1728:tqdm_logging.py:145) 11:29:50.333 [I] Progress on: 2.37kit/3.00kit rate:7.0s/it remaining:1:13:59 elapsed:4:43:39 postfix:- (1728:tqdm_logging.py:145) 11:30:04.433 [I] Progress on: 2.37kit/3.00kit rate:7.0s/it remaining:1:13:46 elapsed:4:43:53 postfix:- (1728:tqdm_logging.py:145) 11:30:18.554 [I] Progress on: 2.37kit/3.00kit rate:7.1s/it remaining:1:13:35 elapsed:4:44:07 postfix:- (1728:tqdm_logging.py:145) 11:30:32.664 [I] Progress on: 2.38kit/3.00kit rate:7.1s/it remaining:1:13:21 elapsed:4:44:21 postfix:- (1728:tqdm_logging.py:145) 11:30:46.785 [I] Progress on: 2.38kit/3.00kit rate:7.1s/it remaining:1:13:09 elapsed:4:44:35 postfix:- (1728:tqdm_logging.py:145) 11:31:00.878 [I] Progress on: 2.38kit/3.00kit rate:7.1s/it remaining:1:12:52 elapsed:4:44:49 postfix:- (1728:tqdm_logging.py:145) 11:31:14.992 [I] Progress on: 2.38kit/3.00kit rate:7.1s/it remaining:1:12:39 elapsed:4:45:03 postfix:- (1728:tqdm_logging.py:145) 11:31:29.102 [I] Progress on: 2.38kit/3.00kit rate:7.1s/it remaining:1:12:25 elapsed:4:45:17 postfix:- (1728:tqdm_logging.py:145) 11:31:43.210 [I] Progress on: 2.39kit/3.00kit rate:7.1s/it remaining:1:12:11 elapsed:4:45:31 postfix:- (1728:tqdm_logging.py:145) 11:31:57.331 [I] Progress on: 2.39kit/3.00kit rate:7.1s/it remaining:1:11:59 elapsed:4:45:46 postfix:- (1728:tqdm_logging.py:145) 11:32:11.437 [I] Progress on: 2.39kit/3.00kit rate:7.1s/it remaining:1:11:43 elapsed:4:46:00 postfix:- (1728:tqdm_logging.py:145) 11:32:25.535 [I] Progress on: 2.39kit/3.00kit rate:7.1s/it remaining:1:11:27 elapsed:4:46:14 postfix:- (1728:tqdm_logging.py:145) 11:32:39.647 [I] Progress on: 2.39kit/3.00kit rate:7.1s/it remaining:1:11:14 elapsed:4:46:28 postfix:- (1728:tqdm_logging.py:145) 11:32:53.755 [I] Progress on: 2.40kit/3.00kit rate:7.1s/it remaining:1:10:59 elapsed:4:46:42 postfix:- (1728:tqdm_logging.py:145) 11:33:07.857 [I] Progress on: 2.40kit/3.00kit rate:7.1s/it remaining:1:10:45 elapsed:4:46:56 postfix:- (1728:tqdm_logging.py:145) 11:33:21.972 [I] Progress on: 2.40kit/3.00kit rate:7.1s/it remaining:1:10:32 elapsed:4:47:10 postfix:- (1728:tqdm_logging.py:145) Step 2400: grad_norm=0.0420, loss=0.0044, param_norm=1804.1060 11:33:35.732 [I] Progress on: 2.40kit/3.00kit rate:7.1s/it remaining:1:10:32 elapsed:4:47:24 postfix:- (1728:tqdm_logging.py:145) 11:33:50.260 [I] Progress on: 2.40kit/3.00kit rate:6.8s/it remaining:1:07:11 elapsed:4:47:39 postfix:- (1728:tqdm_logging.py:145) 11:34:04.369 [I] Progress on: 2.41kit/3.00kit rate:6.9s/it remaining:1:08:26 elapsed:4:47:53 postfix:- (1728:tqdm_logging.py:145) 11:34:18.470 [I] Progress on: 2.41kit/3.00kit rate:7.0s/it remaining:1:08:54 elapsed:4:48:07 postfix:- (1728:tqdm_logging.py:145) 11:34:32.574 [I] Progress on: 2.41kit/3.00kit rate:7.0s/it remaining:1:09:00 elapsed:4:48:21 postfix:- (1728:tqdm_logging.py:145) 11:34:46.667 [I] Progress on: 2.41kit/3.00kit rate:7.0s/it remaining:1:08:54 elapsed:4:48:35 postfix:- (1728:tqdm_logging.py:145) 11:35:00.779 [I] Progress on: 2.41kit/3.00kit rate:7.0s/it remaining:1:08:47 elapsed:4:48:49 postfix:- (1728:tqdm_logging.py:145) 11:35:14.862 [I] Progress on: 2.42kit/3.00kit rate:7.0s/it remaining:1:08:32 elapsed:4:49:03 postfix:- (1728:tqdm_logging.py:145) 11:35:28.957 [I] Progress on: 2.42kit/3.00kit rate:7.0s/it remaining:1:08:19 elapsed:4:49:17 postfix:- (1728:tqdm_logging.py:145) 11:35:43.074 [I] Progress on: 2.42kit/3.00kit rate:7.1s/it remaining:1:08:10 elapsed:4:49:31 postfix:- (1728:tqdm_logging.py:145) 11:35:57.177 [I] Progress on: 2.42kit/3.00kit rate:7.1s/it remaining:1:07:56 elapsed:4:49:45 postfix:- (1728:tqdm_logging.py:145) 11:36:11.284 [I] Progress on: 2.42kit/3.00kit rate:7.1s/it remaining:1:07:42 elapsed:4:50:00 postfix:- (1728:tqdm_logging.py:145) 11:36:25.371 [I] Progress on: 2.43kit/3.00kit rate:7.0s/it remaining:1:07:25 elapsed:4:50:14 postfix:- (1728:tqdm_logging.py:145) 11:36:39.493 [I] Progress on: 2.43kit/3.00kit rate:7.1s/it remaining:1:07:15 elapsed:4:50:28 postfix:- (1728:tqdm_logging.py:145) 11:36:53.608 [I] Progress on: 2.43kit/3.00kit rate:7.1s/it remaining:1:07:01 elapsed:4:50:42 postfix:- (1728:tqdm_logging.py:145) 11:37:07.712 [I] Progress on: 2.43kit/3.00kit rate:7.1s/it remaining:1:06:46 elapsed:4:50:56 postfix:- (1728:tqdm_logging.py:145) 11:37:21.824 [I] Progress on: 2.43kit/3.00kit rate:7.1s/it remaining:1:06:33 elapsed:4:51:10 postfix:- (1728:tqdm_logging.py:145) 11:37:35.930 [I] Progress on: 2.44kit/3.00kit rate:7.1s/it remaining:1:06:18 elapsed:4:51:24 postfix:- (1728:tqdm_logging.py:145) 11:37:50.032 [I] Progress on: 2.44kit/3.00kit rate:7.1s/it remaining:1:06:03 elapsed:4:51:38 postfix:- (1728:tqdm_logging.py:145) 11:38:04.133 [I] Progress on: 2.44kit/3.00kit rate:7.1s/it remaining:1:05:49 elapsed:4:51:52 postfix:- (1728:tqdm_logging.py:145) 11:38:18.732 [I] Progress on: 2.44kit/3.00kit rate:7.2s/it remaining:1:06:57 elapsed:4:52:07 postfix:- (1728:tqdm_logging.py:145) 11:38:32.342 [I] Progress on: 2.44kit/3.00kit rate:7.0s/it remaining:1:05:04 elapsed:4:52:21 postfix:- (1728:tqdm_logging.py:145) 11:38:46.439 [I] Progress on: 2.45kit/3.00kit rate:7.0s/it remaining:1:04:57 elapsed:4:52:35 postfix:- (1728:tqdm_logging.py:145) 11:39:00.521 [I] Progress on: 2.45kit/3.00kit rate:7.0s/it remaining:1:04:45 elapsed:4:52:49 postfix:- (1728:tqdm_logging.py:145) 11:39:14.620 [I] Progress on: 2.45kit/3.00kit rate:7.0s/it remaining:1:04:33 elapsed:4:53:03 postfix:- (1728:tqdm_logging.py:145) Step 2450: grad_norm=0.0408, loss=0.0043, param_norm=1804.1099 11:39:28.369 [I] Progress on: 2.45kit/3.00kit rate:7.0s/it remaining:1:04:33 elapsed:4:53:17 postfix:- (1728:tqdm_logging.py:145) 11:39:42.892 [I] Progress on: 2.45kit/3.00kit rate:6.8s/it remaining:1:01:32 elapsed:4:53:31 postfix:- (1728:tqdm_logging.py:145) 11:39:56.980 [I] Progress on: 2.46kit/3.00kit rate:6.9s/it remaining:1:02:36 elapsed:4:53:45 postfix:- (1728:tqdm_logging.py:145) 11:40:11.086 [I] Progress on: 2.46kit/3.00kit rate:7.0s/it remaining:1:03:03 elapsed:4:53:59 postfix:- (1728:tqdm_logging.py:145) 11:40:25.201 [I] Progress on: 2.46kit/3.00kit rate:7.0s/it remaining:1:03:10 elapsed:4:54:13 postfix:- (1728:tqdm_logging.py:145) 11:40:39.305 [I] Progress on: 2.46kit/3.00kit rate:7.0s/it remaining:1:03:04 elapsed:4:54:28 postfix:- (1728:tqdm_logging.py:145) 11:40:53.414 [I] Progress on: 2.46kit/3.00kit rate:7.0s/it remaining:1:02:56 elapsed:4:54:42 postfix:- (1728:tqdm_logging.py:145) 11:41:07.527 [I] Progress on: 2.47kit/3.00kit rate:7.1s/it remaining:1:02:46 elapsed:4:54:56 postfix:- (1728:tqdm_logging.py:145) 11:41:21.639 [I] Progress on: 2.47kit/3.00kit rate:7.1s/it remaining:1:02:33 elapsed:4:55:10 postfix:- (1728:tqdm_logging.py:145) 11:41:35.737 [I] Progress on: 2.47kit/3.00kit rate:7.1s/it remaining:1:02:17 elapsed:4:55:24 postfix:- (1728:tqdm_logging.py:145) 11:41:49.841 [I] Progress on: 2.47kit/3.00kit rate:7.1s/it remaining:1:02:03 elapsed:4:55:38 postfix:- (1728:tqdm_logging.py:145) 11:42:03.954 [I] Progress on: 2.47kit/3.00kit rate:7.1s/it remaining:1:01:50 elapsed:4:55:52 postfix:- (1728:tqdm_logging.py:145) 11:42:18.071 [I] Progress on: 2.48kit/3.00kit rate:7.1s/it remaining:1:01:37 elapsed:4:56:06 postfix:- (1728:tqdm_logging.py:145) 11:42:32.186 [I] Progress on: 2.48kit/3.00kit rate:7.1s/it remaining:1:01:23 elapsed:4:56:20 postfix:- (1728:tqdm_logging.py:145) 11:42:46.301 [I] Progress on: 2.48kit/3.00kit rate:7.1s/it remaining:1:01:09 elapsed:4:56:35 postfix:- (1728:tqdm_logging.py:145) 11:43:00.412 [I] Progress on: 2.48kit/3.00kit rate:7.1s/it remaining:1:00:55 elapsed:4:56:49 postfix:- (1728:tqdm_logging.py:145) 11:43:14.531 [I] Progress on: 2.48kit/3.00kit rate:7.1s/it remaining:1:00:41 elapsed:4:57:03 postfix:- (1728:tqdm_logging.py:145) 11:43:28.624 [I] Progress on: 2.49kit/3.00kit rate:7.1s/it remaining:1:00:24 elapsed:4:57:17 postfix:- (1728:tqdm_logging.py:145) 11:43:42.734 [I] Progress on: 2.49kit/3.00kit rate:7.1s/it remaining:1:00:11 elapsed:4:57:31 postfix:- (1728:tqdm_logging.py:145) 11:43:56.845 [I] Progress on: 2.49kit/3.00kit rate:7.1s/it remaining:59:57 elapsed:4:57:45 postfix:- (1728:tqdm_logging.py:145) 11:44:10.963 [I] Progress on: 2.49kit/3.00kit rate:7.1s/it remaining:59:45 elapsed:4:57:59 postfix:- (1728:tqdm_logging.py:145) 11:44:25.074 [I] Progress on: 2.49kit/3.00kit rate:7.1s/it remaining:59:30 elapsed:4:58:13 postfix:- (1728:tqdm_logging.py:145) 11:44:39.174 [I] Progress on: 2.50kit/3.00kit rate:7.1s/it remaining:59:14 elapsed:4:58:27 postfix:- (1728:tqdm_logging.py:145) 11:44:53.283 [I] Progress on: 2.50kit/3.00kit rate:7.1s/it remaining:59:01 elapsed:4:58:42 postfix:- (1728:tqdm_logging.py:145) 11:45:07.721 [I] Progress on: 2.50kit/3.00kit rate:7.2s/it remaining:59:36 elapsed:4:58:56 postfix:- (1728:tqdm_logging.py:145) Step 2500: grad_norm=0.0428, loss=0.0043, param_norm=1804.1134 11:45:21.457 [I] Progress on: 2.50kit/3.00kit rate:7.2s/it remaining:59:36 elapsed:4:59:10 postfix:- (1728:tqdm_logging.py:145) 11:45:35.963 [I] Progress on: 2.50kit/3.00kit rate:6.8s/it remaining:56:05 elapsed:4:59:24 postfix:- (1728:tqdm_logging.py:145) 11:45:50.080 [I] Progress on: 2.51kit/3.00kit rate:6.9s/it remaining:57:00 elapsed:4:59:38 postfix:- (1728:tqdm_logging.py:145) 11:46:04.185 [I] Progress on: 2.51kit/3.00kit rate:7.0s/it remaining:57:18 elapsed:4:59:52 postfix:- (1728:tqdm_logging.py:145) 11:46:18.292 [I] Progress on: 2.51kit/3.00kit rate:7.0s/it remaining:57:20 elapsed:5:00:07 postfix:- (1728:tqdm_logging.py:145) 11:46:32.389 [I] Progress on: 2.51kit/3.00kit rate:7.0s/it remaining:57:13 elapsed:5:00:21 postfix:- (1728:tqdm_logging.py:145) 11:46:46.487 [I] Progress on: 2.51kit/3.00kit rate:7.0s/it remaining:57:02 elapsed:5:00:35 postfix:- (1728:tqdm_logging.py:145) 11:47:00.602 [I] Progress on: 2.52kit/3.00kit rate:7.0s/it remaining:56:51 elapsed:5:00:49 postfix:- (1728:tqdm_logging.py:145) 11:47:14.715 [I] Progress on: 2.52kit/3.00kit rate:7.1s/it remaining:56:38 elapsed:5:01:03 postfix:- (1728:tqdm_logging.py:145) 11:47:28.825 [I] Progress on: 2.52kit/3.00kit rate:7.1s/it remaining:56:25 elapsed:5:01:17 postfix:- (1728:tqdm_logging.py:145) 11:47:42.936 [I] Progress on: 2.52kit/3.00kit rate:7.1s/it remaining:56:12 elapsed:5:01:31 postfix:- (1728:tqdm_logging.py:145) 11:47:57.044 [I] Progress on: 2.52kit/3.00kit rate:7.1s/it remaining:55:57 elapsed:5:01:45 postfix:- (1728:tqdm_logging.py:145) 11:48:11.134 [I] Progress on: 2.53kit/3.00kit rate:7.0s/it remaining:55:41 elapsed:5:01:59 postfix:- (1728:tqdm_logging.py:145) 11:48:25.263 [I] Progress on: 2.53kit/3.00kit rate:7.1s/it remaining:55:31 elapsed:5:02:14 postfix:- (1728:tqdm_logging.py:145) 11:48:39.363 [I] Progress on: 2.53kit/3.00kit rate:7.1s/it remaining:55:15 elapsed:5:02:28 postfix:- (1728:tqdm_logging.py:145) 11:48:53.460 [I] Progress on: 2.53kit/3.00kit rate:7.1s/it remaining:54:59 elapsed:5:02:42 postfix:- (1728:tqdm_logging.py:145) 11:49:07.577 [I] Progress on: 2.53kit/3.00kit rate:7.1s/it remaining:54:47 elapsed:5:02:56 postfix:- (1728:tqdm_logging.py:145) 11:49:21.683 [I] Progress on: 2.54kit/3.00kit rate:7.1s/it remaining:54:33 elapsed:5:03:10 postfix:- (1728:tqdm_logging.py:145) 11:49:35.799 [I] Progress on: 2.54kit/3.00kit rate:7.1s/it remaining:54:20 elapsed:5:03:24 postfix:- (1728:tqdm_logging.py:145) 11:49:49.900 [I] Progress on: 2.54kit/3.00kit rate:7.1s/it remaining:54:04 elapsed:5:03:38 postfix:- (1728:tqdm_logging.py:145) 11:50:04.021 [I] Progress on: 2.54kit/3.00kit rate:7.1s/it remaining:53:52 elapsed:5:03:52 postfix:- (1728:tqdm_logging.py:145) 11:50:18.133 [I] Progress on: 2.54kit/3.00kit rate:7.1s/it remaining:53:37 elapsed:5:04:06 postfix:- (1728:tqdm_logging.py:145) 11:50:32.259 [I] Progress on: 2.55kit/3.00kit rate:7.1s/it remaining:53:24 elapsed:5:04:21 postfix:- (1728:tqdm_logging.py:145) 11:50:46.378 [I] Progress on: 2.55kit/3.00kit rate:7.1s/it remaining:53:11 elapsed:5:04:35 postfix:- (1728:tqdm_logging.py:145) 11:51:00.484 [I] Progress on: 2.55kit/3.00kit rate:7.1s/it remaining:52:55 elapsed:5:04:49 postfix:- (1728:tqdm_logging.py:145) Step 2550: grad_norm=0.0416, loss=0.0042, param_norm=1804.1166 11:51:14.223 [I] Progress on: 2.55kit/3.00kit rate:7.1s/it remaining:52:55 elapsed:5:05:03 postfix:- (1728:tqdm_logging.py:145) 11:51:28.759 [I] Progress on: 2.55kit/3.00kit rate:6.8s/it remaining:50:17 elapsed:5:05:17 postfix:- (1728:tqdm_logging.py:145) 11:51:42.878 [I] Progress on: 2.56kit/3.00kit rate:6.9s/it remaining:51:10 elapsed:5:05:31 postfix:- (1728:tqdm_logging.py:145) 11:51:56.999 [I] Progress on: 2.56kit/3.00kit rate:7.0s/it remaining:51:29 elapsed:5:05:45 postfix:- (1728:tqdm_logging.py:145) 11:52:11.111 [I] Progress on: 2.56kit/3.00kit rate:7.0s/it remaining:51:30 elapsed:5:05:59 postfix:- (1728:tqdm_logging.py:145) 11:52:25.219 [I] Progress on: 2.56kit/3.00kit rate:7.0s/it remaining:51:22 elapsed:5:06:14 postfix:- (1728:tqdm_logging.py:145) 11:52:39.330 [I] Progress on: 2.56kit/3.00kit rate:7.0s/it remaining:51:12 elapsed:5:06:28 postfix:- (1728:tqdm_logging.py:145) 11:52:53.455 [I] Progress on: 2.57kit/3.00kit rate:7.1s/it remaining:51:01 elapsed:5:06:42 postfix:- (1728:tqdm_logging.py:145) 11:53:07.563 [I] Progress on: 2.57kit/3.00kit rate:7.1s/it remaining:50:47 elapsed:5:06:56 postfix:- (1728:tqdm_logging.py:145) 11:53:21.678 [I] Progress on: 2.57kit/3.00kit rate:7.1s/it remaining:50:33 elapsed:5:07:10 postfix:- (1728:tqdm_logging.py:145) 11:53:35.801 [I] Progress on: 2.57kit/3.00kit rate:7.1s/it remaining:50:21 elapsed:5:07:24 postfix:- (1728:tqdm_logging.py:145) 11:53:49.890 [I] Progress on: 2.57kit/3.00kit rate:7.1s/it remaining:50:04 elapsed:5:07:38 postfix:- (1728:tqdm_logging.py:145) 11:54:03.998 [I] Progress on: 2.58kit/3.00kit rate:7.1s/it remaining:49:50 elapsed:5:07:52 postfix:- (1728:tqdm_logging.py:145) 11:54:18.127 [I] Progress on: 2.58kit/3.00kit rate:7.1s/it remaining:49:38 elapsed:5:08:06 postfix:- (1728:tqdm_logging.py:145) 11:54:32.236 [I] Progress on: 2.58kit/3.00kit rate:7.1s/it remaining:49:23 elapsed:5:08:21 postfix:- (1728:tqdm_logging.py:145) 11:54:46.335 [I] Progress on: 2.58kit/3.00kit rate:7.1s/it remaining:49:07 elapsed:5:08:35 postfix:- (1728:tqdm_logging.py:145) 11:55:00.458 [I] Progress on: 2.58kit/3.00kit rate:7.1s/it remaining:48:55 elapsed:5:08:49 postfix:- (1728:tqdm_logging.py:145) 11:55:14.575 [I] Progress on: 2.59kit/3.00kit rate:7.1s/it remaining:48:41 elapsed:5:09:03 postfix:- (1728:tqdm_logging.py:145) 11:55:28.685 [I] Progress on: 2.59kit/3.00kit rate:7.1s/it remaining:48:27 elapsed:5:09:17 postfix:- (1728:tqdm_logging.py:145) 11:55:42.803 [I] Progress on: 2.59kit/3.00kit rate:7.1s/it remaining:48:13 elapsed:5:09:31 postfix:- (1728:tqdm_logging.py:145) 11:55:56.923 [I] Progress on: 2.59kit/3.00kit rate:7.1s/it remaining:48:00 elapsed:5:09:45 postfix:- (1728:tqdm_logging.py:145) 11:56:11.028 [I] Progress on: 2.59kit/3.00kit rate:7.1s/it remaining:47:44 elapsed:5:09:59 postfix:- (1728:tqdm_logging.py:145) 11:56:25.155 [I] Progress on: 2.60kit/3.00kit rate:7.1s/it remaining:47:32 elapsed:5:10:13 postfix:- (1728:tqdm_logging.py:145) 11:56:39.265 [I] Progress on: 2.60kit/3.00kit rate:7.1s/it remaining:47:17 elapsed:5:10:28 postfix:- (1728:tqdm_logging.py:145) 11:56:53.377 [I] Progress on: 2.60kit/3.00kit rate:7.1s/it remaining:47:02 elapsed:5:10:42 postfix:- (1728:tqdm_logging.py:145) Step 2600: grad_norm=0.0440, loss=0.0046, param_norm=1804.1196 11:57:07.142 [I] Progress on: 2.60kit/3.00kit rate:7.1s/it remaining:47:02 elapsed:5:10:55 postfix:- (1728:tqdm_logging.py:145) 11:57:21.710 [I] Progress on: 2.60kit/3.00kit rate:6.8s/it remaining:44:43 elapsed:5:11:10 postfix:- (1728:tqdm_logging.py:145) 11:57:35.828 [I] Progress on: 2.61kit/3.00kit rate:6.9s/it remaining:45:27 elapsed:5:11:24 postfix:- (1728:tqdm_logging.py:145) 11:57:49.933 [I] Progress on: 2.61kit/3.00kit rate:7.0s/it remaining:45:39 elapsed:5:11:38 postfix:- (1728:tqdm_logging.py:145) 11:58:04.032 [I] Progress on: 2.61kit/3.00kit rate:7.0s/it remaining:45:37 elapsed:5:11:52 postfix:- (1728:tqdm_logging.py:145) 11:58:18.174 [I] Progress on: 2.61kit/3.00kit rate:7.0s/it remaining:45:34 elapsed:5:12:06 postfix:- (1728:tqdm_logging.py:145) 11:58:32.286 [I] Progress on: 2.61kit/3.00kit rate:7.1s/it remaining:45:21 elapsed:5:12:21 postfix:- (1728:tqdm_logging.py:145) 11:58:46.389 [I] Progress on: 2.62kit/3.00kit rate:7.1s/it remaining:45:08 elapsed:5:12:35 postfix:- (1728:tqdm_logging.py:145) 11:59:00.489 [I] Progress on: 2.62kit/3.00kit rate:7.1s/it remaining:44:53 elapsed:5:12:49 postfix:- (1728:tqdm_logging.py:145) 11:59:14.600 [I] Progress on: 2.62kit/3.00kit rate:7.1s/it remaining:44:40 elapsed:5:13:03 postfix:- (1728:tqdm_logging.py:145) 11:59:28.709 [I] Progress on: 2.62kit/3.00kit rate:7.1s/it remaining:44:26 elapsed:5:13:17 postfix:- (1728:tqdm_logging.py:145) 11:59:42.825 [I] Progress on: 2.62kit/3.00kit rate:7.1s/it remaining:44:12 elapsed:5:13:31 postfix:- (1728:tqdm_logging.py:145) 11:59:56.947 [I] Progress on: 2.63kit/3.00kit rate:7.1s/it remaining:43:59 elapsed:5:13:45 postfix:- (1728:tqdm_logging.py:145) 12:00:11.075 [I] Progress on: 2.63kit/3.00kit rate:7.1s/it remaining:43:47 elapsed:5:13:59 postfix:- (1728:tqdm_logging.py:145) 12:00:25.179 [I] Progress on: 2.63kit/3.00kit rate:7.1s/it remaining:43:31 elapsed:5:14:13 postfix:- (1728:tqdm_logging.py:145) 12:00:39.271 [I] Progress on: 2.63kit/3.00kit rate:7.1s/it remaining:43:14 elapsed:5:14:28 postfix:- (1728:tqdm_logging.py:145) 12:00:53.387 [I] Progress on: 2.63kit/3.00kit rate:7.1s/it remaining:43:01 elapsed:5:14:42 postfix:- (1728:tqdm_logging.py:145) 12:01:07.507 [I] Progress on: 2.64kit/3.00kit rate:7.1s/it remaining:42:49 elapsed:5:14:56 postfix:- (1728:tqdm_logging.py:145) 12:01:21.618 [I] Progress on: 2.64kit/3.00kit rate:7.1s/it remaining:42:34 elapsed:5:15:10 postfix:- (1728:tqdm_logging.py:145) 12:01:35.734 [I] Progress on: 2.64kit/3.00kit rate:7.1s/it remaining:42:20 elapsed:5:15:24 postfix:- (1728:tqdm_logging.py:145) 12:01:49.844 [I] Progress on: 2.64kit/3.00kit rate:7.1s/it remaining:42:06 elapsed:5:15:38 postfix:- (1728:tqdm_logging.py:145) 12:02:03.959 [I] Progress on: 2.64kit/3.00kit rate:7.1s/it remaining:41:52 elapsed:5:15:52 postfix:- (1728:tqdm_logging.py:145) 12:02:18.055 [I] Progress on: 2.65kit/3.00kit rate:7.1s/it remaining:41:36 elapsed:5:16:06 postfix:- (1728:tqdm_logging.py:145) 12:02:32.167 [I] Progress on: 2.65kit/3.00kit rate:7.1s/it remaining:41:22 elapsed:5:16:20 postfix:- (1728:tqdm_logging.py:145) 12:02:46.284 [I] Progress on: 2.65kit/3.00kit rate:7.1s/it remaining:41:09 elapsed:5:16:35 postfix:- (1728:tqdm_logging.py:145) Step 2650: grad_norm=0.0409, loss=0.0045, param_norm=1804.1228 12:03:00.055 [I] Progress on: 2.65kit/3.00kit rate:7.1s/it remaining:41:09 elapsed:5:16:48 postfix:- (1728:tqdm_logging.py:145) 12:03:14.610 [I] Progress on: 2.65kit/3.00kit rate:6.8s/it remaining:39:03 elapsed:5:17:03 postfix:- (1728:tqdm_logging.py:145) 12:03:28.700 [I] Progress on: 2.66kit/3.00kit rate:6.9s/it remaining:39:37 elapsed:5:17:17 postfix:- (1728:tqdm_logging.py:145) 12:03:42.815 [I] Progress on: 2.66kit/3.00kit rate:7.0s/it remaining:39:49 elapsed:5:17:31 postfix:- (1728:tqdm_logging.py:145) 12:03:56.912 [I] Progress on: 2.66kit/3.00kit rate:7.0s/it remaining:39:46 elapsed:5:17:45 postfix:- (1728:tqdm_logging.py:145) 12:04:11.000 [I] Progress on: 2.66kit/3.00kit rate:7.0s/it remaining:39:36 elapsed:5:17:59 postfix:- (1728:tqdm_logging.py:145) 12:04:25.610 [I] Progress on: 2.66kit/3.00kit rate:7.2s/it remaining:40:17 elapsed:5:18:14 postfix:- (1728:tqdm_logging.py:145) 12:04:39.219 [I] Progress on: 2.67kit/3.00kit rate:7.0s/it remaining:39:05 elapsed:5:18:28 postfix:- (1728:tqdm_logging.py:145) 12:04:53.316 [I] Progress on: 2.67kit/3.00kit rate:7.0s/it remaining:38:57 elapsed:5:18:42 postfix:- (1728:tqdm_logging.py:145) 12:05:07.409 [I] Progress on: 2.67kit/3.00kit rate:7.0s/it remaining:38:45 elapsed:5:18:56 postfix:- (1728:tqdm_logging.py:145) 12:05:21.541 [I] Progress on: 2.67kit/3.00kit rate:7.1s/it remaining:38:35 elapsed:5:19:10 postfix:- (1728:tqdm_logging.py:145) 12:05:35.634 [I] Progress on: 2.67kit/3.00kit rate:7.1s/it remaining:38:19 elapsed:5:19:24 postfix:- (1728:tqdm_logging.py:145) 12:05:49.761 [I] Progress on: 2.68kit/3.00kit rate:7.1s/it remaining:38:06 elapsed:5:19:38 postfix:- (1728:tqdm_logging.py:145) 12:06:03.867 [I] Progress on: 2.68kit/3.00kit rate:7.1s/it remaining:37:51 elapsed:5:19:52 postfix:- (1728:tqdm_logging.py:145) 12:06:17.970 [I] Progress on: 2.68kit/3.00kit rate:7.1s/it remaining:37:37 elapsed:5:20:06 postfix:- (1728:tqdm_logging.py:145) 12:06:32.074 [I] Progress on: 2.68kit/3.00kit rate:7.1s/it remaining:37:22 elapsed:5:20:20 postfix:- (1728:tqdm_logging.py:145) 12:06:46.177 [I] Progress on: 2.68kit/3.00kit rate:7.1s/it remaining:37:08 elapsed:5:20:34 postfix:- (1728:tqdm_logging.py:145) 12:07:00.294 [I] Progress on: 2.69kit/3.00kit rate:7.1s/it remaining:36:55 elapsed:5:20:49 postfix:- (1728:tqdm_logging.py:145) 12:07:14.428 [I] Progress on: 2.69kit/3.00kit rate:7.1s/it remaining:36:42 elapsed:5:21:03 postfix:- (1728:tqdm_logging.py:145) 12:07:28.533 [I] Progress on: 2.69kit/3.00kit rate:7.1s/it remaining:36:27 elapsed:5:21:17 postfix:- (1728:tqdm_logging.py:145) 12:07:42.642 [I] Progress on: 2.69kit/3.00kit rate:7.1s/it remaining:36:13 elapsed:5:21:31 postfix:- (1728:tqdm_logging.py:145) 12:07:56.763 [I] Progress on: 2.69kit/3.00kit rate:7.1s/it remaining:35:59 elapsed:5:21:45 postfix:- (1728:tqdm_logging.py:145) 12:08:10.861 [I] Progress on: 2.70kit/3.00kit rate:7.1s/it remaining:35:44 elapsed:5:21:59 postfix:- (1728:tqdm_logging.py:145) 12:08:24.988 [I] Progress on: 2.70kit/3.00kit rate:7.1s/it remaining:35:31 elapsed:5:22:13 postfix:- (1728:tqdm_logging.py:145) 12:08:39.109 [I] Progress on: 2.70kit/3.00kit rate:7.1s/it remaining:35:17 elapsed:5:22:27 postfix:- (1728:tqdm_logging.py:145) Step 2700: grad_norm=0.0426, loss=0.0042, param_norm=1804.1262 12:08:52.858 [I] Progress on: 2.70kit/3.00kit rate:7.1s/it remaining:35:17 elapsed:5:22:41 postfix:- (1728:tqdm_logging.py:145) 12:09:07.433 [I] Progress on: 2.70kit/3.00kit rate:6.8s/it remaining:33:25 elapsed:5:22:56 postfix:- (1728:tqdm_logging.py:145) 12:09:21.531 [I] Progress on: 2.71kit/3.00kit rate:6.9s/it remaining:33:53 elapsed:5:23:10 postfix:- (1728:tqdm_logging.py:145) 12:09:35.650 [I] Progress on: 2.71kit/3.00kit rate:7.0s/it remaining:34:00 elapsed:5:23:24 postfix:- (1728:tqdm_logging.py:145) 12:09:49.780 [I] Progress on: 2.71kit/3.00kit rate:7.0s/it remaining:33:58 elapsed:5:23:38 postfix:- (1728:tqdm_logging.py:145) 12:10:03.908 [I] Progress on: 2.71kit/3.00kit rate:7.0s/it remaining:33:49 elapsed:5:23:52 postfix:- (1728:tqdm_logging.py:145) 12:10:18.037 [I] Progress on: 2.71kit/3.00kit rate:7.1s/it remaining:33:37 elapsed:5:24:06 postfix:- (1728:tqdm_logging.py:145) 12:10:32.161 [I] Progress on: 2.72kit/3.00kit rate:7.1s/it remaining:33:24 elapsed:5:24:20 postfix:- (1728:tqdm_logging.py:145) 12:10:46.266 [I] Progress on: 2.72kit/3.00kit rate:7.1s/it remaining:33:09 elapsed:5:24:35 postfix:- (1728:tqdm_logging.py:145) 12:11:00.376 [I] Progress on: 2.72kit/3.00kit rate:7.1s/it remaining:32:55 elapsed:5:24:49 postfix:- (1728:tqdm_logging.py:145) 12:11:14.501 [I] Progress on: 2.72kit/3.00kit rate:7.1s/it remaining:32:42 elapsed:5:25:03 postfix:- (1728:tqdm_logging.py:145) 12:11:28.607 [I] Progress on: 2.72kit/3.00kit rate:7.1s/it remaining:32:27 elapsed:5:25:17 postfix:- (1728:tqdm_logging.py:145) 12:11:42.736 [I] Progress on: 2.73kit/3.00kit rate:7.1s/it remaining:32:14 elapsed:5:25:31 postfix:- (1728:tqdm_logging.py:145) 12:11:56.847 [I] Progress on: 2.73kit/3.00kit rate:7.1s/it remaining:31:59 elapsed:5:25:45 postfix:- (1728:tqdm_logging.py:145) 12:12:10.960 [I] Progress on: 2.73kit/3.00kit rate:7.1s/it remaining:31:45 elapsed:5:25:59 postfix:- (1728:tqdm_logging.py:145) 12:12:25.087 [I] Progress on: 2.73kit/3.00kit rate:7.1s/it remaining:31:32 elapsed:5:26:13 postfix:- (1728:tqdm_logging.py:145) 12:12:39.216 [I] Progress on: 2.73kit/3.00kit rate:7.1s/it remaining:31:18 elapsed:5:26:28 postfix:- (1728:tqdm_logging.py:145) 12:12:53.325 [I] Progress on: 2.74kit/3.00kit rate:7.1s/it remaining:31:03 elapsed:5:26:42 postfix:- (1728:tqdm_logging.py:145) 12:13:07.452 [I] Progress on: 2.74kit/3.00kit rate:7.1s/it remaining:30:48 elapsed:5:26:56 postfix:- (1728:tqdm_logging.py:145) 12:13:21.569 [I] Progress on: 2.74kit/3.00kit rate:7.1s/it remaining:30:33 elapsed:5:27:10 postfix:- (1728:tqdm_logging.py:145) 12:13:35.670 [I] Progress on: 2.74kit/3.00kit rate:7.1s/it remaining:30:18 elapsed:5:27:24 postfix:- (1728:tqdm_logging.py:145) 12:13:49.792 [I] Progress on: 2.74kit/3.00kit rate:7.1s/it remaining:30:06 elapsed:5:27:38 postfix:- (1728:tqdm_logging.py:145) 12:14:03.929 [I] Progress on: 2.75kit/3.00kit rate:7.1s/it remaining:29:53 elapsed:5:27:52 postfix:- (1728:tqdm_logging.py:145) 12:14:18.021 [I] Progress on: 2.75kit/3.00kit rate:7.1s/it remaining:29:37 elapsed:5:28:06 postfix:- (1728:tqdm_logging.py:145) 12:14:32.141 [I] Progress on: 2.75kit/3.00kit rate:7.1s/it remaining:29:24 elapsed:5:28:20 postfix:- (1728:tqdm_logging.py:145) Step 2750: grad_norm=0.0424, loss=0.0041, param_norm=1804.1294 12:14:45.887 [I] Progress on: 2.75kit/3.00kit rate:7.1s/it remaining:29:24 elapsed:5:28:34 postfix:- (1728:tqdm_logging.py:145) 12:15:00.411 [I] Progress on: 2.75kit/3.00kit rate:6.8s/it remaining:27:44 elapsed:5:28:49 postfix:- (1728:tqdm_logging.py:145) 12:15:14.551 [I] Progress on: 2.76kit/3.00kit rate:6.9s/it remaining:28:08 elapsed:5:29:03 postfix:- (1728:tqdm_logging.py:145) 12:15:28.660 [I] Progress on: 2.76kit/3.00kit rate:7.0s/it remaining:28:11 elapsed:5:29:17 postfix:- (1728:tqdm_logging.py:145) 12:15:42.795 [I] Progress on: 2.76kit/3.00kit rate:7.0s/it remaining:28:06 elapsed:5:29:31 postfix:- (1728:tqdm_logging.py:145) 12:15:56.931 [I] Progress on: 2.76kit/3.00kit rate:7.0s/it remaining:27:57 elapsed:5:29:45 postfix:- (1728:tqdm_logging.py:145) 12:16:11.052 [I] Progress on: 2.76kit/3.00kit rate:7.1s/it remaining:27:44 elapsed:5:29:59 postfix:- (1728:tqdm_logging.py:145) 12:16:25.178 [I] Progress on: 2.77kit/3.00kit rate:7.1s/it remaining:27:31 elapsed:5:30:13 postfix:- (1728:tqdm_logging.py:145) 12:16:39.298 [I] Progress on: 2.77kit/3.00kit rate:7.1s/it remaining:27:17 elapsed:5:30:28 postfix:- (1728:tqdm_logging.py:145) 12:16:53.418 [I] Progress on: 2.77kit/3.00kit rate:7.1s/it remaining:27:03 elapsed:5:30:42 postfix:- (1728:tqdm_logging.py:145) 12:17:07.521 [I] Progress on: 2.77kit/3.00kit rate:7.1s/it remaining:26:48 elapsed:5:30:56 postfix:- (1728:tqdm_logging.py:145) 12:17:21.651 [I] Progress on: 2.77kit/3.00kit rate:7.1s/it remaining:26:35 elapsed:5:31:10 postfix:- (1728:tqdm_logging.py:145) 12:17:35.773 [I] Progress on: 2.78kit/3.00kit rate:7.1s/it remaining:26:21 elapsed:5:31:24 postfix:- (1728:tqdm_logging.py:145) 12:17:49.899 [I] Progress on: 2.78kit/3.00kit rate:7.1s/it remaining:26:07 elapsed:5:31:38 postfix:- (1728:tqdm_logging.py:145) 12:18:04.027 [I] Progress on: 2.78kit/3.00kit rate:7.1s/it remaining:25:53 elapsed:5:31:52 postfix:- (1728:tqdm_logging.py:145) 12:18:18.146 [I] Progress on: 2.78kit/3.00kit rate:7.1s/it remaining:25:39 elapsed:5:32:06 postfix:- (1728:tqdm_logging.py:145) 12:18:32.255 [I] Progress on: 2.78kit/3.00kit rate:7.1s/it remaining:25:24 elapsed:5:32:21 postfix:- (1728:tqdm_logging.py:145) 12:18:46.387 [I] Progress on: 2.79kit/3.00kit rate:7.1s/it remaining:25:10 elapsed:5:32:35 postfix:- (1728:tqdm_logging.py:145) 12:19:00.495 [I] Progress on: 2.79kit/3.00kit rate:7.1s/it remaining:24:56 elapsed:5:32:49 postfix:- (1728:tqdm_logging.py:145) 12:19:14.607 [I] Progress on: 2.79kit/3.00kit rate:7.1s/it remaining:24:41 elapsed:5:33:03 postfix:- (1728:tqdm_logging.py:145) 12:19:28.783 [I] Progress on: 2.79kit/3.00kit rate:7.1s/it remaining:24:31 elapsed:5:33:17 postfix:- (1728:tqdm_logging.py:145) 12:19:42.917 [I] Progress on: 2.79kit/3.00kit rate:7.1s/it remaining:24:17 elapsed:5:33:31 postfix:- (1728:tqdm_logging.py:145) 12:19:56.983 [I] Progress on: 2.80kit/3.00kit rate:7.1s/it remaining:23:59 elapsed:5:33:45 postfix:- (1728:tqdm_logging.py:145) 12:20:11.106 [I] Progress on: 2.80kit/3.00kit rate:7.1s/it remaining:23:45 elapsed:5:33:59 postfix:- (1728:tqdm_logging.py:145) 12:20:25.239 [I] Progress on: 2.80kit/3.00kit rate:7.1s/it remaining:23:31 elapsed:5:34:14 postfix:- (1728:tqdm_logging.py:145) Step 2800: grad_norm=0.0404, loss=0.0038, param_norm=1804.1315 12:20:38.988 [I] Progress on: 2.80kit/3.00kit rate:7.1s/it remaining:23:31 elapsed:5:34:27 postfix:- (1728:tqdm_logging.py:145) 12:20:53.609 [I] Progress on: 2.80kit/3.00kit rate:6.8s/it remaining:22:08 elapsed:5:34:42 postfix:- (1728:tqdm_logging.py:145) 12:21:07.712 [I] Progress on: 2.81kit/3.00kit rate:6.9s/it remaining:22:22 elapsed:5:34:56 postfix:- (1728:tqdm_logging.py:145) 12:21:21.840 [I] Progress on: 2.81kit/3.00kit rate:7.0s/it remaining:22:22 elapsed:5:35:10 postfix:- (1728:tqdm_logging.py:145) 12:21:35.955 [I] Progress on: 2.81kit/3.00kit rate:7.0s/it remaining:22:14 elapsed:5:35:24 postfix:- (1728:tqdm_logging.py:145) 12:21:50.058 [I] Progress on: 2.81kit/3.00kit rate:7.0s/it remaining:22:03 elapsed:5:35:38 postfix:- (1728:tqdm_logging.py:145) 12:22:04.195 [I] Progress on: 2.81kit/3.00kit rate:7.1s/it remaining:21:52 elapsed:5:35:52 postfix:- (1728:tqdm_logging.py:145) 12:22:18.308 [I] Progress on: 2.82kit/3.00kit rate:7.1s/it remaining:21:38 elapsed:5:36:07 postfix:- (1728:tqdm_logging.py:145) 12:22:32.440 [I] Progress on: 2.82kit/3.00kit rate:7.1s/it remaining:21:25 elapsed:5:36:21 postfix:- (1728:tqdm_logging.py:145) 12:22:46.573 [I] Progress on: 2.82kit/3.00kit rate:7.1s/it remaining:21:11 elapsed:5:36:35 postfix:- (1728:tqdm_logging.py:145) 12:23:00.696 [I] Progress on: 2.82kit/3.00kit rate:7.1s/it remaining:20:56 elapsed:5:36:49 postfix:- (1728:tqdm_logging.py:145) 12:23:14.816 [I] Progress on: 2.82kit/3.00kit rate:7.1s/it remaining:20:42 elapsed:5:37:03 postfix:- (1728:tqdm_logging.py:145) 12:23:28.926 [I] Progress on: 2.83kit/3.00kit rate:7.1s/it remaining:20:28 elapsed:5:37:17 postfix:- (1728:tqdm_logging.py:145) 12:23:43.054 [I] Progress on: 2.83kit/3.00kit rate:7.1s/it remaining:20:14 elapsed:5:37:31 postfix:- (1728:tqdm_logging.py:145) 12:23:57.183 [I] Progress on: 2.83kit/3.00kit rate:7.1s/it remaining:20:00 elapsed:5:37:45 postfix:- (1728:tqdm_logging.py:145) 12:24:11.299 [I] Progress on: 2.83kit/3.00kit rate:7.1s/it remaining:19:46 elapsed:5:38:00 postfix:- (1728:tqdm_logging.py:145) 12:24:25.408 [I] Progress on: 2.83kit/3.00kit rate:7.1s/it remaining:19:31 elapsed:5:38:14 postfix:- (1728:tqdm_logging.py:145) 12:24:39.535 [I] Progress on: 2.84kit/3.00kit rate:7.1s/it remaining:19:17 elapsed:5:38:28 postfix:- (1728:tqdm_logging.py:145) 12:24:53.646 [I] Progress on: 2.84kit/3.00kit rate:7.1s/it remaining:19:03 elapsed:5:38:42 postfix:- (1728:tqdm_logging.py:145) 12:25:07.765 [I] Progress on: 2.84kit/3.00kit rate:7.1s/it remaining:18:49 elapsed:5:38:56 postfix:- (1728:tqdm_logging.py:145) 12:25:21.878 [I] Progress on: 2.84kit/3.00kit rate:7.1s/it remaining:18:35 elapsed:5:39:10 postfix:- (1728:tqdm_logging.py:145) 12:25:36.002 [I] Progress on: 2.84kit/3.00kit rate:7.1s/it remaining:18:21 elapsed:5:39:24 postfix:- (1728:tqdm_logging.py:145) 12:25:50.132 [I] Progress on: 2.85kit/3.00kit rate:7.1s/it remaining:18:07 elapsed:5:39:38 postfix:- (1728:tqdm_logging.py:145) 12:26:04.263 [I] Progress on: 2.85kit/3.00kit rate:7.1s/it remaining:17:53 elapsed:5:39:53 postfix:- (1728:tqdm_logging.py:145) 12:26:18.372 [I] Progress on: 2.85kit/3.00kit rate:7.1s/it remaining:17:38 elapsed:5:40:07 postfix:- (1728:tqdm_logging.py:145) Step 2850: grad_norm=0.0450, loss=0.0045, param_norm=1804.1337 12:26:32.158 [I] Progress on: 2.85kit/3.00kit rate:7.1s/it remaining:17:38 elapsed:5:40:20 postfix:- (1728:tqdm_logging.py:145) 12:26:46.694 [I] Progress on: 2.85kit/3.00kit rate:6.8s/it remaining:16:28 elapsed:5:40:35 postfix:- (1728:tqdm_logging.py:145) 12:27:00.823 [I] Progress on: 2.86kit/3.00kit rate:6.9s/it remaining:16:36 elapsed:5:40:49 postfix:- (1728:tqdm_logging.py:145) 12:27:14.942 [I] Progress on: 2.86kit/3.00kit rate:7.0s/it remaining:16:32 elapsed:5:41:03 postfix:- (1728:tqdm_logging.py:145) 12:27:29.068 [I] Progress on: 2.86kit/3.00kit rate:7.0s/it remaining:16:24 elapsed:5:41:17 postfix:- (1728:tqdm_logging.py:145) 12:27:43.187 [I] Progress on: 2.86kit/3.00kit rate:7.0s/it remaining:16:11 elapsed:5:41:31 postfix:- (1728:tqdm_logging.py:145) 12:27:57.305 [I] Progress on: 2.86kit/3.00kit rate:7.1s/it remaining:15:58 elapsed:5:41:46 postfix:- (1728:tqdm_logging.py:145) 12:28:11.423 [I] Progress on: 2.87kit/3.00kit rate:7.1s/it remaining:15:45 elapsed:5:42:00 postfix:- (1728:tqdm_logging.py:145) 12:28:25.546 [I] Progress on: 2.87kit/3.00kit rate:7.1s/it remaining:15:31 elapsed:5:42:14 postfix:- (1728:tqdm_logging.py:145) 12:28:39.643 [I] Progress on: 2.87kit/3.00kit rate:7.1s/it remaining:15:16 elapsed:5:42:28 postfix:- (1728:tqdm_logging.py:145) 12:28:53.782 [I] Progress on: 2.87kit/3.00kit rate:7.1s/it remaining:15:03 elapsed:5:42:42 postfix:- (1728:tqdm_logging.py:145) 12:29:07.909 [I] Progress on: 2.87kit/3.00kit rate:7.1s/it remaining:14:49 elapsed:5:42:56 postfix:- (1728:tqdm_logging.py:145) 12:29:22.040 [I] Progress on: 2.88kit/3.00kit rate:7.1s/it remaining:14:35 elapsed:5:43:10 postfix:- (1728:tqdm_logging.py:145) 12:29:36.160 [I] Progress on: 2.88kit/3.00kit rate:7.1s/it remaining:14:21 elapsed:5:43:24 postfix:- (1728:tqdm_logging.py:145) 12:29:50.269 [I] Progress on: 2.88kit/3.00kit rate:7.1s/it remaining:14:07 elapsed:5:43:39 postfix:- (1728:tqdm_logging.py:145) 12:30:04.382 [I] Progress on: 2.88kit/3.00kit rate:7.1s/it remaining:13:52 elapsed:5:43:53 postfix:- (1728:tqdm_logging.py:145) 12:30:18.514 [I] Progress on: 2.88kit/3.00kit rate:7.1s/it remaining:13:39 elapsed:5:44:07 postfix:- (1728:tqdm_logging.py:145) 12:30:33.186 [I] Progress on: 2.89kit/3.00kit rate:7.2s/it remaining:13:43 elapsed:5:44:21 postfix:- (1728:tqdm_logging.py:145) 12:30:46.778 [I] Progress on: 2.89kit/3.00kit rate:7.0s/it remaining:13:07 elapsed:5:44:35 postfix:- (1728:tqdm_logging.py:145) 12:31:00.910 [I] Progress on: 2.89kit/3.00kit rate:7.1s/it remaining:12:56 elapsed:5:44:49 postfix:- (1728:tqdm_logging.py:145) 12:31:15.024 [I] Progress on: 2.89kit/3.00kit rate:7.1s/it remaining:12:42 elapsed:5:45:03 postfix:- (1728:tqdm_logging.py:145) 12:31:29.129 [I] Progress on: 2.89kit/3.00kit rate:7.1s/it remaining:12:28 elapsed:5:45:17 postfix:- (1728:tqdm_logging.py:145) 12:31:43.252 [I] Progress on: 2.90kit/3.00kit rate:7.1s/it remaining:12:14 elapsed:5:45:32 postfix:- (1728:tqdm_logging.py:145) 12:31:57.363 [I] Progress on: 2.90kit/3.00kit rate:7.1s/it remaining:12:00 elapsed:5:45:46 postfix:- (1728:tqdm_logging.py:145) 12:32:11.470 [I] Progress on: 2.90kit/3.00kit rate:7.1s/it remaining:11:45 elapsed:5:46:00 postfix:- (1728:tqdm_logging.py:145) Step 2900: grad_norm=0.0406, loss=0.0037, param_norm=1804.1359 12:32:25.228 [I] Progress on: 2.90kit/3.00kit rate:7.1s/it remaining:11:45 elapsed:5:46:14 postfix:- (1728:tqdm_logging.py:145) 12:32:39.778 [I] Progress on: 2.90kit/3.00kit rate:6.8s/it remaining:10:50 elapsed:5:46:28 postfix:- (1728:tqdm_logging.py:145) 12:32:53.888 [I] Progress on: 2.91kit/3.00kit rate:6.9s/it remaining:10:50 elapsed:5:46:42 postfix:- (1728:tqdm_logging.py:145) 12:33:08.008 [I] Progress on: 2.91kit/3.00kit rate:7.0s/it remaining:10:43 elapsed:5:46:56 postfix:- (1728:tqdm_logging.py:145) 12:33:22.131 [I] Progress on: 2.91kit/3.00kit rate:7.0s/it remaining:10:32 elapsed:5:47:10 postfix:- (1728:tqdm_logging.py:145) 12:33:36.229 [I] Progress on: 2.91kit/3.00kit rate:7.0s/it remaining:10:19 elapsed:5:47:25 postfix:- (1728:tqdm_logging.py:145) 12:33:50.346 [I] Progress on: 2.91kit/3.00kit rate:7.0s/it remaining:10:06 elapsed:5:47:39 postfix:- (1728:tqdm_logging.py:145) 12:34:04.471 [I] Progress on: 2.92kit/3.00kit rate:7.1s/it remaining:09:52 elapsed:5:47:53 postfix:- (1728:tqdm_logging.py:145) 12:34:18.573 [I] Progress on: 2.92kit/3.00kit rate:7.1s/it remaining:09:38 elapsed:5:48:07 postfix:- (1728:tqdm_logging.py:145) 12:34:32.708 [I] Progress on: 2.92kit/3.00kit rate:7.1s/it remaining:09:24 elapsed:5:48:21 postfix:- (1728:tqdm_logging.py:145) 12:34:46.845 [I] Progress on: 2.92kit/3.00kit rate:7.1s/it remaining:09:11 elapsed:5:48:35 postfix:- (1728:tqdm_logging.py:145) 12:35:00.963 [I] Progress on: 2.92kit/3.00kit rate:7.1s/it remaining:08:56 elapsed:5:48:49 postfix:- (1728:tqdm_logging.py:145) 12:35:15.083 [I] Progress on: 2.93kit/3.00kit rate:7.1s/it remaining:08:42 elapsed:5:49:03 postfix:- (1728:tqdm_logging.py:145) 12:35:29.196 [I] Progress on: 2.93kit/3.00kit rate:7.1s/it remaining:08:28 elapsed:5:49:17 postfix:- (1728:tqdm_logging.py:145) 12:35:43.313 [I] Progress on: 2.93kit/3.00kit rate:7.1s/it remaining:08:14 elapsed:5:49:32 postfix:- (1728:tqdm_logging.py:145) 12:35:57.436 [I] Progress on: 2.93kit/3.00kit rate:7.1s/it remaining:08:00 elapsed:5:49:46 postfix:- (1728:tqdm_logging.py:145) 12:36:11.553 [I] Progress on: 2.93kit/3.00kit rate:7.1s/it remaining:07:45 elapsed:5:50:00 postfix:- (1728:tqdm_logging.py:145) 12:36:25.665 [I] Progress on: 2.94kit/3.00kit rate:7.1s/it remaining:07:31 elapsed:5:50:14 postfix:- (1728:tqdm_logging.py:145) 12:36:39.781 [I] Progress on: 2.94kit/3.00kit rate:7.1s/it remaining:07:17 elapsed:5:50:28 postfix:- (1728:tqdm_logging.py:145) 12:36:53.901 [I] Progress on: 2.94kit/3.00kit rate:7.1s/it remaining:07:03 elapsed:5:50:42 postfix:- (1728:tqdm_logging.py:145) 12:37:08.001 [I] Progress on: 2.94kit/3.00kit rate:7.1s/it remaining:06:49 elapsed:5:50:56 postfix:- (1728:tqdm_logging.py:145) 12:37:22.126 [I] Progress on: 2.94kit/3.00kit rate:7.1s/it remaining:06:35 elapsed:5:51:10 postfix:- (1728:tqdm_logging.py:145) 12:37:36.322 [I] Progress on: 2.95kit/3.00kit rate:7.1s/it remaining:06:22 elapsed:5:51:25 postfix:- (1728:tqdm_logging.py:145) 12:37:50.371 [I] Progress on: 2.95kit/3.00kit rate:7.1s/it remaining:06:06 elapsed:5:51:39 postfix:- (1728:tqdm_logging.py:145) 12:38:04.494 [I] Progress on: 2.95kit/3.00kit rate:7.1s/it remaining:05:52 elapsed:5:51:53 postfix:- (1728:tqdm_logging.py:145) Step 2950: grad_norm=0.0384, loss=0.0036, param_norm=1804.1384 12:38:18.248 [I] Progress on: 2.95kit/3.00kit rate:7.1s/it remaining:05:52 elapsed:5:52:07 postfix:- (1728:tqdm_logging.py:145) 12:38:32.799 [I] Progress on: 2.95kit/3.00kit rate:6.8s/it remaining:05:11 elapsed:5:52:21 postfix:- (1728:tqdm_logging.py:145) 12:38:46.907 [I] Progress on: 2.96kit/3.00kit rate:6.9s/it remaining:05:04 elapsed:5:52:35 postfix:- (1728:tqdm_logging.py:145) 12:39:01.041 [I] Progress on: 2.96kit/3.00kit rate:7.0s/it remaining:04:53 elapsed:5:52:49 postfix:- (1728:tqdm_logging.py:145) 12:39:15.198 [I] Progress on: 2.96kit/3.00kit rate:7.0s/it remaining:04:41 elapsed:5:53:03 postfix:- (1728:tqdm_logging.py:145) 12:39:29.302 [I] Progress on: 2.96kit/3.00kit rate:7.0s/it remaining:04:27 elapsed:5:53:18 postfix:- (1728:tqdm_logging.py:145) 12:39:43.498 [I] Progress on: 2.96kit/3.00kit rate:7.1s/it remaining:04:14 elapsed:5:53:32 postfix:- (1728:tqdm_logging.py:145) 12:39:57.607 [I] Progress on: 2.97kit/3.00kit rate:7.1s/it remaining:04:00 elapsed:5:53:46 postfix:- (1728:tqdm_logging.py:145) 12:40:11.688 [I] Progress on: 2.97kit/3.00kit rate:7.1s/it remaining:03:45 elapsed:5:54:00 postfix:- (1728:tqdm_logging.py:145) 12:40:25.794 [I] Progress on: 2.97kit/3.00kit rate:7.1s/it remaining:03:31 elapsed:5:54:14 postfix:- (1728:tqdm_logging.py:145) 12:40:39.906 [I] Progress on: 2.97kit/3.00kit rate:7.1s/it remaining:03:17 elapsed:5:54:28 postfix:- (1728:tqdm_logging.py:145) 12:40:54.025 [I] Progress on: 2.97kit/3.00kit rate:7.1s/it remaining:03:03 elapsed:5:54:42 postfix:- (1728:tqdm_logging.py:145) 12:41:08.137 [I] Progress on: 2.98kit/3.00kit rate:7.1s/it remaining:02:49 elapsed:5:54:56 postfix:- (1728:tqdm_logging.py:145) 12:41:22.261 [I] Progress on: 2.98kit/3.00kit rate:7.1s/it remaining:02:35 elapsed:5:55:11 postfix:- (1728:tqdm_logging.py:145) 12:41:36.360 [I] Progress on: 2.98kit/3.00kit rate:7.1s/it remaining:02:21 elapsed:5:55:25 postfix:- (1728:tqdm_logging.py:145) 12:41:50.479 [I] Progress on: 2.98kit/3.00kit rate:7.1s/it remaining:02:07 elapsed:5:55:39 postfix:- (1728:tqdm_logging.py:145) 12:42:04.597 [I] Progress on: 2.98kit/3.00kit rate:7.1s/it remaining:01:52 elapsed:5:55:53 postfix:- (1728:tqdm_logging.py:145) 12:42:18.722 [I] Progress on: 2.99kit/3.00kit rate:7.1s/it remaining:01:38 elapsed:5:56:07 postfix:- (1728:tqdm_logging.py:145) 12:42:32.841 [I] Progress on: 2.99kit/3.00kit rate:7.1s/it remaining:01:24 elapsed:5:56:21 postfix:- (1728:tqdm_logging.py:145) 12:42:46.947 [I] Progress on: 2.99kit/3.00kit rate:7.1s/it remaining:01:10 elapsed:5:56:35 postfix:- (1728:tqdm_logging.py:145) 12:43:01.085 [I] Progress on: 2.99kit/3.00kit rate:7.1s/it remaining:00:56 elapsed:5:56:49 postfix:- (1728:tqdm_logging.py:145) 12:43:15.213 [I] Progress on: 2.99kit/3.00kit rate:7.1s/it remaining:00:42 elapsed:5:57:04 postfix:- (1728:tqdm_logging.py:145) 12:43:29.334 [I] Progress on: 3.00kit/3.00kit rate:7.1s/it remaining:00:28 elapsed:5:57:18 postfix:- (1728:tqdm_logging.py:145) 12:43:43.461 [I] Progress on: 3.00kit/3.00kit rate:7.1s/it remaining:00:14 elapsed:5:57:32 postfix:- (1728:tqdm_logging.py:145) 12:43:57.591 [I] [process=0][thread=MainThread][wait_until_finished] No Save Finalize thread to wait for. Returning. (1728:checkpoint_manager.py:1987) 12:43:57.592 [I] [process=0] Saving checkpoint at step 2999 (1728:checkpoint_manager.py:1408) 12:43:57.593 [I] [process=0] Started async saving checkpoint to /workspace/openpi/checkpoints/pi05_mecharm_deep_even10_lora/deep_even10_v1/2999. (1728:async_checkpointer.py:439) 12:43:57.595 [I] Creating tmp directory /workspace/openpi/checkpoints/pi05_mecharm_deep_even10_lora/deep_even10_v1/2999.orbax-checkpoint-tmp-8 (1728:atomicity.py:144) 12:43:57.608 [I] Creating tmp directory /workspace/openpi/checkpoints/pi05_mecharm_deep_even10_lora/deep_even10_v1/2999.orbax-checkpoint-tmp-8/assets.orbax-checkpoint-tmp-9 (1728:atomicity.py:144) 12:43:57.609 [I] Creating tmp directory /workspace/openpi/checkpoints/pi05_mecharm_deep_even10_lora/deep_even10_v1/2999.orbax-checkpoint-tmp-8/params.orbax-checkpoint-tmp-10 (1728:atomicity.py:144) 12:43:57.609 [I] Creating tmp directory /workspace/openpi/checkpoints/pi05_mecharm_deep_even10_lora/deep_even10_v1/2999.orbax-checkpoint-tmp-8/train_state.orbax-checkpoint-tmp-11 (1728:atomicity.py:144) 12:43:57.620 [I] Transferring arrays to host memory with options: use_replica_parallel=True, enable_pinned_host_transfer=False (1728:replica_slices.py:341) 12:44:08.151 [I] Transferring arrays to host memory with options: use_replica_parallel=True, enable_pinned_host_transfer=False (1728:replica_slices.py:341) 12:44:08.154 [I] Array name: 'params.PaliGemma.llm.layers.mlp.gating_einsum.value', global shape: (18, 2, 2048, 16384), write shape: (18, 2, 2048, 16384), chosen chunk shape: (18, 2, 2048, 8192) (1728:tensorstore_utils.py:408) 12:44:08.222 [I] [process=0][thread=array_type_handler] Wrote 71 array_metadata.ArrayMetadata to /workspace/openpi/checkpoints/pi05_mecharm_deep_even10_lora/deep_even10_v1/2999.orbax-checkpoint-tmp-8/params.orbax-checkpoint-tmp-10/array_metadatas/process_0 (1728:array_metadata_store.py:198) 12:44:14.500 [I] [process=0] /jax/checkpoint/write/blocking_bytes_per_sec: 436.7 MiB/s (total bytes: 7.2 GiB) (time elapsed: 16 seconds) (per-host) (1728:base_pytree_checkpoint_handler.py:114) 12:44:14.507 [I] [process=0] /jax/checkpoint/write/blocking_bytes_per_sec: 210.8 MiB/s (total bytes: 3.5 GiB) (time elapsed: 16 seconds) (per-host) (1728:base_pytree_checkpoint_handler.py:114) 12:44:14.515 [I] [process=0][thread=async_save] Background save thread started. (1728:async_checkpointer.py:77) 12:44:14.516 [I] Finished blocking save in 16.92 seconds. Continuing to save asynchronously to /workspace/openpi/checkpoints/pi05_mecharm_deep_even10_lora/deep_even10_v1/2999. (1728:async_checkpointer.py:548) 12:44:14.522 [I] Preserving Checkpoint[step=1000 | time=2026-11-09 08:48:44.411183+00:00]: (Reason: on keep_period=1000). (1728:checkpoint_manager.py:1949) 12:44:14.522 [I] Preserving Checkpoint[step=2000 | time=2026-11-09 10:46:32.891133+00:00]: (Reason: on keep_period=1000). (1728:checkpoint_manager.py:1949) 12:44:14.523 [I] [process=0][thread=MainThread][step=2999] Starting CheckpointManager Save Finalize thread=save_finalize (1728:checkpoint_manager.py:1453) 12:44:14.524 [I] [process=0][thread=save_finalize] Waiting for background save thread=async_save. (1728:async_checkpointer.py:258) 12:44:14.524 [I] {'step': 2999, 'event_type': 'save', 'directory': '/workspace/openpi/checkpoints/pi05_mecharm_deep_even10_lora/deep_even10_v1', 'reached_preemption': False, 'preemption_received_at': None, 'synchronous': False, 'wait_for_prev_start_time': 1794228237.5919025, 'wait_for_prev_duration_secs': 0.0002002716064453125, 'checkpointer_blocking_start_time': 1794228237.5925238, 'checkpointer_blocking_duration_secs': 16.929755926132202, 'get_old_steps_start_time': 1794228254.5223072, 'get_old_steps_duration_secs': 0.0012187957763671875, 'checkpoint_manager_blocking_start_time': 1794228237.5915833, 'checkpoint_manager_blocking_duration_secs': 16.933098077774048} (1728:standard_logger.py:34) 12:44:14.525 [I] Progress on: 3.00kit/3.00kit rate:12.1s/it remaining:00:00 elapsed:5:58:03 postfix:- (1728:tqdm_logging.py:145) 12:44:14.526 [I] Progress on: 3.00kit/3.00kit rate:12.1s/it remaining:00:00 elapsed:5:58:03 postfix:- (1728:tqdm_logging.py:145) 12:44:14.528 [I] Waiting for checkpoint manager to finish (1728:train.py:275) 12:44:14.528 [I] [process=0][thread=MainThread][step=2999][wait_until_finished] Waiting for Save Finalize thread (save_finalize) to complete. (1728:checkpoint_manager.py:1998) 12:44:14.532 [I] [process=0] /jax/checkpoint/write/bytes_per_sec: 435.9 MiB/s (total bytes: 7.2 GiB) (time elapsed: 16 seconds) (per-host) (1728:base_pytree_checkpoint_handler.py:114) 12:44:14.616 [I] [process=0][thread=array_type_handler] Wrote 105 array_metadata.ArrayMetadata to /workspace/openpi/checkpoints/pi05_mecharm_deep_even10_lora/deep_even10_v1/2999.orbax-checkpoint-tmp-8/train_state.orbax-checkpoint-tmp-11/array_metadatas/process_0 (1728:array_metadata_store.py:198) 12:44:17.981 [I] [process=0] /jax/checkpoint/write/bytes_per_sec: 174.8 MiB/s (total bytes: 3.5 GiB) (time elapsed: 20 seconds) (per-host) (1728:base_pytree_checkpoint_handler.py:114) 12:44:17.981 [I] [process=0][thread=async_save] 5 Handler Commit operations completed. (1728:async_checkpointer.py:87) 12:44:17.982 [I] Renaming /workspace/openpi/checkpoints/pi05_mecharm_deep_even10_lora/deep_even10_v1/2999.orbax-checkpoint-tmp-8/assets.orbax-checkpoint-tmp-9 to /workspace/openpi/checkpoints/pi05_mecharm_deep_even10_lora/deep_even10_v1/2999.orbax-checkpoint-tmp-8/assets (1728:atomicity.py:289) 12:44:17.988 [I] [process=0][thread=async_save] Skipped cross-host ArrayMetadata validation because only one process is found: process_index=0. (1728:array_metadata_store.py:362) 12:44:18.001 [I] Renaming /workspace/openpi/checkpoints/pi05_mecharm_deep_even10_lora/deep_even10_v1/2999.orbax-checkpoint-tmp-8/train_state.orbax-checkpoint-tmp-11 to /workspace/openpi/checkpoints/pi05_mecharm_deep_even10_lora/deep_even10_v1/2999.orbax-checkpoint-tmp-8/train_state (1728:atomicity.py:289) 12:44:18.010 [I] [process=0][thread=async_save] Skipped cross-host ArrayMetadata validation because only one process is found: process_index=0. (1728:array_metadata_store.py:362) 12:44:18.021 [I] Renaming /workspace/openpi/checkpoints/pi05_mecharm_deep_even10_lora/deep_even10_v1/2999.orbax-checkpoint-tmp-8/params.orbax-checkpoint-tmp-10 to /workspace/openpi/checkpoints/pi05_mecharm_deep_even10_lora/deep_even10_v1/2999.orbax-checkpoint-tmp-8/params (1728:atomicity.py:289) 12:44:18.022 [I] Renaming /workspace/openpi/checkpoints/pi05_mecharm_deep_even10_lora/deep_even10_v1/2999.orbax-checkpoint-tmp-8 to /workspace/openpi/checkpoints/pi05_mecharm_deep_even10_lora/deep_even10_v1/2999 (1728:atomicity.py:289) 12:44:18.024 [I] [process=0][thread=async_save] Finished saving checkpoint (finalized tmp dir) to `/workspace/openpi/checkpoints/pi05_mecharm_deep_even10_lora/deep_even10_v1/2999`. (1728:atomicity.py:573) 12:44:18.025 [I] Finished asynchronous save (blocking + background) in 20.43 seconds to /workspace/openpi/checkpoints/pi05_mecharm_deep_even10_lora/deep_even10_v1/2999 (1728:async_checkpointer.py:412) 12:44:18.025 [I] [process=0][thread=async_save] Background save thread done. (1728:async_checkpointer.py:143) 12:44:18.025 [I] [process=0][thread=save_finalize] Done with waiting for background save thread=async_save. (1728:async_checkpointer.py:266) 12:44:18.025 [I] [process=0][thread=save_finalize] No errors found in background save thread=async_save. (1728:async_checkpointer.py:276) 12:44:18.026 [I] [process=0][thread=save_finalize][step=2999] CheckpointManager Save Finalize is syncing with other hosts... (1728:checkpoint_manager.py:2095) 12:44:18.026 [I] [process=0][thread=save_finalize][step=2999] CheckpointManager Save Finalize is done on all hosts. (1728:checkpoint_manager.py:2110) 12:44:18.026 [I] [process=0][thread=MainThread][step=2999][wait_until_finished] Done waiting for Save Finalize thread (save_finalize) running at step=2999. (1728:checkpoint_manager.py:2010) ``` ### 学習を始めた時刻と終わった時刻（train_run.log） ```text 2026-11-09 06:44:30 始める 2026-11-09 12:44:22 終わった (終了コード 0) ``` ### 学習中の GPU の記録・30秒ごと（gpu_train.csv） ```text timestamp, name, memory.used [MiB], memory.total [MiB], utilization.gpu [%], temperature.gpu, power.draw [W] 2026/11/09 06:44:30.118, NVIDIA A40, 0 MiB, 46068 MiB, 0 %, 36, 59.69 W 2026/11/09 06:45:00.142, NVIDIA A40, 41221 MiB, 46068 MiB, 0 %, 41, 111.01 W 2026/11/09 06:45:30.143, NVIDIA A40, 41221 MiB, 46068 MiB, 0 %, 42, 111.72 W 2026/11/09 06:46:00.144, NVIDIA A40, 41221 MiB, 46068 MiB, 0 %, 42, 111.99 W 2026/11/09 06:46:30.187, NVIDIA A40, 41229 MiB, 46068 MiB, 0 %, 43, 112.52 W 2026/11/09 06:47:00.187, NVIDIA A40, 41229 MiB, 46068 MiB, 0 %, 43, 112.15 W 2026/11/09 06:47:30.191, NVIDIA A40, 41229 MiB, 46068 MiB, 0 %, 43, 112.25 W 2026/11/09 06:48:00.191, NVIDIA A40, 41229 MiB, 46068 MiB, 0 %, 43, 112.27 W 2026/11/09 06:48:30.192, NVIDIA A40, 41229 MiB, 46068 MiB, 0 %, 43, 112.22 W 2026/11/09 06:49:00.291, NVIDIA A40, 41229 MiB, 46068 MiB, 0 %, 43, 112.15 W 2026/11/09 06:49:30.293, NVIDIA A40, 41273 MiB, 46068 MiB, 98 %, 60, 290.59 W 2026/11/09 06:50:00.293, NVIDIA A40, 41275 MiB, 46068 MiB, 0 %, 48, 114.07 W 2026/11/09 06:50:30.295, NVIDIA A40, 41275 MiB, 46068 MiB, 0 %, 46, 116.01 W 2026/11/09 06:51:00.301, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 62, 295.18 W 2026/11/09 06:51:30.302, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 66, 254.17 W 2026/11/09 06:52:00.303, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 68, 294.02 W 2026/11/09 06:52:30.304, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 70, 295.49 W 2026/11/09 06:53:00.305, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 71, 299.71 W 2026/11/09 06:53:30.306, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 71, 281.90 W 2026/11/09 06:54:00.307, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 72, 296.49 W 2026/11/09 06:54:30.309, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 72, 298.24 W 2026/11/09 06:55:00.310, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 72, 324.46 W 2026/11/09 06:55:30.311, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 301.62 W 2026/11/09 06:56:00.312, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 74, 295.72 W 2026/11/09 06:56:30.313, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 74, 294.92 W 2026/11/09 06:57:00.315, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 74, 299.98 W 2026/11/09 06:57:30.315, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 269.36 W 2026/11/09 06:58:00.316, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 74, 294.70 W 2026/11/09 06:58:30.317, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 297.59 W 2026/11/09 06:59:00.318, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 74, 288.71 W 2026/11/09 06:59:30.318, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 74, 311.76 W 2026/11/09 07:00:00.318, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 292.23 W 2026/11/09 07:00:30.320, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 294.21 W 2026/11/09 07:01:00.321, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 74, 297.95 W 2026/11/09 07:01:30.322, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 74, 292.83 W 2026/11/09 07:02:00.322, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 74, 295.71 W 2026/11/09 07:02:30.323, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 294.06 W 2026/11/09 07:03:00.325, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 74, 302.36 W 2026/11/09 07:03:30.326, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 74, 303.26 W 2026/11/09 07:04:00.327, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 297.14 W 2026/11/09 07:04:30.328, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 292.16 W 2026/11/09 07:05:00.330, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 74, 296.35 W 2026/11/09 07:05:30.331, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 74, 294.74 W 2026/11/09 07:06:00.332, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 74, 271.82 W 2026/11/09 07:06:30.333, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 295.59 W 2026/11/09 07:07:00.334, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 301.77 W 2026/11/09 07:07:30.335, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 74, 303.71 W 2026/11/09 07:08:00.336, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 74, 313.31 W 2026/11/09 07:08:30.337, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 74, 298.05 W 2026/11/09 07:09:00.338, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 74, 295.77 W 2026/11/09 07:09:30.339, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 296.25 W 2026/11/09 07:10:00.340, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 74, 310.81 W 2026/11/09 07:10:30.342, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 74, 291.57 W 2026/11/09 07:11:00.343, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 295.12 W 2026/11/09 07:11:30.344, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 74, 300.56 W 2026/11/09 07:12:00.344, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 261.05 W 2026/11/09 07:12:30.345, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 300.66 W 2026/11/09 07:13:00.347, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 74, 296.94 W 2026/11/09 07:13:30.348, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 74, 297.59 W 2026/11/09 07:14:00.349, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 74, 312.93 W 2026/11/09 07:14:30.350, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 320.79 W 2026/11/09 07:15:00.351, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 74, 298.99 W 2026/11/09 07:15:30.352, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 297.99 W 2026/11/09 07:16:00.353, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 301.47 W 2026/11/09 07:16:30.354, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 74, 328.65 W 2026/11/09 07:17:00.356, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 74, 289.52 W 2026/11/09 07:17:30.356, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 74, 294.79 W 2026/11/09 07:18:00.357, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 74, 291.73 W 2026/11/09 07:18:30.358, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 268.48 W 2026/11/09 07:19:00.358, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 74, 293.00 W 2026/11/09 07:19:30.359, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 299.56 W 2026/11/09 07:20:00.360, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 74, 304.67 W 2026/11/09 07:20:30.361, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 74, 319.80 W 2026/11/09 07:21:00.362, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 74, 282.18 W 2026/11/09 07:21:30.363, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 299.08 W 2026/11/09 07:22:00.364, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 74, 303.95 W 2026/11/09 07:22:30.366, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 74, 305.83 W 2026/11/09 07:23:00.367, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 74, 310.15 W 2026/11/09 07:23:30.368, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 293.29 W 2026/11/09 07:24:00.369, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 74, 297.55 W 2026/11/09 07:24:30.370, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 74, 295.32 W 2026/11/09 07:25:00.370, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 317.05 W 2026/11/09 07:25:30.371, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 291.01 W 2026/11/09 07:26:00.372, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 291.40 W 2026/11/09 07:26:30.373, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 74, 296.51 W 2026/11/09 07:27:00.374, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 74, 319.85 W 2026/11/09 07:27:30.375, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 296.12 W 2026/11/09 07:28:00.376, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 74, 296.26 W 2026/11/09 07:28:30.377, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 74, 300.34 W 2026/11/09 07:29:00.378, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 74, 295.66 W 2026/11/09 07:29:30.378, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 74, 294.98 W 2026/11/09 07:30:00.380, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 74, 296.56 W 2026/11/09 07:30:30.381, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 74, 300.10 W 2026/11/09 07:31:00.382, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 74, 297.81 W 2026/11/09 07:31:30.383, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 74, 299.80 W 2026/11/09 07:32:00.385, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 295.06 W 2026/11/09 07:32:30.386, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 74, 297.72 W 2026/11/09 07:33:00.387, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 74, 305.73 W 2026/11/09 07:33:30.388, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 302.55 W 2026/11/09 07:34:00.389, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 74, 296.94 W 2026/11/09 07:34:30.390, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 74, 287.36 W 2026/11/09 07:35:00.391, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 74, 299.06 W 2026/11/09 07:35:30.392, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 74, 296.29 W 2026/11/09 07:36:00.392, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 74, 293.37 W 2026/11/09 07:36:30.393, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 74, 297.22 W 2026/11/09 07:37:00.395, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 74, 309.52 W 2026/11/09 07:37:30.396, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 74, 317.97 W 2026/11/09 07:38:00.397, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 72, 296.47 W 2026/11/09 07:38:30.398, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 291.45 W 2026/11/09 07:39:00.398, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 288.94 W 2026/11/09 07:39:30.399, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 72, 278.28 W 2026/11/09 07:40:00.400, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 290.99 W 2026/11/09 07:40:30.401, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 291.09 W 2026/11/09 07:41:00.402, NVIDIA A40, 41283 MiB, 46068 MiB, 99 %, 73, 301.77 W 2026/11/09 07:41:30.403, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 315.12 W 2026/11/09 07:42:00.404, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 72, 291.34 W 2026/11/09 07:42:30.405, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 72, 293.10 W 2026/11/09 07:43:00.406, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 295.43 W 2026/11/09 07:43:30.407, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 321.25 W 2026/11/09 07:44:00.408, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 72, 295.61 W 2026/11/09 07:44:30.408, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 72, 288.80 W 2026/11/09 07:45:00.410, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 292.53 W 2026/11/09 07:45:30.411, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 315.43 W 2026/11/09 07:46:00.412, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 72, 290.04 W 2026/11/09 07:46:30.413, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 299.89 W 2026/11/09 07:47:00.414, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 293.38 W 2026/11/09 07:47:30.416, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 260.30 W 2026/11/09 07:48:00.417, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 72, 301.09 W 2026/11/09 07:48:30.419, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 72, 294.54 W 2026/11/09 07:49:00.420, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 301.95 W 2026/11/09 07:49:30.421, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 294.51 W 2026/11/09 07:50:00.422, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 311.60 W 2026/11/09 07:50:30.423, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 72, 294.85 W 2026/11/09 07:51:00.425, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 310.05 W 2026/11/09 07:51:30.426, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 294.98 W 2026/11/09 07:52:00.427, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 72, 306.03 W 2026/11/09 07:52:30.428, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 72, 294.84 W 2026/11/09 07:53:00.428, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 301.15 W 2026/11/09 07:53:30.429, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 272.48 W 2026/11/09 07:54:00.430, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 72, 290.08 W 2026/11/09 07:54:30.431, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 72, 295.33 W 2026/11/09 07:55:00.431, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 295.65 W 2026/11/09 07:55:30.432, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 292.40 W 2026/11/09 07:56:00.433, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 74, 283.16 W 2026/11/09 07:56:30.434, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 74, 296.06 W 2026/11/09 07:57:00.435, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 289.89 W 2026/11/09 07:57:30.435, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 291.93 W 2026/11/09 07:58:00.437, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 262.98 W 2026/11/09 07:58:30.437, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 301.01 W 2026/11/09 07:59:00.438, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 299.78 W 2026/11/09 07:59:30.439, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 74, 295.58 W 2026/11/09 08:00:00.440, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 300.82 W 2026/11/09 08:00:30.441, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 74, 297.11 W 2026/11/09 08:01:00.442, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 292.36 W 2026/11/09 08:01:30.443, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 296.80 W 2026/11/09 08:02:00.445, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 291.90 W 2026/11/09 08:02:30.446, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 292.72 W 2026/11/09 08:03:00.447, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 298.70 W 2026/11/09 08:03:30.448, NVIDIA A40, 41283 MiB, 46068 MiB, 99 %, 73, 297.04 W 2026/11/09 08:04:00.449, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 316.91 W 2026/11/09 08:04:30.451, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 298.32 W 2026/11/09 08:05:00.452, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 298.04 W 2026/11/09 08:05:30.453, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 304.29 W 2026/11/09 08:06:00.454, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 74, 300.16 W 2026/11/09 08:06:30.455, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 293.33 W 2026/11/09 08:07:00.457, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 293.69 W 2026/11/09 08:07:30.458, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 302.93 W 2026/11/09 08:08:00.459, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 72, 292.38 W 2026/11/09 08:08:30.460, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 298.07 W 2026/11/09 08:09:00.461, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 297.23 W 2026/11/09 08:09:30.462, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 294.99 W 2026/11/09 08:10:00.465, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 309.85 W 2026/11/09 08:10:30.466, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 291.69 W 2026/11/09 08:11:00.469, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 72, 293.91 W 2026/11/09 08:11:30.470, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 297.09 W 2026/11/09 08:12:00.471, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 304.08 W 2026/11/09 08:12:30.472, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 72, 293.70 W 2026/11/09 08:13:00.473, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 72, 294.94 W 2026/11/09 08:13:30.473, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 306.47 W 2026/11/09 08:14:00.474, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 258.37 W 2026/11/09 08:14:30.475, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 296.93 W 2026/11/09 08:15:00.478, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 291.36 W 2026/11/09 08:15:30.479, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 299.52 W 2026/11/09 08:16:00.480, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 329.31 W 2026/11/09 08:16:30.481, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 296.45 W 2026/11/09 08:17:00.482, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 72, 297.99 W 2026/11/09 08:17:30.484, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 300.17 W 2026/11/09 08:18:00.485, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 299.34 W 2026/11/09 08:18:30.486, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 300.65 W 2026/11/09 08:19:00.486, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 297.55 W 2026/11/09 08:19:30.486, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 301.62 W 2026/11/09 08:20:00.487, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 257.45 W 2026/11/09 08:20:30.489, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 72, 297.69 W 2026/11/09 08:21:00.490, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 293.22 W 2026/11/09 08:21:30.491, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 291.71 W 2026/11/09 08:22:00.492, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 313.69 W 2026/11/09 08:22:30.493, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 72, 289.57 W 2026/11/09 08:23:00.494, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 292.91 W 2026/11/09 08:23:30.496, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 301.72 W 2026/11/09 08:24:00.496, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 322.22 W 2026/11/09 08:24:30.498, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 72, 300.45 W 2026/11/09 08:25:00.499, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 297.53 W 2026/11/09 08:25:30.500, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 301.69 W 2026/11/09 08:26:00.501, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 258.88 W 2026/11/09 08:26:30.502, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 297.03 W 2026/11/09 08:27:00.503, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 292.42 W 2026/11/09 08:27:30.504, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 295.17 W 2026/11/09 08:28:00.505, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 285.08 W 2026/11/09 08:28:30.506, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 293.36 W 2026/11/09 08:29:00.507, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 297.92 W 2026/11/09 08:29:30.508, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 286.32 W 2026/11/09 08:30:00.509, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 326.90 W 2026/11/09 08:30:30.510, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 298.77 W 2026/11/09 08:31:00.511, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 291.06 W 2026/11/09 08:31:30.512, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 295.12 W 2026/11/09 08:32:00.513, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 315.80 W 2026/11/09 08:32:30.514, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 293.13 W 2026/11/09 08:33:00.515, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 72, 296.45 W 2026/11/09 08:33:30.517, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 293.28 W 2026/11/09 08:34:00.518, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 292.20 W 2026/11/09 08:34:30.520, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 293.88 W 2026/11/09 08:35:00.520, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 298.91 W 2026/11/09 08:35:30.521, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 287.12 W 2026/11/09 08:36:00.522, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 304.52 W 2026/11/09 08:36:30.523, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 72, 296.06 W 2026/11/09 08:37:00.524, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 72, 298.24 W 2026/11/09 08:37:30.527, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 293.46 W 2026/11/09 08:38:00.528, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 288.34 W 2026/11/09 08:38:30.529, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 295.92 W 2026/11/09 08:39:00.530, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 72, 292.82 W 2026/11/09 08:39:30.531, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 309.44 W 2026/11/09 08:40:00.532, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 254.50 W 2026/11/09 08:40:30.533, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 72, 296.55 W 2026/11/09 08:41:00.534, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 72, 299.57 W 2026/11/09 08:41:30.535, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 304.59 W 2026/11/09 08:42:00.536, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 257.21 W 2026/11/09 08:42:30.537, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 72, 294.60 W 2026/11/09 08:43:00.538, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 295.65 W 2026/11/09 08:43:30.539, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 286.23 W 2026/11/09 08:44:00.539, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 313.24 W 2026/11/09 08:44:30.540, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 288.59 W 2026/11/09 08:45:00.605, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 72, 294.05 W 2026/11/09 08:45:30.606, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 302.22 W 2026/11/09 08:46:00.610, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 320.76 W 2026/11/09 08:46:30.611, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 74, 301.00 W 2026/11/09 08:47:00.612, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 296.59 W 2026/11/09 08:47:30.613, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 294.49 W 2026/11/09 08:48:00.614, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 285.60 W 2026/11/09 08:48:30.615, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 295.26 W 2026/11/09 08:49:00.616, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 71, 255.71 W 2026/11/09 08:49:30.617, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 296.89 W 2026/11/09 08:50:00.618, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 74, 295.70 W 2026/11/09 08:50:30.619, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 74, 303.46 W 2026/11/09 08:51:00.620, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 74, 262.88 W 2026/11/09 08:51:30.622, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 74, 295.06 W 2026/11/09 08:52:00.632, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 74, 294.10 W 2026/11/09 08:52:30.633, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 74, 305.36 W 2026/11/09 08:53:00.633, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 74, 283.04 W 2026/11/09 08:53:30.636, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 300.26 W 2026/11/09 08:54:00.637, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 292.20 W 2026/11/09 08:54:30.637, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 298.53 W 2026/11/09 08:55:00.639, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 258.36 W 2026/11/09 08:55:30.640, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 295.89 W 2026/11/09 08:56:00.641, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 293.94 W 2026/11/09 08:56:30.645, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 294.40 W 2026/11/09 08:57:00.657, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 292.03 W 2026/11/09 08:57:30.670, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 297.89 W 2026/11/09 08:58:00.683, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 294.37 W 2026/11/09 08:58:30.695, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 74, 302.50 W 2026/11/09 08:59:00.701, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 74, 318.58 W 2026/11/09 08:59:30.709, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 74, 295.61 W 2026/11/09 09:00:00.724, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 74, 294.26 W 2026/11/09 09:00:30.732, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 70, 279.81 W 2026/11/09 09:01:00.743, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 74, 325.56 W 2026/11/09 09:01:30.756, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 297.38 W 2026/11/09 09:02:00.767, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 298.32 W 2026/11/09 09:02:30.776, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 299.29 W 2026/11/09 09:03:00.786, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 258.78 W 2026/11/09 09:03:30.798, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 74, 296.18 W 2026/11/09 09:04:00.807, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 297.66 W 2026/11/09 09:04:30.817, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 74, 299.74 W 2026/11/09 09:05:00.827, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 74, 299.72 W 2026/11/09 09:05:30.838, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 74, 298.90 W 2026/11/09 09:06:00.850, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 74, 292.68 W 2026/11/09 09:06:30.858, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 74, 300.24 W 2026/11/09 09:07:00.870, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 74, 291.06 W 2026/11/09 09:07:30.881, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 292.56 W 2026/11/09 09:08:00.891, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 74, 296.82 W 2026/11/09 09:08:30.905, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 74, 298.66 W 2026/11/09 09:09:00.915, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 74, 326.41 W 2026/11/09 09:09:30.927, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 298.54 W 2026/11/09 09:10:00.935, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 298.70 W 2026/11/09 09:10:30.947, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 286.92 W 2026/11/09 09:11:00.957, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 288.96 W 2026/11/09 09:11:30.966, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 293.34 W 2026/11/09 09:12:00.975, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 295.35 W 2026/11/09 09:12:30.986, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 72, 299.99 W 2026/11/09 09:13:01.000, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 261.54 W 2026/11/09 09:13:31.012, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 74, 294.58 W 2026/11/09 09:14:01.020, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 72, 287.07 W 2026/11/09 09:14:31.029, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 296.52 W 2026/11/09 09:15:01.039, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 300.93 W 2026/11/09 09:15:31.054, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 298.69 W 2026/11/09 09:16:01.061, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 293.95 W 2026/11/09 09:16:31.071, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 294.31 W 2026/11/09 09:17:01.085, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 74, 301.13 W 2026/11/09 09:17:31.094, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 295.06 W 2026/11/09 09:18:01.107, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 74, 294.44 W 2026/11/09 09:18:31.120, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 294.03 W 2026/11/09 09:19:01.129, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 74, 300.68 W 2026/11/09 09:19:31.155, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 72, 293.08 W 2026/11/09 09:20:01.155, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 74, 297.40 W 2026/11/09 09:20:31.157, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 301.04 W 2026/11/09 09:21:01.169, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 254.62 W 2026/11/09 09:21:31.180, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 294.99 W 2026/11/09 09:22:01.193, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 74, 293.25 W 2026/11/09 09:22:31.202, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 296.72 W 2026/11/09 09:23:01.213, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 255.92 W 2026/11/09 09:23:31.225, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 298.37 W 2026/11/09 09:24:01.234, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 298.36 W 2026/11/09 09:24:31.244, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 298.67 W 2026/11/09 09:25:01.254, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 258.08 W 2026/11/09 09:25:31.264, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 291.96 W 2026/11/09 09:26:01.277, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 301.24 W 2026/11/09 09:26:31.291, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 297.20 W 2026/11/09 09:27:01.297, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 74, 296.69 W 2026/11/09 09:27:31.309, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 72, 292.19 W 2026/11/09 09:28:01.320, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 72, 294.07 W 2026/11/09 09:28:31.328, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 301.08 W 2026/11/09 09:29:01.343, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 306.83 W 2026/11/09 09:29:31.352, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 72, 295.31 W 2026/11/09 09:30:01.360, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 72, 294.61 W 2026/11/09 09:30:31.375, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 297.22 W 2026/11/09 09:31:01.385, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 316.96 W 2026/11/09 09:31:31.393, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 298.72 W 2026/11/09 09:32:01.406, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 298.18 W 2026/11/09 09:32:31.422, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 301.60 W 2026/11/09 09:33:01.427, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 303.10 W 2026/11/09 09:33:31.438, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 296.66 W 2026/11/09 09:34:01.447, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 294.30 W 2026/11/09 09:34:31.459, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 298.15 W 2026/11/09 09:35:01.468, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 283.68 W 2026/11/09 09:35:31.483, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 72, 293.41 W 2026/11/09 09:36:01.495, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 301.64 W 2026/11/09 09:36:31.504, NVIDIA A40, 41283 MiB, 46068 MiB, 99 %, 73, 302.93 W 2026/11/09 09:37:01.514, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 296.43 W 2026/11/09 09:37:31.531, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 72, 295.96 W 2026/11/09 09:38:01.538, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 72, 294.19 W 2026/11/09 09:38:31.545, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 296.52 W 2026/11/09 09:39:01.557, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 301.06 W 2026/11/09 09:39:31.569, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 295.31 W 2026/11/09 09:40:01.584, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 291.74 W 2026/11/09 09:40:31.593, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 304.16 W 2026/11/09 09:41:01.599, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 255.99 W 2026/11/09 09:41:31.614, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 292.34 W 2026/11/09 09:42:01.623, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 293.54 W 2026/11/09 09:42:31.636, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 299.21 W 2026/11/09 09:43:01.644, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 263.91 W 2026/11/09 09:43:31.655, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 297.41 W 2026/11/09 09:44:01.668, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 295.90 W 2026/11/09 09:44:31.674, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 294.53 W 2026/11/09 09:45:01.685, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 286.13 W 2026/11/09 09:45:31.697, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 299.00 W 2026/11/09 09:46:01.709, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 295.98 W 2026/11/09 09:46:31.719, NVIDIA A40, 41283 MiB, 46068 MiB, 99 %, 73, 288.70 W 2026/11/09 09:47:01.728, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 74, 305.23 W 2026/11/09 09:47:31.743, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 74, 299.37 W 2026/11/09 09:48:01.752, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 294.52 W 2026/11/09 09:48:31.765, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 293.16 W 2026/11/09 09:49:01.771, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 306.89 W 2026/11/09 09:49:31.782, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 72, 295.00 W 2026/11/09 09:50:01.792, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 72, 296.59 W 2026/11/09 09:50:31.803, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 285.71 W 2026/11/09 09:51:01.814, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 72, 304.95 W 2026/11/09 09:51:31.825, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 299.04 W 2026/11/09 09:52:01.834, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 292.85 W 2026/11/09 09:52:31.844, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 306.84 W 2026/11/09 09:53:01.854, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 294.86 W 2026/11/09 09:53:31.868, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 300.95 W 2026/11/09 09:54:01.877, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 291.93 W 2026/11/09 09:54:31.888, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 303.67 W 2026/11/09 09:55:01.898, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 262.03 W 2026/11/09 09:55:31.908, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 290.51 W 2026/11/09 09:56:01.919, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 72, 297.36 W 2026/11/09 09:56:31.934, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 297.27 W 2026/11/09 09:57:01.941, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 260.09 W 2026/11/09 09:57:31.953, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 72, 293.56 W 2026/11/09 09:58:01.964, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 74, 294.28 W 2026/11/09 09:58:31.977, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 308.28 W 2026/11/09 09:59:01.986, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 314.41 W 2026/11/09 09:59:31.995, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 297.23 W 2026/11/09 10:00:02.007, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 74, 295.10 W 2026/11/09 10:00:32.015, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 303.07 W 2026/11/09 10:01:02.025, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 318.67 W 2026/11/09 10:01:32.038, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 72, 294.64 W 2026/11/09 10:02:02.053, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 72, 293.82 W 2026/11/09 10:02:32.067, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 286.04 W 2026/11/09 10:03:02.080, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 72, 298.08 W 2026/11/09 10:03:32.084, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 301.13 W 2026/11/09 10:04:02.099, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 292.73 W 2026/11/09 10:04:32.108, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 72, 294.36 W 2026/11/09 10:05:02.116, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 301.46 W 2026/11/09 10:05:32.124, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 299.87 W 2026/11/09 10:06:02.138, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 298.77 W 2026/11/09 10:06:32.147, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 294.82 W 2026/11/09 10:07:02.159, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 297.38 W 2026/11/09 10:07:32.174, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 72, 296.17 W 2026/11/09 10:08:02.183, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 293.57 W 2026/11/09 10:08:32.194, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 310.61 W 2026/11/09 10:09:02.203, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 273.27 W 2026/11/09 10:09:32.215, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 74, 297.05 W 2026/11/09 10:10:02.225, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 295.80 W 2026/11/09 10:10:32.234, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 72, 277.73 W 2026/11/09 10:11:02.246, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 72, 310.39 W 2026/11/09 10:11:32.253, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 299.25 W 2026/11/09 10:12:02.265, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 298.36 W 2026/11/09 10:12:32.274, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 286.27 W 2026/11/09 10:13:02.285, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 301.44 W 2026/11/09 10:13:32.295, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 299.57 W 2026/11/09 10:14:02.304, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 292.83 W 2026/11/09 10:14:32.318, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 288.67 W 2026/11/09 10:15:02.327, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 72, 299.54 W 2026/11/09 10:15:32.344, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 72, 293.21 W 2026/11/09 10:16:02.354, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 72, 296.49 W 2026/11/09 10:16:32.360, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 311.63 W 2026/11/09 10:17:02.371, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 312.13 W 2026/11/09 10:17:32.385, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 294.24 W 2026/11/09 10:18:02.394, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 72, 290.85 W 2026/11/09 10:18:32.403, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 298.48 W 2026/11/09 10:19:02.413, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 72, 291.41 W 2026/11/09 10:19:32.429, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 72, 290.30 W 2026/11/09 10:20:02.439, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 300.24 W 2026/11/09 10:20:32.448, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 296.55 W 2026/11/09 10:21:02.458, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 310.20 W 2026/11/09 10:21:32.468, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 72, 295.84 W 2026/11/09 10:22:02.479, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 286.63 W 2026/11/09 10:22:32.490, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 287.55 W 2026/11/09 10:23:02.501, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 72, 299.11 W 2026/11/09 10:23:32.512, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 72, 296.10 W 2026/11/09 10:24:02.523, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 294.47 W 2026/11/09 10:24:32.533, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 302.93 W 2026/11/09 10:25:02.546, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 289.00 W 2026/11/09 10:25:32.556, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 294.62 W 2026/11/09 10:26:02.567, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 291.90 W 2026/11/09 10:26:32.577, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 304.55 W 2026/11/09 10:27:02.586, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 287.35 W 2026/11/09 10:27:32.595, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 298.58 W 2026/11/09 10:28:02.609, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 291.22 W 2026/11/09 10:28:32.618, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 303.76 W 2026/11/09 10:29:02.630, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 290.97 W 2026/11/09 10:29:32.639, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 298.09 W 2026/11/09 10:30:02.653, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 301.19 W 2026/11/09 10:30:32.709, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 299.17 W 2026/11/09 10:31:02.715, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 72, 294.25 W 2026/11/09 10:31:32.730, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 293.54 W 2026/11/09 10:32:02.737, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 300.45 W 2026/11/09 10:32:32.751, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 258.58 W 2026/11/09 10:33:02.755, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 72, 294.01 W 2026/11/09 10:33:32.767, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 72, 292.33 W 2026/11/09 10:34:02.776, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 291.62 W 2026/11/09 10:34:32.788, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 316.10 W 2026/11/09 10:35:02.799, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 294.98 W 2026/11/09 10:35:32.810, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 72, 291.14 W 2026/11/09 10:36:02.820, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 294.03 W 2026/11/09 10:36:32.833, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 287.67 W 2026/11/09 10:37:02.844, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 293.49 W 2026/11/09 10:37:32.856, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 72, 291.46 W 2026/11/09 10:38:02.868, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 72, 286.98 W 2026/11/09 10:38:32.878, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 324.92 W 2026/11/09 10:39:02.888, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 292.23 W 2026/11/09 10:39:32.901, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 72, 293.26 W 2026/11/09 10:40:02.907, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 296.22 W 2026/11/09 10:40:32.917, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 257.11 W 2026/11/09 10:41:02.929, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 294.25 W 2026/11/09 10:41:32.941, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 72, 292.80 W 2026/11/09 10:42:02.950, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 299.54 W 2026/11/09 10:42:32.959, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 255.56 W 2026/11/09 10:43:02.970, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 72, 294.64 W 2026/11/09 10:43:32.981, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 72, 296.76 W 2026/11/09 10:44:02.999, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 304.45 W 2026/11/09 10:44:33.005, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 306.33 W 2026/11/09 10:45:03.016, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 72, 294.30 W 2026/11/09 10:45:33.125, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 72, 296.31 W 2026/11/09 10:46:03.137, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 295.79 W 2026/11/09 10:46:33.148, NVIDIA A40, 41283 MiB, 46068 MiB, 89 %, 65, 161.35 W 2026/11/09 10:47:03.155, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 287.16 W 2026/11/09 10:47:33.166, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 298.33 W 2026/11/09 10:48:03.180, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 292.71 W 2026/11/09 10:48:33.189, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 74, 293.79 W 2026/11/09 10:49:03.202, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 74, 303.04 W 2026/11/09 10:49:33.212, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 74, 296.17 W 2026/11/09 10:50:03.221, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 294.34 W 2026/11/09 10:50:33.237, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 74, 293.81 W 2026/11/09 10:51:03.243, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 74, 299.21 W 2026/11/09 10:51:33.254, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 74, 294.64 W 2026/11/09 10:52:03.266, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 74, 292.10 W 2026/11/09 10:52:33.275, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 74, 295.96 W 2026/11/09 10:53:03.286, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 74, 323.78 W 2026/11/09 10:53:33.297, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 74, 297.62 W 2026/11/09 10:54:03.305, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 74, 293.48 W 2026/11/09 10:54:33.315, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 74, 297.50 W 2026/11/09 10:55:03.325, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 74, 314.67 W 2026/11/09 10:55:33.337, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 74, 296.45 W 2026/11/09 10:56:03.348, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 75, 298.01 W 2026/11/09 10:56:33.361, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 74, 294.51 W 2026/11/09 10:57:03.373, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 319.62 W 2026/11/09 10:57:33.380, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 299.74 W 2026/11/09 10:58:03.391, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 72, 290.52 W 2026/11/09 10:58:33.401, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 296.20 W 2026/11/09 10:59:03.413, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 310.98 W 2026/11/09 10:59:33.427, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 72, 293.37 W 2026/11/09 11:00:03.435, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 72, 294.89 W 2026/11/09 11:00:33.447, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 298.25 W 2026/11/09 11:01:03.456, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 296.00 W 2026/11/09 11:01:33.468, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 292.99 W 2026/11/09 11:02:03.479, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 72, 297.96 W 2026/11/09 11:02:33.491, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 72, 298.83 W 2026/11/09 11:03:03.502, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 291.61 W 2026/11/09 11:03:33.516, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 294.35 W 2026/11/09 11:04:03.526, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 72, 295.14 W 2026/11/09 11:04:33.538, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 289.35 W 2026/11/09 11:05:03.548, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 328.04 W 2026/11/09 11:05:33.556, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 72, 293.94 W 2026/11/09 11:06:03.570, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 290.12 W 2026/11/09 11:06:33.578, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 297.96 W 2026/11/09 11:07:03.586, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 315.23 W 2026/11/09 11:07:33.601, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 297.98 W 2026/11/09 11:08:03.612, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 301.94 W 2026/11/09 11:08:33.622, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 299.41 W 2026/11/09 11:09:03.635, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 72, 262.53 W 2026/11/09 11:09:33.647, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 289.83 W 2026/11/09 11:10:03.650, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 293.50 W 2026/11/09 11:10:33.660, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 298.60 W 2026/11/09 11:11:03.669, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 72, 258.88 W 2026/11/09 11:11:33.680, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 72, 299.85 W 2026/11/09 11:12:03.690, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 291.80 W 2026/11/09 11:12:33.701, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 315.76 W 2026/11/09 11:13:03.714, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 290.68 W 2026/11/09 11:13:33.725, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 290.29 W 2026/11/09 11:14:03.735, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 294.59 W 2026/11/09 11:14:33.744, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 290.08 W 2026/11/09 11:15:03.755, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 72, 298.82 W 2026/11/09 11:15:33.777, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 295.55 W 2026/11/09 11:16:03.780, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 291.89 W 2026/11/09 11:16:33.788, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 321.38 W 2026/11/09 11:17:03.798, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 292.97 W 2026/11/09 11:17:33.811, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 297.37 W 2026/11/09 11:18:03.823, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 300.12 W 2026/11/09 11:18:33.831, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 297.80 W 2026/11/09 11:19:03.845, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 306.67 W 2026/11/09 11:19:33.861, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 290.58 W 2026/11/09 11:20:03.862, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 298.28 W 2026/11/09 11:20:33.872, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 265.98 W 2026/11/09 11:21:03.885, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 302.48 W 2026/11/09 11:21:33.893, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 72, 293.79 W 2026/11/09 11:22:03.908, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 300.13 W 2026/11/09 11:22:33.918, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 273.78 W 2026/11/09 11:23:03.929, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 293.59 W 2026/11/09 11:23:33.942, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 72, 295.05 W 2026/11/09 11:24:03.954, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 299.12 W 2026/11/09 11:24:33.962, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 289.30 W 2026/11/09 11:25:03.970, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 291.96 W 2026/11/09 11:25:33.980, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 72, 295.92 W 2026/11/09 11:26:03.993, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 306.29 W 2026/11/09 11:26:34.004, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 299.19 W 2026/11/09 11:27:04.013, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 298.03 W 2026/11/09 11:27:34.026, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 293.45 W 2026/11/09 11:28:04.038, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 289.68 W 2026/11/09 11:28:34.047, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 300.21 W 2026/11/09 11:29:04.061, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 294.34 W 2026/11/09 11:29:34.070, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 293.86 W 2026/11/09 11:30:04.082, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 303.56 W 2026/11/09 11:30:34.091, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 273.72 W 2026/11/09 11:31:04.099, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 292.62 W 2026/11/09 11:31:34.110, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 74, 293.38 W 2026/11/09 11:32:04.121, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 301.31 W 2026/11/09 11:32:34.130, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 316.69 W 2026/11/09 11:33:04.141, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 72, 291.66 W 2026/11/09 11:33:34.155, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 72, 300.30 W 2026/11/09 11:34:04.163, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 296.60 W 2026/11/09 11:34:34.174, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 72, 327.17 W 2026/11/09 11:35:04.184, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 72, 294.45 W 2026/11/09 11:35:34.198, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 294.74 W 2026/11/09 11:36:04.209, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 297.56 W 2026/11/09 11:36:34.217, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 319.17 W 2026/11/09 11:37:04.232, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 298.15 W 2026/11/09 11:37:34.241, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 72, 298.84 W 2026/11/09 11:38:04.250, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 293.93 W 2026/11/09 11:38:34.260, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 72, 268.81 W 2026/11/09 11:39:04.273, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 292.83 W 2026/11/09 11:39:34.284, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 72, 293.94 W 2026/11/09 11:40:04.295, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 308.83 W 2026/11/09 11:40:34.307, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 296.18 W 2026/11/09 11:41:04.315, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 72, 298.20 W 2026/11/09 11:41:34.328, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 72, 294.95 W 2026/11/09 11:42:04.337, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 289.91 W 2026/11/09 11:42:34.347, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 74, 300.50 W 2026/11/09 11:43:04.362, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 74, 299.91 W 2026/11/09 11:43:34.371, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 293.72 W 2026/11/09 11:44:04.382, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 305.73 W 2026/11/09 11:44:34.390, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 265.84 W 2026/11/09 11:45:04.399, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 72, 294.50 W 2026/11/09 11:45:34.409, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 293.56 W 2026/11/09 11:46:04.422, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 72, 286.39 W 2026/11/09 11:46:34.436, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 72, 307.18 W 2026/11/09 11:47:04.445, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 72, 290.17 W 2026/11/09 11:47:34.454, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 288.83 W 2026/11/09 11:48:04.468, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 305.11 W 2026/11/09 11:48:34.481, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 72, 306.65 W 2026/11/09 11:49:04.492, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 292.55 W 2026/11/09 11:49:34.500, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 294.94 W 2026/11/09 11:50:04.511, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 290.29 W 2026/11/09 11:50:34.526, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 306.33 W 2026/11/09 11:51:04.536, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 293.33 W 2026/11/09 11:51:34.542, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 293.72 W 2026/11/09 11:52:04.550, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 306.80 W 2026/11/09 11:52:34.559, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 72, 284.43 W 2026/11/09 11:53:04.570, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 72, 298.77 W 2026/11/09 11:53:34.580, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 296.07 W 2026/11/09 11:54:04.591, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 298.18 W 2026/11/09 11:54:34.602, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 74, 283.83 W 2026/11/09 11:55:04.616, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 74, 297.20 W 2026/11/09 11:55:34.628, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 294.02 W 2026/11/09 11:56:04.635, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 314.47 W 2026/11/09 11:56:34.644, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 301.40 W 2026/11/09 11:57:04.656, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 297.70 W 2026/11/09 11:57:34.669, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 298.44 W 2026/11/09 11:58:04.679, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 305.97 W 2026/11/09 11:58:34.690, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 74, 302.36 W 2026/11/09 11:59:04.701, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 74, 297.52 W 2026/11/09 11:59:34.715, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 295.07 W 2026/11/09 12:00:04.724, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 311.99 W 2026/11/09 12:00:34.733, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 72, 270.94 W 2026/11/09 12:01:04.742, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 296.83 W 2026/11/09 12:01:34.757, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 288.50 W 2026/11/09 12:02:04.766, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 298.93 W 2026/11/09 12:02:34.772, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 270.19 W 2026/11/09 12:03:04.786, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 72, 296.69 W 2026/11/09 12:03:34.799, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 300.10 W 2026/11/09 12:04:04.808, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 72, 269.90 W 2026/11/09 12:04:34.818, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 272.72 W 2026/11/09 12:05:04.828, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 74, 299.42 W 2026/11/09 12:05:34.838, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 289.65 W 2026/11/09 12:06:04.849, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 287.27 W 2026/11/09 12:06:34.862, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 72, 287.38 W 2026/11/09 12:07:04.870, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 72, 298.57 W 2026/11/09 12:07:34.883, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 300.23 W 2026/11/09 12:08:04.893, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 286.19 W 2026/11/09 12:08:34.906, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 74, 288.00 W 2026/11/09 12:09:04.917, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 292.08 W 2026/11/09 12:09:34.925, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 74, 299.22 W 2026/11/09 12:10:04.934, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 74, 303.57 W 2026/11/09 12:10:34.947, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 74, 303.95 W 2026/11/09 12:11:04.956, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 74, 293.70 W 2026/11/09 12:11:34.966, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 74, 306.77 W 2026/11/09 12:12:04.976, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 74, 256.69 W 2026/11/09 12:12:34.991, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 74, 293.43 W 2026/11/09 12:13:05.003, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 74, 292.29 W 2026/11/09 12:13:35.014, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 74, 301.02 W 2026/11/09 12:14:05.015, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 74, 292.33 W 2026/11/09 12:14:35.016, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 74, 291.85 W 2026/11/09 12:15:05.017, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 74, 297.24 W 2026/11/09 12:15:35.018, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 74, 299.56 W 2026/11/09 12:16:05.019, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 74, 303.95 W 2026/11/09 12:16:35.020, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 306.58 W 2026/11/09 12:17:05.021, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 289.50 W 2026/11/09 12:17:35.023, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 74, 303.53 W 2026/11/09 12:18:05.024, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 74, 292.48 W 2026/11/09 12:18:35.025, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 74, 290.83 W 2026/11/09 12:19:05.027, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 74, 295.22 W 2026/11/09 12:19:35.028, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 292.93 W 2026/11/09 12:20:05.030, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 74, 325.16 W 2026/11/09 12:20:35.030, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 302.86 W 2026/11/09 12:21:05.031, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 74, 294.44 W 2026/11/09 12:21:35.033, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 74, 293.76 W 2026/11/09 12:22:05.034, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 74, 324.13 W 2026/11/09 12:22:35.035, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 74, 314.76 W 2026/11/09 12:23:05.036, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 294.75 W 2026/11/09 12:23:35.037, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 74, 298.58 W 2026/11/09 12:24:05.039, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 306.02 W 2026/11/09 12:24:35.040, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 74, 279.58 W 2026/11/09 12:25:05.041, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 74, 297.43 W 2026/11/09 12:25:35.043, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 74, 296.26 W 2026/11/09 12:26:05.044, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 74, 304.48 W 2026/11/09 12:26:35.046, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 317.45 W 2026/11/09 12:27:05.047, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 294.54 W 2026/11/09 12:27:35.049, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 74, 297.13 W 2026/11/09 12:28:05.051, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 74, 298.88 W 2026/11/09 12:28:35.052, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 74, 275.29 W 2026/11/09 12:29:05.053, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 74, 294.87 W 2026/11/09 12:29:35.055, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 299.44 W 2026/11/09 12:30:05.056, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 292.17 W 2026/11/09 12:30:35.060, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 286.89 W 2026/11/09 12:31:05.062, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 74, 295.83 W 2026/11/09 12:31:35.063, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 74, 295.82 W 2026/11/09 12:32:05.064, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 74, 315.81 W 2026/11/09 12:32:35.065, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 265.56 W 2026/11/09 12:33:05.067, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 298.74 W 2026/11/09 12:33:35.068, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 296.61 W 2026/11/09 12:34:05.070, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 74, 313.53 W 2026/11/09 12:34:35.070, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 74, 306.57 W 2026/11/09 12:35:05.072, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 298.13 W 2026/11/09 12:35:35.073, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 74, 288.76 W 2026/11/09 12:36:05.074, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 295.89 W 2026/11/09 12:36:35.075, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 297.95 W 2026/11/09 12:37:05.076, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 298.34 W 2026/11/09 12:37:35.077, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 74, 291.45 W 2026/11/09 12:38:05.078, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 74, 299.17 W 2026/11/09 12:38:35.079, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 290.98 W 2026/11/09 12:39:05.080, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 296.92 W 2026/11/09 12:39:35.081, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 74, 293.50 W 2026/11/09 12:40:05.082, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 74, 296.99 W 2026/11/09 12:40:35.083, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 298.85 W 2026/11/09 12:41:05.084, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 296.89 W 2026/11/09 12:41:35.085, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 293.93 W 2026/11/09 12:42:05.087, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 74, 301.72 W 2026/11/09 12:42:35.088, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 73, 268.04 W 2026/11/09 12:43:05.088, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 74, 299.64 W 2026/11/09 12:43:35.089, NVIDIA A40, 41283 MiB, 46068 MiB, 100 %, 74, 295.33 W 2026/11/09 12:44:05.091, NVIDIA A40, 41283 MiB, 46068 MiB, 11 %, 68, 184.43 W 2026/11/09 12:44:35.092, NVIDIA A40, 0 MiB, 46068 MiB, 0 %, 50, 38.19 W 2026/11/09 12:45:05.093, NVIDIA A40, 3 MiB, 46068 MiB, 0 %, 41, 63.31 W 2026/11/09 12:45:35.094, NVIDIA A40, 0 MiB, 46068 MiB, 0 %, 36, 33.93 W 2026/11/09 12:46:05.095, NVIDIA A40, 0 MiB, 46068 MiB, 0 %, 33, 33.04 W 2026/11/09 12:46:35.095, NVIDIA A40, 0 MiB, 46068 MiB, 0 %, 32, 32.81 W 2026/11/09 12:47:05.096, NVIDIA A40, 0 MiB, 46068 MiB, 0 %, 31, 32.57 W 2026/11/09 12:47:35.098, NVIDIA A40, 0 MiB, 46068 MiB, 0 %, 31, 32.56 W 2026/11/09 12:48:05.099, NVIDIA A40, 0 MiB, 46068 MiB, 0 %, 30, 32.42 W ``` ### Pod 側のチェックポイントの一覧（pod_checkpoints.tsv） ```text 1000/_CHECKPOINT_METADATA 426 1000/assets/local/mecharm_redblock_box_deep_even10_v1/norm_stats.json 1772 1000/params/_METADATA 33052 1000/params/_sharding 17773 1000/params/array_metadatas/process_0 12891 1000/params/d/2ec07d8052908f76fcfb80c31dc4166d 2772 1000/params/manifest.ocdbt 118 1000/params/ocdbt.process_0/d/04127d8e9dccaf5b05b64f2907947ca5 265961952 1000/params/ocdbt.process_0/d/0d7f83bfa584a9af3d5e77642b243ec7 69867960 1000/params/ocdbt.process_0/d/10f75ee33c115006c317d0d6bdf0db85 58980366 1000/params/ocdbt.process_0/d/1708648c031d6b01d72c7924f0ea2bdf 945626171 1000/params/ocdbt.process_0/d/1a4e0bd8a4b78482753277ea32044191 497190673 1000/params/ocdbt.process_0/d/1dbd8cd311b97e9751c72488efd555ac 1047 1000/params/ocdbt.process_0/d/20c873c702484d768ed3393f328c757c 361 1000/params/ocdbt.process_0/d/2b95a69171936832f5cfc1b5fadacd52 280 1000/params/ocdbt.process_0/d/2da1f4bee2c1cf6d9b74dbc53aee46d6 610 1000/params/ocdbt.process_0/d/332e8b0316959c40896542f369f4d016 264 1000/params/ocdbt.process_0/d/372495120b1d83a8598ff8f52f39a0f6 94040572 1000/params/ocdbt.process_0/d/372613d51330732f77341d1200634328 496 1000/params/ocdbt.process_0/d/38953bc267bc2fb8ba2659d898b2c648 945631382 1000/params/ocdbt.process_0/d/38de2966a1ff71d68a43a694f881db2e 73233493 1000/params/ocdbt.process_0/d/47524054ca196a9f28569cd9feca2eb6 828933371 1000/params/ocdbt.process_0/d/5651304866d7259afb99c98b218f8107 682763990 1000/params/ocdbt.process_0/d/696a13fc9d356a7e63fdcb5d84f699f7 915 1000/params/ocdbt.process_0/d/6dc94fcb7ec06f796c321383298fff01 874 1000/params/ocdbt.process_0/d/7072d8f438d95be2b672f249cc338c0e 497036653 1000/params/ocdbt.process_0/d/707485d70876eb0828dbe48b11856ba6 118186878 1000/params/ocdbt.process_0/d/753fe4e2b78c53b1886e7cb8c6c01af0 1259 1000/params/ocdbt.process_0/d/8570375dc9d2e4e85500a743a88d0fbf 1176 1000/params/ocdbt.process_0/d/988e0ca20246e7f6e0cebd17392626a4 235536914 1000/params/ocdbt.process_0/d/affbb498cf119f41d938de4b4338d337 1328 1000/params/ocdbt.process_0/d/b7347a078eebae605dc55d5038f51d20 2053 1000/params/ocdbt.process_0/d/bde672f5cb5c41959c60f27de59031ea 811 1000/params/ocdbt.process_0/d/c91e086ca8a806bc65d3d7a80f7f2a37 47336752 1000/params/ocdbt.process_0/d/c94d498002c27c632cf1403384a26b71 213 1000/params/ocdbt.process_0/d/e2381f1804cd53dfa701d8d018284296 1230 1000/params/ocdbt.process_0/d/e38c240d19245dca65d69636a035df75 29598951 1000/params/ocdbt.process_0/d/eb0d386e43b5d7ec1a350295c2bf6960 945879509 1000/params/ocdbt.process_0/d/f3af37c12ffddb347f93dbbf8a06e552 979 1000/params/ocdbt.process_0/d/f9e419fc08b34d7c447a25720007f075 1107 1000/params/ocdbt.process_0/manifest.ocdbt 141 1000/train_state/_METADATA 60265 1000/train_state/_sharding 27701 1000/train_state/array_metadatas/process_0 19941 1000/train_state/d/1503350978b4f6d804ad0688dbbf61e9 2905 1000/train_state/manifest.ocdbt 118 1000/train_state/ocdbt.process_0/d/1176469f6e1c61b826ffefd761313e3b 5503 1000/train_state/ocdbt.process_0/d/1c273227246016819557ca92c8d63e2b 445073791 1000/train_state/ocdbt.process_0/d/1e148b52d5c685d6a52e7bad34dee314 32568999 1000/train_state/ocdbt.process_0/d/27382422adafe2946069abdc472c4195 568 1000/train_state/ocdbt.process_0/d/32973eb9e4f9a6ba5b8a69ecdfc2d0b3 75158571 1000/train_state/ocdbt.process_0/d/33c8beb6153e74db4a34d80f38dd4370 826 1000/train_state/ocdbt.process_0/d/37a275803b0308af0fb4101650268866 721 1000/train_state/ocdbt.process_0/d/3d65adb0cc20ba8de714a8d8c63e0bcb 128305766 1000/train_state/ocdbt.process_0/d/4855da7013f14150af2a5a543372562e 629 1000/train_state/ocdbt.process_0/d/4a8cd511617c5e69272971975139b54d 1134 1000/train_state/ocdbt.process_0/d/55fce2ac12df2e902574d7c6718f9e8b 968 1000/train_state/ocdbt.process_0/d/5a368a89263f0771ee100239f6fa04ea 845 1000/train_state/ocdbt.process_0/d/85a8d58827ff772fae2da4abecacbaf9 778 1000/train_state/ocdbt.process_0/d/871c0a2728f5a871aec851beca91dd84 175428993 1000/train_state/ocdbt.process_0/d/925fde50e8278512f6c316168cdc2aa1 16478269 1000/train_state/ocdbt.process_0/d/93476b2225025215c6cd84167b54a7be 466890971 1000/train_state/ocdbt.process_0/d/93d845f8de2f71c2c53fe9d450825092 171 1000/train_state/ocdbt.process_0/d/9aaa5a74ce44bbf5a0406df8b19ee1cc 133450224 1000/train_state/ocdbt.process_0/d/9db7f8a0d8231ad3dbc662aa07315913 865 1000/train_state/ocdbt.process_0/d/a422f4cf5d362d6a787ce6cc6b6e9bf2 41859382 1000/train_state/ocdbt.process_0/d/a73d5259e8d2fdbe65d7f8e2f0751120 955 1000/train_state/ocdbt.process_0/d/ad58dc56599b5212165994f3d33f939d 790451648 1000/train_state/ocdbt.process_0/d/c2084b224f226e79c4f13d336665db93 588 1000/train_state/ocdbt.process_0/d/cfdc52305b8b49bdddf97e641f56c15b 460769736 1000/train_state/ocdbt.process_0/d/d16f6e09a9f185f56099f2342de5b9f0 510 1000/train_state/ocdbt.process_0/d/d3dd6c167df54977de2f1bd00a8ad915 396 1000/train_state/ocdbt.process_0/d/d8474d290b45f1d7245ac937a2b4b65c 1778 1000/train_state/ocdbt.process_0/d/ebbd3fd238b370191973b4398f4217d5 1156 1000/train_state/ocdbt.process_0/d/ed2da68aedb64eada8434fa6662a82e9 923 1000/train_state/ocdbt.process_0/d/f523cff31aad4206400cb97c48f5bb3f 556 1000/train_state/ocdbt.process_0/d/f63ebf9a9e49c494959e63b6333f5c9f 448699377 1000/train_state/ocdbt.process_0/d/ff4d5ce4e7891dd35f33e0d3cda25be5 494 1000/train_state/ocdbt.process_0/manifest.ocdbt 807 2000/_CHECKPOINT_METADATA 426 2000/assets/local/mecharm_redblock_box_deep_even10_v1/norm_stats.json 1772 2000/params/_METADATA 33052 2000/params/_sharding 17773 2000/params/array_metadatas/process_0 12891 2000/params/d/cde8c4554f67f4c56b646db7aec0a6db 2725 2000/params/manifest.ocdbt 118 2000/params/ocdbt.process_0/d/007d6af0cacf3985043776d39557dafd 945879441 2000/params/ocdbt.process_0/d/1a97eb73f57716c6b3e7aa171957d97d 945630408 2000/params/ocdbt.process_0/d/2a11214a8465657b2bbce519a63dbe94 986 2000/params/ocdbt.process_0/d/432ae62b9aa8bdc12e015db1d1e2108d 90308118 2000/params/ocdbt.process_0/d/5775ef570a36835de4292bd14c4e60cc 117433491 2000/params/ocdbt.process_0/d/6dba56c9230985c29509157bf8d5f78c 502773296 2000/params/ocdbt.process_0/d/7aba5f2961cb0b62e4f967ffeee34905 88705573 2000/params/ocdbt.process_0/d/965d64a902e75d5f604efbb4265d805c 235536865 2000/params/ocdbt.process_0/d/968b3a6845378878795e722ff680fdd8 213 2000/params/ocdbt.process_0/d/98cb113db754e7f3ca26d3a27a012258 497190831 2000/params/ocdbt.process_0/d/9d98bc1197c828ae3fe7d79f1b72b746 102672121 2000/params/ocdbt.process_0/d/a1ecc99a0263f7b84cda3de97c26dfaa 5247613 2000/params/ocdbt.process_0/d/aa2210ace000acbad99634177fb2af39 532808710 2000/params/ocdbt.process_0/d/aadbd3a4cf8d45a0e7e2ca1e4ac158be 828933321 2000/params/ocdbt.process_0/d/b6afb8edbe8b6fdb212b1d7a3ecd88e1 1144 2000/params/ocdbt.process_0/d/c9affd5f6aecfbc69e5d67dd157a58fe 945626943 2000/params/ocdbt.process_0/d/cee1d47d1d79a39ad10ff03e79a489a5 497036109 2000/params/ocdbt.process_0/d/ed13b670d95488046d634145a2cbd33b 1376 2000/params/ocdbt.process_0/manifest.ocdbt 202 2000/train_state/_METADATA 60265 2000/train_state/_sharding 27701 2000/train_state/array_metadatas/process_0 19941 2000/train_state/d/9f0e6ca7acb7cc5e50252ec4a9c3a4fb 2842 2000/train_state/manifest.ocdbt 118 2000/train_state/ocdbt.process_0/d/16a87c445c621e6da286377c07ef508a 1136 2000/train_state/ocdbt.process_0/d/16d13b58232b336d486846c3c3dd636f 461772818 2000/train_state/ocdbt.process_0/d/2f32ac020c14d6b95ba9a85c6fc3f151 1148 2000/train_state/ocdbt.process_0/d/30893951a6ee53cdd51547fe74bfb181 265 2000/train_state/ocdbt.process_0/d/391d1d334d96ed1536080db4a18650a9 568 2000/train_state/ocdbt.process_0/d/4283bc52c7cb21ca7fbd09fe0c0d4828 955 2000/train_state/ocdbt.process_0/d/447230f58fcba8a52a32acd353960173 171 2000/train_state/ocdbt.process_0/d/51d868bcc7e36cb746a45287358649b3 1156 2000/train_state/ocdbt.process_0/d/573b13e1c6fbab67367b0a7cc1236ec8 906 2000/train_state/ocdbt.process_0/d/574d0eca6faca7807b4ec7c8cdb6d06a 1112 2000/train_state/ocdbt.process_0/d/57c5ceb48cd6b3208368e43e902d8eaa 133446099 2000/train_state/ocdbt.process_0/d/59b84788208d31477696f335db346a07 608 2000/train_state/ocdbt.process_0/d/5d8a3f43756969bbc6965693d1033248 494 2000/train_state/ocdbt.process_0/d/5dbbf3d91c11155c4238d7678aa98991 9824 2000/train_state/ocdbt.process_0/d/60c524b9a062febaae7058a63ec502db 1134 2000/train_state/ocdbt.process_0/d/66ac851f9b986091cf6d51fa26bcb9b4 1027 2000/train_state/ocdbt.process_0/d/68e67044f467c25ee41c5bb14baf8655 1144 2000/train_state/ocdbt.process_0/d/87d2410a8869c3fd644f733872a66afa 337 2000/train_state/ocdbt.process_0/d/89e49e07a9268e48dd3565ff5d5f8fc1 706 2000/train_state/ocdbt.process_0/d/8a1339a54e9fb4687961540df3443dd7 396 2000/train_state/ocdbt.process_0/d/943ee4f54f5ebbe8134715f5e55cee56 826 2000/train_state/ocdbt.process_0/d/9757269e38e1492ab7c9276eb52c1eed 539 2000/train_state/ocdbt.process_0/d/9b4b826a3a259597abc34766a317242a 63071747 2000/train_state/ocdbt.process_0/d/9df86df411d47c3707d89e6421a66ec4 1143 2000/train_state/ocdbt.process_0/d/9f09ac676124770bfe1c79325073fd73 1149 2000/train_state/ocdbt.process_0/d/a010329f7f39e3e109cb4bb0b748d466 1744 2000/train_state/ocdbt.process_0/d/a8681bca2a3d275c7aa1c45c2519cb28 263992817 2000/train_state/ocdbt.process_0/d/acbb11ac6b935848b72087f9ca3a8723 448798994 2000/train_state/ocdbt.process_0/d/ae6c115ba7ea969f3a4e883f4a77ef7b 1147 2000/train_state/ocdbt.process_0/d/b9c3827d476eb9d24b91ba1b43fedf01 655111585 2000/train_state/ocdbt.process_0/d/bfb3488b495c37d6c00b2a0d52e7bfe3 1872 2000/train_state/ocdbt.process_0/d/c3ed70b0d82c62226ae325159f9d7da5 82271628 2000/train_state/ocdbt.process_0/d/c4dccac7cd9409399035b8f66154b44f 196249075 2000/train_state/ocdbt.process_0/d/ca01ad7a657ba9c9dfcae0a4e05f5707 319 2000/train_state/ocdbt.process_0/d/d246656b1567c2bc1eeb20b350d26450 441115416 2000/train_state/ocdbt.process_0/d/d4a6033506ab5fc223eeca737f29f925 1150 2000/train_state/ocdbt.process_0/d/e1a5ed74fbc91c847b90691ba1b91862 1147 2000/train_state/ocdbt.process_0/d/e510f4ea0d9540dae2a992d8f711ac98 1147 2000/train_state/ocdbt.process_0/d/e888baa64aafedaafc43f2561300b010 1149 2000/train_state/ocdbt.process_0/d/efd8f085a90eb3661f472d19b4e62509 434150525 2000/train_state/ocdbt.process_0/d/fb67c92edf1dbf7ca74aab3f7ee8c7ef 778 2000/train_state/ocdbt.process_0/d/fb8f21e8f3282e3fd8032a520f886c75 1150 2000/train_state/ocdbt.process_0/d/fcfd8c58282282831de1e769e26a9188 1150 2000/train_state/ocdbt.process_0/d/fd8446383353262cf4e7c75abccd68dc 1149 2000/train_state/ocdbt.process_0/d/fe24ae671a7cc5ea33958f0aaf51e9cd 641 2000/train_state/ocdbt.process_0/manifest.ocdbt 689 2999/_CHECKPOINT_METADATA 426 2999/assets/local/mecharm_redblock_box_deep_even10_v1/norm_stats.json 1772 2999/params/_METADATA 33052 2999/params/_sharding 17773 2999/params/array_metadatas/process_0 12891 2999/params/d/b54e441e6ec634a075db6badcfedcef7 2713 2999/params/manifest.ocdbt 118 2999/params/ocdbt.process_0/d/08cf05d025b550f150365b6fa2db0096 956 2999/params/ocdbt.process_0/d/0dd8585b444297cf7b18711d657cea57 829 2999/params/ocdbt.process_0/d/0e43ede65a0e26b4383cbfcbae981fb5 133035633 2999/params/ocdbt.process_0/d/14cdee8c0f3165256776b0248af09131 549639970 2999/params/ocdbt.process_0/d/1aa4ed0ee7dff4563352e8465238b40e 945630386 2999/params/ocdbt.process_0/d/34129ce9e67ba2bcb0834424b3e59e9b 945626101 2999/params/ocdbt.process_0/d/35914a35cb113012759e4e198faadd5d 1144 2999/params/ocdbt.process_0/d/3cbb3d5e859b5e03e912efbf9a507e9f 945879428 2999/params/ocdbt.process_0/d/4540a892ee6f3e107c5e7c3ea55c1583 1347940843 2999/params/ocdbt.process_0/d/531c232f719787fc92c3abcb20bd8815 1376 2999/params/ocdbt.process_0/d/57f10e154993d8a61be2f0df666c115c 251101228 2999/params/ocdbt.process_0/d/64866c1760cd1011533b76e3d7263176 1258 2999/params/ocdbt.process_0/d/6a35534b8514667d372bc0b1dc653435 103461521 2999/params/ocdbt.process_0/d/79e799b5d9ee70b339132b0ea3308e3e 1328 2999/params/ocdbt.process_0/d/acd1c946899138c517f3011d6d1f49d1 120876 2999/params/ocdbt.process_0/d/bc422d52055c3a708602add154aba084 213 2999/params/ocdbt.process_0/d/bebe802d8dd6db2cfdbf7cb06e4b38ae 69371308 2999/params/ocdbt.process_0/d/c0480948731f0b19ecc9dcf507fe2d35 902 2999/params/ocdbt.process_0/d/c16b308ee95b0cb6af47cba75d762808 538 2999/params/ocdbt.process_0/d/c2ab1a237b99cf1f7a6af1553be9455d 81988328 2999/params/ocdbt.process_0/d/c65fa5c1c9867cd2c390a666932ee198 444 2999/params/ocdbt.process_0/d/d1e1a76c24eeb731405871a6b4200769 828933304 2999/params/ocdbt.process_0/d/d2a0677c9fc1009ba0e46b91ffcc13aa 133038450 2999/params/ocdbt.process_0/d/dce57051dcd1400c417b819474a256d1 1220 2999/params/ocdbt.process_0/manifest.ocdbt 504 2999/train_state/_METADATA 60265 2999/train_state/_sharding 27701 2999/train_state/array_metadatas/process_0 19941 2999/train_state/d/eeda410d3495ed5a7928277f22e27790 2866 2999/train_state/manifest.ocdbt 118 2999/train_state/ocdbt.process_0/d/053d351edcd469c729202b2a014af984 49264463 2999/train_state/ocdbt.process_0/d/0661128dd246e6c4c7080b1adf18c707 557 2999/train_state/ocdbt.process_0/d/126669b6e1ae5bf6064cab91ef5fcac5 1040 2999/train_state/ocdbt.process_0/d/172fa71e4de106fa9f8c7978a257e2e1 9812 2999/train_state/ocdbt.process_0/d/19067628007188fd0809dc6299ecaae9 1156 2999/train_state/ocdbt.process_0/d/25509a9e3849a11b6908b6e8790f2c31 791 2999/train_state/ocdbt.process_0/d/2eb029f4e672c39758e2d4b5fc6dacbd 171 2999/train_state/ocdbt.process_0/d/387a8a0d7d1030f903e9aff812cecdaa 853 2999/train_state/ocdbt.process_0/d/3984a36558388ea7e2d9c693fc3d8a6b 1129 2999/train_state/ocdbt.process_0/d/3ab3db145ad44d3f8d406e6623a943f4 439285758 2999/train_state/ocdbt.process_0/d/3c52d09bd4e10f75c082b2e6e2b729f3 721 2999/train_state/ocdbt.process_0/d/43c4fdcf2e95a27b46d712938e7e51c3 1147 2999/train_state/ocdbt.process_0/d/4551604651333d387601c484e542a66e 1150 2999/train_state/ocdbt.process_0/d/46be0378ccf145682b17ada4e1f8a4bd 1090 2999/train_state/ocdbt.process_0/d/507febcba1e6b9ed272379ff099fabc2 906 2999/train_state/ocdbt.process_0/d/553024dc63fba2abf30253f04a337422 1146 2999/train_state/ocdbt.process_0/d/589c21136aa20616b18bf18670c00efb 445598567 2999/train_state/ocdbt.process_0/d/5b231595ece935a8834808957f097d1d 1149 2999/train_state/ocdbt.process_0/d/5ce036f1525a5265e1b973a4dc196f00 1070 2999/train_state/ocdbt.process_0/d/5f944c870b9d84548e9a3cc356dc400b 1149 2999/train_state/ocdbt.process_0/d/665bb5486fb04c46f1958b3769f9e709 993 2999/train_state/ocdbt.process_0/d/6b6522dded8219412b91edb43c1f2c7a 431984161 2999/train_state/ocdbt.process_0/d/73577614aac54064af9287bfdb2dde5a 1667 2999/train_state/ocdbt.process_0/d/83ef36d21507b210cd7365f6718e271c 524596317 2999/train_state/ocdbt.process_0/d/97c471226746c4cd511413155ff6db20 568 2999/train_state/ocdbt.process_0/d/a0853be65bca5119d8bb1520b2db6f28 804 2999/train_state/ocdbt.process_0/d/a6d42de508d2c5ebc13d38e61f107cb3 1149 2999/train_state/ocdbt.process_0/d/a73ac1dd6b33ec278e90b5bbb53b9c86 396923796 2999/train_state/ocdbt.process_0/d/b1a6c8378d24822214484bc90bed5923 1147 2999/train_state/ocdbt.process_0/d/b220ea1a53d829312449c74a6b7258b8 934 2999/train_state/ocdbt.process_0/d/b3c011486bc907a15a07b002b89b94f3 1147 2999/train_state/ocdbt.process_0/d/b4db77cab98e2b3f4cb3cb76edd74fae 476 2999/train_state/ocdbt.process_0/d/b6ef16d3452f16630ecd56d0ae20d603 826 2999/train_state/ocdbt.process_0/d/bf45a6507a6eaa061f2789f94329b2a5 588 2999/train_state/ocdbt.process_0/d/c64265e31f9544a4544bed1085a8d7a3 175718286 2999/train_state/ocdbt.process_0/d/c76ef312c2f42daad5651a419e2fc043 32546827 2999/train_state/ocdbt.process_0/d/c9b2e313e2923a7aac667b003fd66bc0 1143 2999/train_state/ocdbt.process_0/d/c9bb7065c9a36f89fa20fd2e8ec0c7f6 845 2999/train_state/ocdbt.process_0/d/cd588685c359758a6266bc0385cfdfad 129998017 2999/train_state/ocdbt.process_0/d/d8fe83bbb4e769c658187e42eba92d3e 1149 2999/train_state/ocdbt.process_0/d/de502b26e9c8f0961f6cd5cd6d4a89a6 460431071 2999/train_state/ocdbt.process_0/d/e55e66da7fd77ce72637f1b903e18579 1878 2999/train_state/ocdbt.process_0/d/e926aecaf60c8bf5b9283fe19448f71f 556 2999/train_state/ocdbt.process_0/d/e96fff3012021dedb2929d96336567c1 664 2999/train_state/ocdbt.process_0/d/ea35d913acb1873f43b0c77670054363 84108974 2999/train_state/ocdbt.process_0/d/ff4b502cd05357824b9b9a82cc9eabf3 346 2999/train_state/ocdbt.process_0/manifest.ocdbt 737 ``` ### 上げた 2999 のファイルの一覧（pod_files_2999.tsv） ```text _CHECKPOINT_METADATA 426 assets/local/mecharm_redblock_box_deep_even10_v1/norm_stats.json 1772 params/_METADATA 33052 params/_sharding 17773 params/array_metadatas/process_0 12891 params/d/b54e441e6ec634a075db6badcfedcef7 2713 params/manifest.ocdbt 118 params/ocdbt.process_0/d/08cf05d025b550f150365b6fa2db0096 956 params/ocdbt.process_0/d/0dd8585b444297cf7b18711d657cea57 829 params/ocdbt.process_0/d/0e43ede65a0e26b4383cbfcbae981fb5 133035633 params/ocdbt.process_0/d/14cdee8c0f3165256776b0248af09131 549639970 params/ocdbt.process_0/d/1aa4ed0ee7dff4563352e8465238b40e 945630386 params/ocdbt.process_0/d/34129ce9e67ba2bcb0834424b3e59e9b 945626101 params/ocdbt.process_0/d/35914a35cb113012759e4e198faadd5d 1144 params/ocdbt.process_0/d/3cbb3d5e859b5e03e912efbf9a507e9f 945879428 params/ocdbt.process_0/d/4540a892ee6f3e107c5e7c3ea55c1583 1347940843 params/ocdbt.process_0/d/531c232f719787fc92c3abcb20bd8815 1376 params/ocdbt.process_0/d/57f10e154993d8a61be2f0df666c115c 251101228 params/ocdbt.process_0/d/64866c1760cd1011533b76e3d7263176 1258 params/ocdbt.process_0/d/6a35534b8514667d372bc0b1dc653435 103461521 params/ocdbt.process_0/d/79e799b5d9ee70b339132b0ea3308e3e 1328 params/ocdbt.process_0/d/acd1c946899138c517f3011d6d1f49d1 120876 params/ocdbt.process_0/d/bc422d52055c3a708602add154aba084 213 params/ocdbt.process_0/d/bebe802d8dd6db2cfdbf7cb06e4b38ae 69371308 params/ocdbt.process_0/d/c0480948731f0b19ecc9dcf507fe2d35 902 params/ocdbt.process_0/d/c16b308ee95b0cb6af47cba75d762808 538 params/ocdbt.process_0/d/c2ab1a237b99cf1f7a6af1553be9455d 81988328 params/ocdbt.process_0/d/c65fa5c1c9867cd2c390a666932ee198 444 params/ocdbt.process_0/d/d1e1a76c24eeb731405871a6b4200769 828933304 params/ocdbt.process_0/d/d2a0677c9fc1009ba0e46b91ffcc13aa 133038450 params/ocdbt.process_0/d/dce57051dcd1400c417b819474a256d1 1220 params/ocdbt.process_0/manifest.ocdbt 504 ``` ## 依頼135：安全範囲を50本から作り、次の学習から学習率をステップ数に合わせた ### 依頼135 の依頼文（依頼箱/済み/20261109_135_deep1の安全範囲を50本から作り次の学習率をステップ数に合わせる.md） ```markdown --- 作業フォルダ: /Users/shimazueiichi/r2r2r-scripts 上限時間: 120 --- # 依頼135：deep1 の実機推論の安全範囲を50本から作る・次の学習から学習率の下げ方をステップ数に合わせる 実機は動かさない。エイイチは見ていない。Log27。依頼133 の報告の「決めてほしいこと」1・2 へのエイイチの答え（2026-11-09）。 ## 大事：いま回っている学習を止めない・触らない 依頼133 の本番（lab_train.sh pid 8808・Pod arn304g7xu5wfk・21:40 ごろ終わる見込み）が動いている。Pod・その lab_train.sh のプロセス・KIOXIA の lab/Log27/train/ には手を出さない。lab_train.sh は main の中身を始めに読んでいるので書き換えてよいが、lab_lib.sh・pi05/openpi_addon/ の変更がいまのプロセスに効かないことを確かめてから変える（効くおそれがあれば止めて報告）。 ## やること 1. **安全範囲（決め1）**：lab_start.sh で --model deep1 のときは、安全範囲（関節範囲 ±10°）を作る --raw-dir を mecharm_redblock_box_deep_even10_v1 の raw_episodes にする。trim・corr・corr2・corr3 は今までどおり trim2 のまま。deep1 で作った範囲（各関節の下限・上限）を報告に、trim2 の範囲と並べて書く 2. **学習率の下げ方（決め2）**：次に回す学習から、decay_steps を num_train_steps と同じにする（warmup 300・peak 5e-5・decay_lr 5e-6 はそのまま）。対象は base から学習する deep1 の流れ（add_deep_config.py）。corr〜corr3 の続きの学習の設定は変えない - いま回っている 3,000 ステップの学習は decay_steps 5,000 のまま終わる。由来（train_provenance.json）や記録で、どちらの下げ方で学習したモデルかが分かるようにする（学習の設定の名前を分けるなど、やり方は任せる。ただし lab_start.sh --model deep1 で今回の 2999 を立てられること） 3. テスト：test_lab_start.sh・test_lab_train.sh・pi05/test_merge_and_config.py を通す。足すテスト：deep1 の --raw-dir が deep_even10_v1 になる・ほかのモデルは trim2 のまま・新しい学習では decay_steps がステップ数と同じになる・今回の 2999 も立てられる 4. codex で差分を見てもらう。指摘を直して通るまで 5. 変えたファイルだけを指定して git add・commit（git add . は使わない） ## 報告 claude_code_reports/20261109_135.txt に：1 の範囲の表（deep1 と trim2）、2 の変えた所と今回のモデルの扱い、テストの結果、codex の回数と指摘、commit の番号、いまの学習に影響がなかったことの確かめ（lab_train_*.txt の最後の行の時刻とステップ）、決めてほしいこと。 **報告の最後の1行は必ず次のどれか**：「判定：合格」／「判定：止めた（決めてほしいことあり）」／「判定：失敗」 ``` ### 依頼135 の報告（r2r2r-scripts/claude_code_reports/20261109_135.txt） ```text 依頼135 報告：deep1 の実機推論の安全範囲を50本から作る・次の学習から学習率の下げ方をステップ数に合わせる (2026-11-09) 実機は動かしていない。Pod・lab_train.sh pid 8808・KIOXIA の lab/Log27/train/ には触っていない。 ■ 1. 安全範囲 (決め1) lab_start.sh で --model deep1 (と新しい deep1b) のときは、run_policy_client.py --raw-dir を /Volumes/KIOXIA/r2r2r-data/datasets/mecharm_redblock_box_deep_even10_v1/raw_episodes にした。 trim・corr・corr2・corr3 は今までどおり trim2。位置合わせ (camera_align.py) のお手本はどのモデルも trim2 のまま。 起動時に「[RANGE ] 安全範囲 (学習データ ±10°) は ... から」を出す。deep のデータがなければ API を呼ぶ前に止まる。 範囲 (run_policy_client.py の load_envelope と同じ計算。データの最小・最大 ±10° を物理可動域・J4_SAFE・J1_SAFE_MAX で切る) deep1 (deep_even10_v1 50本・7128行) trim2 (19本・2717行) 関節 データ範囲 安全範囲 データ範囲 安全範囲 j0 -61.7〜+44.7 -71.7〜+54.7 -61.7〜+44.7 -71.7〜+54.7 j1 +1.2〜+87.9 -8.8〜+88.0 +1.3〜+87.9 -8.7〜+88.0 (上限は J1_SAFE_MAX 88°) j2 -34.5〜+42.5 -44.5〜+52.5 -35.7〜+31.4 -45.7〜+41.4 j3 -7.8〜+17.0 -17.8〜+27.0 -7.8〜+17.0 -17.8〜+27.0 j4 -76.6〜 +0.4 -86.6〜+10.4 -88.2〜 +0.4 -90.0〜+10.4 (下限は J4_SAFE -90°) j5 -40.7〜-25.3 -50.7〜-15.3 -40.6〜-23.6 -50.6〜-13.6 違い：j2 の上限が +11.1° 広い (+41.4 → +52.5)。j4 の下限は 3.4° 狭い (-90.0 → -86.6)。j5 の上限は 1.7° 狭い。j0・j3 は同じ。 ■ 2. 学習率の下げ方 (決め2) - add_deep_config.py に --variant (必須) を足した。 v1 = 設定 pi05_mecharm_deep_even10_lora・decay_steps 5_000 (依頼133 の今回の 2999。中身は変えていない) b = 設定 pi05_mecharm_deep_even10_b_lora・decay_steps = --steps (warmup 300・peak 5e-5・decay_lr 5e-6 は同じ) decay_steps が warmup 300 以下なら止まる。v1 と b は名前が違うので同じ config.py に両方入れられる。 - pod_prep_deep.sh は を取る (v1/b 以外なら tar を見る前に止まる)。 - lab_lib.sh / lab_train.sh：新しいモデル名 deep1b を足した (上げる先 PixelstoMotors/pi05-mecharm-deep-even10-b-lora-v1、 KIOXIA の checkpoints/pi05_mecharm_deep_even10_b_lora_v1/<ステップ>、exp deep_even10_v1_b、Pod の /workspace/ckpt/deep_b_v1_2999)。 lab_train.sh --model deep1b --steps N で decay_steps = N。 - 今回のモデル (deep1) の扱い：設定の名前 pi05_mecharm_deep_even10_lora = decay_steps 5000 として残す。 lab_start.sh --model deep1 は今までどおり deep_v1_2999 を立てる (3f は add_deep_config.py --steps 3000 --variant v1)。 lab_train.sh --model deep1 で新しく学習しようとすると止まって deep1b を案内する (同じ置き場所・リポジトリに別の下げ方の モデルが上書きされるのを防ぐため)。--stop (記録の model=deep1) はそのまま使える。 - 由来 (train_provenance.json) の config に lr_schedule {warmup_steps 300, peak_lr 5e-5, decay_steps, decay_lr 5e-6, source "add_deep_config.py --variant v1|b"} を足した (deep1・deep1b だけ。corr〜corr3 には足さない)。 ※いま動いている学習は前のコードで動いているので、その由来には lr_schedule は入らない。 見分けは config.name = pi05_mecharm_deep_even10_lora (= decay_steps 5000) で行う。 - corr〜corr3 の続きの学習の設定は変えていない。 ■ 3. テスト - test_lab_train.sh：OK 240 / NG 0 (deep1b の成功・--steps 2000 で decay 2000・--steps 300 は止まる・deep1 の新しい学習は止まる・ --stop の deep1 は decay 5000 で由来を書く・corr3 の由来に lr_schedule なし・正規化の値の失敗 3 種を deep1b で) - test_lab_start.sh：OK 172 / NG 0 (deep1 の --raw-dir が deep_even10_v1・trim は trim2・deep のデータがなければ止まる・ corr3 は deep のデータがなくても trim2 で進む・deep1 は --variant v1 で 2999 を立てる・deep1b の流れ) - pi05/test_merge_and_config.py：34 件 OK (v1 は trim と同じ decay 5000・b は decay = ステップ数 (3000/2000)・ --variant 必須・decay ≤ 300 は止まる・書き換えられた decay は止まる・pod_prep_deep.sh は知らない variant で先に止まる) ■ 4. codex 1 回。指摘なし (安全範囲・deep1 の残し方・deep1b の区別・いまの学習に影響しないことを確認したとの答え)。 (1 回目の最初の起動は標準入力待ちで止まったので、標準入力を閉じて流し直した。数に入れていない) ■ 5. commit 6715f5e (変えたファイルだけを指定して add) ■ 6. いまの学習に影響がなかったことの確かめ - lab_train.sh は main の中身を始めに読み、lab_lib.sh は始めに source 済み。pi05/ は 3a で Pod へ送り済み、4b (pod_prep_deep.sh) も済み。 終わりに Mac 側で読み直すのは provenance.py だけ (変えていない)。コードの指紋は始める時点で測れている (「測れなかった」の WARN なし)。 - pid 8808 は動いている。lab_train_20261109_153622.txt の最後の行： [TRAIN ] 17:55 step 1060/3000 loss 0.0069 (Step 1050 の行) 経過 2:11 上限まで 4:40 (作業前 17:25 は step 806/3000。5 分おきに進んでいる) ■ 決めてほしいこと 1. 新しいモデルの名前：deep1b・設定 pi05_mecharm_deep_even10_b_lora・Hugging Face PixelstoMotors/pi05-mecharm-deep-even10-b-lora-v1 (非公開で上げる) にした。これでよいか。 2. deep1 の安全範囲は j2 の上限が trim2 より +11.1° 広い (+52.5°)。実機で deep1 を動かす前に、この広がりで良いか確かめてほしい。 3. run_policy_client.py は写り方の確かめの基準画像も --raw-dir の最初のエピソードから取る。deep1・deep1b では基準が deep_even10_v1 の最初の本 (pi05_20261031_175527) になる (lab_start.sh は --ignore-framing を付けているので止まらず警告だけ)。位置合わせ (camera_align.py) は trim2 のまま。これでよいか。 4. lab_start.sh --model deep1b はステップ数 3000 (2999) を前提にしている。deep1b を別のステップ数で学習するときは lab_lib.sh の LAB_DEEP1B_STEPS も合わせて変える必要がある。 判定：止めた（決めてほしいことあり） ``` ### 依頼135 の commit（6715f5e）の説明と変えたファイル ```text commit 6715f5ef63b11dda8f9dd33a8692b2aba3e0960f Date: Mon Nov 9 17:58:08 2026 +0900 deep1 の実機推論の安全範囲 (run_policy_client.py --raw-dir) を mecharm_redblock_box_deep_even10_v1 の raw_episodes から作る (trim・corr〜corr3 は trim2 のまま・位置合わせのお手本はどれも trim2). 次の学習から decay_steps を num_train_steps と同じにする lab_train.sh --model deep1b を足す (add_deep_config.py --variant b・設定 pi05_mecharm_deep_even10_b_lora・exp deep_even10_v1_b・上げる先 PixelstoMotors/pi05-mecharm-deep-even10-b-lora-v1・Pod の /workspace/ckpt/deep_b_v1_2999. warmup 300・peak 5e-5・decay_lr 5e-6 はそのまま). 依頼 133 の deep1 (decay_steps 5000) は --variant v1 として残し, lab_start.sh --model deep1 で 2999 を立てられ, --stop で片付けられるが新しくは学習しない. 由来の config に lr_schedule (decay_steps・--variant) を残す (deep だけ). pod_prep_deep.sh は を取る. corr〜corr3 の設定は変えない. テストを足す (lab_train 240・lab_start 172・test_merge_and_config 34) (依頼 135. codex 1 回) Co-Authored-By: Claude Opus 5.5 lab_lib.sh | 56 ++++++++++++++++------- lab_start.sh | 20 ++++++--- lab_train.sh | 35 +++++++++++---- lab_train_fakes.sh | 8 ++-- pi05/openpi_addon/add_deep_config.py | 86 ++++++++++++++++++++++++++---------- pi05/openpi_addon/pod_prep_deep.sh | 23 ++++++---- pi05/test_merge_and_config.py | 82 ++++++++++++++++++++++++++++++---- test_lab_start.sh | 30 +++++++++++-- test_lab_train.sh | 62 +++++++++++++++++--------- 9 files changed, 299 insertions(+), 103 deletions(-) ``` ### 依頼135 の commit（6715f5e）の差分の全量 ```diff diff --git a/lab_lib.sh b/lab_lib.sh index 6e1e082..33ebd5c 100644 --- a/lab_lib.sh +++ b/lab_lib.sh @@ -277,8 +277,12 @@ lab_cmd_addon() { # 3f. 何度流しても同じ結果 (install_addon. # corr2 (依頼 57) は corr_v1_1499 から trim2_corr_v4 で続きを学習したもの (設定 pi05_mecharm_trim_corr_v4_lora) # corr3 (依頼 97) は corr_v2_1499 から trim2_corr_v5 で続きを学習したもの (設定 pi05_mecharm_trim_corr_v5_lora) # deep1 (依頼 133) は π0.5 base から deep_even10_v1 (位置 1〜5 を各 10 本) で新しく学習したもの (設定 pi05_mecharm_deep_even10_lora. -# 中身は trim と同じ LoRA・batch 32・学習率の予定. 正規化の値はこの 50 本で測り直したもの) -LAB_MODELS="trim corr corr2 corr3 deep1" +# 中身は trim と同じ LoRA・batch 32・学習率の予定 (decay_steps 5_000 のまま 3000 ステップで終えた). 正規化の値はこの 50 本で測り直したもの) +# deep1b (依頼 135) は deep1 と同じデータ・同じ値で, decay_steps だけを num_train_steps と同じにして π0.5 base から学習するもの +# (設定 pi05_mecharm_deep_even10_b_lora). 依頼 135 から deep_even10_v1 を新しく学習するのはこちら (deep1 は学習し直さない) +# 安全範囲 (学習データ ±10°. run_policy_client.py --raw-dir) は datasets//raw_episodes. deep1・deep1b は +# deep_even10_v1, ほかは trim2 (依頼 135) +LAB_MODELS="trim corr corr2 corr3 deep1 deep1b" LAB_CORR_STEPS=1500 LAB_CORR_INIT=/workspace/ckpt/3999/params LAB_CORR2_STEPS=1500 @@ -286,7 +290,9 @@ LAB_CORR2_INIT=/workspace/ckpt/corr_v1_1499/params LAB_CORR3_STEPS=1500 LAB_CORR3_INIT=/workspace/ckpt/corr_v2_1499/params LAB_DEEP1_STEPS=3000 # 依頼 133. add_deep_config.py の --steps (推論では使わないが, 学習と同じ値で入れる) -lab_model_def() { # lab_model_def trim|corr → MODEL_REPO MODEL_CKPT MODEL_CONFIG MODEL_DATA MODEL_DESC. 知らない名前なら 1 +LAB_DEEP1B_STEPS=3000 # 依頼 135. deep1b の学習の既定のステップ数 (lab_train.sh) と, 推論で入れる設定の --steps +lab_model_def() { # lab_model_def trim|corr|... → MODEL_REPO MODEL_CKPT MODEL_CONFIG MODEL_DATA MODEL_DESC MODEL_RANGE_DATA. 知らない名前なら 1 + MODEL_RANGE_DATA=mecharm_redblock_box_teach_v1_trim2 # 安全範囲を作るデータ (依頼 135. deep1・deep1b だけ下で変える) case "$1" in trim) MODEL_REPO=PixelstoMotors/pi05-mecharm-trim-lora-v1 @@ -317,18 +323,27 @@ lab_model_def() { # lab_model_def trim|corr → MODEL_REPO MODEL_CKPT MODEL_CKPT=/workspace/ckpt/deep_v1_2999 MODEL_CONFIG=pi05_mecharm_deep_even10_lora MODEL_DATA=mecharm_redblock_box_deep_even10_v1 # 依頼 131: 位置 1〜5 を各 10 本 (50 本) - MODEL_DESC="π0.5 base から deep_even10_v1 で新しく学習した 2999 (依頼 133)" ;; + MODEL_RANGE_DATA=$MODEL_DATA # 依頼 135: 安全範囲はこの 50 本から + MODEL_DESC="π0.5 base から deep_even10_v1 で新しく学習した 2999 (依頼 133. decay_steps 5000)" ;; + deep1b) + MODEL_REPO=PixelstoMotors/pi05-mecharm-deep-even10-b-lora-v1 + MODEL_CKPT=/workspace/ckpt/deep_b_v1_$((LAB_DEEP1B_STEPS - 1)) + MODEL_CONFIG=pi05_mecharm_deep_even10_b_lora + MODEL_DATA=mecharm_redblock_box_deep_even10_v1 + MODEL_RANGE_DATA=$MODEL_DATA + MODEL_DESC="π0.5 base から deep_even10_v1 で decay_steps をステップ数と同じにして学習した $((LAB_DEEP1B_STEPS - 1)) (依頼 135)" ;; *) return 1 ;; esac } -lab_cmd_addon_model() { # lab_cmd_addon_model trim|corr|corr2|corr3|deep1 → 3f (trim は lab_cmd_addon と同じ. corr〜deep1 は学習の設定と assets も足す) +lab_cmd_addon_model() { # lab_cmd_addon_model trim|corr|corr2|corr3|deep1|deep1b → 3f (trim は lab_cmd_addon と同じ. corr〜deep1b は学習の設定と assets も足す) lab_model_def "$1" || return 1 if [ "$1" = trim ]; then lab_cmd_addon; return; fi local add="add_trim_corr_config.py /workspace/openpi --steps $LAB_CORR_STEPS --init $LAB_CORR_INIT" [ "$1" = corr2 ] && add="add_trim_corr_config.py /workspace/openpi --variant v4 --steps $LAB_CORR2_STEPS --init $LAB_CORR2_INIT" [ "$1" = corr3 ] && add="add_trim_corr_config.py /workspace/openpi --variant v5 --steps $LAB_CORR3_STEPS --init $LAB_CORR3_INIT" - [ "$1" = deep1 ] && add="add_deep_config.py /workspace/openpi --steps $LAB_DEEP1_STEPS" # 依頼 133 + [ "$1" = deep1 ] && add="add_deep_config.py /workspace/openpi --steps $LAB_DEEP1_STEPS --variant v1" # 依頼 133 (decay_steps 5000) + [ "$1" = deep1b ] && add="add_deep_config.py /workspace/openpi --steps $LAB_DEEP1B_STEPS --variant b" # 依頼 135 # assets は hf download が置いたモデルの assets/local/<データ名> を, openpi の assets/<設定>/local/ に写す # (trim の 3f が assets/pi05_mecharm_trim_lora/local/ に写すのと同じ形. corr〜corr3 は trim2 と同じ正規化の値, # deep1 は deep_even10_v1 で測り直した値) @@ -337,10 +352,14 @@ lab_cmd_addon_model() { # lab_cmd_addon_model trim|corr|corr2|corr3|deep1 # ---- lab_train.sh で作るモデル (依頼 57. --model corr | corr2. 依頼 97 で corr3. 依頼 133 で deep1) ---- # 作るモデルの名前は lab_start.sh の --model と同じ. 続きの元 (TRAIN_FROM) も lab_model_def のモデル -# deep1 は続きの元がない (π0.5 base から. TRAIN_FROM・TRAIN_REPO_IN・TRAIN_CKPT_IN・TRAIN_FROM_KIOXIA は空, TRAIN_INIT=base) -LAB_TRAIN_MODELS="corr corr2 corr3 deep1" -lab_train_def() { # lab_train_def corr|corr2|corr3|deep1 → TRAIN_* (知らない名前なら 1) +# deep1・deep1b は続きの元がない (π0.5 base から. TRAIN_FROM・TRAIN_REPO_IN・TRAIN_CKPT_IN・TRAIN_FROM_KIOXIA は空, TRAIN_INIT=base) +# TRAIN_DECAY: 学習率を下げきるステップ (decay_steps). 「steps」は --steps と同じ (依頼 135). 由来に残すのは deep1・deep1b だけ +# TRAIN_RETIRED: 1 なら新しくは学習しない (--stop で片付けるときだけ使う). deep1 は依頼 133 の 2999 (decay_steps 5000) を +# 作った形なので, 依頼 135 からは deep1b で学習する +LAB_TRAIN_MODELS="corr corr2 corr3 deep1 deep1b" +lab_train_def() { # lab_train_def corr|corr2|corr3|deep1|deep1b → TRAIN_* (知らない名前なら 1) TRAIN_STEPS=1500; TRAIN_PREP=pod_prep_trim_corr.sh # ステップ数の既定・4b で流す準備 (依頼 133 で deep1 だけ違う) + TRAIN_DECAY=steps; TRAIN_RETIRED=0 # corr〜corr3 は add_trim_corr_config.py の 1_500 (= 既定のステップ数) case "$1" in corr) # 依頼 49: 3999 から trim2_corr_v3 で続きを学習した 1499 TRAIN_FROM=trim; TRAIN_VARIANT=v3 @@ -351,18 +370,23 @@ lab_train_def() { # lab_train_def corr|corr2|corr3|deep1 → TRAIN_* ( corr3) # 依頼 97: corr2 の 1499 から trim2_corr_v5 で続きを学習する TRAIN_FROM=corr2; TRAIN_VARIANT=v5 TRAIN_FROM_KIOXIA=pi05_mecharm_trim_corr_lora_v2/1499 ;; - deep1) # 依頼 133: π0.5 base から deep_even10_v1 で新しく学習する (正規化の値は Pod で測り直す) - TRAIN_FROM=""; TRAIN_VARIANT=deep + deep1) # 依頼 133: π0.5 base から deep_even10_v1 で新しく学習した (正規化の値は Pod で測り直す) + TRAIN_FROM=""; TRAIN_VARIANT=v1; TRAIN_EXP_DEEP=deep_even10_v1 TRAIN_FROM_KIOXIA="" - TRAIN_STEPS=3000; TRAIN_PREP=pod_prep_deep.sh ;; + TRAIN_STEPS=3000; TRAIN_PREP=pod_prep_deep.sh + TRAIN_DECAY=5000; TRAIN_RETIRED=1 ;; # 依頼 135: 新しくは学習しない (deep1b で学習する) + deep1b) # 依頼 135: deep1 と同じデータ・値で, decay_steps を --steps と同じにして学習する + TRAIN_FROM=""; TRAIN_VARIANT=b; TRAIN_EXP_DEEP=deep_even10_v1_b + TRAIN_FROM_KIOXIA="" + TRAIN_STEPS=$LAB_DEEP1B_STEPS; TRAIN_PREP=pod_prep_deep.sh ;; *) return 1 ;; esac lab_model_def "$1" || return 1 TRAIN_DATA=$MODEL_DATA; TRAIN_CONFIG=$MODEL_CONFIG; TRAIN_REPO_OUT=$MODEL_REPO TRAIN_CKPT_NAME=${MODEL_REPO#*/pi05-}; TRAIN_CKPT_NAME=pi05_${TRAIN_CKPT_NAME//-/_} # 例 pi05_mecharm_trim_corr_lora_v2 TRAIN_EXP=trim_corr_${TRAIN_VARIANT} - if [ -z "$TRAIN_FROM" ]; then # deep1: 続きの元を取らない - TRAIN_EXP=deep_even10_v1 + if [ -z "$TRAIN_FROM" ]; then # deep1・deep1b: 続きの元を取らない + TRAIN_EXP=$TRAIN_EXP_DEEP TRAIN_REPO_IN=""; TRAIN_CKPT_IN=""; TRAIN_INIT=base return 0 fi @@ -371,8 +395,8 @@ lab_train_def() { # lab_train_def corr|corr2|corr3|deep1 → TRAIN_* ( lab_model_def "$1" # MODEL_* は作るモデルのものに戻す } -lab_cmd_addon_train() { # lab_cmd_addon_train corr|corr2|corr3|deep1 → lab_train.sh の 3f - # corr は lab_cmd_addon と同じ (3999 を取っているので trim の assets も写す). corr2・corr3・deep1 は 3999 を取らないので +lab_cmd_addon_train() { # lab_cmd_addon_train corr|corr2|corr3|deep1|deep1b → lab_train.sh の 3f + # corr は lab_cmd_addon と同じ (3999 を取っているので trim の assets も写す). corr2・corr3・deep1・deep1b は 3999 を取らないので # addon と trim2 の設定だけを入れる (学習の設定と正規化の値は pod_prep_trim_corr.sh・pod_prep_deep.sh が置く) if [ "$1" = corr ]; then lab_cmd_addon; return; fi echo "cd /workspace/openpi && python3 /workspace/r2r2r/pi05/openpi_addon/install_addon.py /workspace/openpi && python3 /workspace/r2r2r/add_trim_config.py /workspace/openpi" diff --git a/lab_start.sh b/lab_start.sh index 6b33165..5fe03d2 100755 --- a/lab_start.sh +++ b/lab_start.sh @@ -18,9 +18,12 @@ # deep1 = π0.5 base から deep_even10_v1 (位置 1〜5 を各 10 本の 50 本) で新しく学習した 2999 # (PixelstoMotors/pi05-mecharm-deep-even10-lora-v1. 依頼 133. lab_train.sh --model deep1 で作る. # Pod の /workspace/ckpt/deep_v1_2999. 正規化の値はこの 50 本で測り直したもの (モデルの assets にある)) -# 取る場所・設定・assets・サーバーの起動・つなぎ直しの確かめが選んだモデルに合う. 安全範囲 (学習データ ±10°) -# と位置合わせのお手本はどれも trim2 の raw_episodes (corr_v3・corr_v4 と合わせても範囲はほぼ同じで, trim2 の方が狭い. -# corr_v5 は 11-06 の 184650 の j2 が trim2 の上限 +31.4° を最大 11° 超える) +# deep1b = deep1 と同じデータで decay_steps をステップ数と同じにして学習した 2999 (依頼 135. lab_train.sh --model deep1b. +# PixelstoMotors/pi05-mecharm-deep-even10-b-lora-v1. Pod の /workspace/ckpt/deep_b_v1_2999) +# 取る場所・設定・assets・サーバーの起動・つなぎ直しの確かめが選んだモデルに合う. 安全範囲 (学習データ ±10°. +# run_policy_client.py --raw-dir) は, deep1・deep1b は deep_even10_v1 の raw_episodes (依頼 135), ほかは trim2 の +# raw_episodes (corr_v3・corr_v4 と合わせても範囲はほぼ同じで, trim2 の方が狭い. corr_v5 は 11-06 の 184650 の j2 が +# trim2 の上限 +31.4° を最大 11° 超える). 位置合わせ (camera_align.py) のお手本はどれも trim2 の raw_episodes # つなぎ直しで, 残っている Pod のサーバーが別のモデルで動いていれば, そのことを出して止まる # # 1. Pod を作る (A40×1・Runpod Pytorch 2.8.0・コンテナのディスク 200GB・22/tcp を外へ出す) @@ -68,7 +71,7 @@ while [ $# -gt 0 ]; do --dry-run) DRY_RUN=1; shift ;; --model) [ $# -ge 2 ] || { echo "[ERROR] --model には ${LAB_MODELS// / か } を付ける" >&2; exit 2; } # 値がないと shift 2 が失敗して回り続ける (codex R1) MODEL="$2"; shift 2 ;; - -h|--help) sed -n '2,42p' "$0"; exit 0 ;; + -h|--help) sed -n '2,45p' "$0"; exit 0 ;; *) echo "[ERROR] 知らない引数: $1 (./lab_start.sh --help)" >&2; exit 2 ;; esac done @@ -105,8 +108,11 @@ lab_need_key "$HF_TOKEN_FILE" "Hugging Face の鍵" [ -s "$POD_SSH_KEY.pub" ] || { echo "[ERROR] SSH の公開鍵がありません: $POD_SSH_KEY.pub" >&2; exit 1; } ADD_TRIM=/Volumes/KIOXIA/r2r2r-data/lab/Log11/pod/add_trim_config.py [ -f "$ADD_TRIM" ] || { echo "[ERROR] add_trim_config.py がありません: $ADD_TRIM" >&2; exit 1; } -RAW=/Volumes/KIOXIA/r2r2r-data/datasets/mecharm_redblock_box_teach_v1_trim2/raw_episodes +RAW=/Volumes/KIOXIA/r2r2r-data/datasets/mecharm_redblock_box_teach_v1_trim2/raw_episodes # 位置合わせのお手本 (どのモデルも trim2) [ -d "$RAW/$EPISODE" ] || { echo "[ERROR] お手本がありません: $RAW/$EPISODE" >&2; exit 1; } +RAW_RANGE=/Volumes/KIOXIA/r2r2r-data/datasets/$MODEL_RANGE_DATA/raw_episodes # 安全範囲を作る学習データ (lab_lib.sh の lab_model_def. 依頼 135) +[ -d "$RAW_RANGE" ] || { echo "[ERROR] 安全範囲を作る学習データがありません: $RAW_RANGE" >&2; exit 1; } +echo "[RANGE ] 安全範囲 (学習データ ±10°) は $RAW_RANGE から (--model $MODEL)" lab_ssh_opts HF_REPO=$MODEL_REPO # --model で選んだモデル (lab_lib.sh の lab_model_def) @@ -445,12 +451,12 @@ while :; do [ $rc = 0 ] || [ $rc = 130 ] || fail "camera_align.py が失敗 (終了コード $rc)" stage_done stage "8. 実機推論 ($round 回目)" - python3 pi05/run_policy_client.py --raw-dir "$RAW" --steps-per-chunk 10 --via-stride 3 \ + python3 pi05/run_policy_client.py --raw-dir "$RAW_RANGE" --steps-per-chunk 10 --via-stride 3 \ --settle-retries 0 --state-from-target --execute --ignore-framing --model "$MODEL_TAG" rc=$? else stage "8. 推論のクライアント (ドライラン・--dry-keys. アームには触らない)" - python3 pi05/run_policy_client.py --raw-dir "$RAW" --steps-per-chunk 10 --via-stride 3 \ + python3 pi05/run_policy_client.py --raw-dir "$RAW_RANGE" --steps-per-chunk 10 --via-stride 3 \ --settle-retries 0 --state-from-target --dry-keys --ignore-framing --model "$MODEL_TAG" rc=$? fi diff --git a/lab_train.sh b/lab_train.sh index d09309c..d1f31ca 100755 --- a/lab_train.sh +++ b/lab_train.sh @@ -15,12 +15,15 @@ # (設定 pi05_mecharm_trim_corr_v4_lora. 3c で取るのは 3999 ではなく corr の 1499) # corr3 = corr2 の 1499 から trim2_corr_v5 で続きを学習 → PixelstoMotors/pi05-mecharm-trim-corr-lora-v3 # (依頼 97. 設定 pi05_mecharm_trim_corr_v5_lora. 3c で取るのは corr2 の 1499) -# deep1 = π0.5 base から deep_even10_v1 (位置 1〜5 を各 10 本の 50 本) で新しく学習 -# → PixelstoMotors/pi05-mecharm-deep-even10-lora-v1 (依頼 133. 設定 pi05_mecharm_deep_even10_lora. +# deep1b = π0.5 base から deep_even10_v1 (位置 1〜5 を各 10 本の 50 本) で新しく学習 +# → PixelstoMotors/pi05-mecharm-deep-even10-b-lora-v1 (依頼 135. 設定 pi05_mecharm_deep_even10_b_lora. +# 学習率の予定は trim と同じ warmup 300・peak 5e-5・decay_lr 5e-6 で, decay_steps だけ --steps と同じ. # 3c は飛ばす (続きの元を取らない). 4b は pod_prep_deep.sh で, 正規化の値を Pod で測り直し, # lab//train/norm_stats.json に持ち帰って sha256 を由来に残す) -# --steps / --init / --exp-name 学習 (既定 1500 (deep1 は 3000) / 続きの元の params (deep1 は base) / -# trim_corr_v3・trim_corr_v4・trim_corr_v5・deep_even10_v1) +# deep1 = deep1b と同じで decay_steps 5000 のもの (依頼 133 の 2999 → PixelstoMotors/pi05-mecharm-deep-even10-lora-v1. +# 設定 pi05_mecharm_deep_even10_lora). 依頼 135 から新しくは学習しない (--stop の片付けだけ) +# --steps / --init / --exp-name 学習 (既定 1500 (deep1b は 3000) / 続きの元の params (deep1b は base) / +# trim_corr_v3・trim_corr_v4・trim_corr_v5・deep_even10_v1_b) # 続きの元: corr は /workspace/ckpt/3999/params, corr2 は /workspace/ckpt/corr_v1_1499/params, # corr3 は /workspace/ckpt/corr_v2_1499/params # --poll-min 進みを見る間 (既定 5 分) @@ -70,18 +73,26 @@ while [ $# -gt 0 ]; do --stall-min) lab_need_val 30 "$@"; STALL_MIN="$2"; shift 2 ;; --stop) MODE=stop; shift ;; --dry-run) DRY_RUN=1; shift ;; - -h|--help) sed -n '2,39p' "$0"; exit 0 ;; + -h|--help) sed -n '2,45p' "$0"; exit 0 ;; *) echo "[ERROR] 知らない引数: $1 (./lab_train.sh --help)" >&2; exit 2 ;; esac done [ -n "$LOG" ] || { echo "[ERROR] --log Log17 のように Log 番号を付けてください" >&2; exit 2; } lab_train_def "$MODEL" || { echo "[ERROR] --model は ${LAB_TRAIN_MODELS// / か }: $MODEL" >&2; exit 2; } +if [ $MODE = train ] && [ "$TRAIN_RETIRED" = 1 ]; then # 依頼 135: deep1 (decay_steps 5000) は学習し直さない + echo "[ERROR] --model $MODEL は学習し終えたモデル (decay_steps $TRAIN_DECAY) で, 新しくは学習しません. deep_even10_v1 を学習するなら --model deep1b" >&2 + exit 2 +fi [ -n "$INIT" ] || INIT=$TRAIN_INIT [ -n "$STEPS" ] || STEPS=$TRAIN_STEPS # corr〜corr3 は 1500, deep1 は 3000 (依頼 133) [ -n "$EXP" ] || EXP=$TRAIN_EXP num() { awk -v x="$1" 'BEGIN { exit !(x ~ /^[0-9]+(\.[0-9]+)?$/ && x > 0) }'; } for v in LIMIT_HOURS POLL_MIN STALL_MIN; do num "${!v}" || { echo "[ERROR] $v は 0 より大きい数: ${!v}" >&2; exit 2; }; done [[ "$STEPS" =~ ^[1-9][0-9]*$ ]] || { echo "[ERROR] --steps は 1 以上の整数: $STEPS" >&2; exit 2; } +DECAY_STEPS=$TRAIN_DECAY; [ "$DECAY_STEPS" = steps ] && DECAY_STEPS=$STEPS # 学習率を下げきるステップ (依頼 135) +if [ "$TRAIN_PREP" = pod_prep_deep.sh ] && [ "$DECAY_STEPS" -le 300 ]; then # add_deep_config.py の warmup 300 より後で下げきる + echo "[ERROR] --model $MODEL の decay_steps ($DECAY_STEPS) が warmup 300 以下. --steps を 301 以上にする" >&2; exit 2 +fi [[ "$EXP" =~ ^[A-Za-z0-9_.-]+$ ]] || { echo "[ERROR] --exp-name は英数字と _ . - だけ: $EXP" >&2; exit 2; } [[ "$INIT" =~ ^[A-Za-z0-9_./-]+$ ]] || { echo "[ERROR] --init に使えない文字がある: $INIT" >&2; exit 2; } case "$GPU" in *" "*) GPU_ID="$GPU" ;; *) GPU_ID="NVIDIA $GPU" ;; esac @@ -112,7 +123,7 @@ use_model() { # use_model corr|corr2|corr3|deep1 → 作るモデ CKPT_IN=$TRAIN_CKPT_IN # 3c で続きの元を置く場所 (deep1 は空) CKPT_IN_KIOXIA="" # HF_REPO_IN と同じもの (由来の指紋を取る. 依頼 54. deep1 は空) [ -n "$TRAIN_FROM_KIOXIA" ] && CKPT_IN_KIOXIA="$CKPT_ROOT/$TRAIN_FROM_KIOXIA" - PREP_VARIANT=$TRAIN_VARIANT # pod_prep_trim_corr.sh の 3 つ目 (v3 / v4 / v5) + PREP_VARIANT=$TRAIN_VARIANT # pod_prep_trim_corr.sh の 3 つ目 (v3 / v4 / v5). pod_prep_deep.sh の 2 つ目 (v1 / b. 依頼 135) PREP_SH=$TRAIN_PREP # 4b で流す準備 (deep1 は pod_prep_deep.sh. 依頼 133) } use_model "$MODEL" @@ -216,6 +227,7 @@ prov_spec() { # 由来の spec (lab_train.sh が知っているこ --arg pod "$POD_ID" --arg gpu "$GPU_USED" --arg cph "$COST_PER_HR" --arg yen "$YEN_PER_USD" \ --arg t0 "$(iso_of "$POD_T0")" --arg t1 "$(iso_of "$t_end")" --arg tt "$(iso_of "$TRAIN_T0")" --arg rented "$rented" \ --arg cost_line "${COST_LINE:-}" --arg norm_sha "$NORM_SHA" --arg norm_pod "$NORM_POD" --arg norm_local "$NORM_LOCAL" \ + --arg deep "$([ "$PREP_SH" = pod_prep_deep.sh ] && echo 1)" --arg decay "${DECAY_STEPS:-}" --arg variant "$PREP_VARIANT" \ 'def num: if . == "" then null else tonumber end; def str: if . == "" then null else . end; {producer: {script: $script, argv: $argv, dry_run: ($dry == "1")}, external_code: $openpi, @@ -226,7 +238,10 @@ prov_spec() { # 由来の spec (lab_train.sh が知っているこ config: ({name: $config, exp_name: $exp, num_train_steps: ($steps | num), init_params: ($init | str), data_name: $data, limit_hours: ($limit | num), checkpoint_step: ($step | str)} + (if $norm_sha == "" then {} else {norm_stats: {source: "Pod で compute_norm_stats.py を回して測り直した (依頼 133)", - sha256: $norm_sha, pod_path: ("/workspace/openpi/" + $norm_pod), lab_copy: ($norm_local | str)}} end)), + sha256: $norm_sha, pod_path: ("/workspace/openpi/" + $norm_pod), lab_copy: ($norm_local | str)}} end) + # 依頼 135: deep の学習率の予定 (deep1 = decay_steps 5000 / deep1b = num_train_steps と同じ) + + (if $deep == "1" then {lr_schedule: {warmup_steps: 300, peak_lr: 5e-5, decay_steps: ($decay | num), decay_lr: 5e-6, + source: ("add_deep_config.py --variant " + $variant)}} else {} end)), compute: {provider: "runpod", pod_id: ($pod | str), gpu: ($gpu | str), started_at: ($t0 | str), train_started_at: ($tt | str), ended_at: ($t1 | str), duration_sec: ($rented | num), cost_per_hr_usd: ($cph | num), cost_usd: (if $rented == "" then null else (($rented | tonumber) / 3600 * ($cph | tonumber) * 100 | round / 100) end), @@ -649,6 +664,7 @@ if [ $MODE = stop ]; then STEPS=$(jq -r '.steps // ""' "$STATE"); INIT=$(jq -r '.init // ""' "$STATE"); LIMIT_HOURS=$(jq -r '.limit_hours // ""' "$STATE") PROV_START=$(jq -c '.prov_start // empty' "$STATE") NORM_SHA=$(jq -r '.norm_sha256 // ""' "$STATE"); NORM_POD=$(jq -r '.norm_pod // ""' "$STATE"); NORM_LOCAL=$(jq -r '.norm_local // ""' "$STATE") # 依頼 133 + DECAY_STEPS=$TRAIN_DECAY; [ "$DECAY_STEPS" = steps ] && DECAY_STEPS=$STEPS # 依頼 135 (記録のモデルとステップ数から) [ "$PREP_SH" = pod_prep_deep.sh ] && PREP_LOG=/workspace/logs/prep_deep.log OPENPI_JSON=$(jq -c '.openpi // [{name: "openpi", commit: null, dirty: null, error: "記録にない (--stop)"}]' "$STATE") || OPENPI_JSON='[]' POD_EXP_DIR=/workspace/openpi/checkpoints/$CONFIG/$EXP; TRAIN_LOG=/workspace/logs/train_$EXP.log @@ -660,6 +676,7 @@ fi echo "[LAB ] $(date '+%Y-%m-%d %H:%M:%S') 学習 GPU=$GPU_ID$([ $FALLBACK = 1 ] && echo " (借りられなければ ${FALLBACK_GPUS[*]})")" echo " --model $MODEL (${HF_REPO_IN:-π0.5 base} → $HF_REPO_OUT) データ $DATA_NAME" echo " $CONFIG --exp-name=$EXP steps=$STEPS init=$INIT 上限 $LIMIT_HOURS 時間 記録 $LOGDIR$([ $DRY_RUN = 1 ] && echo ' ドライラン (Pod は借りない)')" +[ "$PREP_SH" = pod_prep_deep.sh ] && echo " 学習率: warmup 300・peak 5e-5・decay_steps ${DECAY_STEPS}・decay_lr 5e-6 (add_deep_config.py --variant ${PREP_VARIANT})" # ---- 0. 送るものと Mac の道具 ---- stage "0. 送るものと Mac の道具を確かめる" @@ -821,10 +838,10 @@ LAB_TO=$LAB_TO_LONG pod_scp "$TAR" "root@$POD_IP:/workspace/data/" || wrap_fail LAB_TO=$LAB_TO_SHORT pssh "ls -l /workspace/data/$(basename "$TAR")" || wrap_fail prep "送った tar が見えない" stage_done if [ "$PREP_SH" = pod_prep_deep.sh ]; then # deep1 (依頼 133): 正規化の値を Pod で測り直す - stage "4b. pod_prep_deep.sh $STEPS (変換・正規化の値を測り直す)" + stage "4b. pod_prep_deep.sh $STEPS $PREP_VARIANT (変換・正規化の値を測り直す)" # 出力は corr と同じくその場で画面とターミナルの記録に流し, Mac 側にも写しを残して NORM の行を読む (codex R1: $(...) に溜めない) PREP_OUT=$(dest_of prep_deep_mac.txt) - LAB_TO=$LAB_TO_LONG pssh "set -o pipefail; bash /workspace/r2r2r/pi05/openpi_addon/pod_prep_deep.sh $STEPS 2>&1 | tee $PREP_LOG" | tee "$PREP_OUT" \ + LAB_TO=$LAB_TO_LONG pssh "set -o pipefail; bash /workspace/r2r2r/pi05/openpi_addon/pod_prep_deep.sh $STEPS $PREP_VARIANT 2>&1 | tee $PREP_LOG" | tee "$PREP_OUT" \ || wrap_fail prep "pod_prep_deep.sh に失敗 (上の出力を見る)" nl=$(grep -E '^NORM sha256=[0-9a-f]{64} path=assets/[A-Za-z0-9_./-]+/norm_stats\.json$' "$PREP_OUT" | tail -1) [ -n "$nl" ] || wrap_fail prep "pod_prep_deep.sh から正規化の値の sha256 が返らない" diff --git a/lab_train_fakes.sh b/lab_train_fakes.sh index 7b86d96..756a95f 100644 --- a/lab_train_fakes.sh +++ b/lab_train_fakes.sh @@ -28,9 +28,9 @@ # hf_same Hugging Face にもう同じ中身がある (上げ直さない. 由来も足さない. 前の回の由来がある. 依頼 54) # hf_prov_hang 由来を Hugging Face へ足すのが返らない (LAB_PROV_TO を小さくして使う. codex R2) # openpi_partial Pod の openpi のコミットの行だけ出て ssh が切れる (codex R1) -# norm_missing deep1: pod_prep_deep.sh が正規化の値の sha256 の行を出さずに終わる (依頼 133) -# norm_fetch_fail deep1: 測り直した正規化の値を 4b で持ち帰れない (学習は続け, あとでチェックポイントの assets から取る. 依頼 133) -# norm_bad deep1: Pod が言う正規化の値の sha256 と, 持ち帰ったもの・チェックポイントの assets の中身が違う (依頼 133) +# norm_missing deep1・deep1b: pod_prep_deep.sh が正規化の値の sha256 の行を出さずに終わる (依頼 133) +# norm_fetch_fail deep1・deep1b: 測り直した正規化の値を 4b で持ち帰れない (学習は続け, あとでチェックポイントの assets から取る. 依頼 133) +# norm_bad deep1・deep1b: Pod が言う正規化の値の sha256 と, 持ち帰ったもの・チェックポイントの assets の中身が違う (依頼 133) LAB_DRY_ROOT=${LAB_DRY_ROOT:-$(mktemp -d /tmp/lab_train_dry.XXXXXX)} LAB_FAKE_SCENARIO=${LAB_FAKE_SCENARIO:-success} @@ -166,7 +166,7 @@ pod_ssh() { *"pod_prep_trim_corr.sh"*) echo "== 5. 準備できました (ドライラン)" ;; *"pod_prep_deep.sh"*) # 依頼 133: 正規化の値を測り直し, sha256 の行を出す echo "== 4. 正規化の値をこの 50 本で測り直す (ドライラン)" - [ "$LAB_FAKE_SCENARIO" = norm_missing ] || echo "NORM sha256=$([ "$LAB_FAKE_SCENARIO" = norm_bad ] && printf 'a%.0s' {1..64} || echo $FAKE_NORM_SHA) path=assets/pi05_mecharm_deep_even10_lora/local/mecharm_redblock_box_deep_even10_v1/norm_stats.json" + [ "$LAB_FAKE_SCENARIO" = norm_missing ] || echo "NORM sha256=$([ "$LAB_FAKE_SCENARIO" = norm_bad ] && printf 'a%.0s' {1..64} || echo $FAKE_NORM_SHA) path=assets/$CONFIG/local/mecharm_redblock_box_deep_even10_v1/norm_stats.json" # 設定の名前は作るモデルのもの (依頼 135) echo "== 5. 準備できました (ドライラン)" ;; *": lt=openpi_rev"*) [ "$LAB_FAKE_SCENARIO" = openpi_unknown ] && { echo "fatal: not a git repository"; return 128; } diff --git a/pi05/openpi_addon/add_deep_config.py b/pi05/openpi_addon/add_deep_config.py index f3409c1..e4c260a 100755 --- a/pi05/openpi_addon/add_deep_config.py +++ b/pi05/openpi_addon/add_deep_config.py @@ -1,44 +1,73 @@ #!/usr/bin/env python3 -"""openpi config.py に deep_even10_v1 (位置 1〜5 を各 10 本にそろえた 50 本) 用の学習設定 pi05_mecharm_deep_even10_lora を足す -(2026-11-09 依頼 133. lab_train.sh --model deep1). +"""openpi config.py に deep_even10_v1 (位置 1〜5 を各 10 本にそろえた 50 本) 用の学習設定を足す +(2026-11-09 依頼 133. lab_train.sh --model deep1. 依頼 135 で --variant b (lab_train.sh --model deep1b) を足した). π0.5 base から新しく学習する (続きの学習ではない). 中身は trim の pi05_mecharm_trim_lora と同じ LoRA・batch 32・ -学習率の予定 (warmup 300 / peak 5e-5 / decay_steps 5_000 / decay_lr 5e-6)・weight_loader (π0.5 base) で, -次の 3 か所だけ変える (add_trim_config.py と同じ方式): - name pi05_mecharm_trim_lora -> pi05_mecharm_deep_even10_lora +学習率の予定 (warmup 300 / peak 5e-5 / decay_lr 5e-6)・weight_loader (π0.5 base) で, 次の所だけ変える +(add_trim_config.py と同じ方式): + name pi05_mecharm_trim_lora -> --variant v1: pi05_mecharm_deep_even10_lora + --variant b: pi05_mecharm_deep_even10_b_lora repo_id local/mecharm_redblock_box_teach_v1_trim2 -> local/mecharm_redblock_box_deep_even10_v1 num_train_steps 4_000 -> --steps (依頼 133 は 3000) + decay_steps 5_000 -> --variant v1: 5_000 のまま (依頼 133 の 2999 はこれで学習した) + --variant b: --steps と同じ (依頼 135. 学習の終わりで decay_lr まで下げきる) 正規化の値は写さず, Pod で compute_norm_stats.py をこの設定で回して測り直す (pod_prep_deep.sh). install_addon.py と add_trim_config.py (lab/Log11/pod/) を実行した後に 1 回実行: - python3 add_deep_config.py /workspace/openpi --steps 3000 -何度実行しても同じ結果 (二重に入れない). 入っている設定と --steps が違うときは止まる. + python3 add_deep_config.py /workspace/openpi --steps 3000 --variant b +何度実行しても同じ結果 (二重に入れない). 入っている設定と --steps が違うときは止まる. v1 と b は名前が違うので両方入れられる. """ import argparse import re import sys from pathlib import Path -NAME = "pi05_mecharm_deep_even10_lora" REPO_ID = "local/mecharm_redblock_box_deep_even10_v1" -MARK_A = " # >>> r2r2r mecharm deep_even10_v1 (train config)\n" -MARK_B = " # <<< r2r2r mecharm deep_even10_v1 (train config)\n" TRIM_A = " # >>> r2r2r mecharm teach_v1_trim2 (train config)\n" TRIM_B = " # <<< r2r2r mecharm teach_v1_trim2 (train config)\n" BASE_LOADER = 'weight_loader=weight_loaders.CheckpointWeightLoader("gs://openpi-assets/checkpoints/pi05_base/params"),' -# trim と同じであることを確かめる値 (写したあとも変わっていないこと) -SAME_AS_TRIM = ("batch_size=32,", "warmup_steps=300,", "peak_lr=5e-5,", "decay_steps=5_000,", "decay_lr=5e-6,", BASE_LOADER) +TRIM_DECAY = "decay_steps=5_000," +WARMUP = 300 # trim と同じ warmup_steps. decay_steps はこれより大きくないと下げる区間がない +# trim と同じであることを確かめる値 (写したあとも変わっていないこと). decay_steps は --variant ごとに見る +SAME_AS_TRIM = ("batch_size=32,", f"warmup_steps={WARMUP},", "peak_lr=5e-5,", "decay_lr=5e-6,", BASE_LOADER) +# --variant ごとの設定の名前・印・decay_steps (None は --steps と同じ) +VARIANTS = { + "v1": {"name": "pi05_mecharm_deep_even10_lora", "mark": "deep_even10_v1", "decay": 5000}, # 依頼 133 (今回の 2999) + "b": {"name": "pi05_mecharm_deep_even10_b_lora", "mark": "deep_even10_v1 b", "decay": None}, # 依頼 135 +} +# 依頼 133 の形を残す (テスト・ほかから読む名前) +NAME = VARIANTS["v1"]["name"] -def make_block(s, steps): +def marks(variant): + m = VARIANTS[variant]["mark"] + return (f" # >>> r2r2r mecharm {m} (train config)\n", f" # <<< r2r2r mecharm {m} (train config)\n") + + +MARK_A, MARK_B = marks("v1") + + +def decay_of(variant, steps): + d = VARIANTS[variant]["decay"] + return steps if d is None else d + + +def decay_text(n): + """openpi の書き方に合わせる (5000 -> 5_000)""" + return f"decay_steps={n:_}," + + +def make_block(s, steps, variant="v1"): assert s.count(TRIM_A) == 1 and s.count(TRIM_B) == 1, "先に install_addon.py と add_trim_config.py を実行する" block = s[s.index(TRIM_A):s.index(TRIM_B) + len(TRIM_B)] + ma, mb = marks(variant) reps = [ - (TRIM_A, MARK_A), - (TRIM_B, MARK_B), - ('name="pi05_mecharm_trim_lora"', f'name="{NAME}"'), + (TRIM_A, ma), + (TRIM_B, mb), + ('name="pi05_mecharm_trim_lora"', f'name="{VARIANTS[variant]["name"]}"'), ('repo_id="local/mecharm_redblock_box_teach_v1_trim2"', f'repo_id="{REPO_ID}"'), ("num_train_steps=4_000,", f"num_train_steps={steps},"), + (TRIM_DECAY, decay_text(decay_of(variant, steps))), ] for a, b in reps: assert block.count(a) == 1, f"見つからない/重複: {a!r}" @@ -52,27 +81,36 @@ def main(argv=None): ap = argparse.ArgumentParser() ap.add_argument("openpi_root") ap.add_argument("--steps", type=int, required=True, help="学習のステップ数 (num_train_steps. 依頼 133 は 3000)") + ap.add_argument("--variant", choices=sorted(VARIANTS), required=True, + help="v1 = decay_steps 5_000 (依頼 133 の 2999) / b = decay_steps を --steps と同じに (依頼 135)") args = ap.parse_args(argv) if args.steps <= 0: sys.exit("[ERROR] --steps は 1 以上") + decay = decay_of(args.variant, args.steps) + if decay <= WARMUP: + sys.exit(f"[ERROR] decay_steps ({decay}) が warmup_steps ({WARMUP}) 以下. --steps を大きくする") + name = VARIANTS[args.variant]["name"] + ma, mb = marks(args.variant) cfg = Path(args.openpi_root).resolve() / "src" / "openpi" / "training" / "config.py" s = cfg.read_text() - if MARK_A in s: - have = s[s.index(MARK_A):] - have = have[:have.index(MARK_B)] + if ma in s: + have = s[s.index(ma):] + have = have[:have.index(mb)] m = re.search(r"num_train_steps=(\d+),", have) - if not (f'name="{NAME}"' in have and f'repo_id="{REPO_ID}"' in have and all(x in have for x in SAME_AS_TRIM)): + if not (f'name="{name}"' in have and f'repo_id="{REPO_ID}"' in have and all(x in have for x in SAME_AS_TRIM)): sys.exit("[ERROR] 入っている設定の名前・データ名・学習の値が違う. openpi の config.py を git checkout してから入れ直す") if not (m and int(m.group(1)) == args.steps): sys.exit("[ERROR] 別の --steps で入っている. openpi の config.py を git checkout してから入れ直す") + if have.count(decay_text(decay)) != 1: + sys.exit(f"[ERROR] 入っている設定の decay_steps が {decay} ではない. openpi の config.py を git checkout してから入れ直す") print("already installed (config.py は変更なし)") return 0 - if f'name="{NAME}"' in s: - sys.exit(f"[ERROR] {NAME} がもう別の形で入っている. openpi の config.py を git checkout してから入れ直す") - block = make_block(s, args.steps) + if f'name="{name}"' in s: + sys.exit(f"[ERROR] {name} がもう別の形で入っている. openpi の config.py を git checkout してから入れ直す") + block = make_block(s, args.steps, args.variant) j = s.index(TRIM_B) + len(TRIM_B) cfg.write_text(s[:j] + block + s[j:]) - print(f"installed: {NAME} (repo_id={REPO_ID}, {args.steps} steps, init=π0.5 base)") + print(f"installed: {name} (repo_id={REPO_ID}, {args.steps} steps, decay_steps {decay}, init=π0.5 base)") return 0 diff --git a/pi05/openpi_addon/pod_prep_deep.sh b/pi05/openpi_addon/pod_prep_deep.sh index 41048a6..51d2fc5 100755 --- a/pi05/openpi_addon/pod_prep_deep.sh +++ b/pi05/openpi_addon/pod_prep_deep.sh @@ -1,6 +1,6 @@ #!/usr/bin/env bash # pod_prep_deep.sh — Pod で deep_even10_v1 (位置 1〜5 を各 10 本にそろえた 50 本) の学習の準備をする (2026-11-09 依頼 133). -# 学習は回さない. lab_train.sh --model deep1 の 4b が呼ぶ. +# 学習は回さない. lab_train.sh --model deep1b (依頼 135) の 4b が呼ぶ (deep1 は依頼 133 で呼んだ形). # # 前提: lab_train.sh 3a〜3f で Pod を準備してある (openpi・install_addon.py・add_trim_config.py). 続きの元のモデルは取らない # (π0.5 base から. 重みは学習の最初に openpi が gs://openpi-assets から取る) @@ -8,12 +8,15 @@ # /workspace/data/mecharm_redblock_box_deep_even10_v1_raw_episodes.tar # /workspace/r2r2r/pi05/ (add_deep_config.py と convert_to_lerobot.py を使う) # -# bash /workspace/r2r2r/pi05/openpi_addon/pod_prep_deep.sh -# 例: bash ... 3000 +# bash /workspace/r2r2r/pi05/openpi_addon/pod_prep_deep.sh +# 例: bash ... 3000 b +# は add_deep_config.py の --variant (依頼 135): +# v1 = 設定 pi05_mecharm_deep_even10_lora・decay_steps 5_000 (依頼 133 の 2999. lab_train.sh --model deep1) +# b = 設定 pi05_mecharm_deep_even10_b_lora・decay_steps を と同じに (lab_train.sh --model deep1b) # # すること (何度流しても同じ結果): # 1. tar を /workspace/data/<データ名>/ に広げ, エピソードの本数を確かめる (Mac で作ったときは 50 本) -# 2. openpi の config.py に pi05_mecharm_deep_even10_lora を足す (--steps) +# 2. openpi の config.py に の設定を足す (--steps --variant) # 3. LeRobot 形式に変換 (repo_id local/<データ名>) # 4. 正規化の値をこの 50 本で測り直す (RUNPOD_TRAIN.md 4. の compute_norm_stats.py をこの設定で). # 置く場所は openpi が学習で読む assets/<設定の名前>/local/<データ名>/norm_stats.json. @@ -22,9 +25,13 @@ set -euo pipefail STEPS="${1:?学習のステップ数 (例 3000)}" +VARIANT="${2:?add_deep_config.py の --variant (v1 か b)}" NAME=mecharm_redblock_box_deep_even10_v1 -CONFIG=pi05_mecharm_deep_even10_lora -EXP=deep_even10_v1 +case "$VARIANT" in # 何かする前に確かめる (依頼 135) + v1) CONFIG=pi05_mecharm_deep_even10_lora; EXP=deep_even10_v1 ;; + b) CONFIG=pi05_mecharm_deep_even10_b_lora; EXP=deep_even10_v1_b ;; + *) echo "[ERROR] は v1 か b: $VARIANT" >&2; exit 2 ;; +esac N_MAC=50 TAR=/workspace/data/${NAME}_raw_episodes.tar DATA=/workspace/data/$NAME @@ -49,8 +56,8 @@ mv "$TMP" "$DATA" echo " エピソード $N 本 (Mac で作ったときは $N_MAC 本)" cd "$OPENPI" -echo "== 2. 学習設定 $CONFIG を足す (steps=$STEPS. π0.5 base から)" -python3 "$ADDON/add_deep_config.py" "$OPENPI" --steps "$STEPS" +echo "== 2. 学習設定 $CONFIG を足す (steps=${STEPS}・variant ${VARIANT}. π0.5 base から)" +python3 "$ADDON/add_deep_config.py" "$OPENPI" --steps "$STEPS" --variant "$VARIANT" echo "== 3. LeRobot 形式に変換 (local/$NAME)" uv run python /workspace/r2r2r/pi05/convert_to_lerobot.py --raw-dir "$DATA/raw_episodes" --repo-id "local/$NAME" diff --git a/pi05/test_merge_and_config.py b/pi05/test_merge_and_config.py index 31cc80c..b6ce35e 100644 --- a/pi05/test_merge_and_config.py +++ b/pi05/test_merge_and_config.py @@ -335,27 +335,78 @@ class TestConfigV5(unittest.TestCase): class TestConfigDeep(unittest.TestCase): - """依頼 133: deep_even10_v1 の設定は trim と同じで, 名前・データ名・ステップ数だけ違う. π0.5 base から.""" + """依頼 133: deep_even10_v1 の設定は trim と同じで, 名前・データ名・ステップ数だけ違う. π0.5 base から. + 依頼 135: --variant v1 は decay_steps 5_000 のまま (依頼 133 の 2999), --variant b は decay_steps を --steps と同じにする.""" def test_same_as_trim_except_data_and_steps(self): with tempfile.TemporaryDirectory() as td, mock.patch("sys.stdout"): cfg = fake_config(Path(td)) self.assertEqual(atc.main([td, "--variant", "v5", "--steps", "1500", "--init", "/workspace/ckpt/corr_v2_1499/params"]), 0) # corr の設定と並べて入る - self.assertEqual(adc.main([td, "--steps", "3000"]), 0) + self.assertEqual(adc.main([td, "--steps", "3000", "--variant", "v1"]), 0) c = configs(cfg) new, trim = c["pi05_mecharm_deep_even10_lora"], c["pi05_mecharm_trim_lora"] self.assertIn("local/mecharm_redblock_box_deep_even10_v1", new["data"]) self.assertEqual(new["num_train_steps"], "3000") self.assertIn("gs://openpi-assets/checkpoints/pi05_base/params", new["weight_loader"]) for k in set(trim) - {"name", "data", "num_train_steps"}: - self.assertEqual(new[k], trim[k], k) + self.assertEqual(new[k], trim[k], k) # v1 は学習率の予定も trim と同じ (decay_steps 5000) + self.assertIn("decay_steps=5000", new["lr_schedule"]) self.assertEqual(new["data"].replace("deep_even10_v1", "teach_v1_trim2"), trim["data"]) self.assertEqual(trim["num_train_steps"], "4000") # trim2 の設定は変えない text = cfg.read_text() - self.assertEqual(adc.main([td, "--steps", "3000"]), 0) # 二重に入れない + self.assertEqual(adc.main([td, "--steps", "3000", "--variant", "v1"]), 0) # 二重に入れない self.assertEqual(cfg.read_text(), text) with self.assertRaises(SystemExit): - adc.main([td, "--steps", "2000"]) # 違うステップ数では止まる + adc.main([td, "--steps", "2000", "--variant", "v1"]) # 違うステップ数では止まる + self.assertEqual(cfg.read_text(), text) + + def test_variant_b_decay_equals_steps(self): + """依頼 135: b は decay_steps = num_train_steps. ほかは v1 (trim) と同じ. v1 と並べて入る""" + for steps in (3000, 2000): + with self.subTest(steps=steps), tempfile.TemporaryDirectory() as td, mock.patch("sys.stdout"): + cfg = fake_config(Path(td)) + self.assertEqual(adc.main([td, "--steps", "3000", "--variant", "v1"]), 0) + self.assertEqual(adc.main([td, "--steps", str(steps), "--variant", "b"]), 0) + c = configs(cfg) + b, v1, trim = (c["pi05_mecharm_deep_even10_b_lora"], c["pi05_mecharm_deep_even10_lora"], + c["pi05_mecharm_trim_lora"]) + self.assertEqual(b["num_train_steps"], str(steps)) + self.assertEqual(b["lr_schedule"], trim["lr_schedule"].replace("decay_steps=5000", f"decay_steps={steps}")) + self.assertIn("warmup_steps=300", b["lr_schedule"]) + self.assertIn("peak_lr=5e-05", b["lr_schedule"]) + self.assertIn("decay_lr=5e-06", b["lr_schedule"]) + self.assertEqual(b["data"], v1["data"]) + for k in set(trim) - {"name", "data", "num_train_steps", "lr_schedule"}: + self.assertEqual(b[k], trim[k], k) + self.assertIn("decay_steps=5000", v1["lr_schedule"]) # v1 (今回の 2999) は変わらない + text = cfg.read_text() + self.assertEqual(adc.main([td, "--steps", str(steps), "--variant", "b"]), 0) # 二重に入れない + self.assertEqual(cfg.read_text(), text) + with self.assertRaises(SystemExit): + adc.main([td, "--steps", str(steps + 500), "--variant", "b"]) # 違うステップ数では止まる + self.assertEqual(cfg.read_text(), text) + + def test_variant_b_refuses_changed_decay(self): + """入っている b の decay_steps がステップ数と違えば止まる (手で書き換えたものを使わない)""" + with tempfile.TemporaryDirectory() as td, mock.patch("sys.stdout"): + cfg = fake_config(Path(td)) + self.assertEqual(adc.main([td, "--steps", "3000", "--variant", "b"]), 0) + cfg.write_text(cfg.read_text().replace("decay_steps=3_000,", "decay_steps=5_000,")) + text = cfg.read_text() + with self.assertRaises(SystemExit): + adc.main([td, "--steps", "3000", "--variant", "b"]) + self.assertEqual(cfg.read_text(), text) + + def test_variant_required_and_warmup(self): + with tempfile.TemporaryDirectory() as td, mock.patch("sys.stdout"), mock.patch("sys.stderr"): + cfg = fake_config(Path(td)) + text = cfg.read_text() + with self.assertRaises(SystemExit): + adc.main([td, "--steps", "3000"]) # --variant なしでは入れない + with self.assertRaises(SystemExit): + adc.main([td, "--steps", "3000", "--variant", "c"]) + with self.assertRaises(SystemExit): + adc.main([td, "--steps", "300", "--variant", "b"]) # decay_steps が warmup 300 以下 self.assertEqual(cfg.read_text(), text) def test_requires_trim_first(self): @@ -363,18 +414,31 @@ class TestConfigDeep(unittest.TestCase): cfg = Path(td) / "src" / "openpi" / "training" / "config.py" cfg.parent.mkdir(parents=True) cfg.write_text("_CONFIGS = []\n") - with self.assertRaises(AssertionError): - adc.main([td, "--steps", "3000"]) + for v in ("v1", "b"): + with self.assertRaises(AssertionError): + adc.main([td, "--steps", "3000", "--variant", v]) def test_pod_prep_deep_computes_norm(self): sh = _HERE / "openpi_addon" / "pod_prep_deep.sh" self.assertEqual(subprocess.run(["bash", "-n", str(sh)]).returncode, 0) code = "\n".join(ln for ln in sh.read_text().splitlines() if not ln.lstrip().startswith("#")) - for x in ('uv run scripts/compute_norm_stats.py --config-name "$CONFIG"', "CONFIG=pi05_mecharm_deep_even10_lora", + for x in ('uv run scripts/compute_norm_stats.py --config-name "$CONFIG"', + "v1) CONFIG=pi05_mecharm_deep_even10_lora; EXP=deep_even10_v1 ;;", + "b) CONFIG=pi05_mecharm_deep_even10_b_lora; EXP=deep_even10_v1_b ;;", "NAME=mecharm_redblock_box_deep_even10_v1", "NORM_DST=assets/$CONFIG/local/$NAME/norm_stats.json", - 'echo "NORM sha256=$SHA path=$NORM_DST"', 'add_deep_config.py" "$OPENPI" --steps "$STEPS"'): + 'echo "NORM sha256=$SHA path=$NORM_DST"', 'add_deep_config.py" "$OPENPI" --steps "$STEPS" --variant "$VARIANT"'): self.assertIn(x, code) + def test_pod_prep_deep_unknown_variant_stops_first(self): + """依頼 135: が v1・b 以外 (またはない) なら, tar を見る前に止まる""" + sh = _HERE / "openpi_addon" / "pod_prep_deep.sh" + r = subprocess.run(["bash", str(sh), "3000", "c"], capture_output=True, text=True) + self.assertEqual(r.returncode, 2) + self.assertIn(" は v1 か b: c", r.stderr) + r = subprocess.run(["bash", str(sh), "3000"], capture_output=True, text=True) + self.assertNotEqual(r.returncode, 0) + self.assertNotIn("tar がない", r.stderr) + class TestPodPrep(unittest.TestCase): """依頼 48: pod_prep_trim_corr.sh は正規化の値を計算し直さず, 3999 の学習の trim2 の値を置く.""" diff --git a/test_lab_start.sh b/test_lab_start.sh index 394bc5c..bcb8528 100755 --- a/test_lab_start.sh +++ b/test_lab_start.sh @@ -24,7 +24,8 @@ setup() { # 1 つの場合ごとに新しい一時フォルダ T=$(mktemp -d) mkdir -p "$T/bin" "$T/api" "$T/keys" "$T/pi05" \ "$T/data/lab/Log11/pod" "$T/data/lab/LogT/pod" \ - "$T/data/datasets/mecharm_redblock_box_teach_v1_trim2/raw_episodes/run_0001_20261019_194818" + "$T/data/datasets/mecharm_redblock_box_teach_v1_trim2/raw_episodes/run_0001_20261019_194818" \ + "$T/data/datasets/mecharm_redblock_box_deep_even10_v1/raw_episodes" echo "x" > "$T/data/lab/Log11/pod/add_trim_config.py" echo "rk" > "$T/keys/runpod"; echo "hk" > "$T/keys/hf"; echo "k" > "$T/keys/id"; echo "k.pub" > "$T/keys/id.pub" sed -e "s#/Volumes/KIOXIA/r2r2r-data#$T/data#g" -e 's#^LISTEN_WAIT=30; LISTEN_POLL=5#LISTEN_WAIT=2; LISTEN_POLL=1#' \ @@ -340,6 +341,7 @@ check "終了コード 0" '[ $RC = 0 ]' check "3999 を取り, trim の設定でサーバーを起動" 'has ssh.log "hf download PixelstoMotors/pi05-mecharm-trim-lora-v1 --local-dir /workspace/ckpt/3999" && has ssh.log "--policy.config=pi05_mecharm_trim_lora --policy.dir=/workspace/ckpt/3999 2>&1"' check "3f は今までと同じ (corr の設定は足さない)" 'has ssh.log "$(. "$HERE/lab_lib.sh"; lab_cmd_addon)" && ! has ssh.log "add_trim_corr_config.py"' check "クライアントに --model trim を渡す・安全範囲は trim2" 'has py.log "--model trim (PixelstoMotors/pi05-mecharm-trim-lora-v1)" && has py.log "datasets/mecharm_redblock_box_teach_v1_trim2/raw_episodes "' +check "trim の安全範囲 (run_policy_client.py --raw-dir) は trim2 のまま・deep_even10_v1 は使わない (依頼 135)" 'has py.log "run_policy_client.py --raw-dir $T/data/datasets/mecharm_redblock_box_teach_v1_trim2/raw_episodes --steps-per-chunk" && ! has py.log "deep_even10_v1" && has out.txt "[RANGE ] 安全範囲 (学習データ ±10°) は $T/data/datasets/mecharm_redblock_box_teach_v1_trim2/raw_episodes から (--model trim)"' check "記録 lab_pod.json に model=trim" '[ "$(jq -r .model "$T/data/lab/LogT/pod/lab_pod.json")" = trim ]' check "lab_start_*.txt にモデルが残る" 'grep -qF "[MODEL ] --model trim" "$T/data/lab/LogT"/lab_start_*.txt && grep -qF "モデル trim (PixelstoMotors/pi05-mecharm-trim-lora-v1)" "$T/data/lab/LogT"/lab_start_*.txt' rm -rf "$T" @@ -416,23 +418,43 @@ echo "== 34d. --model deep1 (依頼 133) で新しく作る → π0.5 base か setup; run --model deep1 check "終了コード 0" '[ $RC = 0 ]' check "deep1 のモデルを /workspace/ckpt/deep_v1_2999 に取る (ほかのモデルは取らない)" 'has ssh.log "hf download PixelstoMotors/pi05-mecharm-deep-even10-lora-v1 --local-dir /workspace/ckpt/deep_v1_2999" && ! has ssh.log "hf download PixelstoMotors/pi05-mecharm-trim-lora-v1 " && ! has ssh.log "hf download PixelstoMotors/pi05-mecharm-trim-corr-lora-v"' -check "3f で学習と同じ deep の設定 (3000 ステップ) を足し, モデルの assets (測り直した正規化の値) を写す" 'has ssh.log "add_deep_config.py /workspace/openpi --steps 3000" && ! has ssh.log "add_trim_corr_config.py" && has ssh.log "mkdir -p assets/pi05_mecharm_deep_even10_lora/local && cp -r /workspace/ckpt/deep_v1_2999/assets/local/mecharm_redblock_box_deep_even10_v1 assets/pi05_mecharm_deep_even10_lora/local/"' +check "3f で学習と同じ deep の設定 (3000 ステップ・--variant v1 = decay_steps 5000) を足し, モデルの assets (測り直した正規化の値) を写す" 'has ssh.log "add_deep_config.py /workspace/openpi --steps 3000 --variant v1 &&" && ! has ssh.log "add_trim_corr_config.py" && has ssh.log "mkdir -p assets/pi05_mecharm_deep_even10_lora/local && cp -r /workspace/ckpt/deep_v1_2999/assets/local/mecharm_redblock_box_deep_even10_v1 assets/pi05_mecharm_deep_even10_lora/local/"' check "deep1 の設定とフォルダでサーバーを起動" 'has ssh.log "--policy.config=pi05_mecharm_deep_even10_lora --policy.dir=/workspace/ckpt/deep_v1_2999 2>&1"' check "クライアントに --model deep1 を渡す" 'has py.log "--model deep1 (PixelstoMotors/pi05-mecharm-deep-even10-lora-v1)"' +check "deep1 の安全範囲 (run_policy_client.py --raw-dir) は deep_even10_v1 の raw_episodes・trim2 は使わない (依頼 135)" 'has py.log "run_policy_client.py --raw-dir $T/data/datasets/mecharm_redblock_box_deep_even10_v1/raw_episodes --steps-per-chunk" && ! has py.log "teach_v1_trim2" && has out.txt "[RANGE ] 安全範囲 (学習データ ±10°) は $T/data/datasets/mecharm_redblock_box_deep_even10_v1/raw_episodes から (--model deep1)"' check "記録に model=deep1" '[ "$(jq -r .model "$T/data/lab/LogT/pod/lab_pod.json")" = deep1 ]' rm -rf "$T" +setup; rmdir "$T/data/datasets/mecharm_redblock_box_deep_even10_v1/raw_episodes"; run --model deep1 +check "deep1 で deep_even10_v1 の raw_episodes がない → API を呼ばずに止まる (依頼 135)" '[ $RC = 1 ] && has out.txt "安全範囲を作る学習データがありません: $T/data/datasets/mecharm_redblock_box_deep_even10_v1/raw_episodes" && [ ! -s "$T/calls.log" ]' +rm -rf "$T" +setup; rmdir "$T/data/datasets/mecharm_redblock_box_deep_even10_v1/raw_episodes"; run --model corr3 +check "corr3 は deep_even10_v1 がなくても trim2 で進む (依頼 135)" '[ $RC = 0 ] && has py.log "run_policy_client.py --raw-dir $T/data/datasets/mecharm_redblock_box_teach_v1_trim2/raw_episodes --steps-per-chunk"' +rm -rf "$T" setup; state $ID_OLD; pod 200 "$RUNNING"; probe 1 1 1 1 1; serve_dir /workspace/ckpt/corr_v3_1499; run --model deep1 check "corr3 のサーバーが動いている Pod に --model deep1 → 止まり, --model corr3 を案内" '[ $RC = 1 ] && has out.txt "別のモデル (--model corr3)で動いています" && ! has ssh.log "tmux new" && ! has ssh.log "pkill"' rm -rf "$T" check "フォルダからモデルの名前 (deep1)" '[ "$(. "$HERE/lab_lib.sh"; lab_model_of_dir /workspace/ckpt/deep_v1_2999)" = deep1 ]' +echo "== 34e. --model deep1b (依頼 135) → decay_steps をステップ数と同じにして学習したモデルを取り, b の設定・assets を足して起動" +setup; run --model deep1b +check "終了コード 0" '[ $RC = 0 ]' +check "deep1b のモデルを /workspace/ckpt/deep_b_v1_2999 に取る (deep1 のモデルは取らない)" 'has ssh.log "hf download PixelstoMotors/pi05-mecharm-deep-even10-b-lora-v1 --local-dir /workspace/ckpt/deep_b_v1_2999" && ! has ssh.log "hf download PixelstoMotors/pi05-mecharm-deep-even10-lora-v1 "' +check "3f で add_deep_config.py --steps 3000 --variant b を足し, モデルの assets を pi05_mecharm_deep_even10_b_lora に写す" 'has ssh.log "add_deep_config.py /workspace/openpi --steps 3000 --variant b &&" && ! has ssh.log "--variant v1" && has ssh.log "mkdir -p assets/pi05_mecharm_deep_even10_b_lora/local && cp -r /workspace/ckpt/deep_b_v1_2999/assets/local/mecharm_redblock_box_deep_even10_v1 assets/pi05_mecharm_deep_even10_b_lora/local/"' +check "deep1b の設定とフォルダでサーバーを起動" 'has ssh.log "--policy.config=pi05_mecharm_deep_even10_b_lora --policy.dir=/workspace/ckpt/deep_b_v1_2999 2>&1"' +check "deep1b の安全範囲も deep_even10_v1 の raw_episodes" 'has py.log "--model deep1b (PixelstoMotors/pi05-mecharm-deep-even10-b-lora-v1)" && has py.log "run_policy_client.py --raw-dir $T/data/datasets/mecharm_redblock_box_deep_even10_v1/raw_episodes --steps-per-chunk"' +rm -rf "$T" +setup; state $ID_OLD; pod 200 "$RUNNING"; probe 1 1 1 1 1; serve_dir /workspace/ckpt/deep_v1_2999; run --model deep1b +check "deep1 のサーバーが動いている Pod に --model deep1b → 止まり, --model deep1 を案内" '[ $RC = 1 ] && has out.txt "別のモデル (--model deep1)で動いています" && ! has ssh.log "tmux new" && ! has ssh.log "pkill"' +rm -rf "$T" +check "フォルダからモデルの名前 (deep1b)" '[ "$(. "$HERE/lab_lib.sh"; lab_model_of_dir /workspace/ckpt/deep_b_v1_2999)" = deep1b ]' + echo "== 35. --model に知らない名前 → 何もせずに止まる" setup; run --model foo -check "終了コード 2・使える名前を出す・API を呼ばない" '[ $RC = 2 ] && has out.txt "--model は trim か corr か corr2 か corr3 か deep1" && [ ! -s "$T/calls.log" ]' +check "終了コード 2・使える名前を出す・API を呼ばない" '[ $RC = 2 ] && has out.txt "--model は trim か corr か corr2 か corr3 か deep1 か deep1b" && [ ! -s "$T/calls.log" ]' rm -rf "$T" setup (cd "$T" && PATH="$T/bin:$PATH" perl -e 'alarm 10; exec @ARGV' ./lab_start.sh --log LogT --dry-run --model < /dev/null > "$T/out.txt" 2>&1); RC=$? -check "--model だけで値がない → 回り続けずに終了コード 2 (codex R1)" '[ $RC = 2 ] && has out.txt "--model には trim か corr か corr2 か corr3 か deep1 を付ける"' +check "--model だけで値がない → 回り続けずに終了コード 2 (codex R1)" '[ $RC = 2 ] && has out.txt "--model には trim か corr か corr2 か corr3 か deep1 か deep1b を付ける"' rm -rf "$T" for a in "--gpu" "--episode" "--log" "--gpu --dry-run" "--episode --log LogT" "--log --dry-run"; do setup diff --git a/test_lab_train.sh b/test_lab_train.sh index e0de0fc..f368a6c 100755 --- a/test_lab_train.sh +++ b/test_lab_train.sh @@ -237,48 +237,66 @@ TD="$T/data/lab/LogT/train" check "--stop (--model なし): 記録の model=corr3 の設定・exp でログと一覧を取る" '[ $(n_call DELETE) = 1 ] && [ -f "$TD/train_trim_corr_v5.log" ] && has ssh.log "cd /workspace/openpi/checkpoints/pi05_mecharm_trim_corr_v5_lora/trim_corr_v5" && [ "$(jq -r .config.data_name "$TD/train_provenance.json")" = mecharm_redblock_box_teach_v1_trim2_corr_v5 ]' rm -rf "$T" -echo "== 12d. --model deep1 (依頼 133): π0.5 base から deep_even10_v1 で新しく学習する (正規化の値は Pod で測り直す)" -run success --model deep1 --limit-hours 7 -CKD="$T/data/checkpoints/pi05_mecharm_deep_even10_lora_v1" +echo "== 12d. --model deep1b (依頼 135. 依頼 133 の deep1 と同じ流れで decay_steps だけステップ数と同じ): π0.5 base から deep_even10_v1 で新しく学習する (正規化の値は Pod で測り直す)" +run success --model deep1b --limit-hours 7 +CKD="$T/data/checkpoints/pi05_mecharm_deep_even10_b_lora_v1" NSHA=c1448b9e612668eedc320cc285053fcee62dc9afb40e435c70948441d486b8e2 -check "終了コード 0・画面に --model deep1 (π0.5 base → 上げる先)・steps 3000 (既定)・上限 7 時間" '[ $RC = 0 ] && has out.txt "--model deep1 (π0.5 base → PixelstoMotors/pi05-mecharm-deep-even10-lora-v1)" && has out.txt "steps=3000 init=base 上限 7 時間"' +check "終了コード 0・画面に --model deep1b (π0.5 base → 上げる先)・steps 3000 (既定)・上限 7 時間・decay_steps 3000" '[ $RC = 0 ] && has out.txt "--model deep1b (π0.5 base → PixelstoMotors/pi05-mecharm-deep-even10-b-lora-v1)" && has out.txt "steps=3000 init=base 上限 7 時間" && has out.txt "decay_steps 3000・decay_lr 5e-6 (add_deep_config.py --variant b)"' check "3c は続きの元を取らない (hf download を Pod で流さない)" '! has ssh.log "hf download" && has out.txt "3c. 続きの元は取らない (π0.5 base から"' check "3f は addon と trim2 の設定だけ (3999 の assets を写さない)" 'has ssh.log "add_trim_config.py /workspace/openpi" && ! has ssh.log "cp -r /workspace/ckpt/"' -check "deep_even10_v1 の tar だけを送り, pod_prep_deep.sh 3000 (pod_prep_trim_corr.sh は流さない)" 'has scp.log "mecharm_redblock_box_deep_even10_v1_raw_episodes.tar root@" && ! has scp.log "corr_v" && has ssh.log "bash /workspace/r2r2r/pi05/openpi_addon/pod_prep_deep.sh 3000 2>&1 | tee /workspace/logs/prep_deep.log" && ! has ssh.log "pod_prep_trim_corr.sh"' -check "測り直した正規化の値を学習の前に持ち帰る (lab//train/norm_stats.json・sha256 が Pod と同じ)" '[ "$(shasum -a 256 "$TD/norm_stats.json" | cut -d" " -f1)" = $NSHA ] && [ $(line_of out.txt "[NORM ] 正規化の値") -lt $(line_of out.txt "=== 5. 学習を始める") ] && [ "$(jq -r .norm_sha256 "$TD/lab_pod.json")" = $NSHA ]' -check "学習は pi05_mecharm_deep_even10_lora --exp-name=deep_even10_v1" 'has ssh.log "uv run scripts/train.py pi05_mecharm_deep_even10_lora --exp-name=deep_even10_v1 --overwrite > /workspace/logs/train_deep_even10_v1.log"' -check "最後は 2999 を Hugging Face の pi05-mecharm-deep-even10-lora-v1 へ上げ, KIOXIA の checkpoints/pi05_mecharm_deep_even10_lora_v1/2999 に assets ごと置く" 'has ssh.log "hf_upload_ckpt.py --dir /workspace/openpi/checkpoints/pi05_mecharm_deep_even10_lora/deep_even10_v1/2999 --repo PixelstoMotors/pi05-mecharm-deep-even10-lora-v1" && [ -f "$CKD/2999/_CHECKPOINT_METADATA" ] && [ -f "$CKD/2999/assets/local/mecharm_redblock_box_deep_even10_v1/norm_stats.json" ] && has out.txt "assets の正規化の値は測り直したものと sha256 が同じ"' -check "記録 lab_pod.json に model・設定・exp・init・steps" '[ "$(jq -r "[.model,.config,.exp,.init,.steps] | join(\" \")" "$TD/lab_pod.json")" = "deep1 pi05_mecharm_deep_even10_lora deep_even10_v1 base 3000" ]' +check "deep_even10_v1 の tar だけを送り, pod_prep_deep.sh 3000 b (pod_prep_trim_corr.sh は流さない)" 'has scp.log "mecharm_redblock_box_deep_even10_v1_raw_episodes.tar root@" && ! has scp.log "corr_v" && has ssh.log "bash /workspace/r2r2r/pi05/openpi_addon/pod_prep_deep.sh 3000 b 2>&1 | tee /workspace/logs/prep_deep.log" && ! has ssh.log "pod_prep_trim_corr.sh"' +check "測り直した正規化の値を学習の前に持ち帰る (lab//train/norm_stats.json・sha256 が Pod と同じ)" '[ "$(shasum -a 256 "$TD/norm_stats.json" | cut -d" " -f1)" = $NSHA ] && [ $(line_of out.txt "[NORM ] 正規化の値") -lt $(line_of out.txt "=== 5. 学習を始める") ] && [ "$(jq -r .norm_sha256 "$TD/lab_pod.json")" = $NSHA ] && [ "$(jq -r .norm_pod "$TD/lab_pod.json")" = assets/pi05_mecharm_deep_even10_b_lora/local/mecharm_redblock_box_deep_even10_v1/norm_stats.json ]' +check "学習は pi05_mecharm_deep_even10_b_lora --exp-name=deep_even10_v1_b (deep1 の設定・exp は使わない)" 'has ssh.log "uv run scripts/train.py pi05_mecharm_deep_even10_b_lora --exp-name=deep_even10_v1_b --overwrite > /workspace/logs/train_deep_even10_v1_b.log" && ! has ssh.log "train.py pi05_mecharm_deep_even10_lora "' +check "最後は 2999 を Hugging Face の pi05-mecharm-deep-even10-b-lora-v1 へ上げ, KIOXIA の checkpoints/pi05_mecharm_deep_even10_b_lora_v1/2999 に assets ごと置く (deep1 の置き場所には置かない)" 'has ssh.log "hf_upload_ckpt.py --dir /workspace/openpi/checkpoints/pi05_mecharm_deep_even10_b_lora/deep_even10_v1_b/2999 --repo PixelstoMotors/pi05-mecharm-deep-even10-b-lora-v1" && [ -f "$CKD/2999/_CHECKPOINT_METADATA" ] && [ -f "$CKD/2999/assets/local/mecharm_redblock_box_deep_even10_v1/norm_stats.json" ] && has out.txt "assets の正規化の値は測り直したものと sha256 が同じ" && [ ! -e "$T/data/checkpoints/pi05_mecharm_deep_even10_lora_v1" ]' +check "記録 lab_pod.json に model・設定・exp・init・steps" '[ "$(jq -r "[.model,.config,.exp,.init,.steps] | join(\" \")" "$TD/lab_pod.json")" = "deep1b pi05_mecharm_deep_even10_b_lora deep_even10_v1_b base 3000" ]' PD="$TD/train_provenance.json" check "由来: base から (pi05_base・gs://)・データの tar の sha256・正規化の値の sha256" '[ "$(jq -r ".inputs[1] | [.kind,.id,.url,.path] | join(\" \")" "$PD")" = "checkpoint pi05_base gs://openpi-assets/checkpoints/pi05_base/params " ] && [ "$(jq -r ".config.init_params" "$PD")" = base ] && [ "$(jq -r ".inputs[0].tar | [.path,.sha256] | join(\" \")" "$PD")" = "datasets/mecharm_redblock_box_deep_even10_v1/mecharm_redblock_box_deep_even10_v1_raw_episodes.tar $(shasum -a 256 "$T/data/datasets/mecharm_redblock_box_deep_even10_v1/mecharm_redblock_box_deep_even10_v1_raw_episodes.tar" | cut -d" " -f1)" ] && [ "$(jq -r ".config.norm_stats.sha256" "$PD")" = $NSHA ]' -check "由来: 出力・チェックポイント側と Hugging Face にも同じもの" '[ "$(jq -r ".outputs[0] | [.path,.hf_repo] | join(\" \")" "$PD")" = "checkpoints/pi05_mecharm_deep_even10_lora_v1/2999 PixelstoMotors/pi05-mecharm-deep-even10-lora-v1" ] && cmp -s "$PD" "$CKD/2999/train_provenance.json" && has calls.log "hf upload PixelstoMotors/pi05-mecharm-deep-even10-lora-v1 train_provenance.json"' +check "由来 (依頼 135): 設定の名前と学習率の予定 (decay_steps = num_train_steps = 3000・warmup 300・peak 5e-5・decay_lr 5e-6・--variant b)" '[ "$(jq -r ".config | [.name, .num_train_steps, .lr_schedule.decay_steps, .lr_schedule.warmup_steps, .lr_schedule.peak_lr, .lr_schedule.decay_lr, .lr_schedule.source] | join(\" \")" "$PD")" = "pi05_mecharm_deep_even10_b_lora 3000 3000 300 0.00005 0.000005 add_deep_config.py --variant b" ]' +check "由来: 出力・チェックポイント側と Hugging Face にも同じもの" '[ "$(jq -r ".outputs[0] | [.path,.hf_repo] | join(\" \")" "$PD")" = "checkpoints/pi05_mecharm_deep_even10_b_lora_v1/2999 PixelstoMotors/pi05-mecharm-deep-even10-b-lora-v1" ] && cmp -s "$PD" "$CKD/2999/train_provenance.json" && has calls.log "hf upload PixelstoMotors/pi05-mecharm-deep-even10-b-lora-v1 train_provenance.json"' check "続きの元のチェックポイント (3999 など) を偽物でも置かない・鍵を出さない" '[ ! -e "$T/data/checkpoints/pi05_mecharm_trim_lora_v1" ] && no_secret' check "見張りは上限 7 時間 + 60 分" '[ "$(jq -r ".watchdog_epoch - .created_epoch" "$TD/lab_pod.json")" = 28800 ]' rm -rf "$T" -run norm_missing --model deep1 -check "deep1: 正規化の値の sha256 が返らない → 学習を始めずログを持ち帰って消す" '[ $RC = 1 ] && has notify.log "正規化の値の sha256 が返らない" && ! has ssh.log "lt=start_train" && [ $(n_call DELETE) = 1 ] && [ -f "$TD/prep_deep.log" ]' +run success --model deep1b --steps 2000 +check "deep1b --steps 2000 → pod_prep_deep.sh 2000 b・由来の decay_steps も 2000 (ステップ数に合わせる. 依頼 135)" '[ $RC = 0 ] && has ssh.log "pod_prep_deep.sh 2000 b 2>&1" && [ "$(jq -r ".config | [.num_train_steps, .lr_schedule.decay_steps] | join(\" \")" "$TD/train_provenance.json")" = "2000 2000" ] && [ -f "$T/data/checkpoints/pi05_mecharm_deep_even10_b_lora_v1/1999/_CHECKPOINT_METADATA" ]' rm -rf "$T" -run norm_fetch_fail --model deep1 -check "deep1: 4b で正規化の値を持ち帰れなくても学習は続け, KIOXIA のチェックポイントの assets から持ち帰る (codex R1)" '[ $RC = 0 ] && has out.txt "正規化の値を持ち帰れなかった" && has out.txt "正規化の値をチェックポイントの assets から持ち帰った" && [ "$(shasum -a 256 "$TD/norm_stats.json" | cut -d" " -f1)" = c1448b9e612668eedc320cc285053fcee62dc9afb40e435c70948441d486b8e2 ] && [ "$(jq -r ".config.norm_stats.lab_copy" "$TD/train_provenance.json")" = "$TD/norm_stats.json" ] && [ "$(jq -r .norm_local "$TD/lab_pod.json")" = "$TD/norm_stats.json" ]' +run success --model deep1b --steps 300 +check "deep1b --steps 300 → decay_steps が warmup 300 以下なので Pod を借りずに終了コード 2" '[ $RC = 2 ] && has out.txt "decay_steps (300) が warmup 300 以下" && [ ! -s "$T/calls.log" ]' rm -rf "$T" -run norm_bad --model deep1 -check "deep1: 正規化の値の中身が Pod の sha256 と違う → 学習のあと KIOXIA に置けたとせず失敗で通知 (codex R1)" '[ $RC = 1 ] && [ ! -e "$TD/norm_stats.json" ] && has notify.log "assets の正規化の値が測り直したもの" && has notify.log "測り直した正規化の値を lab/ に持ち帰れなかった" && [ $(n_call DELETE) = 1 ]' -rm -rf "$T" -PRE='D="$T/data/checkpoints/pi05_mecharm_deep_even10_lora_v1/2999"; while IFS=$(printf "\t") read -r f n; do mkdir -p "$D/$(dirname "$f")"; head -c "$n" /dev/zero > "$D/$f"; done <<< "$(printf "_CHECKPOINT_METADATA\t426\nassets/local/mecharm_redblock_box_deep_even10_v1/norm_stats.json\t2120\nparams/_METADATA\t33052\nparams/d/0123abcd\t1048576\nparams/manifest.ocdbt\t118\n")"; head -c 2120 /dev/zero | tr "\0" x > "$D/assets/local/mecharm_redblock_box_deep_even10_v1/norm_stats.json"' run success --model deep1 +check "deep1 (依頼 133 の decay_steps 5000) は新しくは学習しない → 終了コード 2・deep1b を案内・API を呼ばない (依頼 135)" '[ $RC = 2 ] && has out.txt "--model deep1 は学習し終えたモデル (decay_steps 5000) で, 新しくは学習しません" && has out.txt "--model deep1b" && [ ! -s "$T/calls.log" ]' +rm -rf "$T" +run norm_missing --model deep1b +check "deep1b: 正規化の値の sha256 が返らない → 学習を始めずログを持ち帰って消す" '[ $RC = 1 ] && has notify.log "正規化の値の sha256 が返らない" && ! has ssh.log "lt=start_train" && [ $(n_call DELETE) = 1 ] && [ -f "$TD/prep_deep.log" ]' +rm -rf "$T" +run norm_fetch_fail --model deep1b +check "deep1b: 4b で正規化の値を持ち帰れなくても学習は続け, KIOXIA のチェックポイントの assets から持ち帰る (codex R1)" '[ $RC = 0 ] && has out.txt "正規化の値を持ち帰れなかった" && has out.txt "正規化の値をチェックポイントの assets から持ち帰った" && [ "$(shasum -a 256 "$TD/norm_stats.json" | cut -d" " -f1)" = c1448b9e612668eedc320cc285053fcee62dc9afb40e435c70948441d486b8e2 ] && [ "$(jq -r ".config.norm_stats.lab_copy" "$TD/train_provenance.json")" = "$TD/norm_stats.json" ] && [ "$(jq -r .norm_local "$TD/lab_pod.json")" = "$TD/norm_stats.json" ]' +rm -rf "$T" +run norm_bad --model deep1b +check "deep1b: 正規化の値の中身が Pod の sha256 と違う → 学習のあと KIOXIA に置けたとせず失敗で通知 (codex R1)" '[ $RC = 1 ] && [ ! -e "$TD/norm_stats.json" ] && has notify.log "assets の正規化の値が測り直したもの" && has notify.log "測り直した正規化の値を lab/ に持ち帰れなかった" && [ $(n_call DELETE) = 1 ]' +rm -rf "$T" +PRE='D="$T/data/checkpoints/pi05_mecharm_deep_even10_b_lora_v1/2999"; while IFS=$(printf "\t") read -r f n; do mkdir -p "$D/$(dirname "$f")"; head -c "$n" /dev/zero > "$D/$f"; done <<< "$(printf "_CHECKPOINT_METADATA\t426\nassets/local/mecharm_redblock_box_deep_even10_v1/norm_stats.json\t2120\nparams/_METADATA\t33052\nparams/d/0123abcd\t1048576\nparams/manifest.ocdbt\t118\n")"; head -c 2120 /dev/zero | tr "\0" x > "$D/assets/local/mecharm_redblock_box_deep_even10_v1/norm_stats.json"' +run success --model deep1b PRE="" -check "deep1: KIOXIA に 2999 がもとからあり, 名前・バイト数は同じでも正規化の値の中身が違う → 合ったとせず上書きもしない (codex R1)" '[ $RC = 1 ] && has notify.log "名前・バイト数は Pod 側と同じだが assets の正規化の値が測り直したもの" && [ "$(head -c 1 "$T/data/checkpoints/pi05_mecharm_deep_even10_lora_v1/2999/assets/local/mecharm_redblock_box_deep_even10_v1/norm_stats.json")" = x ]' +check "deep1b: KIOXIA に 2999 がもとからあり, 名前・バイト数は同じでも正規化の値の中身が違う → 合ったとせず上書きもしない (codex R1)" '[ $RC = 1 ] && has notify.log "名前・バイト数は Pod 側と同じだが assets の正規化の値が測り直したもの" && [ "$(head -c 1 "$T/data/checkpoints/pi05_mecharm_deep_even10_b_lora_v1/2999/assets/local/mecharm_redblock_box_deep_even10_v1/norm_stats.json")" = x ]' rm -rf "$T" rm -rf "$T" -run success --model deep1 --init /workspace/ckpt/3999/params -check "deep1 に --init で続きの元を渡す → Pod を借りずに止まる" '[ $RC = 1 ] && has out.txt "π0.5 base から学習する (--init /workspace/ckpt/3999/params は使えない)" && [ $(n_call "POST /pods") = 0 ]' +run success --model deep1b --init /workspace/ckpt/3999/params +check "deep1b に --init で続きの元を渡す → Pod を借りずに止まる" '[ $RC = 1 ] && has out.txt "π0.5 base から学習する (--init /workspace/ckpt/3999/params は使えない)" && [ $(n_call "POST /pods") = 0 ]' rm -rf "$T" T=$(mktemp -d); mkdir -p "$T/data/lab/LogT/train" "$T/fake" jq -n '{id:"dryrunpod0001", kind:"train", model:"deep1", config:"pi05_mecharm_deep_even10_lora", gpu:"NVIDIA A40", costPerHr:0.44, created_epoch:1793449178, created_at:"2026-10-31 21:19:38", exp:"deep_even10_v1", steps:3000, init:"base", train_started_epoch:1793449300, norm_sha256:"c1448b9e612668eedc320cc285053fcee62dc9afb40e435c70948441d486b8e2", norm_pod:"assets/pi05_mecharm_deep_even10_lora/local/mecharm_redblock_box_deep_even10_v1/norm_stats.json", norm_local:""}' > "$T/data/lab/LogT/train/lab_pod.json" echo "NVIDIA A40" > "$T/fake/created" (cd "$HERE" && LAB_DRY_ROOT="$T" ./lab_train.sh --log LogT --dry-run --stop < /dev/null > "$T/out.txt" 2>&1); RC=$? TD="$T/data/lab/LogT/train" -check "--stop (--model なし): 記録の model=deep1 の設定・exp・準備のログでログと一覧を取り, 由来に正規化の値の sha256" '[ $(n_call DELETE) = 1 ] && [ -f "$TD/train_deep_even10_v1.log" ] && [ -f "$TD/prep_deep.log" ] && has ssh.log "cd /workspace/openpi/checkpoints/pi05_mecharm_deep_even10_lora/deep_even10_v1" && [ "$(jq -r .config.data_name "$TD/train_provenance.json")" = mecharm_redblock_box_deep_even10_v1 ] && [ "$(jq -r .config.norm_stats.sha256 "$TD/train_provenance.json")" = c1448b9e612668eedc320cc285053fcee62dc9afb40e435c70948441d486b8e2 ]' +check "--stop (--model なし): 記録の model=deep1 (依頼 133 で始めた学習) の設定・exp・準備のログでログと一覧を取り, 由来に正規化の値の sha256" '[ $(n_call DELETE) = 1 ] && [ -f "$TD/train_deep_even10_v1.log" ] && [ -f "$TD/prep_deep.log" ] && has ssh.log "cd /workspace/openpi/checkpoints/pi05_mecharm_deep_even10_lora/deep_even10_v1" && [ "$(jq -r .config.data_name "$TD/train_provenance.json")" = mecharm_redblock_box_deep_even10_v1 ] && [ "$(jq -r .config.norm_stats.sha256 "$TD/train_provenance.json")" = c1448b9e612668eedc320cc285053fcee62dc9afb40e435c70948441d486b8e2 ]' +check "--stop の deep1: 由来の学習率の予定は decay_steps 5000 (--variant v1. 依頼 135)" '[ "$(jq -r ".config | [.name, .num_train_steps, .lr_schedule.decay_steps, .lr_schedule.source] | join(\" \")" "$TD/train_provenance.json")" = "pi05_mecharm_deep_even10_lora 3000 5000 add_deep_config.py --variant v1" ]' +rm -rf "$T" +T=$(mktemp -d); mkdir -p "$T/data/lab/LogT/train" "$T/fake" +jq -n '{id:"dryrunpod0001", kind:"train", model:"corr3", config:"pi05_mecharm_trim_corr_v5_lora", gpu:"NVIDIA A40", costPerHr:0.44, created_epoch:1793449178, created_at:"2026-10-31 21:19:38", exp:"trim_corr_v5", steps:1500, train_started_epoch:1793449300}' > "$T/data/lab/LogT/train/lab_pod.json" +echo "NVIDIA A40" > "$T/fake/created" +(cd "$HERE" && LAB_DRY_ROOT="$T" ./lab_train.sh --log LogT --dry-run --stop < /dev/null > "$T/out.txt" 2>&1); RC=$? +TD="$T/data/lab/LogT/train" +check "corr3 の由来には学習率の予定を足さない (corr〜corr3 は変えない. 依頼 135)" '[ $(n_call DELETE) = 1 ] && [ "$(jq -r ".config | has(\"lr_schedule\")" "$TD/train_provenance.json")" = false ]' rm -rf "$T" echo "== 13. 見守る途中で止められても Pod を残さない (TERM. Ctrl-C・HUP も同じ扱い)" ``` ## 依頼137：写り方の確かめの基準画像を trim2 にそろえた ### 依頼137 の依頼文（依頼箱/済み/20261109_137_deep1の写り方の確かめの基準をtrim2にそろえる.md） ```markdown --- 作業フォルダ: /Users/shimazueiichi/r2r2r-scripts 上限時間: 60 --- # 依頼137：deep1・deep1b の写り方の確かめの基準画像を trim2 にそろえる 実機は動かさない。エイイチは見ていない。Log27。依頼135 の報告の「決めてほしいこと」へのエイイチの答え（2026-11-09）。 ## エイイチの決め - 3：写り方の確かめの基準画像は、位置合わせ（camera_align.py）と同じ trim2 にそろえる。安全範囲は依頼135 のとおり deep_even10_v1 のまま - 1：新しいモデルの名前は deep1b のままでよい - 2：j2 の上限 +52.5° はこのままでよい（初回の実機で目で見る） ## 大事：いま回っている学習を止めない・触らない 依頼133 の本番（lab_train.sh pid 8808・21:45 ごろ学習が終わり、そのあと持ち帰りと Pod の削除）が動いている。Pod・そのプロセス・KIOXIA の lab/Log27/train/ には手を出さない。lab_train.sh・lab_lib.sh・provenance.py は変えない（lab_start.sh・run_policy_client.py 側で足りるはず。足りなければ止めて報告）。 ## やること 1. run_policy_client.py の写り方の確かめの基準画像を、安全範囲の --raw-dir とは別に指定できるようにする（例 --framing-raw-dir。指定がなければ今までどおり --raw-dir から） 2. lab_start.sh：deep1・deep1b では基準画像を trim2 のお手本から取る（安全範囲は deep_even10_v1 のまま）。ほかのモデルは今までどおり。起動時の [RANGE ] の行の近くに、基準画像をどこから取ったかを1行出す 3. テスト（test_lab_start.sh ほか関係するもの）を足して通す：deep1・deep1b で安全範囲は deep・基準画像は trim2・ほかのモデルは今までどおり 4. codex で差分を見てもらう。指摘を直して通るまで 5. 変えたファイルだけを指定して git add・commit（git add . は使わない） ## 報告 claude_code_reports/20261109_137.txt に：変えた所、テストの結果、codex の回数と指摘、commit の番号、いまの学習に影響がなかったことの確かめ（lab_train_*.txt の最後の行）、決めてほしいこと。 **報告の最後の1行は必ず次のどれか**：「判定：合格」／「判定：止めた（決めてほしいことあり）」／「判定：失敗」 ``` ### 依頼137 の報告（r2r2r-scripts/claude_code_reports/20261109_137.txt） ```text 依頼137 報告：deep1・deep1b の写り方の確かめの基準画像を trim2 にそろえる (2026-11-09) 実機は動かしていない。Pod・lab_train.sh pid 8808・KIOXIA の lab/Log27/train/ には触っていない。 lab_train.sh・lab_lib.sh・provenance.py は変えていない (lab_start.sh と run_policy_client.py 側で足りた)。 ■ 1. 変えた所 - pi05/run_policy_client.py --framing-raw-dir を足した。写り方の確かめの基準画像 (load_reference. 最初のエピソードの 1 枚目) はここから取る。 指定がなければ今までどおり --raw-dir から。安全範囲 (load_envelope) は --raw-dir のまま。 log.jsonl の頭 (type=model) に framing_raw_dir を残す (基準画像のパスは今までどおり type=framing の reference にも残る)。 - lab_start.sh run_policy_client.py に --framing-raw-dir "$RAW" (位置合わせと同じ trim2 の raw_episodes) を渡す (--execute・--dry-keys の両方)。 deep1・deep1b：安全範囲 deep_even10_v1・基準画像 trim2。 trim・corr〜corr3：安全範囲・基準画像とも trim2 (今まで --raw-dir から取っていた基準画像と同じフォルダ。動きは変わらない。 引数の見た目に --framing-raw-dir が 1 つ増えるだけ)。 起動時の [RANGE ] の行の次に「[FRAME ] 写り方の確かめの基準画像は の最初のエピソードから (位置合わせと同じ trim2. 依頼 137)」を出す。 頭の説明の文も合わせた。 - test_lab_start.sh：trim・corr3・deep1・deep1b で、--framing-raw-dir が trim2・[FRAME ] の行が出る・deep1/deep1b の --raw-dir は deep のまま (trim2 を --raw-dir に渡さない・deep を --framing-raw-dir に渡さない) を確かめる行を足した。 deep1 の前からの行「! has py.log teach_v1_trim2」は、基準画像で trim2 が出るようになったので「--raw-dir に trim2 を渡さない」に直した。 - pi05/test_framing_raw_dir.py (新規)：main を load_reference まで流し、指定ありなら基準画像は --framing-raw-dir・安全範囲は --raw-dir、 指定なしなら両方 --raw-dir、log.jsonl の頭に framing_raw_dir が残ることを確かめる (アーム・カメラ・サーバーには触らない)。 - pi05/test_rewind_correct.py：git HEAD の版と比べるテストで、新しく増えた framing_raw_dir を比べる対象から外した (項目が増えるだけを許す既存の仕組みに 1 つ足した)。 ■ 2. テストの結果 - test_lab_start.sh：OK 176 / NG 0 (前回 172 → 4 行増) - pi05/test_framing_raw_dir.py：2 件 OK - pi05/test_rewind_correct.py：69 件 OK (直す前は test_unchanged_without_b_c が framing_raw_dir の増加で 1 件落ちた → 上のとおり直して通った) - run_policy_client を読み込むほかのテスト：test_pad_correction 4・test_pad_build 3・test_recorder_rpc 4・test_recorder_currents 15・test_teach_currents すべて OK ■ 3. codex 1 回。指摘なし (安全範囲は --raw-dir、基準画像だけ --framing-raw-dir、未指定時の互換性、deep1・deep1b とほかのモデルの設定、[FRAME ] の表示、 変更禁止の 3 ファイルが差分にないこと、テストに重大な抜けがないことを確認したとの答え)。 ■ 4. commit 3310ac0 (変えた 5 ファイルだけを指定して add) ■ 5. いまの学習に影響がなかったことの確かめ - 変えたのは lab_start.sh・run_policy_client.py とテストだけ。lab_train.sh・lab_lib.sh・provenance.py・pi05/openpi_addon は触っていない。 - pid 8808 (./lab_train.sh --log Log27 --model deep1 --steps 3000 --limit-hours 7) は動いている。 lab_train_20261109_153622.txt の最後の行： [TRAIN ] 21:17 step 2780/3000 loss 0.0041 (Step 2750 の行) 経過 5:33 上限まで 1:18 ■ 決めてほしいこと なし 判定：合格 ``` ### 依頼137 の commit（3310ac0）の説明と変えたファイル ```text commit 3310ac0aed7fe95dcb6f94b67d217b97ecb04541 Date: Mon Nov 9 21:20:30 2026 +0900 実機推論の写り方の確かめの基準画像を安全範囲の --raw-dir と別に指定できる run_policy_client.py --framing-raw-dir を足す (未指定なら今までどおり --raw-dir. log.jsonl の頭に framing_raw_dir を残す). lab_start.sh はどのモデルでも位置合わせ (camera_align.py) と同じ trim2 の raw_episodes を渡し, deep1・deep1b の基準画像も trim2 にそろえる (安全範囲は deep_even10_v1 のまま. trim・corr〜corr3 は今までどおり trim2). 起動時に [FRAME ] で基準画像の取り先を出す. テストを足す (lab_start 176・test_framing_raw_dir 2・test_rewind_correct は新しい項目を比べる対象から外す) (依頼 137. codex 1 回) Co-Authored-By: Claude Opus 5.5 lab_start.sh | 8 ++-- pi05/run_policy_client.py | 11 +++++- pi05/test_framing_raw_dir.py | 91 ++++++++++++++++++++++++++++++++++++++++++++ pi05/test_rewind_correct.py | 2 +- test_lab_start.sh | 6 ++- 5 files changed, 112 insertions(+), 6 deletions(-) ``` ### 依頼137 の commit（3310ac0）の差分の全量 ```diff diff --git a/lab_start.sh b/lab_start.sh index 5fe03d2..8c18032 100755 --- a/lab_start.sh +++ b/lab_start.sh @@ -23,7 +23,8 @@ # 取る場所・設定・assets・サーバーの起動・つなぎ直しの確かめが選んだモデルに合う. 安全範囲 (学習データ ±10°. # run_policy_client.py --raw-dir) は, deep1・deep1b は deep_even10_v1 の raw_episodes (依頼 135), ほかは trim2 の # raw_episodes (corr_v3・corr_v4 と合わせても範囲はほぼ同じで, trim2 の方が狭い. corr_v5 は 11-06 の 184650 の j2 が -# trim2 の上限 +31.4° を最大 11° 超える). 位置合わせ (camera_align.py) のお手本はどれも trim2 の raw_episodes +# trim2 の上限 +31.4° を最大 11° 超える). 位置合わせ (camera_align.py) のお手本と, 実機推論の写り方の確かめの基準画像 +# (run_policy_client.py --framing-raw-dir. 依頼 137) はどれも trim2 の raw_episodes # つなぎ直しで, 残っている Pod のサーバーが別のモデルで動いていれば, そのことを出して止まる # # 1. Pod を作る (A40×1・Runpod Pytorch 2.8.0・コンテナのディスク 200GB・22/tcp を外へ出す) @@ -113,6 +114,7 @@ RAW=/Volumes/KIOXIA/r2r2r-data/datasets/mecharm_redblock_box_teach_v1_trim2/raw_ RAW_RANGE=/Volumes/KIOXIA/r2r2r-data/datasets/$MODEL_RANGE_DATA/raw_episodes # 安全範囲を作る学習データ (lab_lib.sh の lab_model_def. 依頼 135) [ -d "$RAW_RANGE" ] || { echo "[ERROR] 安全範囲を作る学習データがありません: $RAW_RANGE" >&2; exit 1; } echo "[RANGE ] 安全範囲 (学習データ ±10°) は $RAW_RANGE から (--model $MODEL)" +echo "[FRAME ] 写り方の確かめの基準画像は $RAW の最初のエピソードから (位置合わせと同じ trim2. 依頼 137)" lab_ssh_opts HF_REPO=$MODEL_REPO # --model で選んだモデル (lab_lib.sh の lab_model_def) @@ -452,12 +454,12 @@ while :; do stage_done stage "8. 実機推論 ($round 回目)" python3 pi05/run_policy_client.py --raw-dir "$RAW_RANGE" --steps-per-chunk 10 --via-stride 3 \ - --settle-retries 0 --state-from-target --execute --ignore-framing --model "$MODEL_TAG" + --settle-retries 0 --state-from-target --execute --ignore-framing --model "$MODEL_TAG" --framing-raw-dir "$RAW" rc=$? else stage "8. 推論のクライアント (ドライラン・--dry-keys. アームには触らない)" python3 pi05/run_policy_client.py --raw-dir "$RAW_RANGE" --steps-per-chunk 10 --via-stride 3 \ - --settle-retries 0 --state-from-target --dry-keys --ignore-framing --model "$MODEL_TAG" + --settle-retries 0 --state-from-target --dry-keys --ignore-framing --model "$MODEL_TAG" --framing-raw-dir "$RAW" rc=$? fi echo "[CLIENT] run_policy_client.py が終わった (終了コード $rc. モデル $MODEL_TAG)" diff --git a/pi05/run_policy_client.py b/pi05/run_policy_client.py index 23fbd21..8ac31ec 100644 --- a/pi05/run_policy_client.py +++ b/pi05/run_policy_client.py @@ -1082,6 +1082,11 @@ def apply_wait_args(args): f"{GRIP_OPEN_DAEMON_WAIT:.1f}s) / 止まるのを待つ目安 {SETTLE_WAIT_SEC:.1f}s") +def framing_raw_dir(args) -> Path: + """写り方の確かめの基準画像を取るお手本. --framing-raw-dir がなければ --raw-dir (依頼 137).""" + return Path(args.framing_raw_dir or args.raw_dir) + + def main(): ap = argparse.ArgumentParser() ap.add_argument("--host", default="localhost") @@ -1093,6 +1098,9 @@ def main(): ap.add_argument("--max-move-deg", type=float, default=20.0, help="1 回の移動で各関節が動ける上限") ap.add_argument("--raw-dir", required=True, help="安全範囲の算出に使う学習エピソード (必須. 学習に使った raw_episodes を指定する)") + ap.add_argument("--framing-raw-dir", default=None, + help="写り方の確かめの基準画像を取るお手本 (最初のエピソードの 1 枚目). " + "未指定なら --raw-dir から (依頼 137. lab_start.sh は位置合わせと同じ trim2 を渡す)") ap.add_argument("--prompt", default=TASK) ap.add_argument("--model", default=None, help="サーバーが使っているモデルの名前 (記録に残すだけ. lab_start.sh が --model に合わせて渡す. 依頼 52)") @@ -1147,6 +1155,7 @@ def main(): print(f"[MODEL ] {args.model or '(指定なし)'}") # どのモデルで推論したかを log.jsonl の頭に残す ({run_stem}.json には args.model で残る. 依頼 52) log_f.write(json.dumps({"type": "model", "model": args.model, "raw_dir": str(args.raw_dir), + "framing_raw_dir": str(framing_raw_dir(args)), "host": args.host, "port": args.port, "t": time.time()}, ensure_ascii=False) + "\n") log_f.flush() @@ -1183,7 +1192,7 @@ def main(): # (082645 は dy=+37.5 のまま実機を動かした. その 1 分前のドライラン 082512 は dy=-7.8). # アームは HOME に置いた状態で, 1 枚目を お手本の 1 枚目 (同じく HOME) と比べる. # **この確認が通るまで daemon にも触らない** (アームには何も送らない). - ref_bgr, ref_path = load_reference(Path(args.raw_dir)) + ref_bgr, ref_path = load_reference(framing_raw_dir(args)) bgr0 = None for _ in range(10): _rgb0, bgr0 = grab_rgb(cap) diff --git a/pi05/test_framing_raw_dir.py b/pi05/test_framing_raw_dir.py new file mode 100644 index 0000000..82121a8 --- /dev/null +++ b/pi05/test_framing_raw_dir.py @@ -0,0 +1,91 @@ +#!/usr/bin/env python3 +"""test_framing_raw_dir.py — run_policy_client.py の写り方の確かめの基準画像の取り先 (依頼 137). + + - --framing-raw-dir を付けると, 基準画像 (load_reference) はそこから取り, 安全範囲 (load_envelope) は --raw-dir のまま + - 付けなければ今までどおり --raw-dir から取る + - log.jsonl の頭 (type=model) に framing_raw_dir が残る + +アーム・カメラ・daemon・サーバーには触らない. load_reference を偽物にして, そこで止める. + + python3 pi05/test_framing_raw_dir.py +""" +import json +import sys +import tempfile +import unittest +from pathlib import Path +from unittest import mock + +import numpy as np + +_HERE = Path(__file__).resolve().parent +sys.path.insert(0, str(_HERE)) + +import run_policy_client as rpc # noqa: E402 + + +class _Stop(Exception): + pass + + +class _FakeCap: + def isOpened(self): + return True + + def read(self): + return True, np.zeros((4, 4, 3), np.uint8) + + def release(self): + pass + + +def _run(argv): + """main を load_reference まで流す. 戻り値: (load_envelope の引数, load_reference の引数, log.jsonl の頭).""" + seen = {} + + def fake_envelope(p): + seen["envelope"] = Path(p) + return np.full(6, -10.0), np.full(6, 10.0), 1 + + def fake_reference(p, *a, **k): + seen["reference"] = Path(p) + raise _Stop() + + with tempfile.TemporaryDirectory() as td: + td = Path(td) + with mock.patch.object(rpc, "DATA_ROOT", td), \ + mock.patch.object(rpc, "LOG_ROOT", td / "logs"), \ + mock.patch.object(rpc, "load_envelope", fake_envelope), \ + mock.patch.object(rpc, "load_reference", fake_reference), \ + mock.patch.object(rpc.camera_resolver, "resolve_cameras", lambda **k: {"top": 0}), \ + mock.patch.object(rpc.cv2, "VideoCapture", lambda *a, **k: _FakeCap()), \ + mock.patch.object(sys, "argv", ["run_policy_client.py"] + argv), \ + mock.patch("builtins.print"): + try: + rpc.main() + except _Stop: + pass + logs = sorted((td / "logs").glob("pi05_*/log.jsonl")) + head = json.loads(logs[0].read_text().splitlines()[0]) + return seen["envelope"], seen["reference"], head + + +class TestFramingRawDir(unittest.TestCase): + def test_framing_raw_dir_given(self): + env, ref, head = _run(["--raw-dir", "/x/deep_even10_v1/raw_episodes", + "--framing-raw-dir", "/x/trim2/raw_episodes", "--dry-keys", "--model", "deep1"]) + self.assertEqual(env, Path("/x/deep_even10_v1/raw_episodes")) # 安全範囲は --raw-dir のまま + self.assertEqual(ref, Path("/x/trim2/raw_episodes")) # 基準画像は --framing-raw-dir + self.assertEqual(head["type"], "model") + self.assertEqual(head["raw_dir"], "/x/deep_even10_v1/raw_episodes") + self.assertEqual(head["framing_raw_dir"], "/x/trim2/raw_episodes") + + def test_framing_raw_dir_default(self): + env, ref, head = _run(["--raw-dir", "/x/trim2/raw_episodes", "--dry-keys", "--model", "trim"]) + self.assertEqual(env, Path("/x/trim2/raw_episodes")) + self.assertEqual(ref, Path("/x/trim2/raw_episodes")) # 付けなければ今までどおり --raw-dir + self.assertEqual(head["framing_raw_dir"], "/x/trim2/raw_episodes") + + +if __name__ == "__main__": + unittest.main() diff --git a/pi05/test_rewind_correct.py b/pi05/test_rewind_correct.py index 353003c..0df4063 100644 --- a/pi05/test_rewind_correct.py +++ b/pi05/test_rewind_correct.py @@ -597,7 +597,7 @@ class TestClientFlow(unittest.TestCase): self.assertEqual(w_old.sends, w_new.sends, argv) self.assertEqual(w_old.grips, w_new.grips, argv) self.assertEqual(len(l_old), len(l_new)) - new_keys = ("t_exec_start", "t_exec_end", "sent_targets") + new_keys = ("t_exec_start", "t_exec_end", "sent_targets", "framing_raw_dir") # framing_raw_dir は依頼 137 # HEAD が依頼 25 より前なら new_keys は新しい方にだけある. 依頼 25 以降なら両方にある (両方から外す) drop = ("t", "latency_s", "reference") + new_keys so, sn = strip(l_old, drop=drop), strip(l_new, drop=drop) diff --git a/test_lab_start.sh b/test_lab_start.sh index bcb8528..c98ed67 100755 --- a/test_lab_start.sh +++ b/test_lab_start.sh @@ -342,6 +342,7 @@ check "3999 を取り, trim の設定でサーバーを起動" 'has ssh.log "hf check "3f は今までと同じ (corr の設定は足さない)" 'has ssh.log "$(. "$HERE/lab_lib.sh"; lab_cmd_addon)" && ! has ssh.log "add_trim_corr_config.py"' check "クライアントに --model trim を渡す・安全範囲は trim2" 'has py.log "--model trim (PixelstoMotors/pi05-mecharm-trim-lora-v1)" && has py.log "datasets/mecharm_redblock_box_teach_v1_trim2/raw_episodes "' check "trim の安全範囲 (run_policy_client.py --raw-dir) は trim2 のまま・deep_even10_v1 は使わない (依頼 135)" 'has py.log "run_policy_client.py --raw-dir $T/data/datasets/mecharm_redblock_box_teach_v1_trim2/raw_episodes --steps-per-chunk" && ! has py.log "deep_even10_v1" && has out.txt "[RANGE ] 安全範囲 (学習データ ±10°) は $T/data/datasets/mecharm_redblock_box_teach_v1_trim2/raw_episodes から (--model trim)"' +check "trim の写り方の基準画像 (--framing-raw-dir) も今までどおり trim2 (依頼 137)" 'has py.log "--ignore-framing --model trim (PixelstoMotors/pi05-mecharm-trim-lora-v1) --framing-raw-dir $T/data/datasets/mecharm_redblock_box_teach_v1_trim2/raw_episodes" && has out.txt "[FRAME ] 写り方の確かめの基準画像は $T/data/datasets/mecharm_redblock_box_teach_v1_trim2/raw_episodes の最初のエピソードから"' check "記録 lab_pod.json に model=trim" '[ "$(jq -r .model "$T/data/lab/LogT/pod/lab_pod.json")" = trim ]' check "lab_start_*.txt にモデルが残る" 'grep -qF "[MODEL ] --model trim" "$T/data/lab/LogT"/lab_start_*.txt && grep -qF "モデル trim (PixelstoMotors/pi05-mecharm-trim-lora-v1)" "$T/data/lab/LogT"/lab_start_*.txt' rm -rf "$T" @@ -421,7 +422,8 @@ check "deep1 のモデルを /workspace/ckpt/deep_v1_2999 に取る (ほかの check "3f で学習と同じ deep の設定 (3000 ステップ・--variant v1 = decay_steps 5000) を足し, モデルの assets (測り直した正規化の値) を写す" 'has ssh.log "add_deep_config.py /workspace/openpi --steps 3000 --variant v1 &&" && ! has ssh.log "add_trim_corr_config.py" && has ssh.log "mkdir -p assets/pi05_mecharm_deep_even10_lora/local && cp -r /workspace/ckpt/deep_v1_2999/assets/local/mecharm_redblock_box_deep_even10_v1 assets/pi05_mecharm_deep_even10_lora/local/"' check "deep1 の設定とフォルダでサーバーを起動" 'has ssh.log "--policy.config=pi05_mecharm_deep_even10_lora --policy.dir=/workspace/ckpt/deep_v1_2999 2>&1"' check "クライアントに --model deep1 を渡す" 'has py.log "--model deep1 (PixelstoMotors/pi05-mecharm-deep-even10-lora-v1)"' -check "deep1 の安全範囲 (run_policy_client.py --raw-dir) は deep_even10_v1 の raw_episodes・trim2 は使わない (依頼 135)" 'has py.log "run_policy_client.py --raw-dir $T/data/datasets/mecharm_redblock_box_deep_even10_v1/raw_episodes --steps-per-chunk" && ! has py.log "teach_v1_trim2" && has out.txt "[RANGE ] 安全範囲 (学習データ ±10°) は $T/data/datasets/mecharm_redblock_box_deep_even10_v1/raw_episodes から (--model deep1)"' +check "deep1 の安全範囲 (run_policy_client.py --raw-dir) は deep_even10_v1 の raw_episodes・trim2 は使わない (依頼 135)" 'has py.log "run_policy_client.py --raw-dir $T/data/datasets/mecharm_redblock_box_deep_even10_v1/raw_episodes --steps-per-chunk" && ! has py.log "--raw-dir $T/data/datasets/mecharm_redblock_box_teach_v1_trim2/raw_episodes" && has out.txt "[RANGE ] 安全範囲 (学習データ ±10°) は $T/data/datasets/mecharm_redblock_box_deep_even10_v1/raw_episodes から (--model deep1)"' +check "deep1 の写り方の基準画像 (--framing-raw-dir) は trim2・deep_even10_v1 ではない (依頼 137)" 'has py.log "--ignore-framing --model deep1 (PixelstoMotors/pi05-mecharm-deep-even10-lora-v1) --framing-raw-dir $T/data/datasets/mecharm_redblock_box_teach_v1_trim2/raw_episodes" && ! has py.log "--framing-raw-dir $T/data/datasets/mecharm_redblock_box_deep_even10_v1/raw_episodes" && has out.txt "[FRAME ] 写り方の確かめの基準画像は $T/data/datasets/mecharm_redblock_box_teach_v1_trim2/raw_episodes の最初のエピソードから" && has out.txt "[RANGE ] 安全範囲 (学習データ ±10°) は $T/data/datasets/mecharm_redblock_box_deep_even10_v1/raw_episodes から (--model deep1)"' check "記録に model=deep1" '[ "$(jq -r .model "$T/data/lab/LogT/pod/lab_pod.json")" = deep1 ]' rm -rf "$T" setup; rmdir "$T/data/datasets/mecharm_redblock_box_deep_even10_v1/raw_episodes"; run --model deep1 @@ -429,6 +431,7 @@ check "deep1 で deep_even10_v1 の raw_episodes がない → API を呼ばず rm -rf "$T" setup; rmdir "$T/data/datasets/mecharm_redblock_box_deep_even10_v1/raw_episodes"; run --model corr3 check "corr3 は deep_even10_v1 がなくても trim2 で進む (依頼 135)" '[ $RC = 0 ] && has py.log "run_policy_client.py --raw-dir $T/data/datasets/mecharm_redblock_box_teach_v1_trim2/raw_episodes --steps-per-chunk"' +check "corr3 の写り方の基準画像も今までどおり trim2 (依頼 137)" 'has py.log "--model corr3 (PixelstoMotors/pi05-mecharm-trim-corr-lora-v3) --framing-raw-dir $T/data/datasets/mecharm_redblock_box_teach_v1_trim2/raw_episodes" && has out.txt "[FRAME ] 写り方の確かめの基準画像は $T/data/datasets/mecharm_redblock_box_teach_v1_trim2/raw_episodes の最初のエピソードから"' rm -rf "$T" setup; state $ID_OLD; pod 200 "$RUNNING"; probe 1 1 1 1 1; serve_dir /workspace/ckpt/corr_v3_1499; run --model deep1 check "corr3 のサーバーが動いている Pod に --model deep1 → 止まり, --model corr3 を案内" '[ $RC = 1 ] && has out.txt "別のモデル (--model corr3)で動いています" && ! has ssh.log "tmux new" && ! has ssh.log "pkill"' @@ -442,6 +445,7 @@ check "deep1b のモデルを /workspace/ckpt/deep_b_v1_2999 に取る (deep1 check "3f で add_deep_config.py --steps 3000 --variant b を足し, モデルの assets を pi05_mecharm_deep_even10_b_lora に写す" 'has ssh.log "add_deep_config.py /workspace/openpi --steps 3000 --variant b &&" && ! has ssh.log "--variant v1" && has ssh.log "mkdir -p assets/pi05_mecharm_deep_even10_b_lora/local && cp -r /workspace/ckpt/deep_b_v1_2999/assets/local/mecharm_redblock_box_deep_even10_v1 assets/pi05_mecharm_deep_even10_b_lora/local/"' check "deep1b の設定とフォルダでサーバーを起動" 'has ssh.log "--policy.config=pi05_mecharm_deep_even10_b_lora --policy.dir=/workspace/ckpt/deep_b_v1_2999 2>&1"' check "deep1b の安全範囲も deep_even10_v1 の raw_episodes" 'has py.log "--model deep1b (PixelstoMotors/pi05-mecharm-deep-even10-b-lora-v1)" && has py.log "run_policy_client.py --raw-dir $T/data/datasets/mecharm_redblock_box_deep_even10_v1/raw_episodes --steps-per-chunk"' +check "deep1b の写り方の基準画像は trim2・安全範囲は deep_even10_v1 (依頼 137)" 'has py.log "--model deep1b (PixelstoMotors/pi05-mecharm-deep-even10-b-lora-v1) --framing-raw-dir $T/data/datasets/mecharm_redblock_box_teach_v1_trim2/raw_episodes" && ! has py.log "--framing-raw-dir $T/data/datasets/mecharm_redblock_box_deep_even10_v1/raw_episodes" && ! has py.log "--raw-dir $T/data/datasets/mecharm_redblock_box_teach_v1_trim2/raw_episodes" && has out.txt "[FRAME ] 写り方の確かめの基準画像は $T/data/datasets/mecharm_redblock_box_teach_v1_trim2/raw_episodes の最初のエピソードから" && has out.txt "[RANGE ] 安全範囲 (学習データ ±10°) は $T/data/datasets/mecharm_redblock_box_deep_even10_v1/raw_episodes から (--model deep1b)"' rm -rf "$T" setup; state $ID_OLD; pod 200 "$RUNNING"; probe 1 1 1 1 1; serve_dir /workspace/ckpt/deep_v1_2999; run --model deep1b check "deep1 のサーバーが動いている Pod に --model deep1b → 止まり, --model deep1 を案内" '[ $RC = 1 ] && has out.txt "別のモデル (--model deep1)で動いています" && ! has ssh.log "tmux new" && ! has ssh.log "pkill"' ``` ## 依頼137 のあとの、base から学習する2つのファイルの全体 ### 学習の設定を openpi に足すコード（r2r2r-scripts/pi05/openpi_addon/add_deep_config.py） ```python #!/usr/bin/env python3 """openpi config.py に deep_even10_v1 (位置 1〜5 を各 10 本にそろえた 50 本) 用の学習設定を足す (2026-11-09 依頼 133. lab_train.sh --model deep1. 依頼 135 で --variant b (lab_train.sh --model deep1b) を足した). π0.5 base から新しく学習する (続きの学習ではない). 中身は trim の pi05_mecharm_trim_lora と同じ LoRA・batch 32・ 学習率の予定 (warmup 300 / peak 5e-5 / decay_lr 5e-6)・weight_loader (π0.5 base) で, 次の所だけ変える (add_trim_config.py と同じ方式): name pi05_mecharm_trim_lora -> --variant v1: pi05_mecharm_deep_even10_lora --variant b: pi05_mecharm_deep_even10_b_lora repo_id local/mecharm_redblock_box_teach_v1_trim2 -> local/mecharm_redblock_box_deep_even10_v1 num_train_steps 4_000 -> --steps (依頼 133 は 3000) decay_steps 5_000 -> --variant v1: 5_000 のまま (依頼 133 の 2999 はこれで学習した) --variant b: --steps と同じ (依頼 135. 学習の終わりで decay_lr まで下げきる) 正規化の値は写さず, Pod で compute_norm_stats.py をこの設定で回して測り直す (pod_prep_deep.sh). install_addon.py と add_trim_config.py (lab/Log11/pod/) を実行した後に 1 回実行: python3 add_deep_config.py /workspace/openpi --steps 3000 --variant b 何度実行しても同じ結果 (二重に入れない). 入っている設定と --steps が違うときは止まる. v1 と b は名前が違うので両方入れられる. """ import argparse import re import sys from pathlib import Path REPO_ID = "local/mecharm_redblock_box_deep_even10_v1" TRIM_A = " # >>> r2r2r mecharm teach_v1_trim2 (train config)\n" TRIM_B = " # <<< r2r2r mecharm teach_v1_trim2 (train config)\n" BASE_LOADER = 'weight_loader=weight_loaders.CheckpointWeightLoader("gs://openpi-assets/checkpoints/pi05_base/params"),' TRIM_DECAY = "decay_steps=5_000," WARMUP = 300 # trim と同じ warmup_steps. decay_steps はこれより大きくないと下げる区間がない # trim と同じであることを確かめる値 (写したあとも変わっていないこと). decay_steps は --variant ごとに見る SAME_AS_TRIM = ("batch_size=32,", f"warmup_steps={WARMUP},", "peak_lr=5e-5,", "decay_lr=5e-6,", BASE_LOADER) # --variant ごとの設定の名前・印・decay_steps (None は --steps と同じ) VARIANTS = { "v1": {"name": "pi05_mecharm_deep_even10_lora", "mark": "deep_even10_v1", "decay": 5000}, # 依頼 133 (今回の 2999) "b": {"name": "pi05_mecharm_deep_even10_b_lora", "mark": "deep_even10_v1 b", "decay": None}, # 依頼 135 } # 依頼 133 の形を残す (テスト・ほかから読む名前) NAME = VARIANTS["v1"]["name"] def marks(variant): m = VARIANTS[variant]["mark"] return (f" # >>> r2r2r mecharm {m} (train config)\n", f" # <<< r2r2r mecharm {m} (train config)\n") MARK_A, MARK_B = marks("v1") def decay_of(variant, steps): d = VARIANTS[variant]["decay"] return steps if d is None else d def decay_text(n): """openpi の書き方に合わせる (5000 -> 5_000)""" return f"decay_steps={n:_}," def make_block(s, steps, variant="v1"): assert s.count(TRIM_A) == 1 and s.count(TRIM_B) == 1, "先に install_addon.py と add_trim_config.py を実行する" block = s[s.index(TRIM_A):s.index(TRIM_B) + len(TRIM_B)] ma, mb = marks(variant) reps = [ (TRIM_A, ma), (TRIM_B, mb), ('name="pi05_mecharm_trim_lora"', f'name="{VARIANTS[variant]["name"]}"'), ('repo_id="local/mecharm_redblock_box_teach_v1_trim2"', f'repo_id="{REPO_ID}"'), ("num_train_steps=4_000,", f"num_train_steps={steps},"), (TRIM_DECAY, decay_text(decay_of(variant, steps))), ] for a, b in reps: assert block.count(a) == 1, f"見つからない/重複: {a!r}" block = block.replace(a, b) for x in SAME_AS_TRIM: assert block.count(x) == 1, f"trim の設定に見つからない: {x!r}" return block def main(argv=None): ap = argparse.ArgumentParser() ap.add_argument("openpi_root") ap.add_argument("--steps", type=int, required=True, help="学習のステップ数 (num_train_steps. 依頼 133 は 3000)") ap.add_argument("--variant", choices=sorted(VARIANTS), required=True, help="v1 = decay_steps 5_000 (依頼 133 の 2999) / b = decay_steps を --steps と同じに (依頼 135)") args = ap.parse_args(argv) if args.steps <= 0: sys.exit("[ERROR] --steps は 1 以上") decay = decay_of(args.variant, args.steps) if decay <= WARMUP: sys.exit(f"[ERROR] decay_steps ({decay}) が warmup_steps ({WARMUP}) 以下. --steps を大きくする") name = VARIANTS[args.variant]["name"] ma, mb = marks(args.variant) cfg = Path(args.openpi_root).resolve() / "src" / "openpi" / "training" / "config.py" s = cfg.read_text() if ma in s: have = s[s.index(ma):] have = have[:have.index(mb)] m = re.search(r"num_train_steps=(\d+),", have) if not (f'name="{name}"' in have and f'repo_id="{REPO_ID}"' in have and all(x in have for x in SAME_AS_TRIM)): sys.exit("[ERROR] 入っている設定の名前・データ名・学習の値が違う. openpi の config.py を git checkout してから入れ直す") if not (m and int(m.group(1)) == args.steps): sys.exit("[ERROR] 別の --steps で入っている. openpi の config.py を git checkout してから入れ直す") if have.count(decay_text(decay)) != 1: sys.exit(f"[ERROR] 入っている設定の decay_steps が {decay} ではない. openpi の config.py を git checkout してから入れ直す") print("already installed (config.py は変更なし)") return 0 if f'name="{name}"' in s: sys.exit(f"[ERROR] {name} がもう別の形で入っている. openpi の config.py を git checkout してから入れ直す") block = make_block(s, args.steps, args.variant) j = s.index(TRIM_B) + len(TRIM_B) cfg.write_text(s[:j] + block + s[j:]) print(f"installed: {name} (repo_id={REPO_ID}, {args.steps} steps, decay_steps {decay}, init=π0.5 base)") return 0 if __name__ == "__main__": sys.exit(main()) ``` ### Pod でデータを変換し正規化の値を測り直すスクリプト（r2r2r-scripts/pi05/openpi_addon/pod_prep_deep.sh） ```bash #!/usr/bin/env bash # pod_prep_deep.sh — Pod で deep_even10_v1 (位置 1〜5 を各 10 本にそろえた 50 本) の学習の準備をする (2026-11-09 依頼 133). # 学習は回さない. lab_train.sh --model deep1b (依頼 135) の 4b が呼ぶ (deep1 は依頼 133 で呼んだ形). # # 前提: lab_train.sh 3a〜3f で Pod を準備してある (openpi・install_addon.py・add_trim_config.py). 続きの元のモデルは取らない # (π0.5 base から. 重みは学習の最初に openpi が gs://openpi-assets から取る) # Mac から次の 2 つを送ってある: # /workspace/data/mecharm_redblock_box_deep_even10_v1_raw_episodes.tar # /workspace/r2r2r/pi05/ (add_deep_config.py と convert_to_lerobot.py を使う) # # bash /workspace/r2r2r/pi05/openpi_addon/pod_prep_deep.sh # 例: bash ... 3000 b # は add_deep_config.py の --variant (依頼 135): # v1 = 設定 pi05_mecharm_deep_even10_lora・decay_steps 5_000 (依頼 133 の 2999. lab_train.sh --model deep1) # b = 設定 pi05_mecharm_deep_even10_b_lora・decay_steps を と同じに (lab_train.sh --model deep1b) # # すること (何度流しても同じ結果): # 1. tar を /workspace/data/<データ名>/ に広げ, エピソードの本数を確かめる (Mac で作ったときは 50 本) # 2. openpi の config.py に の設定を足す (--steps --variant) # 3. LeRobot 形式に変換 (repo_id local/<データ名>) # 4. 正規化の値をこの 50 本で測り直す (RUNPOD_TRAIN.md 4. の compute_norm_stats.py をこの設定で). # 置く場所は openpi が学習で読む assets/<設定の名前>/local/<データ名>/norm_stats.json. # 最後に「NORM sha256=<64 桁> path=」の1行を出す (lab_train.sh が由来に残し, 持ち帰る) # 5. 学習を始めるコマンドを出す (流さない) set -euo pipefail STEPS="${1:?学習のステップ数 (例 3000)}" VARIANT="${2:?add_deep_config.py の --variant (v1 か b)}" NAME=mecharm_redblock_box_deep_even10_v1 case "$VARIANT" in # 何かする前に確かめる (依頼 135) v1) CONFIG=pi05_mecharm_deep_even10_lora; EXP=deep_even10_v1 ;; b) CONFIG=pi05_mecharm_deep_even10_b_lora; EXP=deep_even10_v1_b ;; *) echo "[ERROR] は v1 か b: $VARIANT" >&2; exit 2 ;; esac N_MAC=50 TAR=/workspace/data/${NAME}_raw_episodes.tar DATA=/workspace/data/$NAME OPENPI=/workspace/openpi ADDON=/workspace/r2r2r/pi05/openpi_addon NORM_DST=assets/$CONFIG/local/$NAME/norm_stats.json # $OPENPI からの相対 [ -f "$TAR" ] || { echo "[ERROR] tar がない: $TAR" >&2; exit 1; } # shellcheck disable=SC1091 [ -f "$HOME/.local/bin/env" ] && source "$HOME/.local/bin/env" echo "== 1. tar を広げる → $DATA" # 前に広げたものへ上書きしない (pod_prep_trim_corr.sh と同じ. 空の一時フォルダに広げて確かめてから入れ替える) TMP="$DATA.tmp.$$" rm -rf "$TMP" mkdir -p "$TMP" tar xf "$TAR" -C "$TMP" N=$(find "$TMP/raw_episodes" -mindepth 2 -maxdepth 2 -name episode.npz | wc -l | tr -d ' ') [ "$N" = "$N_MAC" ] || { echo "[ERROR] tar のエピソードが $N 本 (Mac で作ったときは $N_MAC 本): $TAR" >&2; rm -rf "$TMP"; exit 1; } rm -rf "$DATA" mv "$TMP" "$DATA" echo " エピソード $N 本 (Mac で作ったときは $N_MAC 本)" cd "$OPENPI" echo "== 2. 学習設定 $CONFIG を足す (steps=${STEPS}・variant ${VARIANT}. π0.5 base から)" python3 "$ADDON/add_deep_config.py" "$OPENPI" --steps "$STEPS" --variant "$VARIANT" echo "== 3. LeRobot 形式に変換 (local/$NAME)" uv run python /workspace/r2r2r/pi05/convert_to_lerobot.py --raw-dir "$DATA/raw_episodes" --repo-id "local/$NAME" echo "== 4. 正規化の値をこの $N 本で測り直す (compute_norm_stats.py --config-name $CONFIG)" rm -f "$NORM_DST" # 前の回の値を残さない (測れなかったときに古い値で学習しない) uv run scripts/compute_norm_stats.py --config-name "$CONFIG" [ -s "$NORM_DST" ] || { echo "[ERROR] 正規化の値ができていない: $OPENPI/$NORM_DST" >&2; exit 1; } SHA=$(sha256sum "$NORM_DST" | cut -d' ' -f1) echo " $OPENPI/$NORM_DST (sha256 ${SHA:0:12}…)" echo "NORM sha256=$SHA path=$NORM_DST" echo "== 5. 準備できました. 学習はこのコマンドで始める (このスクリプトでは流さない):" echo " cd $OPENPI && XLA_PYTHON_CLIENT_MEM_FRACTION=0.9 uv run scripts/train.py $CONFIG --exp-name=$EXP --overwrite 2>&1 | tee /workspace/logs/train_$EXP.log" ```