Created by: stephenroller
Patch Description Small QOL change. Adds a new log line saying the full nodelist in SLURM format.
Also silence some torch.distributed log lines.
Testing steps Internal training launch. Initialization now looks like this (with other internal change):
commit 380d8bc41351410399797f7e68fbe005acdaa4f2
Submitted batch job 82764
2022-08-05 08:44:41 | INFO | metaseq.distributed.utils | initialized host fairwus3-1-htc-337 as rank 0
2022-08-05 08:44:41 | INFO | metaseq.distributed.utils | initialized host fairwus3-1-htc-339 as rank 16
2022-08-05 08:44:41 | INFO | metaseq.distributed.utils | initialized host fairwus3-1-htc-339 as rank 23
2022-08-05 08:44:41 | INFO | metaseq.distributed.utils | initialized host fairwus3-1-htc-339 as rank 21
2022-08-05 08:44:41 | INFO | metaseq.distributed.utils | initialized host fairwus3-1-htc-339 as rank 18
2022-08-05 08:44:41 | INFO | metaseq.distributed.utils | initialized host fairwus3-1-htc-339 as rank 22
2022-08-05 08:44:41 | INFO | metaseq.distributed.utils | initialized host fairwus3-1-htc-339 as rank 20
2022-08-05 08:44:41 | INFO | metaseq.distributed.utils | initialized host fairwus3-1-htc-337 as rank 1
2022-08-05 08:44:41 | INFO | metaseq.distributed.utils | initialized host fairwus3-1-htc-337 as rank 5
2022-08-05 08:44:41 | INFO | metaseq.distributed.utils | initialized host fairwus3-1-htc-337 as rank 4
2022-08-05 08:44:41 | INFO | metaseq.distributed.utils | initialized host fairwus3-1-htc-340 as rank 26
2022-08-05 08:44:41 | INFO | metaseq.distributed.utils | initialized host fairwus3-1-htc-337 as rank 2
2022-08-05 08:44:41 | INFO | metaseq.distributed.utils | initialized host fairwus3-1-htc-337 as rank 6
2022-08-05 08:44:41 | INFO | metaseq.distributed.utils | initialized host fairwus3-1-htc-337 as rank 7
2022-08-05 08:44:41 | INFO | metaseq.distributed.utils | initialized host fairwus3-1-htc-340 as rank 27
2022-08-05 08:44:41 | INFO | metaseq.distributed.utils | initialized host fairwus3-1-htc-340 as rank 25
2022-08-05 08:44:41 | INFO | metaseq.distributed.utils | initialized host fairwus3-1-htc-338 as rank 11
2022-08-05 08:44:41 | INFO | metaseq.distributed.utils | initialized host fairwus3-1-htc-338 as rank 12
2022-08-05 08:44:41 | INFO | metaseq.distributed.utils | initialized host fairwus3-1-htc-338 as rank 10
2022-08-05 08:44:41 | INFO | metaseq.distributed.utils | initialized host fairwus3-1-htc-340 as rank 30
2022-08-05 08:44:41 | INFO | metaseq.distributed.utils | initialized host fairwus3-1-htc-340 as rank 24
2022-08-05 08:44:41 | INFO | metaseq.distributed.utils | initialized host fairwus3-1-htc-338 as rank 13
2022-08-05 08:44:41 | INFO | metaseq.distributed.utils | initialized host fairwus3-1-htc-338 as rank 8
2022-08-05 08:44:41 | INFO | metaseq.distributed.utils | initialized host fairwus3-1-htc-338 as rank 9
2022-08-05 08:44:41 | INFO | metaseq.distributed.utils | initialized host fairwus3-1-htc-340 as rank 29
2022-08-05 08:44:41 | INFO | metaseq.distributed.utils | initialized host fairwus3-1-htc-338 as rank 14
2022-08-05 08:44:41 | INFO | metaseq.distributed.utils | initialized host fairwus3-1-htc-338 as rank 15
2022-08-05 08:44:41 | INFO | metaseq.distributed.utils | initialized host fairwus3-1-htc-340 as rank 31
2022-08-05 08:44:41 | INFO | metaseq.distributed.utils | initialized host fairwus3-1-htc-337 as rank 3
2022-08-05 08:44:41 | INFO | metaseq.distributed.utils | initialized host fairwus3-1-htc-339 as rank 19
2022-08-05 08:44:41 | INFO | metaseq.distributed.utils | initialized host fairwus3-1-htc-340 as rank 28
2022-08-05 08:44:41 | INFO | metaseq.distributed.utils | initialized host fairwus3-1-htc-339 as rank 17
NCCL version 2.10.3+cuda11.6
2022-08-05 08:44:50 | INFO | metaseq.distributed.utils | SLURM nodelist: fairwus3-1-htc-[337-340]
> initializing tensor model parallel with size 2
> initializing pipeline model parallel with size 1