Skip to content
GitLab
Closed
Issue created by Administrator@rootOwner

Unable to run interactive_hosted locally

Created by: suchenzang

To reproduce:

  • ssh to node 131 on cluster
  • confirm that mp 175B shards exist in /mnt/scratch/175B/reshard_no_os/
  • permissions on dir has been opened to all (chmod 777)
  • metaseq is on main

Resulting stacktrace:

(metaseq-test-apr7) susanz@<node 131 on cluster>:~/staging/metaseq$ python -m metaseq_cli.interactive_hosted
2022-05-08 20:09:42 | INFO | metaseq_cli.interactive | Local checkpoint copy already exists, skipping copy
2022-05-08 20:09:42 | INFO | metaseq.tasks.language_modeling | dictionary: 50272 types
2022-05-08 20:09:42 | INFO | metaseq.hub_utils | loading model(s) from /mnt/scratch/175B/reshard_no_os/reshard.pt
Traceback (most recent call last):
  File "/shared/home/susanz/miniconda3/envs/metaseq-test-apr7/lib/python3.8/runpy.py", line 194, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/shared/home/susanz/miniconda3/envs/metaseq-test-apr7/lib/python3.8/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/shared/home/susanz/staging/metaseq/metaseq_cli/interactive_hosted.py", line 304, in <module>
    cli_main()
  File "/shared/home/susanz/staging/metaseq/metaseq_cli/interactive_hosted.py", line 300, in cli_main
    dist_utils.call_main(cfg, worker_main, namespace_args=args)
  File "/shared/home/susanz/staging/metaseq/metaseq/distributed/utils.py", line 263, in call_main
    return main(cfg, **kwargs)
  File "/shared/home/susanz/staging/metaseq/metaseq_cli/interactive_hosted.py", line 156, in worker_main
    models = generator.load_model()  # noqa: F841
  File "/shared/home/susanz/staging/metaseq/metaseq/hub_utils.py", line 485, in load_model
    models, _model_args, _task = checkpoint_utils.load_model_ensemble_and_task(
  File "/shared/home/susanz/staging/metaseq/metaseq/checkpoint_utils.py", line 473, in load_model_ensemble_and_task
    state = load_checkpoint_to_cpu(filename, arg_overrides)
  File "/shared/home/susanz/staging/metaseq/metaseq/checkpoint_utils.py", line 408, in load_checkpoint_to_cpu
    paths_to_load = get_paths_to_load(local_path, suffix="shard")
  File "/shared/home/susanz/staging/metaseq/metaseq/checkpoint_utils.py", line 348, in get_paths_to_load
    if not _is_checkpoint_sharded(checkpoint_files):
  File "/shared/home/susanz/staging/metaseq/metaseq/checkpoint_utils.py", line 339, in _is_checkpoint_sharded
    size_ratio = max(sizes) / min(sizes)
ValueError: max() arg is an empty sequence

@stephenroller last time we saw this, it seemed to be a read permissions issue?