continue opt175b download from the latest available file

Created by: Barqawiz

🚀 Feature Request

In the case of an internet issue or interruption, the download will stop, and the available opt175b download script will download from scratch.

Motivation

opt175b files are enormous, and we need a way to continue the download from the last milestone. After two days of downloading the opt175b files, my two years old daughter unplugged the internet! And when I tried to re-run the download script, it started from scratch.

Pitch

I create a script that downloads the file only if it does not exist or the file size is zero bytes and the script is ready to merge.

Alternatives

I checked the script files and I did not find a script to continue opt175b from latest milestone

Additional context

The solution is available below as a new file if this feature is approved: https://github.com/Barqawiz/metaseq/blob/main/metaseq/scripts/download_checkpoint_opt175b.sh

related pull request: #289