bunch of changes for sequence parallel, seletive activation checkpointing, gelu fc2 merge, offloading activations in new stream

* Add eos mode for src tgt dataset

* lint

* Addressed comments

* lint

* fix docker build fails

* revert metaseq branch tomain in dockerfile

* limit max version of hydra-core

Co-authored-by: qiushuo <qiushuo@microsoft.com>

* remove unused build_progress_bar method

* remove noop/none logging as a log format option

* make log_format default to json and remove default_log_format arg

* move progress_bar to submodule

* rename to base_progress_bar

* module rename

* move out json progress bar

* move out tensorboard progress bar

* move out wandb progress wrapper

* split out helpers to utils, avoid circular import

* fix broken init

* remove module rename for progress bar

* move get_precise_epoch to higher level utils

* lint

* remove no progress bar flag

* fix more imports

* cleanup

* remove utils

* add license

* pin omegaconf to 2.1.1

* pin tensorboard to 2.8.0

* protobuf pin to 3.20.1

Based on the setup instructions

* Add maximum for example proportional sampling

* lint

* Addressed comments

* include diff with black check

* fix lint

* set defaults for scale window, and init scale

* remove more default logic for scale window

* also decrease scale window with loss scale

* remove floating point error for hitting min loss scale

* also log out scale_window

* 0.03125 -> 2 ** -5

* add mypy as a dependency

* add mypy.ini

* add mypy ci

* allow mypy ci error to exit sucessfully

* change python version to 3.8 in mypy.ini

* Loss only on target tokens

Co-authored-by: Ramakanth Pasunuru <ramakanth.1729@gmail.com>

* [scripts] Convert resharded MP checkpoints to unflattened.

* Black

* Check it out ma

* Black

* Update metaseq/scripts/convert_to_singleton.py

Co-authored-by: Thomas Wang <24695242+thomasw21@users.noreply.github.com>

* Recursively unwrap fully sharded model in `convert_to_singleton.py` (#72)

* Recursively unwrap fully sharded model

* Update metaseq/scripts/convert_to_singleton.py

Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com>

Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com>

* temp

* Whoops spawn too many

* Add logging.

* Black

Co-authored-by: Thomas Wang <24695242+thomasw21@users.noreply.github.com>
Co-authored-by: Patrick von Platen <patrick.v.platen@gmail.com>

* remove dict.txt generation

* remove dictionary and targets from LanguageModelingTask

* remove LanguageModelingTask build_model, default to LegacyTask build_model

* cleanup, continue removing targets

* removed targets arg from LanguageModelingTask

* remove targets from LanguageModelingInferenceForModelsTrainedWithStreamingTask

* lint

* add in args from streaming lm config

* fix

* missing final_vocab_size arg from LM task config

This commit updates the documentation for downloading the OPT-175B parameters with the space required to store all of the files. This is super helpful for any practitioner planning to actually download the data.

This reverts commit 2d5e7535.

* CLI interactive mode.

* Whoops

* more

* Nits.

* Black

* Lint.

* replace fairseq refs

* remove private path example

* nit

* changed metaseq_internal to metaseq

* fixed naming of script in docs, added resharding script without slurm

* [checkpoint] Copy over consolidate script

* Copyright

Fixes #23.

* update to facebookresearch links

* adding reshard scripts

* script to download opt175b

* add initial instructions for OPT-175B access

* adding gpt2 merges, vocab files