Created by: dianaml0
Patch Description Creating this PR off of #511, so it can be reviewed by @stephenroller
The last commit (3d709dba) removes some changes from the sequence parallel code which enabled testing with world size of 1. CI is not currently running the test anyway because CI needs to be updated for the test to run.
The forward and backward tests are passing right now. However in some cases, about .2% of the elements fail
Testing steps
Unit Test gpu_tests/test_sequence_parallel_transformer_layer.py