Created by: stephenroller
Patch Description Adds support for CLI interactive.
This involves a major change to distributed initialization, which I copied from ParlAI: Instead of having the main process spawn N subprocesses all running the distributed code, and then idle until they wrap up, we have it spawn N-1 processes and then we also run the distributed code on the main process.
This means that exceptions or signals like KeyboardInterrupts in the main thread can be caught and propagated to the spawns to be terminated. It also means we don't need to do any hacks to reroute stdin/stdout in the rank0 process.
Testing steps
Also tested that I can cleanly ctrl-c now, and also several exceptions due to standard iterations were correctly propagated.