Jobs can sometimes be processed after a queue is paused locally.

Created by: josephwarrick

Jobs can sometimes be processed after a queue is paused locally.

This happens because we wait up to five seconds for a job to become available when we try to get the next job.

This is the sequence of events when this happens:

The queue does queue.bclient.brpoplpush to get the next job, but none are available yet.

We locally pause the queue; queue.bclient.brpoplpush is still pending. Since some jobs are still actively processing, we wait for them to finish before resolving the pause.

A new job is pushed to the queue.

queue.bclient.blpoprpush returns with the new job, and our queue starts processing it.

The original set of active jobs all complete and queue.pause(true) resolves. The new job is still processing.

So we think our queue is paused, but we're actually still processing a job.

If we kill the queue, the active job may be terminated before it can complete.