Skip to content
GitLab
Closed
Issue created by Administrator@rootContributor

Python interpreter hangs on exit when using 'AUTO' thread_type on Linux

Created by: Breakthrough

Overview

Setting thread_type = 'AUTO' prevents the Python interpreter from gracefully exiting and can hang indefinitely. I have reproduced the issue with PyAV version 8 and 9, on Ubuntu 16.04, 18.04, and 20.04 (see reproduction below). I also believe the issue affects the threading example in the cookbook, which makes no mention of the possible issues nor workarounds that exist.

I have not been able to reproduce the issue on Windows, and I am unable to test this on OSX.

Expected behavior

PyAV should not prevent the program from terminating when using AUTO threading mode, or if this is unavoidable, the threading example in the cookbook should better indicate that this can happen and how it can be mitigated.

The caveats section implies this issue only affects subinterpreters, but I'm not sure that's the case. I'm also curious if it would be possible to disable logging by default when PyAV is shutting down which would likely resolve most user facing issues.

Actual behavior

Any application using PyAV on Linux can occasionally hang on termination when using thread_type = 'AUTO' so long as at least one frame has been decoded.

Reproduction

Verified using the following minimum working example:

import av
print('Starting test.')
container = av.open('video.mp4')
container.streams.video[0].thread_type = 'AUTO'
# Have to decode at least one frame to trigger the bug.
next(container.decode(video=0))
print('Test complete.')

# No longer hangs after uncommenting the following line:
#av.logging.restore_default_callback()

Running this in a loop via bash will output something like:

$ while true; do python3 test_av_hang.py; done
Starting test.
Test complete.
Starting test.
Test complete.
Starting test.
Test complete.

The output will then stop suddenly.

I discovered the issue on Travis CI, and was able to reproduce locally in both a VM and on an actual Ubuntu distro. I had difficulty reproducing the issue with only a single-core VM, but it seems to fail rather quickly on any multicore Linux machine. Adding av.logging.restore_default_callback() at the end of the program resolves the issue.

This issue also occurs with the threading example in the cookbook, which makes no mention of this problem.

The caveats page implies this only affects sub-interpreters so I initially overlooked that during my initial analysis. Disabling logging before the program terminates seems to resolve the issue (e.g. calling av.logging.restore_default_callback() before the program terminates). Thus I'd like to ask, is it possible to automatically do this when PyAV is being destroyed, or can a timeout be set in the event things deadlock? Or can the documentation be updated to reflect that this can happen in normal usage, not just subprocesses/subinterpreters?

If my understanding of this is incorrect I do apologize.

Versions

  • OS: Ubuntu 16.04, 18.04, 20.04. Unable to reproduce on Windows. Did not test OSX.
  • Python: Reproducible on 3.6, 3.7, 3.8, and 3.9.
  • PyAV runtime: tested both 8 and 9 with binary wheels (default from using pip install av as this is what most end users of the project will do)

P.S. Thank you all for the awesome project!