Skip to content
GitLab

Allow NaNs for MAD

Merged Administrator requested to merge github/fork/ingonader/allow-nans-for-mad into development

Created by: ingonader

All Submissions Basics:

  • Have you followed the guidelines in our Contributing document?
  • Have you checked to ensure there aren't other open Pull Requests for the same update/change?
  • Have you checked all Issues to tie the PR to a specific one?

All Submissions Cores:

  • Have you added an explanation of what your changes do and why you'd like us to include them?
  • Have you written new tests for your core changes, as applicable?
  • Have you successfully ran tests with your changes locally?
  • Does your submission pass tests, including CircleCI, Travis CI, and AppVeyor?
  • Does your submission have appropriate code coverage? The cutoff threshold is 95% by Coversall.

Changes

Related Issue: https://github.com/yzhao062/pyod/issues/376

This PR aims to allow np.NaN and np.infty values in the MAD outlier detection method. The method as previously implemented seemes to have some functionality to deal with NaNs (using np.nanmedian() at least in some calculations), but the check_array() function, as currently parametrized, throws an error.

This PR uses check_array() with force_all_finite=False and also changes the calculation of self.median_diff in _mad() to np.nanmedian(). Hence, the model is no able to cope with missing and infinite values.

In addition, some of the existing tests have been added (based on existing tests with addition of missing and infinite values).

Comments to submission checklist

Have you successfully ran tests with your changes locally?

All tests for MAD (python -m unittest pyod/test/test_mad.py) ran successfully. However, out of the total test suite (python -m unittest discover ./pyod/test/), 5 tests failed. All of the failing tests are unrelated to MAD, and all of the failing tests are related to keras. I suspect this is some kind of local problem with my installation.

Gist from test logs:

ERROR: test_auto_encoder (unittest.loader._FailedTest)
ImportError: cannot import name 'tf2'
ERROR: test_deepsvdd (unittest.loader._FailedTest)
ModuleNotFoundError: DeepSVDD runs only with TensorFlow 2.0+
ERROR: test_mo_gaal (unittest.loader._FailedTest)
ImportError: cannot import name 'tf2'
ERROR: test_so_gaal (unittest.loader._FailedTest)
ImportError: cannot import name 'tf2'
ERROR: test_vae (unittest.loader._FailedTest)
ImportError: cannot import name 'tf2'
Ran 527 tests in 571.009s
FAILED (errors=5)

Does your submission pass tests, including CircleCI, Travis CI, and AppVeyor?

I have only tested via Travis CI, and I hope this does not pose a problem. In Travis, 4 of 5 build passed, but the build für Python 3.3 failed when installing the requirements (which are unchanged in this PR). So I hope this is still fine.

Gist from Travis logs:

$ pip install -r requirements_ci.txt
Collecting combo
  Downloading combo-0.1.2.tar.gz (37 kB)
  Preparing metadata (setup.py) ... error
  error: subprocess-exited-with-error
  × python setup.py egg_info did not run successfully.
  [...]
        File "/home/travis/virtualenv/python3.7.1/lib/python3.7/site-packages/setuptools/_vendor/importlib_metadata/__init__.py", line 886, in find_distributions
          found = self._search_paths(context.name, context.path)
      AttributeError: 'str' object has no attribute 'name'
      [end of output]
  note: This error originates from a subprocess, and is likely not a problem with pip.