Skip to content
This repository has been archived by the owner on Nov 17, 2023. It is now read-only.

test_loss.test_ce_loss has fixed seed that can mask flakiness #11692

Closed
szha opened this issue Jul 13, 2018 · 4 comments
Closed

test_loss.test_ce_loss has fixed seed that can mask flakiness #11692

szha opened this issue Jul 13, 2018 · 4 comments

Comments

@szha
Copy link
Member

szha commented Jul 13, 2018

The unit test in title have been using fixed seed to mask flakiness. Suggested action:

  1. Evaluate whether the test is flaky without fixed seed. If not, remove seed. Else move to 2
  2. If test is flaky, determine whether it's an actual uncaught edge case. If so, fix the operator. Else move to 3
  3. If numerical instability is inevitable, adjust tolerance level appropriately.
@apeforest
Copy link
Contributor

Thanks for filing this issue. We will investigate this Flaky test

@lanking520
Copy link
Member

Able to reproduce MXNET_SEED: 1327713000

@lanking520
Copy link
Member

After adding Xavier initializer, problem solved:

python tools/flakiness_checker.py -n 10000 test_operator_gpu.test_ce_loss
INFO:root:Testing: /home/ubuntu/incubator-mxnet/tests/python/gpu/test_operator_gpu.py:test_ce_loss
INFO:root:No test seed provided, using random seed
/usr/local/lib/python3.5/dist-packages/nose/util.py:453: DeprecationWarning: inspect.getargspec() is deprecated, use inspect.signature() instead
  inspect.getargspec(func)
[INFO] Setting module np/mx/python random seeds, use MXNET_MODULE_SEED=449509033 to reproduce.
test_operator_gpu.test_ce_loss ... ok

----------------------------------------------------------------------
Ran 1 test in 9277.876s

OK
INFO:root:Nosetests terminated with exit code 0

@haojin2
Copy link
Contributor

haojin2 commented Aug 16, 2018

@sandeep-krishnamurthy Related fix is merged, should be good to close.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

No branches or pull requests

5 participants