Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

CIFAR-10 eval fails with error TypeError: Input 'predictions' of 'InTopKV2' Op has type float16 that does not match expected type of float32 #7225

Closed
chrismattmann opened this issue Jul 16, 2019 · 3 comments

Comments

@chrismattmann
Copy link
Contributor

chrismattmann commented Jul 16, 2019

System information

  • What is the top-level directory of the model you are using:
    tutorials/image/cifar10/cifar10_eval.py
  • Have I written custom code (as opposed to using a stock example script provided in TensorFlow):
    no
  • OS Platform and Distribution (e.g., Linux Ubuntu 16.04): Linux jupyter-mattmann-40usc-2eedu 4.15.15-1.el7.x86_64 initial commit, simple, separated models #1 SMP Thu Oct 4 07:42:41 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux
  • TensorFlow installed from (source or binary): used PIP (binary)
  • TensorFlow version (use command below): 1.13.1, tensorflow-datasets 1.0.2
  • Bazel version (if compiling from source): N/A
  • CUDA/cuDNN version: N/A
  • GPU model and memory: 4 GPUs
  • Exact command to reproduce:
    python3 cifar_eval.py
== env ==========================================================
LD_LIBRARY_PATH /usr/local/nvidia/lib:/usr/local/nvidia/lib64
DYLD_LIBRARY_PATH is unset

== nvidia-smi ===================================================
Tue Jul 16 15:59:26 2019       
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 430.14       Driver Version: 430.14       CUDA Version: 10.2     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GeForce RTX 208...  Off  | 00000000:1B:00.0 Off |                  N/A |
| 31%   32C    P0    85W / 250W |      0MiB / 11019MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   1  GeForce RTX 208...  Off  | 00000000:1E:00.0 Off |                  N/A |
|  0%   30C    P8    22W / 250W |      0MiB / 11019MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   2  GeForce RTX 208...  Off  | 00000000:61:00.0 Off |                  N/A |
|  0%   30C    P0    65W / 250W |      0MiB / 11019MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   3  GeForce RTX 208...  Off  | 00000000:63:00.0 Off |                  N/A |
| 29%   29C    P0    62W / 250W |      0MiB / 11019MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
                                                                               
+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|  No running processes found                                                 |
+-----------------------------------------------------------------------------+

== cuda libs  ===================================================
/usr/local/cuda-10.0/targets/x86_64-linux/lib/libcudart_static.a
/usr/local/cuda-10.0/targets/x86_64-linux/lib/libcudart.so.10.0.130

== tensorflow installed from info ==================

== python version  ==============================================
(major, minor, micro, releaselevel, serial)
(3, 7, 3, 'final', 0)

== bazel version  ===============================================
jovyan@jupyter-mattmann-40usc-2eedu:~/models/tutorials/image/cifar10$ 

Describe the problem

CIFAR-10 eval script fails. I think the error is related to tensorflow/tensorflow#165. I'll work up a PR to fix.

Source code / logs

Will send a PR.

@chrismattmann
Copy link
Contributor Author

Another bug later on after fixing this one:

Traceback (most recent call last):
  File "cifar10_eval.py", line 156, in <module>
    tf.app.run()
  File "/opt/conda/lib/python3.7/site-packages/tensorflow/python/platform/app.py", line 40, in run
    _run(main=main, argv=argv, flags_parser=_parse_flags_tolerate_undef)
  File "/opt/conda/lib/python3.7/site-packages/absl/app.py", line 300, in run
    _run_main(main, args)
  File "/opt/conda/lib/python3.7/site-packages/absl/app.py", line 251, in _run_main
    sys.exit(main(argv))
  File "cifar10_eval.py", line 152, in main
    evaluate()
  File "cifar10_eval.py", line 128, in evaluate
    top_k_op = tf.nn.in_top_k(logits, labels, 1)
  File "/opt/conda/lib/python3.7/site-packages/tensorflow/python/ops/nn_ops.py", line 4784, in in_top_k
    return gen_nn_ops.in_top_kv2(predictions, targets, k, name=name)
  File "/opt/conda/lib/python3.7/site-packages/tensorflow/python/ops/gen_nn_ops.py", line 5040, in in_top_kv2
    "InTopKV2", predictions=predictions, targets=targets, k=k, name=name)
  File "/opt/conda/lib/python3.7/site-packages/tensorflow/python/framework/op_def_library.py", line 626, in _apply_op_helper
    param_name=input_name)
  File "/opt/conda/lib/python3.7/site-packages/tensorflow/python/framework/op_def_library.py", line 60, in _SatisfiesTypeConstraint
    ", ".join(dtypes.as_dtype(x).name for x in allowed_list)))
TypeError: Value passed to parameter 'targets' has DataType float16 not in list of allowed values: int32, int64

Fixing this cast as well.

chrismattmann added a commit to chrismattmann/models that referenced this issue Jul 16, 2019
…put 'predictions' of 'InTopKV2' Op has type float16 that contributed by mattmann.
chrismattmann added a commit to chrismattmann/models that referenced this issue Jul 16, 2019
…put 'predictions' of 'InTopKV2' Op has type float16 that contributed by mattmann.
@chrismattmann
Copy link
Contributor Author

Added a PR #7227 that fixes this.

tfboyd pushed a commit that referenced this issue Jul 18, 2019
…ctions' of 'InTopKV2' Op has type float16 that contributed by mattmann. (#7227)
@chrismattmann
Copy link
Contributor Author

Committed by @tfboyd in 63605b9 thanks!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant