Projector plugin can't visualize some data protos

[svsgoogle]

The projector plugin freezes with "Computing PCA..." for some data protos, e.g. one where all the embeddings are zero:

{
  "tensor": [
    0.0, 
    0.0, 
    0.0, 
    0.0, 
    0.0, 
    0.0, 
    0.0, 
    0.0, 
    0.0, 
    0.0, 
    0.0, 
    0.0, 
    0.0, 
    0.0, 
    0.0, 
    0.0, 
    0.0, 
    0.0, 
    0.0, 
    0.0, 
    0.0, 
    0.0, 
    0.0, 
    0.0, 
    0.0, 
    0.0, 
    0.0, 
    0.0, 
    0.0, 
    0.0, 
    0.0, 
    0.0, 
    0.0, 
    0.0, 
    0.0, 
    0.0, 
    0.0, 
    0.0, 
    0.0, 
    0.0, 
    0.0, 
    0.0, 
    0.0, 
    0.0, 
    0.0, 
    0.0, 
    0.0, 
    0.0, 
    0.0, 
    0.0, 
    0.0, 
    0.0, 
    0.0, 
    0.0, 
    0.0, 
    0.0, 
    0.0, 
    0.0, 
    0.0, 
    0.0, 
    0.0, 
    0.0, 
    0.0, 
    0.0, 
    0.0, 
    0.0, 
    0.0, 
    0.0, 
    0.0, 
    0.0, 
    0.0, 
    0.0, 
    0.0, 
    0.0, 
    0.0, 
    0.0, 
    0.0, 
    0.0, 
    0.0, 
    0.0, 
    0.0, 
    0.0, 
    0.0, 
    0.0, 
    0.0, 
    0.0, 
    0.0, 
    0.0, 
    0.0, 
    0.0, 
    0.0, 
    0.0, 
    0.0, 
    0.0, 
    0.0, 
    0.0, 
    0.0, 
    0.0, 
    0.0, 
    0.0
  ], 
  "metadata": {
    "columns": [
      {
        "name": "words", 
        "stringValues": [
          "cat", 
          "hat"
        ]
      }
    ]
  }, 
  "shape": [
    2, 
    50
  ]
}

[wchargin]

Thanks for the report—cc @dsmilkov who would know best here.

(I edited your post to add code formatting. In the future, you can place three backticks ``` on a line by themselves before and after your code to format it as above.)

[chihuahua]

Thanks to @wchargin for making sure this bug doesn't get dropped. I can repro with

import os

from tensorflow.contrib.tensorboard.plugins import projector
import tensorflow as tf

value_var = tf.get_variable('my_embedding', initializer=tf.zeros([2, 50]))

# Format: tensorflow/tensorboard/plugins/projector/projector_config.proto
config = projector.ProjectorConfig()

# You can add multiple embeddings. Here we add only one.
embedding = config.embeddings.add()
embedding.tensor_name = value_var.name

LOG_DIR = 'foo'

# Use the same LOG_DIR where you stored your checkpoint.
summary_writer = tf.summary.FileWriter(LOG_DIR)

# The next line writes a projector_config.pbtxt in the LOG_DIR. TensorBoard will
# read this file during startup.
projector.visualize_embeddings(summary_writer, config)

session = tf.Session()
session.run(tf.global_variables_initializer())

saver = tf.train.Saver()
saver.save(session, os.path.join(LOG_DIR, "model.ckpt"), 0)
session.run(value_var)

The problem is that the projector converts all your 0 values to NaN when it attempts to normalize:

tensorboard/tensorboard/plugins/projector/vz_projector/vz-projector.ts

Line 383 in 3a85599

if ((ds != null) && this.normalizeData) {

The projector then computes PCA across vectors of all NaNs (without telling you that it's operating over NaNs). Derp, that's no fun. Arbitrary bad things can happen based on the tensor shape - it's undefined behavior.

I made 2 changes. #473 prevents the projector from taking the unit norm of vectors with all 0s. To make errors clearer to users in the future, I also created #472.

I split into 2 PRs partly because #472 is a bit controversial - it hurts performance a bit.