Text plugin only renders some + most recent events

[sleighsoft]

I have a neural network that generates text during ~356.000 steps.

Every 50.000 steps I save a summary containing a table of text (100 rows, 5 columns).

When running tensorboard it only displays the results of the last step.

I create the output like this:

from tensorboard import summary as tb_summary

summary_protobuf = tb_summary.text_pb('label', [['c0', ''c1', 'c2', 'c3', 'c4']] * 100)
summary_writer.add_summary(summary_protobuf, global_step=step)
summary_writer.flush()

Sometimes tensorboard also throws this on the commandline:

Exception ignored in: <function _get_module_lock.<locals>.cb at 0x7f48cbe09378>
Traceback (most recent call last):
  File "<frozen importlib._bootstrap>", line 174, in cb
KeyError: ('markdown.extensions.tables',)

[nfelt]

Hm, is it just displaying the last step or also "some" events in addition? If you run tensorboard with the --inspect flag against the same data, what does it show?

Re: the KeyError, that's pretty strange. Is that the full stacktrace? At what point in running tensorboard does that appear?

Also, if you can provide your TensorBoard version (what it prints out when it starts up) that would help.

[sleighsoft]

1. Let's say I have 350000 steps in total and I log every 10k steps. What I see in tensorboard is something similar to 20k, 150k, 350k. But it seems completely arbitrary. I can try with --inspect soon.
2. That's all there is. It appears when changing to the Text plugin tab.
3. Tensorboard 1.5.0

[nfelt]

Re: 1, that's probably just the result of reservoir sampling - it is "working as intended" at least right now but we are open to feature requests to make it more customizable. See #83.

Re: 2/3, I'll look into a bit. What platform/OS are you using?

[sleighsoft]

I am on Windows 10.

[nfelt]

Re: the KeyError, it looks like perhaps that's related to https://bugs.python.org/issue31070 versus something specific to the text plugin. What is your full 3-digit python version? If it's less than 3.6.3, upgrading to 3.6.3 or higher might fix your issue.

[sleighsoft]

3.6.2. I'll give it a try

[sleighsoft]

Haven't had that issue with 3.6.2

[nfelt]

Did you mean 3.6.3? I'll go ahead and close this since it sounds like the issue may be resolved, but feel free to reopen if that's not the case.

[nfelt]

FYI, PR #1138 added a --samples_per_plugin flag that can be used to set the number of samples retained on a per-plugin basis. So e.g. --samples_per_plugin=text=100 should set the text dashboard to retain 100 samples for each series.