Negative Log Sigmoid Log Likelihood Loss?

[marco-rudolph]

Hi Denis!
According to your code, you use some "negative log sigmoid log likelihood loss":

cflow-ad/train.py

Line 76 in 4d6ec47

decoder_log_prob = get_logp(C, z, log_jac_det)

cflow-ad/train.py

Line 78 in 4d6ec47

loss = -log_theta(log_prob)

with

cflow-ad/train.py

Line 18 in 4d6ec47

log_theta = torch.nn.LogSigmoid()

What is the motivation behind using this kind of loss?

Thanks in advance,
Marco

[gudovskiy]

@marco-rudolph well, flows can minimize KL-divergence up to the constant or, alternatively, can have log-likelihoods larger than 0.0 which is not desirable. So, to prevent this and to avoid negative loss I added sigmoid. In practice, it can help to improve performance a bit.

[marco-rudolph]

Why are likelihoods larger than 0.0 not desireable? Negative loss itself should not be a problem as long as it does not lead to instabilities which is not really the case here.
But it is an interesting finding that this improves the performance :)