dvae training resulting in an irregular latent space

[hlp-pls]

Hello! I'm not sure whether this should be raised as an issue or it is a fault completely on my side. But I've reached a point where I can't seem to figure it out on my own, so I hope someone could enlighten me.

I'm trying to train the DiscreteVAE with some custom dataset, but the trained model seems to fail in learning a regular latent space.

For instance, when I generate from a codebook index decoded from one of my dataset images, the output image seems fine, but when I try to interpolate between two indices, the latent space between two indices result in completely unrecognizable images.

I am told that the kl loss value have something to do with the regularizing of the latent space, but according to some issues raised before, this does not seem to be a usable option.

Is there a known reason for this kind of irregularity in the latent space? Or rather, has anyone succeeded in smooth latent interpolation while training DiscreteVAE model? It would be really helpful if someone has succeeded and can tell me about the relevant parameters.