How to distinguish sos token(default = 0) and quantified image token zero ?

[JJJYmmm]

Since the transformer take in the quantified image token generated by VQGAN, which codebook has indices (0~n_embed-1), and transformer’s sos token is also set to zero defaultly. Could you tell me why we don't distinguish codebook vector0 and the sos token when training transformer?

[srasoulzadeh]

This was also my Q as well. I believe the reason is that the transformer uses positional encodings to add information about the position of each token in the sequence. The token at position 0 is always the sos_token, and this positional information helps the model distinguish between the sos_token and the codebook's 0 index.

[JJJYmmm]

This was also my Q as well. I believe the reason is that the transformer uses positional encodings to add information about the position of each token in the sequence. The token at position 0 is always the sos_token, and this positional information helps the model distinguish between the sos_token and the codebook's 0 index.

Sounds reasonable!
thx!