How to get embedding without deleting duplicates？

[WangyuchenCS]

Hi,
I wonder How to get embedding without deleting duplicates？ as I found that the output .h5 file return a result that did not match the input .txt length, and it dropped duplicates.

[mjeensung]

Hi @WangyuchenCS

Could you try --keep_text_order True when running the script?

[WangyuchenCS]

Thanks a lot , but it cause an error that has not occurred before

[mjeensung]

Thanks for reporting the error.

Could you replace line 13--16 as follows?

entity_id = str(i)
entity_name = f[entity_id].attrs['text']
embedding = f[entity_id]['embedding'][:]