while embedding_concat() runs, I got "Killed"

[wonchul-kim]

I think I got too much number of data....?

what do you think?

[haobo827]

I think I got too much number of data....?

what do you think?

Try cropsize=224

[michaelstuffer98]

If it is still relevant, this might be if you run out of RAM. Happened for me when having large datasets.

[RichardChangCA]

If I use this model in a larger dataset, out of RAM is a big issue. Can anybody solve it? I guess this is the natural shortcoming of this method.

[michaelstuffer98]

@RichardChangCA what I did was repeatedly running the embedding_concat() after a fixed amount of samples have been loaded, such that you can perform the index-selection during loading the data which reduces memory usage. Wrote some sample function that you have to call after e.g. every 1000 samples have been processed. Define that inside the main such that it can access the idx tensor and the embeddings=[] list that you have to declare yourself in the main function. Reinitialize empty the train_outputs/test_outputs dict after running store embeddings to free the memory.

def store_embeddings(model_outputs):
            for k, v in model_outputs.items():
                model_outputs[k] = torch.cat(v, 0)
            embedding_vectors = model_outputs[0]
            for layer_name in layers[1:]:
                    embedding_vectors = embedding_concat(embedding_vectors, model_outputs[layer_name])
            embedding_vectors = torch.index_select(embedding_vectors, 1, idx)
            embeddings.append(embedding_vectors)

[RichardChangCA]

@RichardChangCA what I did was repeatedly running the embedding_concat() after a fixed amount of samples have been loaded, such that you can perform the index-selection during loading the data which reduces memory usage. Wrote some sample function that you have to call after e.g. every 1000 samples have been processed. Define that inside the main such that it can access the idx tensor and the embeddings=[] list that you have to declare yourself in the main function. Reinitialize empty the train_outputs/test_outputs dict after running store embeddings to free the memory.

def store_embeddings(model_outputs):
            for k, v in model_outputs.items():
                model_outputs[k] = torch.cat(v, 0)
            embedding_vectors = model_outputs[0]
            for layer_name in layers[1:]:
                    embedding_vectors = embedding_concat(embedding_vectors, model_outputs[layer_name])
            embedding_vectors = torch.index_select(embedding_vectors, 1, idx)
            embeddings.append(embedding_vectors)

Thanks @michaelstuffer98 Do you know how to calculate the covariance matrix if the dataset is too large? I optimized the cpu usage for other parts, only left the covariance matrix calculation. I have to store all embeddings for all normal training data to calculate the covariance matrix, but my cpu cannot stand it.