Running BicleanerAI with TPUs

[charliekocsis]

Hi Bitextor team,

I'm using Bicleaner AI to clean up Opus corpora. As part of that I was looking at using TPUs for the clean up process.

On Kaggle, there is documentation on using TPUs with TF and Keras as follows:

# detect and init the TPU
tpu = tf.distribute.cluster_resolver.TPUClusterResolver()

# instantiate a distribution strategy
tf.tpu.experimental.initialize_tpu_system(tpu)
tpu_strategy = tf.distribute.TPUStrategy(tpu)

# instantiating the model in the strategy scope creates the model on the TPU
with tpu_strategy.scope():
    model = tf.keras.Sequential( … ) # define your model normally
    model.compile( … )

# train model normally
model.fit(training_dataset, epochs=EPOCHS, steps_per_epoch=…)

But I am not sure how I would compile and run a bicleaner-ai model with TPUs. Is there a simple way to integrate Bicleaner AI with TPU support?

Thanks for your help in advance.

[ZJaume]

I guess that if you need to set the TPU scope during model loading (not exactly as the example, because that is for training, though), you will need to do it here:

bicleaner-ai/src/bicleaner_ai/models.py

Lines 225 to 230 in 0547fa3

	try:
	self.model = load_model(self.dir+'/'+self.settings["model_file"],
	custom_objects=deps, compile=False)
	except ValueError:
	self.model = self.build_model(compile=False)
	self.model.load_weights(self.dir+'/'+self.settings["model_file"])

But never tried using TPUs with Bicleaner AI, not sure if it will work.