Optimize for CPU inference

[justinsalamon]

Inference on CPU is very slow right now (often too slow for practical application).

I think TensorFlow already uses as many cpu cores as it has access to when running in cpu mode (?), so I'm not sure whether e.g. splitting the audio track and parallelizing inference via e.g. multiprocessing or jobilb would make any difference.

But, it might be worth checking out TF guide on performance, such as the performance guide or the info on model quantization.

[jongwook]

Does Intel CPU have int8-optimized operations? I thought they are specific to CUDA.

Tensorflow already uses all CPU cores by default, and I'm not sure if there's practically any performance to gain unless we compromise the prediction frequency (using time step > 10 ms) or using a smaller model (like the one I used in the web demo)

[jongwook]

It's not precisely optimize but you might be interested in some of these numbers reported here

[justinsalamon]

We should add this table to the README file in the section where we explain about the different models available, ideally also including performance drop with respect to the full model (i.e. use a track from MDB for which we have a reference annotation and evaluate each model not only in terms of time, but also accuracy).

e.g. something similar to table 1 here: https://www.tensorflow.org/performance/quantization

[jongwook]

Yeah that's why I haven't put the table on README yet!