Throughput/Inference Throughput issue in benchmark.py

[brm738]

When running benchmark.py values of throughput printed in the terminal during program execution does not correspond to inference throughput saved in the result file <model_name>_.csv (example patchcore_gpu.csv or patchcore_cpu.csv).
Values printed in the terminal seems to be OK. Values saved in the file however differs quite significantly from them, especially when running testing/inference on GPUs
My setup machine consist of 4GPUs Nvidia A100-SXM4-40GB.

[ashwinvaidya17]

@brm738 thanks for noticing this discrepancy! The models use a callback to calculate inference time and throughput. That's what gets printed on the terminal. The benchmarking script computes time taken without using the callback. Also, The models report batch throughput while the benchmarking reports throughput on sequential images. This is because currently the torch and openvino inferencers are used on streaming/individual images. I'll update the benchmarking script to remove this ambiguity.

[ashwinvaidya17]

Closing as it is addressed in PR #221