Optimize the inference performance on Cloud Run

[Aniket-Parlikar]

We are currently using Google Cloud Run to obtain inferences on the map files generated using the Google Earth Engine API. However, for sending in a large number of concurrent requests(>175) we are experiencing a decline in performance of the Cloud Run services. Some of the indicators of such performance consist of

• request latencies,.
• CPU and memory utilization.
• container startup time.
• Total inference time of all the concurrent requests.

While deploying a service on the Google Cloud Run, we need to configure the following parameters( Container CPU and Memory, Request time-out and # of concurrent requests to be sent.)

We have developed a python utility(src/inference_profiling/profile_program.py) to send desired number of concurrent requests to the Cloud Run API and obtain total inference time for those requests.

Hence, can we get a better understanding of the performance by tuning the the Cloud Run configuration parameters and also be able to optimize it?

[Aniket-Parlikar]

@ivanzvonkov can you please let me know if this looks good or anything needs to be added

[ivanzvonkov]

@Aniket-Parlikar this looks good! How does #201 relate and help with this?
Thanks for all the detail!