Rollout model with istio and/or ambassador

[lluunn]

Problem:
#1823 adds support for rolling out a new version of model, and send X% of traffic to it.
The request path is

ingress --> ambassador --> tfserving (version1 or version2)

However, it actually doesn't work because ambassador is not in istio service mesh, and Istio traffic management needs the client side to talk to Istio Pilot.
(so for ambassador --> tfserving, the client is ambassador)
Ambassador can deploy with Istio,
but we cannot inject istio sidecar to ambassador. (see issue)

So I think the options are

1. Use ambassador to do canary rollout (see doc)
2. Add an additional hop between ambassador and tfserving, which is in the istio mesh.
3. Remove ambassador. We need to figure out how to do (dynamic) routing with istio then.

I think we don't want 1 as we are moving toward Istio.
2 is not ideal.
I will test if 3 can be done by Istio's VirtualService

cc @jlewi

[lluunn]

related #1309

[lluunn]

I think Istio's way to do edge routing is to use Gateway, and bind VirtualService to it.

[jlewi]

It looks like Ambassador has ISTIO integration.
https://www.getambassador.io/user-guide/with-istio/

Does that work?

[lluunn]

The integration means using ambassador at the edge in front of the Istio mesh (which is what we are doing now).
Problem is ambassador is not in service mesh

[lluunn]

ambassador has an open issue on this: emissary-ingress/emissary#255

[jlewi]

Could we use the ISTIO service Gateway as our dynamic reverse proxy rather than Ambassador?

I looked at this blog post about Ambassador and ISTIO
https://itnext.io/ambassador-and-istio-edge-proxy-and-service-mesh-814aac9f23df

I think the current way Ambassador works with ISTIO is that Ambassador routes traffic to a service inside the mesh (in the example the product page app). The product page then uses ISTIO to route traffic to its various backends.

In our case we want to expose outside the cluster an API that will serve models and split traffic between different model versions. In this case we don't have a service inside the mesh that can be a backend for Ambassador (e.g. we don't have a web app sending requests to the different version of the models).

/cc @kflynn

@aroundthecode any thoughts?

[lluunn]

Yeah that's the direction I am exploring

[ukclivecox]

We have an example in Seldon for using istio to route traffic to two versions (canary)
See docs and notebook link here

This is using Ambassador with Istio.

[jlewi]

Talked to @lluunn; we have a solution and instructions.

Next step would be simplifying/automating the ISTIO install process. There is a separate issue #1909 for that.

[vaibhavrtk]

@jlewi Can you please elaborate on the solution that you finally used.