Add container image management doc #1421
Conversation
| # Why container image management | ||
|
|
||
| - After we support container upgrade via k8s, for the container image, we have two problem to be fixed. | ||
| - After we upgrade one container to a higher version via k8s, in this time, k8s is down or something wrong, container need to go back to local mode, local mode container should be the higher version other than the original version. |
| @@ -0,0 +1,40 @@ | |||
| # Why container image management | |||
|
|
|||
| - After we support container upgrade via k8s, for the container image, we have two problem to be fixed. | |||
There was a problem hiding this comment.
why container need to go to local mode if k8s is down ? It's a daemonset pod, right ?
|
|
||
| - After we support container upgrade via k8s, for the container image, we have two problem to be fixed. | ||
| - After we upgrade one container to a higher version via k8s, in this time, k8s is down or something wrong, container need to go back to local mode, local mode container should be the higher version other than the original version. | ||
| - Upgrade will bring a new version container image to sonic, after many times upgrade, the disk will be full, need to avoid this. |
There was a problem hiding this comment.
I think here we need to mention how k8s worker resolves this problem in general, and why we did use that
|
|
||
| # How to handle the two problems | ||
| - Tag latest | ||
| - Each time we do upgrade, after we think the new version is good, we tag the image version to latest. When need to go back to local, local mode will use the latest version image |
There was a problem hiding this comment.
it's better to add description to introduce how the local image looks like and how kube image does, and in local mode the sonic feature service will always use the lastest image.
| @@ -0,0 +1,40 @@ | |||
| # Why container image management | |||
|
|
|||
| - After we support container upgrade via k8s, for the container image, we have two problem to be fixed. | |||
There was a problem hiding this comment.
it's better to mention the two mode: local and kube. k8s will only ugprade the image used in kube mode, there will need to a way to upgrade the image used in local mode.
|
|
||
|
|
||
| # How to handle the two problems | ||
| - Tag latest |
There was a problem hiding this comment.
How to upgrade tag for local container image
| # How to handle the two problems | ||
| - Tag latest | ||
| - Each time we do upgrade, after we think the new version is good, we tag the image version to latest. When need to go back to local, local mode will use the latest version image | ||
| - How to "think the new version is good", after the new version container is running, we will check the container is still running or not in ten minutes, still running mean the new version is good so that we can do tag latest. Not running means fallback happens or something wrong, we can't tag the new version image to latest |
There was a problem hiding this comment.
if the new version of container has been running more than 10 minutes, it means it's stable, and we will tag it as latest.
There was a problem hiding this comment.
do we need to coordinate with upgrade process in k8s controller ?
| ``` | ||
| docker-sonic-telemetry:latest(v2) | ||
| docker-sonic-telemetry:v1 | ||
| sonick8scue.azurecr.io/docker-sonic-telemetry:v1 |
There was a problem hiding this comment.
you can add a result after clean up for comparison
| - Why are there these two cases, find answer in next section. | ||
|
|
||
| # Kube container will not replace the local container when their image version are the same.(Suppose the versions are both v1) | ||
| - Why: when they are in the same version, local container should be more stable and no need to restart the container so no down time; when we rollout, we will enable k8s on many devices, we don't want the containers restart at the same time. |
There was a problem hiding this comment.
this may be confusing, let's focus on one device. Replacing the mode during join may cause service interruption.
| - Why: when they are in the same version, local container should be more stable and no need to restart the container so no down time; when we rollout, we will enable k8s on many devices, we don't want the containers restart at the same time. | ||
| - The kube container will dry-run, no real services start inside the container. But it will still download the sonick8scue.azurecr.io/docker-sonic-telemetry:v1 image. | ||
| - Actually docker-sonic-telemetry:v1 and sonick8scue.azurecr.io/docker-sonic-telemetry:v1 have the defferent image id, but docker-sonic-telemetry:v1 really runs before and the latter not. So for the Clean-Up case-1, we will remove latter, because it never really run before, and tag the docker-sonic-telemetry:v1 to sonick8scue.azurecr.io/docker-sonic-telemetry:v1 and remove docker-sonic-telemetry:v1 to get last latest image prepared for fallback | ||
| - How Clean-Up case-2 comes? Suppose we have a device group, all devices' containers running on v2 version in kube mode, now one device reimaged, it's sonic version is v3, once the device rejoin to k8s cluster, k8s scheduled v2 container will replace the local v3 container. V3 image can't be the last latest image version, so clean up the v3 image directly. |
There was a problem hiding this comment.
it doesn't explain why v1 is deleted directly
| # Why container image management | ||
|
|
||
| After we support container upgrade via k8s, for the container image, we have two problem to be fixed. | ||
| - After we upgrade one container to a higher version via k8s, if k8s control plane is down and worker reboot, container need to go back to local mode, local mode container should run the higher version other than the original version. |
|
|
||
| So we need to manage the container image to fix these two problems | ||
|
|
||
| # Existing designs need to know first |
There was a problem hiding this comment.
don't need a dedicated section for this, we can just have a very short and concise introduction for the local/kube mode.
| when the kube container and local container are in the same version, local container should be more stable, becasue it has run for some time. And when device first join k8s cluster as worker, k8s will try to deploy a same version container with local, if we choose replace, the container will has a down time, actually no upgrade happens, down time is not necessary. | ||
| ##### one note | ||
| The kube container will dry-run, no real services start inside the container. But it still downloads the sonick8scue.azurecr.io/docker-sonic-telemetry:v1 image on device. | ||
| # How to handle the two problems |
There was a problem hiding this comment.
let's focus on the specific issue. 1) How to update tag the local contain image 2) How to clean up obsolete container image
| # How to handle the two problems | ||
| For the first problem, since the systemd service will alway use image with latest tag, we can tag the newest version image with latest. How to Tag-Latest: | ||
| - When: If the new version of container has been running for more than 10 minutes, it means it's stable, and we will tag it as latest. | ||
| - Our customized k8s controller will complete the upgrade process within 10 minutes, if fallback happens or something wrong, it's impossible that the container run for more than 10 minutes, for this case, we can't tag the new version image to latest |
There was a problem hiding this comment.
let's use k8s controller, and discuss the general solution, the audience or user are public, not only for us.
| # How to handle the two problems | ||
| For the first problem, since the systemd service will alway use image with latest tag, we can tag the newest version image with latest. How to Tag-Latest: | ||
| - When: If the new version of container has been running for more than 10 minutes, it means it's stable, and we will tag it as latest. | ||
| - Our customized k8s controller will complete the upgrade process within 10 minutes, if fallback happens or something wrong, it's impossible that the container run for more than 10 minutes, for this case, we can't tag the new version image to latest |
There was a problem hiding this comment.
not tagging it in fallback case is the expected behavior.
| - Our customized k8s controller will complete the upgrade process within 10 minutes, if fallback happens or something wrong, it's impossible that the container run for more than 10 minutes, for this case, we can't tag the new version image to latest | ||
| - After we tag latest, we need to remove the previous stopped local container, otherwise when go back to local, systemd service will not create new local container with new version image and will start the stopped container which running with a previous version image. | ||
|
|
||
| For the second problem, we need to remove the old version images in time. How to Clean-Up: |
| - When: Tag latest will trigger Clean-Up, once the tag latest successfully finished, we will do Clean-Up | ||
| - Clean-Up will clean up all old version images beside the last latest version image(for fallback usage) | ||
| - For image removal, we have three cases to handle: | ||
| - Case 1: after device(v1) joined k8s cluster, k8s will deploy v1 container to device, kube v1 container will dry-run. Due to no local to kube replacement happens, Tag-Latest and Clean-Up will not happen. Then we upgrade container to v2 via k8s, kube v2 container will really run and replace the local one. Then the images on device after Tag-Latest should be like below: |
There was a problem hiding this comment.
I'm afraid the term "really run and replace the local one" would be very confusing to people.
| ``` | ||
| This case is most ofen happened | ||
|
|
||
| - one note is that kube v1 image and local v1 image's image id maybe not same. |
| This case is most ofen happened | ||
|
|
||
| - one note is that kube v1 image and local v1 image's image id maybe not same. | ||
| - Why don't we leverage k8s image clean up function for worker. Kubelet will try to remove all docker images which is not used, our upgrade only support telemetry now, we don't want k8s touch other container's image. |
There was a problem hiding this comment.
let's consider it more general, some container may not enable kube management.
| sonick8scue.azurecr.io/docker-sonic-telemetry:v2 image_id: kube_v2 | ||
| ``` | ||
|
|
||
| - Case-3, proceed from the case-1, we upgrade container to v3 via k8s, kube v3 container will really run and replace the last kube v2 container. Then the images on device after Tag-Latest should be like below: |
There was a problem hiding this comment.
The principle for image cleanup is : keep the version used by kube, and delete others, you can describe the case from this perspective. Not because "it is not really run"
Add container image management doc