Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update Kubernetes test #162

Merged
merged 6 commits into from
Mar 22, 2021
Merged

Update Kubernetes test #162

merged 6 commits into from
Mar 22, 2021

Conversation

pothos
Copy link
Member

@pothos pothos commented Mar 19, 2021

  • kola/tests/kubernetes: do not require Flannel to start before Docker
    Now that flannel itself is started with Docker, we can't start Flannel
    before Docker which results in a deadlock but instead have to rely on
    Flannel to restart Docker to apply its settings.
  • kola/tests/kubernetes: ensure /run/flannel exists and simplify setup
    Placing the symlink in /run/flannel/ could fail if the directory did
    not exist before. Instead of setting up a symlink, directly write the
    template to /run/flannel/ which takes care of creating the directory.
  • kola/tests/kubernetes: use coreos-metadata to find the IP address
    The nodes were assumed to use coreos-cloudinit when an empty
    configuation is given and in result, set up /etc/environment.
    Use coreos-metadata (afterburn) explicitly to find the IP address
    on any cloud environment. Also, add a few more comments about the
    set up and force that no local etcd can start on the K8s controller.
  • kola/tests/kubernetes: switch to 192.168.0.0/16 network
    The 10.x.0.0/16 networks overlapped with the private network 10.0.0.0/8
    used in Equinix Metal and can also overlap with Digital Ocean private
    IP addresses. The 192.168.0.0/16 network seems to be not used by AWS
    nor the internal Docker network at the moment.
  • kola/tests/kubernetes: test on more platforms
    The tests were only running on GCE and by enabling it on more platforms
    we have a better chance to catch errors early.
    For VMware in our setup there are 4 tests in parallel for 12 slots and
    thus we have to limit the cluster size to 3 nodes.
  • kola/tests/kubernetes: drop kubelet-wrapper test
    The kubelet-wrapper script will be removed.

Can be done in a follow-up PR:
TODO: Replace running kubelet-wrapper with Docker, remove supporting rkt as K8s runtime
TODO: Try to fix running on esx and packet platforms

How to use/testing done

Tested on AWS, DO, GCE

Now that flannel itself is started with Docker, we can't start Flannel
before Docker which results in a deadlock but instead have to rely on
Flannel to restart Docker to apply its settings.
The nodes were assumed to use coreos-cloudinit when an empty
configuation is given and in result, set up /etc/environment.
Use coreos-metadata (afterburn) explicitly to find the IP address
on any cloud environment. Also, add a few more comments about the
set up and force that no local etcd can start on the K8s controller.
Placing the symlink in /run/flannel/ could fail if the directory did
not exist before. Instead of setting up a symlink, directly write the
template to /run/flannel/ which takes care of creating the directory.
The 10.x.0.0/16 networks overlapped with the private network 10.0.0.0/8
used in Equinix Metal and can also overlap with Digital Ocean private
IP addresses. The 192.168.0.0/16 network seems to be not used by AWS
nor the internal Docker network at the moment.
The tests were only running on GCE and by enabling it on more platforms
we have a better chance to catch errors early.
For VMware in our setup there are 4 tests in parallel for 12 slots and
thus we have to limit the cluster size to 3 nodes.
The kubelet-wrapper script will be removed.
@pothos pothos marked this pull request as ready for review March 19, 2021 21:05
@pothos pothos requested a review from a team March 19, 2021 21:08
Copy link
Member

@t-lo t-lo left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, thank you.

@pothos pothos force-pushed the kai/kubernetes-test branch 2 times, most recently from 1dc1525 to 184fdbf Compare March 22, 2021 18:37
@pothos
Copy link
Member Author

pothos commented Mar 22, 2021

The test should still work with rkt instead of docker, otherwise we can add ExcludeChannels: []string{"beta", "stable", "lts"}, and remove each channel when a > 2801 release comes in it.

@pothos pothos merged commit 3b29a57 into flatcar-master Mar 22, 2021
@pothos pothos deleted the kai/kubernetes-test branch March 22, 2021 18:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants